Mock Storage Issues

The MockStorageDriver lets you run end-to-end Trilio Site Recovery workflows — including failover, failback, and DR drills — without physical Pure Storage FlashArray hardware. It simulates FlashArray replication behavior using a local SQLite database, making it suitable for development environments, CI pipelines, and operator training. This page covers the three most common problems you will encounter when working with the mock driver: VM recreation failures during failover, stale state leaking between test runs, and the use_mock_storage flag not taking effect.

Before working through these issues, confirm the following:

• Trilio Site Recovery (Protector) is installed on both the primary and secondary OpenStack sites, with protector-api and protector-engine running on each.
• You have access to protector.conf on both sites with sufficient privileges to edit and reload it.
• The protectorclient OSC CLI plugin is installed on the host you use to issue commands.
• You have shell access to the file system at /var/lib/protector/ on both sites (needed to inspect and reset mock state).
• Both sites share a Glance image registry configuration that is relevant to your test setup (see the failover VM recreation issue below).
• systemctl or an equivalent service manager is available to restart protector-engine after configuration changes.

The MockStorageDriver ships with the Protector package — no separate installation is required. You enable it through configuration rather than by installing additional software.

Step 1 — Edit protector.conf on each site

Open /etc/protector/protector.conf in your editor:

sudo vi /etc/protector/protector.conf

Step 2 — Add use_mock_storage to the [DEFAULT] section

Locate the [DEFAULT] section and add the flag there. Do not place it under [storage] or any other section — the driver reads this key only from [DEFAULT]:

[DEFAULT]
debug = True
log_dir = /var/log/protector
state_path = /var/lib/protector
use_mock_storage = True

Step 3 — Create the mock state directory

The mock driver writes its SQLite database to /var/lib/protector/mock_storage/. Create this directory if it does not already exist, and ensure the protector service user owns it:

sudo mkdir -p /var/lib/protector/mock_storage
sudo chown -R protector:protector /var/lib/protector/mock_storage

Step 4 — Restart protector-engine on both sites

The engine process reads configuration at startup. Restart it to pick up the change:

sudo systemctl restart protector-engine

Verify the service came back up cleanly:

sudo systemctl status protector-engine

Step 5 — Confirm mock mode is active

Check the engine log for a line indicating the mock driver has been loaded:

sudo tail -n 50 /var/log/protector/protector-engine.log | grep -i mock

You should see output similar to:

INFO protector.engine.storage [-] Using MockStorageDriver (use_mock_storage=True)

The mock driver has one required configuration key and one implicit path convention:

use_mock_storage

When set to True, the engine substitutes the MockStorageDriver for all Pure Storage FlashArray operations. Protection groups, consistency groups, replication intervals, and failover snapshot promotion are all simulated in the local SQLite database. Setting this to False (or omitting it) causes the engine to use the real Pure FlashArray driver, which requires primary_fa_url, primary_fa_api_token, secondary_fa_url, and secondary_fa_api_token to be set in your replication policy.

Why [DEFAULT] matters: The Protector engine loads storage driver configuration from the [DEFAULT] section before section-specific config is applied. If use_mock_storage appears under any other section (for example [storage]), the engine does not see it and silently falls back to the real driver — which then fails if no FlashArray credentials are configured.

Mock state path

The SQLite database file lives under the state_path directory defined in [DEFAULT]. If you customise state_path, the mock database moves with it. The path is not separately overridable — changing state_path changes where all Protector state files, including the mock database, are stored.

state_path

Sets the root directory for all Protector runtime state, including the mock storage database. Ensure the protector service user has read/write access to this path.

Once mock mode is active, you interact with Trilio Site Recovery exactly as you would against a real FlashArray-backed environment. The protectorclient OSC plugin and Horizon dashboard are unaware of the underlying driver.

Typical test cycle

1. Register your two OpenStack sites, create a protection group with a replication-enabled volume type, add VMs, and configure a replication policy. Because no real FlashArray credentials are needed, you can supply placeholder values for primary_fa_url and related policy fields — the mock driver ignores them.
2. Trigger a test failover (DR drill), planned failover, or unplanned failover as you normally would.
3. Verify that VMs are recreated on the secondary site and that the protection group status transitions correctly.
4. After each test run, delete the mock database to return to a clean state before the next run (see the reset procedure in the Troubleshooting section).

What the mock driver simulates

• Protection group and consistency group creation and deletion.
• Replication snapshot scheduling and promotion during failover.
• Volume creation from replicated snapshots on the secondary site.
• The full DR operation state machine (pending → running → completed or failed).

What the mock driver does not simulate

• Real data replication. No block data moves between sites.
• RPO enforcement. The mock does not measure or enforce replication intervals.
• FlashArray-specific error conditions (array full, replication lag, etc.). Use a real or staged FlashArray environment to test those paths.

Example 1 — Verify mock mode is enabled

Check that the engine loaded the mock driver after you edited protector.conf:

sudo grep -i 'mock' /var/log/protector/protector-engine.log | tail -5

Expected output:

INFO protector.engine.storage [-] Using MockStorageDriver (use_mock_storage=True)

If you see no output, the flag is not in [DEFAULT] or the engine has not been restarted. See the troubleshooting entry for use_mock_storage not taking effect.

Example 2 — Inspect the mock state database

List the tables the mock driver has created after at least one protection group has been set up:

sqlite3 /var/lib/protector/mock_storage/mock_storage.db ".tables"

Expected output:

pg_snapshots  protection_groups  volume_snapshots  volumes

Query the simulated protection groups:

sqlite3 /var/lib/protector/mock_storage/mock_storage.db \
  "SELECT id, name, status FROM protection_groups;"

Example 3 — Reset mock state between test runs

Delete the SQLite database to clear all simulated replication state before starting a new test scenario:

sudo systemctl stop protector-engine
sudo rm -rf /var/lib/protector/mock_storage/
sudo mkdir -p /var/lib/protector/mock_storage
sudo chown -R protector:protector /var/lib/protector/mock_storage
sudo systemctl start protector-engine

The engine recreates the database on next startup. You do not need to recreate the Protector protection groups themselves — those records live in the MariaDB relational database, not in the mock storage database.

Example 4 — Upload a required Glance image to the secondary site

Before running a failover test, ensure the boot image referenced by your source VMs exists on the secondary site with the exact same name:

# On the primary site — find the image name used by your test VM
export OS_AUTH_URL=http://site-a:5000/v3
source ~/site-a-openrc
openstack server show <vm-uuid> -f value -c image
# Example output: ubuntu-22.04-server

# Download the image from the primary site
openstack image save --file /tmp/ubuntu-22.04-server.raw ubuntu-22.04-server

# On the secondary site — upload with the identical name
export OS_AUTH_URL=http://site-b:5000/v3
source ~/site-b-openrc
openstack image create \
  --file /tmp/ubuntu-22.04-server.raw \
  --disk-format raw \
  --container-format bare \
  --public \
  ubuntu-22.04-server

The name must match exactly, including case and spacing.

Issue 1 — Failover VM recreation fails

Symptom: The failover DR operation reaches the instance recreation phase and then transitions to failed. The operation detail (from openstack protector operation show <op-id>) includes an error such as Image not found or No image with name '<image-name>' exists on target site.

Cause: During failover, Protector recreates VMs on the secondary site using the VM metadata captured at the time the VM was added to the protection group. For VMs that boot from a Glance image (rather than booting entirely from a Cinder volume), Protector looks up the image on the secondary site by name. If the image does not exist on the secondary site, or exists under a different name, VM recreation fails.

Fix:

1. Identify the Glance image name used by each VM in the protection group. On the primary site, run openstack server show <vm-uuid> -f value -c image for each member VM.
2. On the secondary site, check whether those image names exist: openstack image list --name '<image-name>'.
3. For any missing images, upload them to the secondary site using the exact same name as on the primary site. See Example 4 above for the upload procedure.
4. Re-run the failover operation.

This requirement applies to both test failover (DR drill) and production failover. For VMs that boot from a Cinder volume (boot-from-volume), the image is not required on the secondary site because the boot volume itself is replicated.

Issue 2 — Mock state persists between test runs

Symptom: After completing a failover or failback test, a subsequent test run behaves unexpectedly — operations skip steps, volumes appear to already exist, or the protection group reports an incorrect status.

Cause: The mock driver records all simulated storage state — protection group snapshots, volume replicas, and replication history — in a SQLite database at /var/lib/protector/mock_storage/. This database is not automatically cleared between test runs. Stale entries from a previous run can conflict with or mislead a new run.

Fix: Stop the engine, delete the mock storage directory, recreate it with correct ownership, and restart the engine:

sudo systemctl stop protector-engine
sudo rm -rf /var/lib/protector/mock_storage/
sudo mkdir -p /var/lib/protector/mock_storage
sudo chown -R protector:protector /var/lib/protector/mock_storage
sudo systemctl start protector-engine

This only removes the mock driver's SQLite state. Protection groups, members, replication policies, and DR operation history stored in MariaDB are unaffected. If you also want to reset those, you must do so through the Protector API or OSC CLI (openstack protector protection-group delete, etc.) before or after clearing the mock database.

Issue 3 — use_mock_storage not taking effect

Symptom: After setting use_mock_storage = True and restarting protector-engine, the engine log shows no mention of the mock driver, and replication policy validation fails with errors about missing FlashArray credentials.

Cause: The use_mock_storage key is only read from the [DEFAULT] section of protector.conf. If you placed it under [storage], [engine], or any other section, the engine does not find it and silently falls through to the real Pure FlashArray driver.

Fix:

1. Open /etc/protector/protector.conf and search for use_mock_storage:

   grep -n 'use_mock_storage' /etc/protector/protector.conf
   

2. If the output shows a section header other than [DEFAULT] above the key, move the line into the [DEFAULT] block:

   [DEFAULT]
   debug = True
   log_dir = /var/log/protector
   state_path = /var/lib/protector
   use_mock_storage = True
   

3. Confirm there is no duplicate entry of use_mock_storage set to False elsewhere in the file.
4. Restart protector-engine:

   sudo systemctl restart protector-engine
   

5. Verify the mock driver is now loaded:

   sudo tail -n 50 /var/log/protector/protector-engine.log | grep -i mock
   

   You should see: INFO protector.engine.storage [-] Using MockStorageDriver (use_mock_storage=True)