Testing with Mock Storage

The MockStorageDriver lets you run the full Trilio Site Recovery workflow — including Protection Group creation, replication policy configuration, failover, failback, and DR drills — without physical Pure Storage FlashArray hardware. It simulates FlashArray behavior using a SQLite-backed state store, making it suitable for development iteration, CI/CD pipelines, pre-production DR workflow validation, and training environments. Because mock mode follows identical code paths to production mode for VM recreation, resource cleanup, and operation status tracking, workflows you validate here translate directly to production deployments.

Before enabling mock storage, ensure the following are in place:

• Trilio Site Recovery (Protector) is installed on both the primary and secondary OpenStack sites — see the Deployment Guide for installation steps.
• Both sites have the protector-api and protector-engine services running.
• The protector service user and directories (/var/lib/protector, /var/log/protector) exist with correct ownership.
• Both OpenStack sites have Cinder volume types configured with replication_enabled='<is> True' and the appropriate replication_type property, even in mock mode — the protection group creation workflow validates these properties regardless of the storage driver in use.
• A Glance image with an identical name (for example, cirros) exists on both sites. Mock mode simulates volume replication by creating new bootable volumes from this image on the target site; if the image is missing or named differently on either side, failover will fail at the instance recreation phase.
• Python 3.8+ and all Protector package dependencies are installed.
• You have write access to /etc/protector/protector.conf on both sites.

Mock storage is built into the Protector package — no separate installation is required. To activate it, you modify protector.conf and create the mock data directory on each site where you want mock behavior.

Step 1: Create the mock data directory

Run this on both the primary and secondary site controllers:

mkdir -p /var/lib/protector/mock_storage
chown -R protector:protector /var/lib/protector/mock_storage

Step 2: Enable mock storage in protector.conf

Open /etc/protector/protector.conf on each site and add use_mock_storage = True to the [DEFAULT] section:

[DEFAULT]
debug = True
log_dir = /var/log/protector
state_path = /var/lib/protector
use_mock_storage = True

Leave all other sections ([database], [keystone_authtoken], etc.) unchanged. Mock mode replaces only the Pure Storage driver calls; all OpenStack service interactions remain real.

Step 3: Restart Protector services on both sites

systemctl restart protector-engine
systemctl restart protector-api

Step 4: Confirm mock mode is active

Check the engine log for a line confirming the mock driver was loaded:

grep -i mock /var/log/protector/protector-engine.log | tail -5

You should see output similar to:

INFO protector.engine.storage MockStorageDriver initialized. State path: /var/lib/protector/mock_storage/

If you see Pure Storage connection attempts or API token errors instead, verify that use_mock_storage = True is in the [DEFAULT] section (not a subsection) and that the service was fully restarted.

Mock storage behavior is controlled by a single flag in protector.conf, plus the implied dependency on Glance image availability.

What mock storage simulates

The MockStorageDriver maintains SQLite state tables that mirror the FlashArray concepts used in production:

• Pods — used for sync replication (ActiveCluster) state
• Protection groups — used for async replication state
• Snapshot creation and replication — snapshot records are written to SQLite; no data is actually copied
• Promotion and demotion operations — state transitions are applied to the SQLite tables
• Sync and async failover and failback state transitions — the full state machine is exercised

Critical limitation: volume replication behavior

In production, failover restores volumes on the secondary site from replicated Pure Storage snapshots. In mock mode, there are no real snapshots to restore from. Instead, the MockStorageDriver simulates volume replication by creating new bootable volumes from a Glance image on the target site.

This means:

• The same Glance image must exist on both sites with identical names (for example, both sites must have an image named cirros).
• The restored volumes will not contain the actual data that was on the primary — they will be fresh volumes booted from the image.
• This is intentional: mock mode validates the DR workflow and orchestration, not data fidelity.

Once mock storage is enabled, you run every Trilio Site Recovery workflow exactly as you would in production. The CLI commands, API calls, and Horizon interactions are identical — the mock driver is transparent to the coordination layer.

Typical mock environment workflow

Configure your clouds.yaml to point at both sites:

clouds:
  site-a:
    auth:
      auth_url: http://site-a-controller:5000/v3
      project_name: admin
      username: admin
      password: password
      user_domain_name: Default
      project_domain_name: Default
    region_name: RegionOne
  site-b:
    auth:
      auth_url: http://site-b-controller:5000/v3
      project_name: admin
      username: admin
      password: password
      user_domain_name: Default
      project_domain_name: Default
    region_name: RegionOne

Register both sites with the Protector service:

openstack protector site create \
  --name site-a \
  --site-type primary \
  --auth-url http://site-a-controller:5000/v3

openstack protector site create \
  --name site-b \
  --site-type secondary \
  --auth-url http://site-b-controller:5000/v3

Create a Protection Group using a replication-enabled volume type:

openstack protector protection-group create \
  --name mock-test-pg \
  --replication-type async \
  --primary-site site-a \
  --secondary-site site-b \
  --volume-type replicated-ssd

Add a VM to the Protection Group (the VM's volumes must use the replication-enabled volume type):

openstack protector protection-group member-add mock-test-pg \
  --instance-id <your-vm-uuid>

Configure a replication policy. In mock mode, the FlashArray URLs and API tokens are validated for format but never actually contacted:

openstack protector protection-group policy-create mock-test-pg \
  --primary-fa-url https://mock-array-a.example.com \
  --primary-fa-token "T-mock-primary-token" \
  --secondary-fa-url https://mock-array-b.example.com \
  --secondary-fa-token "T-mock-secondary-token" \
  --pure-pg-name "mock-pg" \
  --replication-interval 300 \
  --rpo-minutes 15

Execute a test failover (DR drill):

openstack protector protection-group test-failover mock-test-pg \
  --retain-primary \
  --network-mapping net-primary=net-secondary

Execute a planned failover:

openstack protector protection-group failover mock-test-pg \
  --network-mapping net-primary=net-secondary

Execute a failback:

openstack protector protection-group failback mock-test-pg \
  --reverse-replication \
  --network-mapping net-secondary=net-primary

Monitor any operation:

openstack protector operation list
openstack protector operation show <operation-id>

Using mock mode in CI/CD pipelines

Because mock mode requires no FlashArray hardware, you can run end-to-end DR workflow tests in any CI environment that can spin up two OpenStack instances (for example, two DevStack deployments). The mock SQLite state resets automatically when the protector-engine service restarts, giving you a clean state for each test run. If you want persistent state across runs, back up and restore the SQLite file at /var/lib/protector/mock_storage/.

Example 1: Full failover and failback cycle in mock mode

This example demonstrates creating a Protection Group, adding a VM, triggering a failover, and then failing back. It assumes both sites are registered and use_mock_storage = True is set on both.

# Create the protection group
openstack protector protection-group create \
  --name pg-cirros-demo \
  --replication-type async \
  --primary-site site-a \
  --secondary-site site-b \
  --volume-type replicated-ssd

Expected output:

+------------------------+--------------------------------------+
| Field                  | Value                                |
+------------------------+--------------------------------------+
| id                     | pg-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx |
| name                   | pg-cirros-demo                      |
| status                 | active                              |
| replication_type       | async                               |
| primary_site           | site-a                              |
| secondary_site         | site-b                              |
+------------------------+--------------------------------------+

# Add a VM to the protection group
openstack protector protection-group member-add pg-cirros-demo \
  --instance-id a1b2c3d4-1234-5678-abcd-ef0123456789

Expected output:

+------------------------+--------------------------------------+
| Field                  | Value                                |
+------------------------+--------------------------------------+
| id                     | member-xxxx-...                     |
| instance_id            | a1b2c3d4-1234-5678-abcd-ef0123456789 |
| instance_name          | cirros-test-vm                      |
| status                 | protected                           |
| volumes_added          | 1                                   |
+------------------------+--------------------------------------+

# Execute failover
openstack protector protection-group failover pg-cirros-demo \
  --network-mapping net-a=net-b

Expected output:

+------------------------+--------------------------------------+
| Field                  | Value                                |
+------------------------+--------------------------------------+
| operation_id           | op-yyyyyyyy-yyyy-yyyy-yyyy-yyyyyyyy |
| operation_type         | failover                            |
| status                 | running                             |
| progress               | 10%                                 |
+------------------------+--------------------------------------+

# Poll until complete
openstack protector operation show op-yyyyyyyy-yyyy-yyyy-yyyy-yyyyyyyy

Expected output when complete:

+------------------------+--------------------------------------+
| Field                  | Value                                |
+------------------------+--------------------------------------+
| operation_id           | op-yyyyyyyy-yyyy-yyyy-yyyy-yyyyyyyy |
| operation_type         | failover                            |
| status                 | completed                           |
| progress               | 100%                                |
| instances_created      | 1                                   |
| instances_failed       | 0                                   |
+------------------------+--------------------------------------+

# Verify the protection group has flipped
openstack protector protection-group show pg-cirros-demo

Expected output:

+------------------------+--------------------------------------+
| Field                  | Value                                |
+------------------------+--------------------------------------+
| status                 | failed_over                         |
| current_primary_site   | site-b                              |
| failover_count         | 1                                   |
+------------------------+--------------------------------------+

# Fail back to the original primary
openstack protector protection-group failback pg-cirros-demo \
  --reverse-replication \
  --network-mapping net-b=net-a

After the failback operation completes, current_primary_site returns to site-a and status returns to active.

Example 2: Inspecting the mock SQLite state directly

You can inspect the raw mock state for debugging or verification.

sqlite3 /var/lib/protector/mock_storage/mock_storage.db ".tables"

Expected output:

pods  protection_groups  snapshots

sqlite3 /var/lib/protector/mock_storage/mock_storage.db \
  "SELECT * FROM protection_groups;"

This shows all simulated Pure Storage protection groups and their current state, which is useful when diagnosing why a failover or failback state transition did not behave as expected.

Example 3: Resetting mock state between CI test runs

To guarantee a clean slate at the start of each CI pipeline run:

# On both sites
systemctl stop protector-engine
rm -f /var/lib/protector/mock_storage/mock_storage.db
systemctl start protector-engine

The engine recreates the SQLite database on startup. No manual schema migration is required.

Issue: Failover fails with Glance image not found or instance creation error on the target site

Symptom: The failover operation reaches the instance recreation phase (progress ~60–90%) and then fails. The operation error message references a missing image.

Likely cause: The Glance image that mock mode uses to create bootable volumes on the target site does not exist on the secondary site, or its name differs between sites.

Fix: Verify that the same image exists on both sites with identical names:

openstack --os-cloud site-a image list
openstack --os-cloud site-b image list

If the image is missing from the secondary site, upload it:

openstack --os-cloud site-b image create cirros \
  --file cirros-0.5.2-x86_64-disk.img \
  --disk-format qcow2 \
  --container-format bare \
  --public

Ensure the image name on site-b exactly matches the name on site-a, including case.

Issue: protector-engine log shows Pure Storage connection attempts despite use_mock_storage = True

Symptom: After restarting the engine, logs show FlashArray API connection errors or token authentication failures.

Likely cause: The use_mock_storage = True setting was placed in a section other than [DEFAULT], or the service was not fully restarted.

Fix: Open /etc/protector/protector.conf and confirm the flag is in the [DEFAULT] section, not under [api], [engine], or any other stanza:

[DEFAULT]
use_mock_storage = True

Then perform a hard restart (not reload):

systemctl stop protector-engine && systemctl start protector-engine

Confirm with:

grep -i mock /var/log/protector/protector-engine.log | tail -5

Issue: sqlite3.OperationalError: unable to open database file in engine logs

Symptom: The engine starts but immediately logs a SQLite error and mock operations fail.

Likely cause: The /var/lib/protector/mock_storage/ directory does not exist or is not owned by the protector user.

Fix:

mkdir -p /var/lib/protector/mock_storage
chown -R protector:protector /var/lib/protector/mock_storage
systemctl restart protector-engine

Issue: Protection Group creation fails with volume type validation error even though use_mock_storage = True

Symptom: Creating a Protection Group returns an error stating the volume type does not support replication.

Likely cause: Mock mode does not bypass OpenStack service validation. The Cinder volume type on one or both sites is missing the replication_enabled='<is> True' or replication_type extra specs.

Fix: Set the required properties on the volume type on both sites:

openstack volume type set replicated-ssd \
  --property replication_enabled='<is> True' \
  --property replication_type='<in> async'

Run the same command against both site clouds.

Issue: Mock SQLite state is stale from a previous test run, causing unexpected operation errors

Symptom: DR operations fail with errors referencing protection groups or pods that no longer exist in the Protector database, or the mock state and Protector database are out of sync.

Likely cause: The mock SQLite database was not cleared between test runs while Protector database records were reset (or vice versa).

Fix: Reset both the mock state and Protector operational state together:

systemctl stop protector-engine
rm -f /var/lib/protector/mock_storage/mock_storage.db
systemctl start protector-engine

If you also reset the Protector MariaDB database between runs, ensure both are cleared in the same step to keep them consistent.