ocf-managed-container/docs/lab-role-tests.md

Staged lab role tests

All five live stages passed according to operator-supplied terminal output. The sequence followed the successful ELITEDESK stop/start cycle. Only the two-node postgres-ra-test lab is supported; maintenance must remain true throughout. DV6 retains the frozen RA and ELITEDESK has the plugin candidate. Neither implementation is installed or changed by this tool.

The operator confirmed that this lab has no fencing; fencing is available only in production. This checkpoint covers controlled, manual role transitions with a reachable primary. Abrupt-primary-loss automatic failover is not validated for the plugin candidate. The earlier production fencing report applies to the earlier implementation, not this extraction.

During maintenance, Pacemaker's recurring role monitors are paused. The harness invokes the RA directly, so its action results do not update Pacemaker's recorded operation history. A Cockpit cluster role display can therefore remain stale even while the RA explicitly updates promotion scores. Database roles were verified using direct SQL; systemd/container running state remains distinct from PostgreSQL role. Maintenance was not disabled to refresh the display.

scripts/test-lab-roles.py is a portable, operator-run harness with embedded hashes from both lab snapshots. It needs root locally and noninteractive SSH as david to the peer for read-only observations. SSH host keys must already be verified. Passwords are entered in the operator's terminal, never in chat.

Run sudo python3 /path/to/test-lab-roles.py inspect on each node first. Inspection checks cluster name, membership, maintenance, resource parameters, installed hashes, SQL readiness, roles, system identity and maintenance service. It changes no database or cluster state. Do not proceed if either check fails.

Each stage requires a separate invocation and review of its successful output:

Order	Host	Stage argument	Expected result
1	DV6	demote	DV6 streams from ELITEDESK; both are standbys
2	ELITEDESK	promote	ELITEDESK primary, DV6 streaming standby
3	ELITEDESK	demote	ELITEDESK streams from DV6; both are standbys
4	DV6	promote	DV6 primary, ELITEDESK streaming standby
5	ELITEDESK	rejoin	Stop PostgreSQL only, run rewind-only rejoin to DV6, verify streaming

The demote stage submits a synthetic lab pre-demote notification through the RA notify action, then invokes demote. This tests the notification consumer and controlled demotion; it does not prove Pacemaker generates/orders notifications. The RA retains its own catch-up and successor checks. The harness refuses promote unless the peer is already in recovery and streaming from the local standby. Between demote and promote there is deliberately no writable primary. Arrange both terminals before starting; do not perform other manual database operations during the sequence. Scores on both standbys can fluctuate during this interval.

The rejoin stage is restricted to the restored ELITEDESK standby. It explicitly sets basebackup fallback=false. It tests the stopped-PostgreSQL precondition, maintenance container survival, source authentication, pg_rewind invocation, recovery configuration, PostgreSQL restart, streaming and score restoration. Because it does not create divergent data, it does not validate divergent former-primary rewind repair. No fencing, data replacement fallback, VIP moves, Pacemaker-managed switchover or automatic failover are tested here.

Each modifying invocation writes root-only results/action logs beneath /var/backups/managed-container-roles-lab-*. Failures halt without automatic retry/recovery. A failure after demotion can leave no primary; a rejoin failure can leave ELITEDESK stopped. Stop the sequence and inspect the evidence rather than running the next stage. Do not clear failed actions or change maintenance.

The checks sample state; they cannot prevent a separate administrator from changing roles concurrently. Controlled demotion of a reachable primary is the basis for these manual lab promotions. This is not a procedure for promoting after an unreachable primary failure; that requires fencing and Pacemaker's ordering under the operating policy.

Completed live sequence

The operator supplied successful output for each stage below. Evidence directories are on the host named in the table, under /var/backups/. Codex has not read the root-only results.json or action logs; these are operator-reported passes, not independently witnessed live actions.

Host	Stage	Result	Evidence directory
DV6	demote	rc=0; standby monitor=0; score=100	managed-container-roles-lab-o1xjobi2
ELITEDESK	promote	rc=0; primary monitor=8; score=1000	managed-container-roles-lab-qdz57g70
ELITEDESK	demote	rc=0; standby monitor=0; score=100	managed-container-roles-lab-nwbzdxnl
DV6	promote	rc=0; primary monitor=8; score=1000	managed-container-roles-lab-r69amyxg
ELITEDESK	rejoin	stopped monitor=7; rejoin=0; standby monitor=0; score=100	managed-container-roles-lab-7c4vw7_p

Each stage reported preserved database identity and maintenance container, verified roles and streaming, and maintenance=true. The final reported state is DV6 primary and ELITEDESK streaming standby. The rejoin harness also checks the stopped score is -INFINITY before invoking rejoin, with basebackup disabled. This establishes successful pg_rewind invocation, not repair of deliberately divergent timelines or a measured amount of rewound data.

The supplied inspection after ELITEDESK demotion showed both nodes in recovery, streaming from each other, with replay LSN 0/A02B560 and system identifier 7684451577990111248. An earlier inspection in the same paste showed DV6 primary; the later observation was used to resolve the role confusion before restoring DV6.

Both-way SSH access was established and independently checked before the role sequence. The installed code was pinned to the existing lab snapshots. No RA installation, maintenance change, fencing, VIP move or failure cleanup was part of these tests. The limitations above continue to apply.

Offline original-function comparisons and 97 tests passed before the live sequence (the 11 scheduler tests required execution outside the sandbox). Six additional role-harness safety-gate tests passed before staging the harness.

Commit checkpoint verification

On 2026-09-13, all 103 tests in python3 -m unittest discover -s tests -p 'test_*.py' passed, including offline scheduler simulations. python3 tests/verify_refactor.py also passed: frozen reference hashes, all 29 original functions after exact adapter-call reversal, syntax, metadata and 22 differential lifecycle cases. No additional live actions were run for this commit checkpoint.