# Staged lab role tests

**All five live stages passed according to operator-supplied terminal output.**
The sequence followed the successful ELITEDESK stop/start cycle. Only the two-node postgres-ra-test lab is supported; maintenance
must remain true throughout. DV6 retains the frozen RA and ELITEDESK has the
plugin candidate. Neither implementation is installed or changed by this tool.

The operator confirmed that this lab has no fencing; fencing is available only
in production. This checkpoint covers controlled, manual role transitions with
a reachable primary. Abrupt-primary-loss automatic failover is not validated
for the plugin candidate. The earlier production fencing report applies to the
earlier implementation, not this extraction.

During maintenance, Pacemaker's recurring role monitors are paused. The harness
invokes the RA directly, so its action results do not update Pacemaker's recorded
operation history. A Cockpit cluster role display can therefore remain stale
even while the RA explicitly updates promotion scores. Database roles were
verified using direct SQL; systemd/container running state remains distinct
from PostgreSQL role. Maintenance was not disabled to refresh the display.

`scripts/test-lab-roles.py` is a portable, operator-run harness with embedded
hashes from both lab snapshots. It needs root locally and noninteractive SSH
as david to the peer for read-only observations. SSH host keys must already be
verified. Passwords are entered in the operator's terminal, never in chat.

Run `sudo python3 /path/to/test-lab-roles.py inspect` on each node first.
Inspection checks cluster name, membership, maintenance, resource parameters,
installed hashes, SQL readiness, roles, system identity and maintenance service.
It changes no database or cluster state. Do not proceed if either check fails.

Each stage requires a separate invocation and review of its successful output:

| Order | Host | Stage argument | Expected result |
| --- | --- | --- | --- |
| 1 | DV6 | demote | DV6 streams from ELITEDESK; both are standbys |
| 2 | ELITEDESK | promote | ELITEDESK primary, DV6 streaming standby |
| 3 | ELITEDESK | demote | ELITEDESK streams from DV6; both are standbys |
| 4 | DV6 | promote | DV6 primary, ELITEDESK streaming standby |
| 5 | ELITEDESK | rejoin | Stop PostgreSQL only, run rewind-only rejoin to DV6, verify streaming |

The demote stage submits a **synthetic lab pre-demote notification** through the
RA notify action, then invokes demote. This tests the notification consumer and
controlled demotion; it does not prove Pacemaker generates/orders notifications.
The RA retains its own catch-up and successor checks. The harness refuses promote
unless the peer is already in recovery and streaming from the local standby.
Between demote and promote there is deliberately no writable primary. Arrange
both terminals before starting; do not perform other manual database operations
during the sequence. Scores on both standbys can fluctuate during this interval.

The rejoin stage is restricted to the restored ELITEDESK standby. It explicitly
sets basebackup fallback=false. It tests the stopped-PostgreSQL precondition,
maintenance container survival, source authentication, pg_rewind invocation,
recovery configuration, PostgreSQL restart, streaming and score restoration.
Because it does not create divergent data, it does **not** validate divergent
former-primary rewind repair. No fencing, data replacement fallback, VIP moves,
Pacemaker-managed switchover or automatic failover are tested here.

Each modifying invocation writes root-only results/action logs beneath
`/var/backups/managed-container-roles-lab-*`. Failures halt without automatic
retry/recovery. A failure after demotion can leave no primary; a rejoin failure
can leave ELITEDESK stopped. Stop the sequence and inspect the evidence rather
than running the next stage. Do not clear failed actions or change maintenance.

The checks sample state; they cannot prevent a separate administrator from
changing roles concurrently. Controlled demotion of a reachable primary is the
basis for these manual lab promotions. This is not a procedure for promoting
after an unreachable primary failure; that requires fencing and Pacemaker's
ordering under the operating policy.

## Completed live sequence

The operator supplied successful output for each stage below. Evidence directories
are on the host named in the table, under `/var/backups/`. Codex has not read the
root-only results.json or action logs; these are operator-reported passes, not
independently witnessed live actions.

| Host | Stage | Result | Evidence directory |
| --- | --- | --- | --- |
| DV6 | demote | rc=0; standby monitor=0; score=100 | managed-container-roles-lab-o1xjobi2 |
| ELITEDESK | promote | rc=0; primary monitor=8; score=1000 | managed-container-roles-lab-qdz57g70 |
| ELITEDESK | demote | rc=0; standby monitor=0; score=100 | managed-container-roles-lab-nwbzdxnl |
| DV6 | promote | rc=0; primary monitor=8; score=1000 | managed-container-roles-lab-r69amyxg |
| ELITEDESK | rejoin | stopped monitor=7; rejoin=0; standby monitor=0; score=100 | managed-container-roles-lab-7c4vw7_p |

Each stage reported preserved database identity and maintenance container,
verified roles and streaming, and maintenance=true. The final reported state is
DV6 primary and ELITEDESK streaming standby. The rejoin harness also checks the
stopped score is -INFINITY before invoking rejoin, with basebackup disabled.
This establishes successful pg_rewind invocation, not repair of deliberately
divergent timelines or a measured amount of rewound data.

The supplied inspection after ELITEDESK demotion showed both nodes in recovery,
streaming from each other, with replay LSN 0/A02B560 and system identifier
7684451577990111248. An earlier inspection in the same paste showed DV6 primary;
the later observation was used to resolve the role confusion before restoring DV6.

Both-way SSH access was established and independently checked before the role
sequence. The installed code was pinned to the existing lab snapshots. No RA
installation, maintenance change, fencing, VIP move or failure cleanup was part
of these tests. The limitations above continue to apply.

Offline original-function comparisons and 97 tests passed before the live
sequence (the 11 scheduler tests required execution outside the sandbox).
Six additional role-harness safety-gate tests passed before staging the harness.

## Commit checkpoint verification

On 2026-09-13, all 103 tests in `python3 -m unittest discover -s tests -p
'test_*.py'` passed, including offline scheduler simulations.
`python3 tests/verify_refactor.py` also passed: frozen reference hashes, all 29
original functions after exact adapter-call reversal, syntax, metadata and
22 differential lifecycle cases. No additional live actions were run for this
commit checkpoint.