Lyncoln
ec2077f979
Extract PostgreSQL, Podman runtime and systemd lifecycle adapters while preserving existing lifecycle and scoring behavior. Update RPM layout and add guarded operator-run lab harnesses. Record successful manual stop/start, promotion, demotion and rewind-only rejoin with pinned installed hashes. The unfenced lab does not validate automatic failover, Pacemaker-managed transitions or divergent-data rewind. Validation: 103 offline tests, original-function differential checks, checksum manifest and whitespace checks passed.
114 lines
6.7 KiB
Markdown
114 lines
6.7 KiB
Markdown
# Staged lab role tests
|
|
|
|
**All five live stages passed according to operator-supplied terminal output.**
|
|
The sequence followed the successful ELITEDESK stop/start cycle. Only the two-node postgres-ra-test lab is supported; maintenance
|
|
must remain true throughout. DV6 retains the frozen RA and ELITEDESK has the
|
|
plugin candidate. Neither implementation is installed or changed by this tool.
|
|
|
|
The operator confirmed that this lab has no fencing; fencing is available only
|
|
in production. This checkpoint covers controlled, manual role transitions with
|
|
a reachable primary. Abrupt-primary-loss automatic failover is not validated
|
|
for the plugin candidate. The earlier production fencing report applies to the
|
|
earlier implementation, not this extraction.
|
|
|
|
During maintenance, Pacemaker's recurring role monitors are paused. The harness
|
|
invokes the RA directly, so its action results do not update Pacemaker's recorded
|
|
operation history. A Cockpit cluster role display can therefore remain stale
|
|
even while the RA explicitly updates promotion scores. Database roles were
|
|
verified using direct SQL; systemd/container running state remains distinct
|
|
from PostgreSQL role. Maintenance was not disabled to refresh the display.
|
|
|
|
`scripts/test-lab-roles.py` is a portable, operator-run harness with embedded
|
|
hashes from both lab snapshots. It needs root locally and noninteractive SSH
|
|
as david to the peer for read-only observations. SSH host keys must already be
|
|
verified. Passwords are entered in the operator's terminal, never in chat.
|
|
|
|
Run `sudo python3 /path/to/test-lab-roles.py inspect` on each node first.
|
|
Inspection checks cluster name, membership, maintenance, resource parameters,
|
|
installed hashes, SQL readiness, roles, system identity and maintenance service.
|
|
It changes no database or cluster state. Do not proceed if either check fails.
|
|
|
|
Each stage requires a separate invocation and review of its successful output:
|
|
|
|
| Order | Host | Stage argument | Expected result |
|
|
| --- | --- | --- | --- |
|
|
| 1 | DV6 | demote | DV6 streams from ELITEDESK; both are standbys |
|
|
| 2 | ELITEDESK | promote | ELITEDESK primary, DV6 streaming standby |
|
|
| 3 | ELITEDESK | demote | ELITEDESK streams from DV6; both are standbys |
|
|
| 4 | DV6 | promote | DV6 primary, ELITEDESK streaming standby |
|
|
| 5 | ELITEDESK | rejoin | Stop PostgreSQL only, run rewind-only rejoin to DV6, verify streaming |
|
|
|
|
The demote stage submits a **synthetic lab pre-demote notification** through the
|
|
RA notify action, then invokes demote. This tests the notification consumer and
|
|
controlled demotion; it does not prove Pacemaker generates/orders notifications.
|
|
The RA retains its own catch-up and successor checks. The harness refuses promote
|
|
unless the peer is already in recovery and streaming from the local standby.
|
|
Between demote and promote there is deliberately no writable primary. Arrange
|
|
both terminals before starting; do not perform other manual database operations
|
|
during the sequence. Scores on both standbys can fluctuate during this interval.
|
|
|
|
The rejoin stage is restricted to the restored ELITEDESK standby. It explicitly
|
|
sets basebackup fallback=false. It tests the stopped-PostgreSQL precondition,
|
|
maintenance container survival, source authentication, pg_rewind invocation,
|
|
recovery configuration, PostgreSQL restart, streaming and score restoration.
|
|
Because it does not create divergent data, it does **not** validate divergent
|
|
former-primary rewind repair. No fencing, data replacement fallback, VIP moves,
|
|
Pacemaker-managed switchover or automatic failover are tested here.
|
|
|
|
Each modifying invocation writes root-only results/action logs beneath
|
|
`/var/backups/managed-container-roles-lab-*`. Failures halt without automatic
|
|
retry/recovery. A failure after demotion can leave no primary; a rejoin failure
|
|
can leave ELITEDESK stopped. Stop the sequence and inspect the evidence rather
|
|
than running the next stage. Do not clear failed actions or change maintenance.
|
|
|
|
The checks sample state; they cannot prevent a separate administrator from
|
|
changing roles concurrently. Controlled demotion of a reachable primary is the
|
|
basis for these manual lab promotions. This is not a procedure for promoting
|
|
after an unreachable primary failure; that requires fencing and Pacemaker's
|
|
ordering under the operating policy.
|
|
|
|
## Completed live sequence
|
|
|
|
The operator supplied successful output for each stage below. Evidence directories
|
|
are on the host named in the table, under `/var/backups/`. Codex has not read the
|
|
root-only results.json or action logs; these are operator-reported passes, not
|
|
independently witnessed live actions.
|
|
|
|
| Host | Stage | Result | Evidence directory |
|
|
| --- | --- | --- | --- |
|
|
| DV6 | demote | rc=0; standby monitor=0; score=100 | managed-container-roles-lab-o1xjobi2 |
|
|
| ELITEDESK | promote | rc=0; primary monitor=8; score=1000 | managed-container-roles-lab-qdz57g70 |
|
|
| ELITEDESK | demote | rc=0; standby monitor=0; score=100 | managed-container-roles-lab-nwbzdxnl |
|
|
| DV6 | promote | rc=0; primary monitor=8; score=1000 | managed-container-roles-lab-r69amyxg |
|
|
| ELITEDESK | rejoin | stopped monitor=7; rejoin=0; standby monitor=0; score=100 | managed-container-roles-lab-7c4vw7_p |
|
|
|
|
Each stage reported preserved database identity and maintenance container,
|
|
verified roles and streaming, and maintenance=true. The final reported state is
|
|
DV6 primary and ELITEDESK streaming standby. The rejoin harness also checks the
|
|
stopped score is -INFINITY before invoking rejoin, with basebackup disabled.
|
|
This establishes successful pg_rewind invocation, not repair of deliberately
|
|
divergent timelines or a measured amount of rewound data.
|
|
|
|
The supplied inspection after ELITEDESK demotion showed both nodes in recovery,
|
|
streaming from each other, with replay LSN 0/A02B560 and system identifier
|
|
7684451577990111248. An earlier inspection in the same paste showed DV6 primary;
|
|
the later observation was used to resolve the role confusion before restoring DV6.
|
|
|
|
Both-way SSH access was established and independently checked before the role
|
|
sequence. The installed code was pinned to the existing lab snapshots. No RA
|
|
installation, maintenance change, fencing, VIP move or failure cleanup was part
|
|
of these tests. The limitations above continue to apply.
|
|
|
|
Offline original-function comparisons and 97 tests passed before the live
|
|
sequence (the 11 scheduler tests required execution outside the sandbox).
|
|
Six additional role-harness safety-gate tests passed before staging the harness.
|
|
|
|
## Commit checkpoint verification
|
|
|
|
On 2026-09-13, all 103 tests in `python3 -m unittest discover -s tests -p
|
|
'test_*.py'` passed, including offline scheduler simulations.
|
|
`python3 tests/verify_refactor.py` also passed: frozen reference hashes, all 29
|
|
original functions after exact adapter-call reversal, syntax, metadata and
|
|
22 differential lifecycle cases. No additional live actions were run for this
|
|
commit checkpoint.
|