Files
ocf-managed-container/docs/lab-plugin-basic-test.md
Lyncoln ec2077f979 Freeze lab-tested PostgreSQL plugin engine
Extract PostgreSQL, Podman runtime and systemd lifecycle adapters while preserving existing lifecycle and scoring behavior. Update RPM layout and add guarded operator-run lab harnesses.

Record successful manual stop/start, promotion, demotion and rewind-only rejoin with pinned installed hashes. The unfenced lab does not validate automatic failover, Pacemaker-managed transitions or divergent-data rewind.

Validation: 103 offline tests, original-function differential checks, checksum manifest and whitespace checks passed.
2026-09-13 01:57:04 -04:00

6.2 KiB

ELITEDESK plugin installation and basic checks

Status: installed on DAVID-ELITEDESK; operator-reported basic checks passed. All 15 installed code files were subsequently verified read-only against the working candidate and matched byte-for-byte. No production or peer-node actions are part of this procedure.

Read-only preflight on DAVID-ELITEDESK found its PostgreSQL container running as an asynchronous streaming standby of 192.168.1.2 using ha_david_elitedesk. All 11 installed RA/module/helper files match source freeze 488a167; their exact hashes are captured in lab-installed-before-plugin.sha256. This is a lab snapshot, not the outstanding production snapshot. Current cluster maintenance state could not be read without sudo; the installer verifies it live before any replacement.

Run in the operator's terminal:

sudo python3 /home/david/codex-postgres-ha-refactor/scripts/test-elitedesk-plugin.py

The command requires the existing postgres-ra-test cluster, exactly the two lab nodes, pg-ha under pg-ha-clone, expected local container/database settings, and maintenance-mode already true. It does not enable maintenance mode or modify CIB configuration. It copies parameters from the live resource and rejects ambiguous parameter sets. Native sudo prompts stay in the terminal; never share a password.

It verifies source checksums, stages root-owned code, backs up the installed agent and modules under /var/backups/managed-container-plugin-lab-*, validates staged metadata and configuration, then replaces the complete module tree and entrypoint. The installation remains in place after success. A failed post-install check restores the old agent and module tree and retains diagnostic files.

Basic checks are metadata (16 parameters), validate-all, monitor, status, an idempotent start on the already-running standby, and another monitor. Before and after checks require the same PostgreSQL system identifier, postmaster start time, recovery role, streaming sender and slot. No stop, promote, demote, rejoin or fencing is invoked. Normal monitors/start may refresh local promotion evidence and the pg-ha promotion score; those expected side effects are part of testing. A score of -INFINITY is recorded rather than mistaken for a failed role monitor; strict catch-up can fluctuate. Investigate that result before broader HA testing.

The script prints its backup/result directory and action results. It records results.json and installed-SHA256SUMS there with root-only access. Return the terminal output before progressing to disruptive stop/start or switchover tests.

Preparation validation: current structural/differential tests passed; two isolated installer tests passed, covering basic-action-only success and restoration of both agent and modules on an injected installed-monitor failure. These tests mock privilege/cluster boundaries and touch only temporary directories; they do not claim live installation success.

Completed basic lab run

The operator supplied successful installer output for backup/result directory /var/backups/managed-container-plugin-lab-mow_t6zi:

  • Staged metadata and validate-all: rc=0.
  • Installed validate-all, monitor, status, idempotent start, monitor: rc=0.
  • Final promotion score: 100.
  • The installer reported PostgreSQL was not restarted and maintenance remained true.
  • No stop, promote, demote, rejoin, fencing, or peer-node actions ran.

Codex independently compared every installed entrypoint/module/helper file with the working candidate using read-only filesystem access; all 15 matched. Exact installed SHA-256 values are recorded in lab-installed-plugin.sha256. The root-only results.json was not independently read; the action results above are attributed to the operator-supplied output. This validates the installation and non-disruptive standby path, not actual stop/start, switchover or failover.

Installed-file comparison captured at 2026-09-13T03:01:11.114949+00:00 UTC.

Controlled standby stop/start — operator-reported pass

The operator ran:

sudo python3 /home/david/codex-postgres-ha-refactor/scripts/test-elitedesk-cycle.py

This briefly interrupts only the ELITEDESK standby. The script requires the same lab identity, maintenance=true, streaming sender/slot and exact installed hashes. It validates/monitors, runs RA stop, checks systemd inactive and the container stopped/absent, expects monitor=7 and score=-INFINITY, then runs RA start. It requires the same database identifier, a new postmaster start time, recovery and streaming from DV6, monitor/status=0 and score=100 within the verification window. Each RA action has a 60-second timeout with a 5-second kill grace. Post-start streaming/eligibility checks have a 60-second retry window (an in-flight bounded check can finish after the deadline). No source files or cluster configuration are replaced, and no promotion, demotion, rejoin, fencing or peer action runs.

Failures halt without automatic recovery and may leave the standby stopped. Inspect the printed /var/backups/managed-container-cycle-lab-* result directory before proceeding. It contains root-only results.json and per-action output. Three mocked harness tests passed: successful cycle, stop failure without start, and start failure without retry. Those are preparation checks, not live results.

Completed cycle result

The operator supplied successful output with evidence directory /var/backups/managed-container-cycle-lab-b1048r6q:

Check Result
Initial validate-all and monitor rc=0
RA stop rc=0
Monitor while stopped rc=7
Stopped lifecycle state systemd inactive; container stopped/absent
Stopped promotion score -INFINITY
RA start, subsequent monitor and status rc=0
Database identity Same system identifier; new postmaster start time
Replication after restart Streaming from DV6; score 100
Cluster maintenance Remained true

These results are attributed to the operator-supplied script output; Codex has not independently read the root-only results.json. No promotion, demotion, rejoin, fencing or peer-node actions ran. The basic standby lifecycle now has live lab coverage; this does not establish switchover/failover behavior for the plugin extraction or constitute production deployment validation.