Files
ocf-managed-container/docs/testing.md
T
Lyncoln ec2077f979 Freeze lab-tested PostgreSQL plugin engine
Extract PostgreSQL, Podman runtime and systemd lifecycle adapters while preserving existing lifecycle and scoring behavior. Update RPM layout and add guarded operator-run lab harnesses.

Record successful manual stop/start, promotion, demotion and rewind-only rejoin with pinned installed hashes. The unfenced lab does not validate automatic failover, Pacemaker-managed transitions or divergent-data rewind.

Validation: 103 offline tests, original-function differential checks, checksum manifest and whitespace checks passed.
2026-09-13 01:57:04 -04:00

6.8 KiB

These production reports and source hashes describe the 488a167 freeze. The current undeployed extraction is described in plugin-engine.md.

Validation and freeze-point evidence

Reported production fencing and recovery test

Evidence source: operator-supplied production handoff received during the 2026-09-12 documentation review. The operator reports a successful real three-node Pacemaker/Corosync test with SBD fencing. Codex did not execute or independently witness this production test. Its exact start/end timestamps and raw cluster logs were not supplied; no timestamps or measurements are invented.

  1. david-eve was primary and was fenced successfully through SBD.
  2. david-prodesk promoted automatically; the PostgreSQL VIP moved with it.
  3. david-macbookpro remained/reconnected as a streaming standby.
  4. david-eve rebooted with former-primary PGDATA. Ordinary resource start correctly refused it because standby.signal was absent.
  5. The operator manually started its maintenance container with PostgreSQL stopped.
  6. The private rejoin action used the PostgreSQL VIP as rewind source. pg_rewind detected divergence, found the common checkpoint and completed.
  7. david-eve started as a streaming standby. Only after recovery, Pacemaker cleanup allowed the cluster to adopt it as Unpromoted.

Reported final state:

Node Role Score
david-prodesk Promoted; PostgreSQL VIP local 1000
david-macbookpro Unpromoted; asynchronous streaming 100
david-eve Unpromoted; asynchronous streaming 100

No PostgreSQL failed actions remained. The successful run demonstrates the reported fencing/failover/VIP/rejoin sequence, not every possible partition, latency, concurrent-write, or recovery failure. The intermediate disconnected score and lost-transaction count were not supplied. The quoting/authentication incident and required fix are documented in operations.

Validated production snapshot — exact deployed hashes pending

The production outcome is operator-reported validated; the byte-level production snapshot is not yet established. Exact hashes from david-prodesk, david-macbookpro and david-eve have not been supplied or collected. Repository hashes below must not be represented as deployed hashes. A later capture also requires operator confirmation that those bytes are the ones used during the test.

Artifact on each production node Snapshot status
/usr/lib/ocf/resource.d/loxthelion/managed-container Pending production SHA-256
managed-container.d/*.sh (all nine modules) Pending production SHA-256
managed-container.d/promotion_state.py Pending production SHA-256
Running container /usr/local/sbin/postgres-ha-bootstrap Pending production SHA-256

Run the following read-only commands on each production node, using the rootless container owner for podman. They hash code, not environment or secrets:

hostname
date -u '+%Y-%m-%dT%H:%M:%SZ'
sha256sum /usr/lib/ocf/resource.d/loxthelion/managed-container \
  /usr/lib/ocf/resource.d/loxthelion/managed-container.d/*.sh \
  /usr/lib/ocf/resource.d/loxthelion/managed-container.d/promotion_state.py
podman exec postgres sha256sum /usr/local/sbin/postgres-ha-bootstrap
podman inspect --format '{{.Image}}' postgres

If permissions require it, run only the host sha256sum command with sudo. Do not run rootful podman instead of the actual container owner. If the installed bootstrap path/container name differs, record the actual path/name. Capture node identity, UTC capture time, all filenames/digests, immutable image ID, and whether any code changed since the test. Hash the running container's bootstrap; a host snapshot or image tag alone is not equivalent. Do not dump environment, secret files, or pgpass. Compare every node individually, preserving mismatches.

Repository freeze snapshot (488a167)

The exact repository code hashes appear below and in the reviewed checksum manifest. These identify the source freeze only, pending production comparison. The bootstrap is a saved runtime-context artifact, not part of the RA RPM and not yet confirmed identical to the production bootstrap.

Repository file SHA-256
refactored/managed-container a4b775a0ddcabdb0558da9b34c7fe2a883c07d0ef308c1c7224526abf53f63a0
refactored/managed-container.d/auth.sh 93191771d3cb8aefadb2a16d2f89d64a8b6c584071a473439be818237bdf7f17
refactored/managed-container.d/common.sh 97e93d9164e854e5f3688cf9d3cc656b001eb30ef3c2073c907450ca43588f3d
refactored/managed-container.d/container.sh ac6bc363a2301f1491d162ef609103dbfbceb3b099a3556614c6165952309374
refactored/managed-container.d/demote.sh 0af3bb362a303017372c631354ba94ef9e723bf14e64d2eff91b6662e54a5be0
refactored/managed-container.d/notify.sh 6a98fcdda82cea9365db95b4ee779be5c567b3d846f48b90fbc8c254af697670
refactored/managed-container.d/postgres.sh d2779ce6e7d1f426ffca2e41bbb8de77503005dc38d658d69f184020365796eb
refactored/managed-container.d/promote.sh 3085ec9b4410905ab9d4370786ed776eac3470a0415aa1b4f608b083945aeb58
refactored/managed-container.d/rejoin.sh 21beea72bcb7f24a51c17ea48666ac87df9436702ad77b166b45f4647e28e8ad
refactored/managed-container.d/replication.sh 6b2021887e3b4cd6a12490e7a8dff98320b2133b472ab25a22a0eead46137547
refactored/managed-container.d/promotion_state.py 5166fc3370f49ab14cd975ded4b03a74b9bd7dbff5762b41a15552c42f934ea7
runtime-context/postgres-ha-bootstrap 3f1a15f7054fb69b9aa92204de30306fd6053c63560310f4231f314b148e2dcf

Non-production checks

The frozen source retained all 29 original lifecycle function bodies and original parameter defaults. Added scoring changes public dispatch and metadata; it is not a claim of identical total behavior to the monolithic reference.

Run from the repository root:

python3 tests/verify_refactor.py
python3 tests/test_promotion_scores.py
python3 tests/test_disconnected_state.py
python3 tests/test_promotion_scheduler.py
python3 tests/test_promotion_cleanup.py
sha256sum -c notes/REFACTOR-SHA256SUMS
git diff --check

The baseline suite covers 22 differential lifecycle cases, 25 scoring tests, 16 state/monitor tests, 11 real scheduler simulations and 4 planner tests. Scheduler failure simulations use synthetic CIBs, enable fencing, and assert fencing prerequisites for promotion. They do not operate a live cluster.

tests/verify_promotion_sql.py explicitly creates/removes a disposable PostgreSQL 18 container and checks nine strict/observation SQL scenarios. Previous candidate RPM build/install and root-helper CLI verification passed in disposable Fedora containers. They are separate from the operator's production validation. Current freeze review results are recorded in review.md.