david/ocf-managed-container
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Freeze HA scoring and document reported production failover

Browse Source
This commit is contained in:
Lyncoln
2026-09-12 18:07:28 -04:00
parent c8636ed6d4
commit 488a1672d7
29 changed files with 2285 additions and 46 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/operations.md
+130
View File
@@ -0,0 +1,130 @@
# PostgreSQL HA operating policy
This document describes the intentional production design. Historical two-node
lab snapshots and old installation notes are not production runbooks. Keep the
three-node production clone's **target-role=Promoted** and **promoted-max=1**.
Promotion scores select an eligible node; the target role expresses the desired
promoted service. This project requires that explicit configuration even though
Pacemaker can permit promotion with other default target-role settings.
The VIP must be colocated with the Promoted instance and ordered to start after
promotion, with reverse shutdown ordering. Scores do not create those constraints.
## Ownership and lifecycles
Pacemaker invokes the root OCF RA. Its `runuser` boundary enters the rootless
service owner's home and user-systemd runtime, then invokes systemd/Podman with
argument arrays. Quadlet constructs the maintenance container and supplies its
environment, mounts and bootstrap. The bootstrap gates initialization/adoption,
then executes `sleep infinity`; Podman may use an init shim as PID 1. PostgreSQL
is not the container lifetime process.
The RA independently controls PostgreSQL with `pg_ctl`. During demotion and
rejoin, the maintenance container stays available even when PostgreSQL is
stopped. This is required for inspection, `pg_rewind`, and optional basebackup.
The OCF stop action stops both PostgreSQL and the maintenance service/container.
Starting the container alone is not equivalent to starting the OCF resource.
The RA reads PGDATA and intentional replication settings from the running
container. It does not parse or source host PostgreSQL env/config files. Quadlet
may intentionally consume a host env-file when constructing that container.
The bootstrap manages its secret-file/pgpass interfaces separately; the RA's
replication commands read `POSTGRES_REPLICATION_PASSWORD` inside the container
and set PGPASSWORD only for the child process.
## Actions and role reporting
| Action | Implementation contract |
| --- | --- |
| start | Starts or verifies an Unpromoted standby. Refuses an already-primary instance; when PostgreSQL is stopped, refuses PGDATA without standby.signal. |
| monitor/status | Reports healthy standby 0, healthy primary 8, stopped service/container/PostgreSQL 7, and unhealthy/failed query 1. Promotion eligibility does not change these role codes. |
| promote | Promotes a healthy standby and reconciles local HA slots/rewind grants; an existing primary reconciles those invariants. |
| demote | Controlled switchover: uses fresh pre-demote notification and Corosync successor address, verifies successor standby/catch-up, stops local PostgreSQL and rejoins it as a cascading standby. |
| stop | Stops PostgreSQL, then the maintenance service/container. |
| notify | Persists Pacemaker transition context; not promotion eligibility or fencing proof. |
| rejoin | Private/manual recovery action, deliberately absent from advertised OCF actions. Requires a running maintenance container, stopped PostgreSQL, source and slot. Rewinds then verifies streaming recovery. |
| validate-all/meta-data | Validate dependencies/configuration and report metadata. Validation can return 5 for missing utilities or 6 for invalid configuration. |
Lifecycle success is 0; monitor's 8 is a role report, not a generic lifecycle
success. Promoted monitoring must expect the correct role. Metadata advertises
13 parameters and eight actions; status is an alias and rejoin stays private.
The scoring wrappers withdraw scores before lifecycle actions and reassess after
successful actions. They do not change the original 29 function bodies.
## Scoring and fencing
Healthy primary: 1000. Strictly verified streaming standby: 100. Recently verified
eligible disconnected standby: 50. Unsafe or missing evidence: -INFINITY.
Disconnected evidence is bounded by `disconnected_max_age` (default 300 seconds),
bound to the PostgreSQL incarnation and local boot, and recalculated each monitor.
See [the detailed state model](../notes/DISCONNECTED-STANDBY.md).
An eligible score is not a fencing certificate. Pacemaker/SBD must isolate the
failed primary and order promotion safely. Never bypass fencing by manually
calling promote, creating standby.signal on divergent data, or deleting data.
Asynchronous replication accepts possible loss of acknowledged transactions;
the freshness window does not bound the number of lost transactions.
Controlled switchover differs from abrupt failover: demote requires a reachable,
caught-up successor and transition notification. Former-primary rejoin is a
separate recovery procedure after fencing, not an automatic start workaround.
## Former-primary recovery order
1. Establish that fencing completed and identify the surviving writable primary
and its VIP. Coordinate cluster management so Pacemaker cannot race the manual
repair. Do not clear the former primary's failed start yet.
2. Bring up only its maintenance container using the existing Quadlet as the
service owner. Verify PostgreSQL is stopped. Preserve the existing data.
3. Confirm the running container's replication credentials are correct without
printing them. Supply the existing resource instance/parameters and a unique
local replication slot to the private rejoin action, using the PostgreSQL VIP
as `rejoin_source_host`. This documentation does not invent missing live IDs.
4. Rejoin first attempts pg_rewind. If it fails, stop and inspect the error.
Basebackup fallback remains disabled unless explicitly authorized to replace
divergent data. It preserves the old directory but is still a destructive
recovery decision; a failed rewind is not guaranteed to leave usable data.
5. Require rejoin success AND verify `pg_is_in_recovery() = true`, a streaming WAL
receiver connected to the intended primary, and the expected slot/application
identity. Verify the maintenance container remained available.
6. **Only now** clear the relevant failed actions for that resource on that node,
and allow Pacemaker to adopt it as Unpromoted. Recheck roles, scores, streaming,
VIP placement, and failed actions. Do not use broad cleanup before recovery.
Safe inspection commands, run on the appropriate node, include:
```sh
sudo pcs status --full
sudo pcs resource config postgresql-ha-clone
sudo pcs constraint config --full
```
Inspect output locally; raw CIB, environment dumps and logs may contain sensitive
configuration and should not be committed. The optional preference planner reads
an operator-supplied CIB and prints only a narrowly matched temporary preference
removal. It preserves target-role=Promoted and all unrelated constraints. It does
not perform resource failure cleanup or make a node safe to rejoin.
## Podman env-file incident
The production test exposed a quoting mismatch. Podman's env-file parser retained
surrounding quote characters, while the generated pgpass had the intended value.
Streaming worked, but rejoin's direct environment password failed authentication.
These are intentionally fake examples, not deployment values:
```text
# Incorrect if the intended value contains no quotation marks:
POSTGRES_REPLICATION_PASSWORD="EXAMPLE_ONLY_NOT_A_SECRET"
# Correct representation of that same intended value:
POSTGRES_REPLICATION_PASSWORD=EXAMPLE_ONLY_NOT_A_SECRET
```
Podman env-file syntax is not shell assignment syntax; do not source it to
validate container behavior. Correct the secret locally and apply it through the
planned Quadlet/container lifecycle: editing a file alone does not alter an
already-running container's environment. Do not blindly restart a primary to
apply this change. Keep real values, .pgpass contents, secret files, and raw
container environment inspection out of Git and shared logs.
References: [OCF role semantics](https://github.com/ClusterLabs/pacemaker/blob/main/doc/sphinx/Pacemaker_Administration/agents.rst),
[Pacemaker configuration and ordering](https://clusterlabs.org/projects/pacemaker/doc/3.0/Pacemaker_Explained/pdf/Pacemaker_Explained.pdf),
and [Podman run options](https://docs.podman.io/en/latest/markdown/podman-run.1.html).
docs/review.md
+115
View File
@@ -0,0 +1,115 @@
# Freeze review — 2026-09-12
This review preserves the implementation reported as production-tested by the
operator. No production systems were changed. It is a source/documentation
freeze, not certification of every failure mode or a byte-level deployment
attestation. Production hash collection remains outstanding in [testing.md](testing.md).
## Corrections and their reasons
- Added operations.md to distinguish container lifecycle, PostgreSQL lifecycle,
Pacemaker state, scores, fencing, controlled switchover and former-primary rejoin.
- Added testing.md to record the operator's SBD failover/VIP/rewind/rejoin sequence
and final three-node roles/scores, separately from local automated checks.
Recorded exact repository RA/module/helper/bootstrap hashes without inventing
deployed hashes. Included read-only per-node hash collection commands.
- Required target-role=Promoted, promoted-max=1, Promoted-role VIP colocation and
promotion-before-VIP-start ordering in the operating policy. Compatibility
tests for absent/Started target roles are not recommendations to change production.
- Fixed plan-promotion-cleanup.py to emit only temporary node-preference removal
and preserve target-role=Promoted. Removed its obsolete inherited-Started
check because it no longer clears the direct target role. Updated its exact
expected-output test; ambiguity and unrelated-rule protections remain. Added
an explicit Promoted-target scheduler case and used that target in fencing
scenarios; compatibility cases remain.
- Explicitly prohibited clearing former-primary failed actions before successful
rejoin and verification of streaming recovery. Documented rewind before any
explicitly authorized basebackup fallback, and that failed rewind may require
further recovery rather than a normal restart.
- Documented literal env-file quotes and the mismatch between generated pgpass
and the RA's direct container environment password. All examples use obvious
fake values; no actual password from the handoff was copied into the repository.
- Updated README and historical handoffs/notes to point to current operations
and evidence. Corrected the PID 1 simplification: bootstrap executes the
maintenance command, possibly behind Podman's init shim.
- Included current operating/testing/review documents in source/RPM documentation.
Retained the old ELITEDESK installer as a clearly marked historical lab tool:
its hostname/cluster/hash guards reject the scoring implementation. It is not
a production installer and was not executed.
- Ignored generated review patches; the commit records the actual files instead
of duplicating obsolete proposed commands in patch attachments. Refreshed the
checksum manifest. Existing local patch files remain available locally.
The staged scoring, state helper, tests and RPM workflow from the preceding
implementation work are included in this freeze commit. This review did not
modify RA/module/scoring-helper or bootstrap code. The frozen original remains
unchanged, and all 29 original lifecycle bodies still compare byte-for-byte.
## Checks completed in this review
- Structural/default/metadata verification and all 22 differential lifecycle cases.
- 25 promotion-scoring tests, 16 disconnected-state/monitor tests, 11 synthetic
Pacemaker scheduler tests and 4 corrected preference-planner tests: all pass.
- Nine SQL scenarios against a disposable PostgreSQL 18 container: all pass for
the original strict predicate and current observation query. Container removed.
- Bash syntax and Python AST parsing across repository code and historical installer:
pass. ShellCheck and Gitleaks were not installed; no claim of running them is made.
- Repository/diff review for private keys, recognizable tokens, credential-bearing
URLs, and credential assignments, plus manual inspection of matches: no actual
secrets found. Matches were variable references, redacted snapshots, or explicit
fake examples. No CIB, secret file, .pgpass content or raw environment dump was added.
- A disposable Podman test reproduced literal env-file quotes using only a fake
value. No production credentials or environments were inspected.
- Fedora 43 development RPM build with the new docs passed; artifacts are in
dist/documentation-freeze/ and the log is /tmp/mc-freeze-build.log. The extra
Promoted-target scheduler case was subsequently verified separately.
- Checksum manifest and git diff whitespace checks are verified immediately before
commit.
## Privilege and safety boundaries reviewed
The RA does not source host-side PostgreSQL environment/configuration. Its
root-to-service-user boundary preserves argument arrays, and container commands
consume intentional injected settings. Authentication secrets are not placed in
RA command arguments or host-side pgpass files. The stored bootstrap is separate
and has a different provisioning interface.
The scoring helper uses bounded subprocesses, a root-only runtime directory,
strictly parsed JSON, per-resource locks and atomic replacement. Its identity
includes postmaster start time, so a restarted/replaced data instance must stream
successfully before receiving disconnected eligibility. Notification state uses
a separate root-owned /run directory and relies on Pacemaker serializing actions
for the same resource; it is not a distributed locking or fencing mechanism.
Original SQL/shell construction in some lifecycle helpers assumes trusted,
conventional resource configuration (identifiers, paths, connection fields).
It is not a general escaping layer for arbitrary user-supplied text. Restrict CIB
write access and container ownership; do not expose RA parameters to untrusted
input. No new untrusted input interface was introduced by this review.
## Remaining concerns and explicit limits
- The three production nodes' exact deployed hashes and confirmation that those
bytes were used in the fencing test are missing. Source hashes are not a
substitute. The runtime-context bootstrap may differ from deployed production.
- The saved bootstrap's fresh-primary provisioning path passes a replication
password through a psql --set argument. That can expose it to process-argument
inspection under local /proc permissions. This is a concrete existing exposure,
not a claim that the RA rejoin path leaks a password. It must be reviewed against
the actual deployed bootstrap before calling provisioning secret-safe; no
bootstrap rewrite or production secret change was made in this freeze.
- Existing bootstrap pgpass generation assumes values needing no additional
colon/backslash escaping. Do not treat the documented quote correction as a
universal password-encoding solution. Inspect the deployed bootstrap before
changing provisioning behavior.
- Scores are observations, not fencing authorization or atomic promotion guards.
IPC failure can leave a prior score; monitor cadence/fencing latency can exceed
freshness. Two disconnected candidates are not ranked by cross-node WAL history.
- Asynchronous replication can lose acknowledged transactions. Neither the test
nor the freshness threshold proves a numerical data-loss bound.
- The reported fenced-failure/rejoin success does not prove every concurrent-write
controlled-switchover scenario or optional destructive basebackup fallback.
- License provenance remains unresolved as documented in packaging/PROVENANCE.md.
This source freeze is not a license grant or public-release authorization.
No push or production rollout is part of this review.
docs/testing.md
+122
View File
@@ -0,0 +1,122 @@
# Validation and freeze-point evidence
## Reported production fencing and recovery test
Evidence source: operator-supplied production handoff received during the
2026-09-12 documentation review. The operator reports a successful real
three-node Pacemaker/Corosync test with SBD fencing. Codex did not execute or
independently witness this production test. Its exact start/end timestamps and
raw cluster logs were not supplied; no timestamps or measurements are invented.
1. david-eve was primary and was fenced successfully through SBD.
2. david-prodesk promoted automatically; the PostgreSQL VIP moved with it.
3. david-macbookpro remained/reconnected as a streaming standby.
4. david-eve rebooted with former-primary PGDATA. Ordinary resource start
correctly refused it because standby.signal was absent.
5. The operator manually started its maintenance container with PostgreSQL stopped.
6. The private rejoin action used the PostgreSQL VIP as rewind source.
pg_rewind detected divergence, found the common checkpoint and completed.
7. david-eve started as a streaming standby. Only after recovery, Pacemaker
cleanup allowed the cluster to adopt it as Unpromoted.
Reported final state:
| Node | Role | Score |
| --- | --- | ---: |
| david-prodesk | Promoted; PostgreSQL VIP local | 1000 |
| david-macbookpro | Unpromoted; asynchronous streaming | 100 |
| david-eve | Unpromoted; asynchronous streaming | 100 |
No PostgreSQL failed actions remained. The successful run demonstrates the
reported fencing/failover/VIP/rejoin sequence, not every possible partition,
latency, concurrent-write, or recovery failure. The intermediate disconnected
score and lost-transaction count were not supplied. The quoting/authentication
incident and required fix are documented in [operations](operations.md).
## Validated production snapshot — exact deployed hashes pending
The production outcome is operator-reported validated; **the byte-level production
snapshot is not yet established**. Exact hashes from david-prodesk,
david-macbookpro and david-eve have not been supplied or collected. Repository
hashes below must not be represented as deployed hashes. A later capture also
requires operator confirmation that those bytes are the ones used during the test.
| Artifact on each production node | Snapshot status |
| --- | --- |
| /usr/lib/ocf/resource.d/loxthelion/managed-container | Pending production SHA-256 |
| managed-container.d/*.sh (all nine modules) | Pending production SHA-256 |
| managed-container.d/promotion_state.py | Pending production SHA-256 |
| Running container /usr/local/sbin/postgres-ha-bootstrap | Pending production SHA-256 |
Run the following read-only commands on **each production node**, using the
rootless container owner for podman. They hash code, not environment or secrets:
```sh
hostname
date -u '+%Y-%m-%dT%H:%M:%SZ'
sha256sum /usr/lib/ocf/resource.d/loxthelion/managed-container \
/usr/lib/ocf/resource.d/loxthelion/managed-container.d/*.sh \
/usr/lib/ocf/resource.d/loxthelion/managed-container.d/promotion_state.py
podman exec postgres sha256sum /usr/local/sbin/postgres-ha-bootstrap
podman inspect --format '{{.Image}}' postgres
```
If permissions require it, run only the host sha256sum command with sudo. Do not
run rootful podman instead of the actual container owner. If the installed
bootstrap path/container name differs, record the actual path/name. Capture
node identity, UTC capture time, all filenames/digests, immutable image ID, and
whether any code changed since the test. Hash the running container's bootstrap;
a host snapshot or image tag alone is not equivalent. Do not dump environment,
secret files, or pgpass. Compare every node individually, preserving mismatches.
## Repository freeze snapshot
The exact repository code hashes appear below and in the reviewed checksum
manifest. These identify the source freeze only, pending production comparison.
The bootstrap is a saved runtime-context artifact, not part of the RA RPM and
not yet confirmed identical to the production bootstrap.
| Repository file | SHA-256 |
| --- | --- |
| `refactored/managed-container` | `a4b775a0ddcabdb0558da9b34c7fe2a883c07d0ef308c1c7224526abf53f63a0` |
| `refactored/managed-container.d/auth.sh` | `93191771d3cb8aefadb2a16d2f89d64a8b6c584071a473439be818237bdf7f17` |
| `refactored/managed-container.d/common.sh` | `97e93d9164e854e5f3688cf9d3cc656b001eb30ef3c2073c907450ca43588f3d` |
| `refactored/managed-container.d/container.sh` | `ac6bc363a2301f1491d162ef609103dbfbceb3b099a3556614c6165952309374` |
| `refactored/managed-container.d/demote.sh` | `0af3bb362a303017372c631354ba94ef9e723bf14e64d2eff91b6662e54a5be0` |
| `refactored/managed-container.d/notify.sh` | `6a98fcdda82cea9365db95b4ee779be5c567b3d846f48b90fbc8c254af697670` |
| `refactored/managed-container.d/postgres.sh` | `d2779ce6e7d1f426ffca2e41bbb8de77503005dc38d658d69f184020365796eb` |
| `refactored/managed-container.d/promote.sh` | `3085ec9b4410905ab9d4370786ed776eac3470a0415aa1b4f608b083945aeb58` |
| `refactored/managed-container.d/rejoin.sh` | `21beea72bcb7f24a51c17ea48666ac87df9436702ad77b166b45f4647e28e8ad` |
| `refactored/managed-container.d/replication.sh` | `6b2021887e3b4cd6a12490e7a8dff98320b2133b472ab25a22a0eead46137547` |
| `refactored/managed-container.d/promotion_state.py` | `5166fc3370f49ab14cd975ded4b03a74b9bd7dbff5762b41a15552c42f934ea7` |
| `runtime-context/postgres-ha-bootstrap` | `3f1a15f7054fb69b9aa92204de30306fd6053c63560310f4231f314b148e2dcf` |
## Non-production checks
The source retains all 29 original lifecycle function bodies and original
parameter defaults. Added scoring changes public dispatch and metadata; it is
not a claim of identical total behavior to the monolithic reference.
Run from the repository root:
```sh
python3 tests/verify_refactor.py
python3 tests/test_promotion_scores.py
python3 tests/test_disconnected_state.py
python3 tests/test_promotion_scheduler.py
python3 tests/test_promotion_cleanup.py
sha256sum -c notes/REFACTOR-SHA256SUMS
git diff --check
```
The baseline suite covers 22 differential lifecycle cases, 25 scoring tests,
16 state/monitor tests, 11 real scheduler simulations and 4 planner tests.
Scheduler failure simulations use synthetic CIBs, enable fencing, and assert
fencing prerequisites for promotion. They do not operate a live cluster.
`tests/verify_promotion_sql.py` explicitly creates/removes a disposable PostgreSQL
18 container and checks nine strict/observation SQL scenarios. Previous candidate
RPM build/install and root-helper CLI verification passed in disposable Fedora
containers. They are separate from the operator's production validation.
Current freeze review results are recorded in [review.md](review.md).