david/ocf-managed-container
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Freeze HA scoring and document reported production failover

Browse Source
This commit is contained in:
Lyncoln
2026-09-12 18:07:28 -04:00
parent c8636ed6d4
commit 488a1672d7
29 changed files with 2285 additions and 46 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/operations.md
+130
View File
@@ -0,0 +1,130 @@
# PostgreSQL HA operating policy
This document describes the intentional production design. Historical two-node
lab snapshots and old installation notes are not production runbooks. Keep the
three-node production clone's **target-role=Promoted** and **promoted-max=1**.
Promotion scores select an eligible node; the target role expresses the desired
promoted service. This project requires that explicit configuration even though
Pacemaker can permit promotion with other default target-role settings.
The VIP must be colocated with the Promoted instance and ordered to start after
promotion, with reverse shutdown ordering. Scores do not create those constraints.
## Ownership and lifecycles
Pacemaker invokes the root OCF RA. Its `runuser` boundary enters the rootless
service owner's home and user-systemd runtime, then invokes systemd/Podman with
argument arrays. Quadlet constructs the maintenance container and supplies its
environment, mounts and bootstrap. The bootstrap gates initialization/adoption,
then executes `sleep infinity`; Podman may use an init shim as PID 1. PostgreSQL
is not the container lifetime process.
The RA independently controls PostgreSQL with `pg_ctl`. During demotion and
rejoin, the maintenance container stays available even when PostgreSQL is
stopped. This is required for inspection, `pg_rewind`, and optional basebackup.
The OCF stop action stops both PostgreSQL and the maintenance service/container.
Starting the container alone is not equivalent to starting the OCF resource.
The RA reads PGDATA and intentional replication settings from the running
container. It does not parse or source host PostgreSQL env/config files. Quadlet
may intentionally consume a host env-file when constructing that container.
The bootstrap manages its secret-file/pgpass interfaces separately; the RA's
replication commands read `POSTGRES_REPLICATION_PASSWORD` inside the container
and set PGPASSWORD only for the child process.
## Actions and role reporting
| Action | Implementation contract |
| --- | --- |
| start | Starts or verifies an Unpromoted standby. Refuses an already-primary instance; when PostgreSQL is stopped, refuses PGDATA without standby.signal. |
| monitor/status | Reports healthy standby 0, healthy primary 8, stopped service/container/PostgreSQL 7, and unhealthy/failed query 1. Promotion eligibility does not change these role codes. |
| promote | Promotes a healthy standby and reconciles local HA slots/rewind grants; an existing primary reconciles those invariants. |
| demote | Controlled switchover: uses fresh pre-demote notification and Corosync successor address, verifies successor standby/catch-up, stops local PostgreSQL and rejoins it as a cascading standby. |
| stop | Stops PostgreSQL, then the maintenance service/container. |
| notify | Persists Pacemaker transition context; not promotion eligibility or fencing proof. |
| rejoin | Private/manual recovery action, deliberately absent from advertised OCF actions. Requires a running maintenance container, stopped PostgreSQL, source and slot. Rewinds then verifies streaming recovery. |
| validate-all/meta-data | Validate dependencies/configuration and report metadata. Validation can return 5 for missing utilities or 6 for invalid configuration. |
Lifecycle success is 0; monitor's 8 is a role report, not a generic lifecycle
success. Promoted monitoring must expect the correct role. Metadata advertises
13 parameters and eight actions; status is an alias and rejoin stays private.
The scoring wrappers withdraw scores before lifecycle actions and reassess after
successful actions. They do not change the original 29 function bodies.
## Scoring and fencing
Healthy primary: 1000. Strictly verified streaming standby: 100. Recently verified
eligible disconnected standby: 50. Unsafe or missing evidence: -INFINITY.
Disconnected evidence is bounded by `disconnected_max_age` (default 300 seconds),
bound to the PostgreSQL incarnation and local boot, and recalculated each monitor.
See [the detailed state model](../notes/DISCONNECTED-STANDBY.md).
An eligible score is not a fencing certificate. Pacemaker/SBD must isolate the
failed primary and order promotion safely. Never bypass fencing by manually
calling promote, creating standby.signal on divergent data, or deleting data.
Asynchronous replication accepts possible loss of acknowledged transactions;
the freshness window does not bound the number of lost transactions.
Controlled switchover differs from abrupt failover: demote requires a reachable,
caught-up successor and transition notification. Former-primary rejoin is a
separate recovery procedure after fencing, not an automatic start workaround.
## Former-primary recovery order
1. Establish that fencing completed and identify the surviving writable primary
and its VIP. Coordinate cluster management so Pacemaker cannot race the manual
repair. Do not clear the former primary's failed start yet.
2. Bring up only its maintenance container using the existing Quadlet as the
service owner. Verify PostgreSQL is stopped. Preserve the existing data.
3. Confirm the running container's replication credentials are correct without
printing them. Supply the existing resource instance/parameters and a unique
local replication slot to the private rejoin action, using the PostgreSQL VIP
as `rejoin_source_host`. This documentation does not invent missing live IDs.
4. Rejoin first attempts pg_rewind. If it fails, stop and inspect the error.
Basebackup fallback remains disabled unless explicitly authorized to replace
divergent data. It preserves the old directory but is still a destructive
recovery decision; a failed rewind is not guaranteed to leave usable data.
5. Require rejoin success AND verify `pg_is_in_recovery() = true`, a streaming WAL
receiver connected to the intended primary, and the expected slot/application
identity. Verify the maintenance container remained available.
6. **Only now** clear the relevant failed actions for that resource on that node,
and allow Pacemaker to adopt it as Unpromoted. Recheck roles, scores, streaming,
VIP placement, and failed actions. Do not use broad cleanup before recovery.
Safe inspection commands, run on the appropriate node, include:
```sh
sudo pcs status --full
sudo pcs resource config postgresql-ha-clone
sudo pcs constraint config --full
```
Inspect output locally; raw CIB, environment dumps and logs may contain sensitive
configuration and should not be committed. The optional preference planner reads
an operator-supplied CIB and prints only a narrowly matched temporary preference
removal. It preserves target-role=Promoted and all unrelated constraints. It does
not perform resource failure cleanup or make a node safe to rejoin.
## Podman env-file incident
The production test exposed a quoting mismatch. Podman's env-file parser retained
surrounding quote characters, while the generated pgpass had the intended value.
Streaming worked, but rejoin's direct environment password failed authentication.
These are intentionally fake examples, not deployment values:
```text
# Incorrect if the intended value contains no quotation marks:
POSTGRES_REPLICATION_PASSWORD="EXAMPLE_ONLY_NOT_A_SECRET"
# Correct representation of that same intended value:
POSTGRES_REPLICATION_PASSWORD=EXAMPLE_ONLY_NOT_A_SECRET
```
Podman env-file syntax is not shell assignment syntax; do not source it to
validate container behavior. Correct the secret locally and apply it through the
planned Quadlet/container lifecycle: editing a file alone does not alter an
already-running container's environment. Do not blindly restart a primary to
apply this change. Keep real values, .pgpass contents, secret files, and raw
container environment inspection out of Git and shared logs.
References: [OCF role semantics](https://github.com/ClusterLabs/pacemaker/blob/main/doc/sphinx/Pacemaker_Administration/agents.rst),
[Pacemaker configuration and ordering](https://clusterlabs.org/projects/pacemaker/doc/3.0/Pacemaker_Explained/pdf/Pacemaker_Explained.pdf),
and [Podman run options](https://docs.podman.io/en/latest/markdown/podman-run.1.html).