# PostgreSQL HA operating policy This document describes the intentional production design. Historical two-node lab snapshots and old installation notes are not production runbooks. Keep the three-node production clone's **target-role=Promoted** and **promoted-max=1**. Promotion scores select an eligible node; the target role expresses the desired promoted service. This project requires that explicit configuration even though Pacemaker can permit promotion with other default target-role settings. The VIP must be colocated with the Promoted instance and ordered to start after promotion, with reverse shutdown ordering. Scores do not create those constraints. ## Ownership and lifecycles Pacemaker invokes the root OCF RA. Its `runuser` boundary enters the rootless service owner's home and user-systemd runtime, then invokes systemd/Podman with argument arrays. Quadlet constructs the maintenance container and supplies its environment, mounts and bootstrap. The bootstrap gates initialization/adoption, then executes `sleep infinity`; Podman may use an init shim as PID 1. PostgreSQL is not the container lifetime process. The RA independently controls PostgreSQL with `pg_ctl`. During demotion and rejoin, the maintenance container stays available even when PostgreSQL is stopped. This is required for inspection, `pg_rewind`, and optional basebackup. The OCF stop action stops both PostgreSQL and the maintenance service/container. Starting the container alone is not equivalent to starting the OCF resource. The RA reads PGDATA and intentional replication settings from the running container. It does not parse or source host PostgreSQL env/config files. Quadlet may intentionally consume a host env-file when constructing that container. The bootstrap manages its secret-file/pgpass interfaces separately; the RA's replication commands read `POSTGRES_REPLICATION_PASSWORD` inside the container and set PGPASSWORD only for the child process. ## Actions and role reporting | Action | Implementation contract | | --- | --- | | start | Starts or verifies an Unpromoted standby. Refuses an already-primary instance; when PostgreSQL is stopped, refuses PGDATA without standby.signal. | | monitor/status | Reports healthy standby 0, healthy primary 8, stopped service/container/PostgreSQL 7, and unhealthy/failed query 1. Promotion eligibility does not change these role codes. | | promote | Promotes a healthy standby and reconciles local HA slots/rewind grants; an existing primary reconciles those invariants. | | demote | Controlled switchover: uses fresh pre-demote notification and Corosync successor address, verifies successor standby/catch-up, stops local PostgreSQL and rejoins it as a cascading standby. | | stop | Stops PostgreSQL, then the maintenance service/container. | | notify | Persists Pacemaker transition context; not promotion eligibility or fencing proof. | | rejoin | Private/manual recovery action, deliberately absent from advertised OCF actions. Requires a running maintenance container, stopped PostgreSQL, source and slot. Rewinds then verifies streaming recovery. | | validate-all/meta-data | Validate dependencies/configuration and report metadata. Validation can return 5 for missing utilities or 6 for invalid configuration. | Lifecycle success is 0; monitor's 8 is a role report, not a generic lifecycle success. Promoted monitoring must expect the correct role. Metadata advertises 13 parameters and eight actions; status is an alias and rejoin stays private. The scoring wrappers withdraw scores before lifecycle actions and reassess after successful actions. They do not change the original 29 function bodies. ## Scoring and fencing Healthy primary: 1000. Strictly verified streaming standby: 100. Recently verified eligible disconnected standby: 50. Unsafe or missing evidence: -INFINITY. Disconnected evidence is bounded by `disconnected_max_age` (default 300 seconds), bound to the PostgreSQL incarnation and local boot, and recalculated each monitor. See [the detailed state model](../notes/DISCONNECTED-STANDBY.md). An eligible score is not a fencing certificate. Pacemaker/SBD must isolate the failed primary and order promotion safely. Never bypass fencing by manually calling promote, creating standby.signal on divergent data, or deleting data. Asynchronous replication accepts possible loss of acknowledged transactions; the freshness window does not bound the number of lost transactions. Controlled switchover differs from abrupt failover: demote requires a reachable, caught-up successor and transition notification. Former-primary rejoin is a separate recovery procedure after fencing, not an automatic start workaround. ## Former-primary recovery order 1. Establish that fencing completed and identify the surviving writable primary and its VIP. Coordinate cluster management so Pacemaker cannot race the manual repair. Do not clear the former primary's failed start yet. 2. Bring up only its maintenance container using the existing Quadlet as the service owner. Verify PostgreSQL is stopped. Preserve the existing data. 3. Confirm the running container's replication credentials are correct without printing them. Supply the existing resource instance/parameters and a unique local replication slot to the private rejoin action, using the PostgreSQL VIP as `rejoin_source_host`. This documentation does not invent missing live IDs. 4. Rejoin first attempts pg_rewind. If it fails, stop and inspect the error. Basebackup fallback remains disabled unless explicitly authorized to replace divergent data. It preserves the old directory but is still a destructive recovery decision; a failed rewind is not guaranteed to leave usable data. 5. Require rejoin success AND verify `pg_is_in_recovery() = true`, a streaming WAL receiver connected to the intended primary, and the expected slot/application identity. Verify the maintenance container remained available. 6. **Only now** clear the relevant failed actions for that resource on that node, and allow Pacemaker to adopt it as Unpromoted. Recheck roles, scores, streaming, VIP placement, and failed actions. Do not use broad cleanup before recovery. Safe inspection commands, run on the appropriate node, include: ```sh sudo pcs status --full sudo pcs resource config postgresql-ha-clone sudo pcs constraint config --full ``` Inspect output locally; raw CIB, environment dumps and logs may contain sensitive configuration and should not be committed. The optional preference planner reads an operator-supplied CIB and prints only a narrowly matched temporary preference removal. It preserves target-role=Promoted and all unrelated constraints. It does not perform resource failure cleanup or make a node safe to rejoin. ## Podman env-file incident The production test exposed a quoting mismatch. Podman's env-file parser retained surrounding quote characters, while the generated pgpass had the intended value. Streaming worked, but rejoin's direct environment password failed authentication. These are intentionally fake examples, not deployment values: ```text # Incorrect if the intended value contains no quotation marks: POSTGRES_REPLICATION_PASSWORD="EXAMPLE_ONLY_NOT_A_SECRET" # Correct representation of that same intended value: POSTGRES_REPLICATION_PASSWORD=EXAMPLE_ONLY_NOT_A_SECRET ``` Podman env-file syntax is not shell assignment syntax; do not source it to validate container behavior. Correct the secret locally and apply it through the planned Quadlet/container lifecycle: editing a file alone does not alter an already-running container's environment. Do not blindly restart a primary to apply this change. Keep real values, .pgpass contents, secret files, and raw container environment inspection out of Git and shared logs. References: [OCF role semantics](https://github.com/ClusterLabs/pacemaker/blob/main/doc/sphinx/Pacemaker_Administration/agents.rst), [Pacemaker configuration and ordering](https://clusterlabs.org/projects/pacemaker/doc/3.0/Pacemaker_Explained/pdf/Pacemaker_Explained.pdf), and [Podman run options](https://docs.podman.io/en/latest/markdown/podman-run.1.html).