Lyncoln
ec2077f979
Extract PostgreSQL, Podman runtime and systemd lifecycle adapters while preserving existing lifecycle and scoring behavior. Update RPM layout and add guarded operator-run lab harnesses. Record successful manual stop/start, promotion, demotion and rewind-only rejoin with pinned installed hashes. The unfenced lab does not validate automatic failover, Pacemaker-managed transitions or divergent-data rewind. Validation: 103 offline tests, original-function differential checks, checksum manifest and whitespace checks passed.
132 lines
8.1 KiB
Markdown
132 lines
8.1 KiB
Markdown
# PostgreSQL HA operating policy
|
|
|
|
This document describes the intentional production design. Historical two-node
|
|
lab snapshots and old installation notes are not production runbooks. Keep the
|
|
three-node production clone's **target-role=Promoted** and **promoted-max=1**.
|
|
Promotion scores select an eligible node; the target role expresses the desired
|
|
promoted service. This project requires that explicit configuration even though
|
|
Pacemaker can permit promotion with other default target-role settings.
|
|
The VIP must be colocated with the Promoted instance and ordered to start after
|
|
promotion, with reverse shutdown ordering. Scores do not create those constraints.
|
|
|
|
## Ownership and lifecycles
|
|
|
|
Pacemaker invokes the root OCF RA. Its `runuser` boundary enters the rootless
|
|
service owner's home and user-systemd runtime, then invokes systemd/Podman with
|
|
argument arrays. Quadlet constructs the maintenance container and supplies its
|
|
environment, mounts and bootstrap. The bootstrap gates initialization/adoption,
|
|
then executes `sleep infinity`; Podman may use an init shim as PID 1. PostgreSQL
|
|
is not the container lifetime process.
|
|
|
|
The RA independently controls PostgreSQL with `pg_ctl`. During demotion and
|
|
rejoin, the maintenance container stays available even when PostgreSQL is
|
|
stopped. This is required for inspection, `pg_rewind`, and optional basebackup.
|
|
The OCF stop action stops both PostgreSQL and the maintenance service/container.
|
|
Starting the container alone is not equivalent to starting the OCF resource.
|
|
|
|
The RA reads PGDATA and intentional replication settings from the running
|
|
container. It does not parse or source host PostgreSQL env/config files. Quadlet
|
|
may intentionally consume a host env-file when constructing that container.
|
|
The bootstrap manages its secret-file/pgpass interfaces separately; the RA's
|
|
replication commands read `POSTGRES_REPLICATION_PASSWORD` inside the container
|
|
and set PGPASSWORD only for the child process.
|
|
|
|
## Actions and role reporting
|
|
|
|
| Action | Implementation contract |
|
|
| --- | --- |
|
|
| start | Starts or verifies an Unpromoted standby. Refuses an already-primary instance; when PostgreSQL is stopped, refuses PGDATA without standby.signal. |
|
|
| monitor/status | Reports healthy standby 0, healthy primary 8, stopped service/container/PostgreSQL 7, and unhealthy/failed query 1. Promotion eligibility does not change these role codes. |
|
|
| promote | Promotes a healthy standby and reconciles local HA slots/rewind grants; an existing primary reconciles those invariants. |
|
|
| demote | Controlled switchover: uses fresh pre-demote notification and Corosync successor address, verifies successor standby/catch-up, stops local PostgreSQL and rejoins it as a cascading standby. |
|
|
| stop | Stops PostgreSQL, then the maintenance service/container. |
|
|
| notify | Persists Pacemaker transition context; not promotion eligibility or fencing proof. |
|
|
| rejoin | Private/manual recovery action, deliberately absent from advertised OCF actions. Requires a running maintenance container, stopped PostgreSQL, source and slot. Rewinds then verifies streaming recovery. |
|
|
| validate-all/meta-data | Validate dependencies/configuration and report metadata. Validation can return 5 for missing utilities or 6 for invalid configuration. |
|
|
|
|
Lifecycle success is 0; monitor's 8 is a role report, not a generic lifecycle
|
|
success. Promoted monitoring must expect the correct role. Metadata advertises
|
|
16 parameters and eight actions; status is an alias and rejoin stays private.
|
|
The scoring wrappers withdraw scores before lifecycle actions and reassess after
|
|
successful actions. The engine extraction preserves the original 29 functions with explicit
|
|
adapter-call substitutions; see plugin-engine.md.
|
|
|
|
## Scoring and fencing
|
|
|
|
Healthy primary: 1000. Strictly verified streaming standby: 100. Recently verified
|
|
eligible disconnected standby: 50. Unsafe or missing evidence: -INFINITY.
|
|
Disconnected evidence is bounded by `disconnected_max_age` (default 300 seconds),
|
|
bound to the PostgreSQL incarnation and local boot, and recalculated each monitor.
|
|
See [the detailed state model](../notes/DISCONNECTED-STANDBY.md).
|
|
|
|
An eligible score is not a fencing certificate. Pacemaker/SBD must isolate the
|
|
failed primary and order promotion safely. Never bypass fencing by manually
|
|
calling promote, creating standby.signal on divergent data, or deleting data.
|
|
Asynchronous replication accepts possible loss of acknowledged transactions;
|
|
the freshness window does not bound the number of lost transactions.
|
|
|
|
Controlled switchover differs from abrupt failover: demote requires a reachable,
|
|
caught-up successor and transition notification. Former-primary rejoin is a
|
|
separate recovery procedure after fencing, not an automatic start workaround.
|
|
|
|
## Former-primary recovery order
|
|
|
|
1. Establish that fencing completed and identify the surviving writable primary
|
|
and its VIP. Coordinate cluster management so Pacemaker cannot race the manual
|
|
repair. Do not clear the former primary's failed start yet.
|
|
2. Bring up only its maintenance container using the existing Quadlet as the
|
|
service owner. Verify PostgreSQL is stopped. Preserve the existing data.
|
|
3. Confirm the running container's replication credentials are correct without
|
|
printing them. Supply the existing resource instance/parameters and a unique
|
|
local replication slot to the private rejoin action, using the PostgreSQL VIP
|
|
as `rejoin_source_host`. This documentation does not invent missing live IDs.
|
|
4. Rejoin first attempts pg_rewind. If it fails, stop and inspect the error.
|
|
Basebackup fallback remains disabled unless explicitly authorized to replace
|
|
divergent data. It preserves the old directory but is still a destructive
|
|
recovery decision; a failed rewind is not guaranteed to leave usable data.
|
|
5. Require rejoin success AND verify `pg_is_in_recovery() = true`, a streaming WAL
|
|
receiver connected to the intended primary, and the expected slot/application
|
|
identity. Verify the maintenance container remained available.
|
|
6. **Only now** clear the relevant failed actions for that resource on that node,
|
|
and allow Pacemaker to adopt it as Unpromoted. Recheck roles, scores, streaming,
|
|
VIP placement, and failed actions. Do not use broad cleanup before recovery.
|
|
|
|
Safe inspection commands, run on the appropriate node, include:
|
|
|
|
```sh
|
|
sudo pcs status --full
|
|
sudo pcs resource config postgresql-ha-clone
|
|
sudo pcs constraint config --full
|
|
```
|
|
|
|
Inspect output locally; raw CIB, environment dumps and logs may contain sensitive
|
|
configuration and should not be committed. The optional preference planner reads
|
|
an operator-supplied CIB and prints only a narrowly matched temporary preference
|
|
removal. It preserves target-role=Promoted and all unrelated constraints. It does
|
|
not perform resource failure cleanup or make a node safe to rejoin.
|
|
|
|
## Podman env-file incident
|
|
|
|
The production test exposed a quoting mismatch. Podman's env-file parser retained
|
|
surrounding quote characters, while the generated pgpass had the intended value.
|
|
Streaming worked, but rejoin's direct environment password failed authentication.
|
|
These are intentionally fake examples, not deployment values:
|
|
|
|
```text
|
|
# Incorrect if the intended value contains no quotation marks:
|
|
POSTGRES_REPLICATION_PASSWORD="EXAMPLE_ONLY_NOT_A_SECRET"
|
|
# Correct representation of that same intended value:
|
|
POSTGRES_REPLICATION_PASSWORD=EXAMPLE_ONLY_NOT_A_SECRET
|
|
```
|
|
|
|
Podman env-file syntax is not shell assignment syntax; do not source it to
|
|
validate container behavior. Correct the secret locally and apply it through the
|
|
planned Quadlet/container lifecycle: editing a file alone does not alter an
|
|
already-running container's environment. Do not blindly restart a primary to
|
|
apply this change. Keep real values, .pgpass contents, secret files, and raw
|
|
container environment inspection out of Git and shared logs.
|
|
|
|
References: [OCF role semantics](https://github.com/ClusterLabs/pacemaker/blob/main/doc/sphinx/Pacemaker_Administration/agents.rst),
|
|
[Pacemaker configuration and ordering](https://clusterlabs.org/projects/pacemaker/doc/3.0/Pacemaker_Explained/pdf/Pacemaker_Explained.pdf),
|
|
and [Podman run options](https://docs.podman.io/en/latest/markdown/podman-run.1.html).
|