Freeze HA scoring and document reported production failover
This commit is contained in:
Lyncoln
@@ -0,0 +1,198 @@
|
||||
> Production fencing/rejoin was subsequently reported successful by the operator.
|
||||
> See [../docs/testing.md](../docs/testing.md) for evidence and snapshot status,
|
||||
> and [../docs/operations.md](../docs/operations.md) for current operating policy.
|
||||
> Deployment statements below describe the original development session only.
|
||||
|
||||
# Disconnected standby implementation and original candidate validation
|
||||
|
||||
This revision uses the accepted promotion-scoring candidate as its baseline.
|
||||
The user explicitly accepts asynchronous replication data-loss risk after an
|
||||
unexpected primary failure, provided Pacemaker fences the failed primary before
|
||||
promotion. No installed RA, live cluster attributes, constraints, fencing
|
||||
configuration, or PostgreSQL instance was changed during this revision.
|
||||
|
||||
## State model and scores
|
||||
|
||||
Every monitor first runs the original role/health monitor. Primary remains
|
||||
1000 with OCF return 8; stopped, unready, and failed role detection publish
|
||||
-INFINITY with their original return codes. Healthy standbys report OCF return
|
||||
0 regardless of eligibility. Each standby monitor collects a new local SQL
|
||||
observation and evaluates one of these states:
|
||||
|
||||
| State | Evidence | Score |
|
||||
| --- | --- | --- |
|
||||
| Connected and verified | In recovery; replay unpaused; receiver streaming; flush non-NULL; replay >= flush | 100 |
|
||||
| Disconnected and verified recently | Same instance; recovery and replay healthy; no streaming receiver; valid fresh stored observation; current replay >= saved replay | 50 |
|
||||
| Ineligible | Missing, expired, invalid or incompatible evidence; replay paused/regressed; connected strict check failed; query/helper failure | -INFINITY |
|
||||
|
||||
The strict connected predicate has not been relaxed. A streaming receiver with
|
||||
lag or NULL flush cannot fall back to disconnected eligibility. A missing
|
||||
receiver, or a PostgreSQL receiver in starting/waiting/restarting/stopping state,
|
||||
can use the disconnected path. A valid connected observation replaces the
|
||||
record atomically; disconnected observations never renew its timestamp.
|
||||
Reconnection must pass the strict checks to create fresh evidence.
|
||||
|
||||
Score publication still uses bounded `crm_attribute --promotion` query/update
|
||||
calls. A missing transient attribute is recreated from current eligibility,
|
||||
including disconnected eligibility. This is not retention of an old +100.
|
||||
Lifecycle wrappers continue to withdraw scores before actions and re-evaluate
|
||||
after successful actions. The original 29 function bodies, defaults, and all
|
||||
original safety checks remain byte-identical. An added metadata wrapper
|
||||
advertises the new parameter without modifying the original metadata function.
|
||||
|
||||
## Exact persistence and ownership
|
||||
|
||||
Host-side directory: `/run/managed-container/promotion/`.
|
||||
Both `/run/managed-container` and its `promotion` child are root-owned mode 0700.
|
||||
Each record is `<sha256>.json`, root-owned mode 0600, with a corresponding
|
||||
root-owned mode 0600 `<sha256>.lock`. The key is SHA-256 of the UTF-8 compact JSON
|
||||
array `[OCF_RESOURCE_INSTANCE, service_user, service_name, container_name]`.
|
||||
An instance suffix change conservatively selects a different record.
|
||||
|
||||
The RA's root process runs `managed-container.d/promotion_state.py` with Python
|
||||
3; the rootless service user cannot read/write the host state directory. There
|
||||
are no credentials, connection strings, passwords, or upstream secrets in the
|
||||
record. Contents are schema version, resource context, host boot ID, PostgreSQL
|
||||
system identifier, actual SQL `data_directory`, postmaster start timestamp,
|
||||
verified replay and flush LSNs, wall-clock observation timestamp for audit,
|
||||
and boot-relative observation time for expiry.
|
||||
|
||||
A nonblocking per-resource lock serializes evaluation and same-directory atomic
|
||||
replacement. Temporary files use mode 0600; JSON is bounded and parsed as data,
|
||||
with duplicate keys, invalid types, invalid LSNs, and extra/missing fields
|
||||
rejected. Unsafe directory/file modes, ownership or symlinks fail closed.
|
||||
Failed persistence withdraws eligibility. Runtime state survives separate RA
|
||||
invocations and Pacemaker transient score loss, but not host reboot; boot ID
|
||||
also guards against restored state from a previous boot.
|
||||
|
||||
## Freshness and identity policy
|
||||
|
||||
OCF parameter `disconnected_max_age`: integer seconds 0–99999, default **300**.
|
||||
Zero disables disconnected eligibility. Ages from zero through the threshold
|
||||
are accepted; negative, future, or greater ages are rejected. Invalid parameter
|
||||
values fail validation and cannot yield a positive standby score.
|
||||
|
||||
Expiry uses Linux CLOCK_BOOTTIME, which includes suspended time. Changing the
|
||||
wall clock does not extend or prematurely expire an observation. Choose the
|
||||
threshold to cover the monitoring, failure detection, fencing, and scheduling
|
||||
time needed by the cluster. Slow fencing may outlast this window and leave no
|
||||
eligible candidate; freshness is deliberately not extended in response.
|
||||
|
||||
The data identity is `(system_identifier, data_directory, postmaster_start_time)`
|
||||
plus host boot/resource context. A new PostgreSQL system ID or data path
|
||||
invalidates the observation. **Any PostgreSQL restart also requires a new strict
|
||||
streaming verification**, even when restarting the same data. This is deliberate:
|
||||
a basebackup or same-cluster replacement can preserve the system identifier,
|
||||
but cannot preserve the running postmaster incarnation. Normal reinitialization,
|
||||
rewind/rejoin restart, or container restart therefore cannot inherit eligibility.
|
||||
Current replay must also be at least the stored replay LSN.
|
||||
|
||||
Malformed, stale or mismatching records are deleted when observed by the
|
||||
state evaluator. A subsequent strict streaming verification creates a new
|
||||
record. A stopped PostgreSQL process is immediately scored -INFINITY; its file
|
||||
may remain until the next standby evaluation, but its replacement/restarted
|
||||
process cannot use it. A healthy primary keeps 1000 and does not consume standby
|
||||
evidence. Stale unused resource-key files disappear at host reboot.
|
||||
|
||||
## Candidate selection and fencing
|
||||
|
||||
100 ranks currently verified connected evidence above recent disconnected
|
||||
proof at 50, while preserving primary 1000. Two disconnected candidates receive
|
||||
equal 50. Pacemaker resolves ties using its configured scheduling rules; no
|
||||
preferred hostname is hardcoded. Tests show repeatable selection for an
|
||||
identical synthetic CIB, not a guarantee of a particular host in every cluster.
|
||||
|
||||
We intentionally do not map LSNs or local observation timestamps into unequal
|
||||
scores. A 64-bit LSN cannot be represented faithfully in Pacemaker's bounded
|
||||
score range, and comparison needs compatible timeline/history context. Local
|
||||
observation age is not a measure of WAL completeness: a more recent observation
|
||||
can contain less WAL than an older observation on another node. Cross-node
|
||||
clock timestamps introduce further uncertainty. Replay and flush LSNs are
|
||||
stored for a future explicit comparison policy; this revision has no cross-node
|
||||
shell access or invented distributed ranking mechanism.
|
||||
|
||||
Fencing remains mandatory and is entirely Pacemaker's responsibility. The RA
|
||||
does not inspect fencing completion or create/modify constraints. A score of 50
|
||||
means locally eligible under the accepted asynchronous policy; it is not
|
||||
permission to run the promote action manually against an unfenced primary.
|
||||
Configured fencing, quorum, and action ordering must be validated in the lab.
|
||||
The synthetic failure tests use fencing enabled and check that every promotion
|
||||
has a fencing action as a transitive prerequisite in the scheduler graph.
|
||||
|
||||
## Verification and review artifacts
|
||||
|
||||
- Original frozen hashes and all 29 original function bodies/defaults: unchanged.
|
||||
- All 22 original differential lifecycle cases pass.
|
||||
- All 25 accepted scoring/action tests pass; mock boundary and metadata handler
|
||||
were adjusted for the added helper without removing existing assertions.
|
||||
- 16 state/monitor tests cover streaming-disconnect, never verified, exact
|
||||
expiry/disabled policy, malformed and unsafe state, system/path/postmaster
|
||||
changes, reinitialization replacement, replay pause/regression, connected
|
||||
failures, reconnect, clock changes, stopped PostgreSQL, score loss/reprobe, and query/helper failures.
|
||||
- 10 real `crm_simulate` synthetic-CIB tests pass, including the original six,
|
||||
unexpected primary failure with one/two disconnected candidates, mandatory
|
||||
fencing dependencies, all-stale refusal, and connected preference.
|
||||
- All 4 cleanup-planner tests pass. No generated cleanup commands were executed.
|
||||
- PostgreSQL 18 integration: all nine original strict SQL cases pass, and the
|
||||
new JSON observation query agrees with each. Only a disposable container with
|
||||
tmpfs data and no network/host ports was used.
|
||||
- The root helper CLI also passed in a disposable Fedora container, using the
|
||||
actual fixed /run path: never-verified refusal, 100 -> 50, ownership/modes,
|
||||
unchanged disconnected timestamp, malformed-state deletion and identity change.
|
||||
- Fedora 43 RPM build and disposable installation validation passed, recorded in
|
||||
`/tmp/mc-disconnected-rpm.log`; artifacts are in `dist/disconnected-review/`.
|
||||
- Incremental diff from the accepted candidate: `DISCONNECTED-STANDBY.patch`.
|
||||
Full diff from committed baseline: `PROMOTION-SCORING.patch`.
|
||||
|
||||
Run the offline suite with:
|
||||
|
||||
```sh
|
||||
python3 tests/verify_refactor.py
|
||||
python3 tests/test_promotion_scores.py
|
||||
python3 tests/test_disconnected_state.py
|
||||
python3 tests/test_promotion_scheduler.py
|
||||
python3 tests/test_promotion_cleanup.py
|
||||
```
|
||||
|
||||
The optional disposable SQL test is `python3 tests/verify_promotion_sql.py`.
|
||||
During the original candidate-development session, no live HA transitions or
|
||||
deployment were performed. Subsequent operator-reported production results are
|
||||
recorded in ../docs/testing.md. The previous scoring
|
||||
report is retained as historical context; its strict-disconnection blocker is
|
||||
superseded by this policy. Existing lab/production cleanup prerequisites remain.
|
||||
|
||||
## Remaining limitations
|
||||
|
||||
Asynchronous replication does not guarantee zero data loss. The 300-second
|
||||
window bounds observation age, **not lost bytes or transactions**, and does not
|
||||
prove that a standby has the failed primary's last acknowledged transaction.
|
||||
Nor does this design guarantee selection of the most advanced of two eligible
|
||||
standbys. It does not establish common timelines across peers.
|
||||
|
||||
The SQL user must be allowed to execute `pg_control_system()` and read the
|
||||
required monitoring fields. Insufficient privileges fail closed; no grants are
|
||||
changed automatically. Restarting a disconnected standby forfeits eligibility
|
||||
until it streams again. Loss of /run state likewise requires fresh verification.
|
||||
Replacing files underneath a running PostgreSQL process or deliberately restoring
|
||||
fabricated identity/state is outside supported PostgreSQL operation; root and
|
||||
the database monitoring boundary are trusted.
|
||||
|
||||
The scoring SQL retains its 3-second external bound (1-second kill grace) and
|
||||
2-second statement timeout. State evaluation adds a 1-second external bound
|
||||
(1-second kill grace); crm_attribute retains each 2-second bound and 1-second
|
||||
grace. Original lifecycle/role queries and operation timeouts are unchanged;
|
||||
validate the aggregate monitor budget on actual hardware.
|
||||
|
||||
Eligibility is sampled, not an atomic transaction with future promotion.
|
||||
Publication/IPC failure can leave an old Pacemaker score until the next
|
||||
successful update, as in the accepted baseline. Scheduling and fencing can
|
||||
also outlast an observation's expiry before the next monitor. This revision
|
||||
preserves the existing promote implementation rather than adding a new final
|
||||
promotion guard. Lab testing must cover detection/fencing latency, failover in
|
||||
both directions, reconnect/rejoin, monitoring load, and score updates during
|
||||
failure before a production rollout is approved.
|
||||
|
||||
Sources: PostgreSQL documents the system identifier and server start timestamp
|
||||
in [system information functions](https://www.postgresql.org/docs/18/functions-info.html).
|
||||
Pacemaker's fencing, resource requirements and promotion scheduling are described
|
||||
in [Pacemaker Explained](https://clusterlabs.org/projects/pacemaker/doc/3.0/Pacemaker_Explained/pdf/Pacemaker_Explained.pdf).
|
||||
Reference in New Issue
Block a user