199 lines
12 KiB
Markdown
199 lines
12 KiB
Markdown
> Production fencing/rejoin was subsequently reported successful by the operator.
|
||
> See [../docs/testing.md](../docs/testing.md) for evidence and snapshot status,
|
||
> and [../docs/operations.md](../docs/operations.md) for current operating policy.
|
||
> Deployment statements below describe the original development session only.
|
||
|
||
# Disconnected standby implementation and original candidate validation
|
||
|
||
This revision uses the accepted promotion-scoring candidate as its baseline.
|
||
The user explicitly accepts asynchronous replication data-loss risk after an
|
||
unexpected primary failure, provided Pacemaker fences the failed primary before
|
||
promotion. No installed RA, live cluster attributes, constraints, fencing
|
||
configuration, or PostgreSQL instance was changed during this revision.
|
||
|
||
## State model and scores
|
||
|
||
Every monitor first runs the original role/health monitor. Primary remains
|
||
1000 with OCF return 8; stopped, unready, and failed role detection publish
|
||
-INFINITY with their original return codes. Healthy standbys report OCF return
|
||
0 regardless of eligibility. Each standby monitor collects a new local SQL
|
||
observation and evaluates one of these states:
|
||
|
||
| State | Evidence | Score |
|
||
| --- | --- | --- |
|
||
| Connected and verified | In recovery; replay unpaused; receiver streaming; flush non-NULL; replay >= flush | 100 |
|
||
| Disconnected and verified recently | Same instance; recovery and replay healthy; no streaming receiver; valid fresh stored observation; current replay >= saved replay | 50 |
|
||
| Ineligible | Missing, expired, invalid or incompatible evidence; replay paused/regressed; connected strict check failed; query/helper failure | -INFINITY |
|
||
|
||
The strict connected predicate has not been relaxed. A streaming receiver with
|
||
lag or NULL flush cannot fall back to disconnected eligibility. A missing
|
||
receiver, or a PostgreSQL receiver in starting/waiting/restarting/stopping state,
|
||
can use the disconnected path. A valid connected observation replaces the
|
||
record atomically; disconnected observations never renew its timestamp.
|
||
Reconnection must pass the strict checks to create fresh evidence.
|
||
|
||
Score publication still uses bounded `crm_attribute --promotion` query/update
|
||
calls. A missing transient attribute is recreated from current eligibility,
|
||
including disconnected eligibility. This is not retention of an old +100.
|
||
Lifecycle wrappers continue to withdraw scores before actions and re-evaluate
|
||
after successful actions. The original 29 function bodies, defaults, and all
|
||
original safety checks remain byte-identical. An added metadata wrapper
|
||
advertises the new parameter without modifying the original metadata function.
|
||
|
||
## Exact persistence and ownership
|
||
|
||
Host-side directory: `/run/managed-container/promotion/`.
|
||
Both `/run/managed-container` and its `promotion` child are root-owned mode 0700.
|
||
Each record is `<sha256>.json`, root-owned mode 0600, with a corresponding
|
||
root-owned mode 0600 `<sha256>.lock`. The key is SHA-256 of the UTF-8 compact JSON
|
||
array `[OCF_RESOURCE_INSTANCE, service_user, service_name, container_name]`.
|
||
An instance suffix change conservatively selects a different record.
|
||
|
||
The RA's root process runs `managed-container.d/promotion_state.py` with Python
|
||
3; the rootless service user cannot read/write the host state directory. There
|
||
are no credentials, connection strings, passwords, or upstream secrets in the
|
||
record. Contents are schema version, resource context, host boot ID, PostgreSQL
|
||
system identifier, actual SQL `data_directory`, postmaster start timestamp,
|
||
verified replay and flush LSNs, wall-clock observation timestamp for audit,
|
||
and boot-relative observation time for expiry.
|
||
|
||
A nonblocking per-resource lock serializes evaluation and same-directory atomic
|
||
replacement. Temporary files use mode 0600; JSON is bounded and parsed as data,
|
||
with duplicate keys, invalid types, invalid LSNs, and extra/missing fields
|
||
rejected. Unsafe directory/file modes, ownership or symlinks fail closed.
|
||
Failed persistence withdraws eligibility. Runtime state survives separate RA
|
||
invocations and Pacemaker transient score loss, but not host reboot; boot ID
|
||
also guards against restored state from a previous boot.
|
||
|
||
## Freshness and identity policy
|
||
|
||
OCF parameter `disconnected_max_age`: integer seconds 0–99999, default **300**.
|
||
Zero disables disconnected eligibility. Ages from zero through the threshold
|
||
are accepted; negative, future, or greater ages are rejected. Invalid parameter
|
||
values fail validation and cannot yield a positive standby score.
|
||
|
||
Expiry uses Linux CLOCK_BOOTTIME, which includes suspended time. Changing the
|
||
wall clock does not extend or prematurely expire an observation. Choose the
|
||
threshold to cover the monitoring, failure detection, fencing, and scheduling
|
||
time needed by the cluster. Slow fencing may outlast this window and leave no
|
||
eligible candidate; freshness is deliberately not extended in response.
|
||
|
||
The data identity is `(system_identifier, data_directory, postmaster_start_time)`
|
||
plus host boot/resource context. A new PostgreSQL system ID or data path
|
||
invalidates the observation. **Any PostgreSQL restart also requires a new strict
|
||
streaming verification**, even when restarting the same data. This is deliberate:
|
||
a basebackup or same-cluster replacement can preserve the system identifier,
|
||
but cannot preserve the running postmaster incarnation. Normal reinitialization,
|
||
rewind/rejoin restart, or container restart therefore cannot inherit eligibility.
|
||
Current replay must also be at least the stored replay LSN.
|
||
|
||
Malformed, stale or mismatching records are deleted when observed by the
|
||
state evaluator. A subsequent strict streaming verification creates a new
|
||
record. A stopped PostgreSQL process is immediately scored -INFINITY; its file
|
||
may remain until the next standby evaluation, but its replacement/restarted
|
||
process cannot use it. A healthy primary keeps 1000 and does not consume standby
|
||
evidence. Stale unused resource-key files disappear at host reboot.
|
||
|
||
## Candidate selection and fencing
|
||
|
||
100 ranks currently verified connected evidence above recent disconnected
|
||
proof at 50, while preserving primary 1000. Two disconnected candidates receive
|
||
equal 50. Pacemaker resolves ties using its configured scheduling rules; no
|
||
preferred hostname is hardcoded. Tests show repeatable selection for an
|
||
identical synthetic CIB, not a guarantee of a particular host in every cluster.
|
||
|
||
We intentionally do not map LSNs or local observation timestamps into unequal
|
||
scores. A 64-bit LSN cannot be represented faithfully in Pacemaker's bounded
|
||
score range, and comparison needs compatible timeline/history context. Local
|
||
observation age is not a measure of WAL completeness: a more recent observation
|
||
can contain less WAL than an older observation on another node. Cross-node
|
||
clock timestamps introduce further uncertainty. Replay and flush LSNs are
|
||
stored for a future explicit comparison policy; this revision has no cross-node
|
||
shell access or invented distributed ranking mechanism.
|
||
|
||
Fencing remains mandatory and is entirely Pacemaker's responsibility. The RA
|
||
does not inspect fencing completion or create/modify constraints. A score of 50
|
||
means locally eligible under the accepted asynchronous policy; it is not
|
||
permission to run the promote action manually against an unfenced primary.
|
||
Configured fencing, quorum, and action ordering must be validated in the lab.
|
||
The synthetic failure tests use fencing enabled and check that every promotion
|
||
has a fencing action as a transitive prerequisite in the scheduler graph.
|
||
|
||
## Verification and review artifacts
|
||
|
||
- Original frozen hashes and all 29 original function bodies/defaults: unchanged.
|
||
- All 22 original differential lifecycle cases pass.
|
||
- All 25 accepted scoring/action tests pass; mock boundary and metadata handler
|
||
were adjusted for the added helper without removing existing assertions.
|
||
- 16 state/monitor tests cover streaming-disconnect, never verified, exact
|
||
expiry/disabled policy, malformed and unsafe state, system/path/postmaster
|
||
changes, reinitialization replacement, replay pause/regression, connected
|
||
failures, reconnect, clock changes, stopped PostgreSQL, score loss/reprobe, and query/helper failures.
|
||
- 10 real `crm_simulate` synthetic-CIB tests pass, including the original six,
|
||
unexpected primary failure with one/two disconnected candidates, mandatory
|
||
fencing dependencies, all-stale refusal, and connected preference.
|
||
- All 4 cleanup-planner tests pass. No generated cleanup commands were executed.
|
||
- PostgreSQL 18 integration: all nine original strict SQL cases pass, and the
|
||
new JSON observation query agrees with each. Only a disposable container with
|
||
tmpfs data and no network/host ports was used.
|
||
- The root helper CLI also passed in a disposable Fedora container, using the
|
||
actual fixed /run path: never-verified refusal, 100 -> 50, ownership/modes,
|
||
unchanged disconnected timestamp, malformed-state deletion and identity change.
|
||
- Fedora 43 RPM build and disposable installation validation passed, recorded in
|
||
`/tmp/mc-disconnected-rpm.log`; artifacts are in `dist/disconnected-review/`.
|
||
- Incremental diff from the accepted candidate: `DISCONNECTED-STANDBY.patch`.
|
||
Full diff from committed baseline: `PROMOTION-SCORING.patch`.
|
||
|
||
Run the offline suite with:
|
||
|
||
```sh
|
||
python3 tests/verify_refactor.py
|
||
python3 tests/test_promotion_scores.py
|
||
python3 tests/test_disconnected_state.py
|
||
python3 tests/test_promotion_scheduler.py
|
||
python3 tests/test_promotion_cleanup.py
|
||
```
|
||
|
||
The optional disposable SQL test is `python3 tests/verify_promotion_sql.py`.
|
||
During the original candidate-development session, no live HA transitions or
|
||
deployment were performed. Subsequent operator-reported production results are
|
||
recorded in ../docs/testing.md. The previous scoring
|
||
report is retained as historical context; its strict-disconnection blocker is
|
||
superseded by this policy. Existing lab/production cleanup prerequisites remain.
|
||
|
||
## Remaining limitations
|
||
|
||
Asynchronous replication does not guarantee zero data loss. The 300-second
|
||
window bounds observation age, **not lost bytes or transactions**, and does not
|
||
prove that a standby has the failed primary's last acknowledged transaction.
|
||
Nor does this design guarantee selection of the most advanced of two eligible
|
||
standbys. It does not establish common timelines across peers.
|
||
|
||
The SQL user must be allowed to execute `pg_control_system()` and read the
|
||
required monitoring fields. Insufficient privileges fail closed; no grants are
|
||
changed automatically. Restarting a disconnected standby forfeits eligibility
|
||
until it streams again. Loss of /run state likewise requires fresh verification.
|
||
Replacing files underneath a running PostgreSQL process or deliberately restoring
|
||
fabricated identity/state is outside supported PostgreSQL operation; root and
|
||
the database monitoring boundary are trusted.
|
||
|
||
The scoring SQL retains its 3-second external bound (1-second kill grace) and
|
||
2-second statement timeout. State evaluation adds a 1-second external bound
|
||
(1-second kill grace); crm_attribute retains each 2-second bound and 1-second
|
||
grace. Original lifecycle/role queries and operation timeouts are unchanged;
|
||
validate the aggregate monitor budget on actual hardware.
|
||
|
||
Eligibility is sampled, not an atomic transaction with future promotion.
|
||
Publication/IPC failure can leave an old Pacemaker score until the next
|
||
successful update, as in the accepted baseline. Scheduling and fencing can
|
||
also outlast an observation's expiry before the next monitor. This revision
|
||
preserves the existing promote implementation rather than adding a new final
|
||
promotion guard. Lab testing must cover detection/fencing latency, failover in
|
||
both directions, reconnect/rejoin, monitoring load, and score updates during
|
||
failure before a production rollout is approved.
|
||
|
||
Sources: PostgreSQL documents the system identifier and server start timestamp
|
||
in [system information functions](https://www.postgresql.org/docs/18/functions-info.html).
|
||
Pacemaker's fencing, resource requirements and promotion scheduling are described
|
||
in [Pacemaker Explained](https://clusterlabs.org/projects/pacemaker/doc/3.0/Pacemaker_Explained/pdf/Pacemaker_Explained.pdf).
|