ocf-managed-container/notes/DISCONNECTED-STANDBY.md

Production fencing/rejoin was subsequently reported successful by the operator. See ../docs/testing.md for evidence and snapshot status, and ../docs/operations.md for current operating policy. Deployment statements below describe the original development session only.

Disconnected standby implementation and original candidate validation

This revision uses the accepted promotion-scoring candidate as its baseline. The user explicitly accepts asynchronous replication data-loss risk after an unexpected primary failure, provided Pacemaker fences the failed primary before promotion. No installed RA, live cluster attributes, constraints, fencing configuration, or PostgreSQL instance was changed during this revision.

State model and scores

Every monitor first runs the original role/health monitor. Primary remains 1000 with OCF return 8; stopped, unready, and failed role detection publish -INFINITY with their original return codes. Healthy standbys report OCF return 0 regardless of eligibility. Each standby monitor collects a new local SQL observation and evaluates one of these states:

State	Evidence	Score
Connected and verified	In recovery; replay unpaused; receiver streaming; flush non-NULL; replay >= flush	100
Disconnected and verified recently	Same instance; recovery and replay healthy; no streaming receiver; valid fresh stored observation; current replay >= saved replay	50
Ineligible	Missing, expired, invalid or incompatible evidence; replay paused/regressed; connected strict check failed; query/helper failure	-INFINITY

The strict connected predicate has not been relaxed. A streaming receiver with lag or NULL flush cannot fall back to disconnected eligibility. A missing receiver, or a PostgreSQL receiver in starting/waiting/restarting/stopping state, can use the disconnected path. A valid connected observation replaces the record atomically; disconnected observations never renew its timestamp. Reconnection must pass the strict checks to create fresh evidence.

Score publication still uses bounded crm_attribute --promotion query/update calls. A missing transient attribute is recreated from current eligibility, including disconnected eligibility. This is not retention of an old +100. Lifecycle wrappers continue to withdraw scores before actions and re-evaluate after successful actions. The original 29 function bodies, defaults, and all original safety checks remain byte-identical. An added metadata wrapper advertises the new parameter without modifying the original metadata function.

Exact persistence and ownership

Host-side directory: /run/managed-container/promotion/. Both /run/managed-container and its promotion child are root-owned mode 0700. Each record is <sha256>.json, root-owned mode 0600, with a corresponding root-owned mode 0600 <sha256>.lock. The key is SHA-256 of the UTF-8 compact JSON array [OCF_RESOURCE_INSTANCE, service_user, service_name, container_name]. An instance suffix change conservatively selects a different record.

The RA's root process runs managed-container.d/promotion_state.py with Python 3; the rootless service user cannot read/write the host state directory. There are no credentials, connection strings, passwords, or upstream secrets in the record. Contents are schema version, resource context, host boot ID, PostgreSQL system identifier, actual SQL data_directory, postmaster start timestamp, verified replay and flush LSNs, wall-clock observation timestamp for audit, and boot-relative observation time for expiry.

A nonblocking per-resource lock serializes evaluation and same-directory atomic replacement. Temporary files use mode 0600; JSON is bounded and parsed as data, with duplicate keys, invalid types, invalid LSNs, and extra/missing fields rejected. Unsafe directory/file modes, ownership or symlinks fail closed. Failed persistence withdraws eligibility. Runtime state survives separate RA invocations and Pacemaker transient score loss, but not host reboot; boot ID also guards against restored state from a previous boot.

Freshness and identity policy

OCF parameter disconnected_max_age: integer seconds 0–99999, default 300. Zero disables disconnected eligibility. Ages from zero through the threshold are accepted; negative, future, or greater ages are rejected. Invalid parameter values fail validation and cannot yield a positive standby score.

Expiry uses Linux CLOCK_BOOTTIME, which includes suspended time. Changing the wall clock does not extend or prematurely expire an observation. Choose the threshold to cover the monitoring, failure detection, fencing, and scheduling time needed by the cluster. Slow fencing may outlast this window and leave no eligible candidate; freshness is deliberately not extended in response.

The data identity is (system_identifier, data_directory, postmaster_start_time) plus host boot/resource context. A new PostgreSQL system ID or data path invalidates the observation. Any PostgreSQL restart also requires a new strict streaming verification, even when restarting the same data. This is deliberate: a basebackup or same-cluster replacement can preserve the system identifier, but cannot preserve the running postmaster incarnation. Normal reinitialization, rewind/rejoin restart, or container restart therefore cannot inherit eligibility. Current replay must also be at least the stored replay LSN.

Malformed, stale or mismatching records are deleted when observed by the state evaluator. A subsequent strict streaming verification creates a new record. A stopped PostgreSQL process is immediately scored -INFINITY; its file may remain until the next standby evaluation, but its replacement/restarted process cannot use it. A healthy primary keeps 1000 and does not consume standby evidence. Stale unused resource-key files disappear at host reboot.

Candidate selection and fencing

100 ranks currently verified connected evidence above recent disconnected proof at 50, while preserving primary 1000. Two disconnected candidates receive equal 50. Pacemaker resolves ties using its configured scheduling rules; no preferred hostname is hardcoded. Tests show repeatable selection for an identical synthetic CIB, not a guarantee of a particular host in every cluster.

We intentionally do not map LSNs or local observation timestamps into unequal scores. A 64-bit LSN cannot be represented faithfully in Pacemaker's bounded score range, and comparison needs compatible timeline/history context. Local observation age is not a measure of WAL completeness: a more recent observation can contain less WAL than an older observation on another node. Cross-node clock timestamps introduce further uncertainty. Replay and flush LSNs are stored for a future explicit comparison policy; this revision has no cross-node shell access or invented distributed ranking mechanism.

Fencing remains mandatory and is entirely Pacemaker's responsibility. The RA does not inspect fencing completion or create/modify constraints. A score of 50 means locally eligible under the accepted asynchronous policy; it is not permission to run the promote action manually against an unfenced primary. Configured fencing, quorum, and action ordering must be validated in the lab. The synthetic failure tests use fencing enabled and check that every promotion has a fencing action as a transitive prerequisite in the scheduler graph.

Verification and review artifacts

• Original frozen hashes and all 29 original function bodies/defaults: unchanged.
• All 22 original differential lifecycle cases pass.
• All 25 accepted scoring/action tests pass; mock boundary and metadata handler were adjusted for the added helper without removing existing assertions.
• 16 state/monitor tests cover streaming-disconnect, never verified, exact expiry/disabled policy, malformed and unsafe state, system/path/postmaster changes, reinitialization replacement, replay pause/regression, connected failures, reconnect, clock changes, stopped PostgreSQL, score loss/reprobe, and query/helper failures.
• 10 real crm_simulate synthetic-CIB tests pass, including the original six, unexpected primary failure with one/two disconnected candidates, mandatory fencing dependencies, all-stale refusal, and connected preference.
• All 4 cleanup-planner tests pass. No generated cleanup commands were executed.
• PostgreSQL 18 integration: all nine original strict SQL cases pass, and the new JSON observation query agrees with each. Only a disposable container with tmpfs data and no network/host ports was used.
• The root helper CLI also passed in a disposable Fedora container, using the actual fixed /run path: never-verified refusal, 100 -> 50, ownership/modes, unchanged disconnected timestamp, malformed-state deletion and identity change.
• Fedora 43 RPM build and disposable installation validation passed, recorded in /tmp/mc-disconnected-rpm.log; artifacts are in dist/disconnected-review/.
• Incremental diff from the accepted candidate: DISCONNECTED-STANDBY.patch. Full diff from committed baseline: PROMOTION-SCORING.patch.

Run the offline suite with:

python3 tests/verify_refactor.py
python3 tests/test_promotion_scores.py
python3 tests/test_disconnected_state.py
python3 tests/test_promotion_scheduler.py
python3 tests/test_promotion_cleanup.py

The optional disposable SQL test is python3 tests/verify_promotion_sql.py. During the original candidate-development session, no live HA transitions or deployment were performed. Subsequent operator-reported production results are recorded in ../docs/testing.md. The previous scoring report is retained as historical context; its strict-disconnection blocker is superseded by this policy. Existing lab/production cleanup prerequisites remain.

Remaining limitations

Asynchronous replication does not guarantee zero data loss. The 300-second window bounds observation age, not lost bytes or transactions, and does not prove that a standby has the failed primary's last acknowledged transaction. Nor does this design guarantee selection of the most advanced of two eligible standbys. It does not establish common timelines across peers.

The SQL user must be allowed to execute pg_control_system() and read the required monitoring fields. Insufficient privileges fail closed; no grants are changed automatically. Restarting a disconnected standby forfeits eligibility until it streams again. Loss of /run state likewise requires fresh verification. Replacing files underneath a running PostgreSQL process or deliberately restoring fabricated identity/state is outside supported PostgreSQL operation; root and the database monitoring boundary are trusted.

The scoring SQL retains its 3-second external bound (1-second kill grace) and 2-second statement timeout. State evaluation adds a 1-second external bound (1-second kill grace); crm_attribute retains each 2-second bound and 1-second grace. Original lifecycle/role queries and operation timeouts are unchanged; validate the aggregate monitor budget on actual hardware.

Eligibility is sampled, not an atomic transaction with future promotion. Publication/IPC failure can leave an old Pacemaker score until the next successful update, as in the accepted baseline. Scheduling and fencing can also outlast an observation's expiry before the next monitor. This revision preserves the existing promote implementation rather than adding a new final promotion guard. Lab testing must cover detection/fencing latency, failover in both directions, reconnect/rejoin, monitoring load, and score updates during failure before a production rollout is approved.

Sources: PostgreSQL documents the system identifier and server start timestamp in system information functions. Pacemaker's fencing, resource requirements and promotion scheduling are described in Pacemaker Explained.