ocf-managed-container/notes/PROMOTION-SCORING.md

Current production report and required operating policy: ../docs/testing.md and ../docs/operations.md. The historical lab plan below is not a production runbook.

Historical accepted scoring baseline. The current disconnected policy, tests, and limitations are in DISCONNECTED-STANDBY.md. The pending decision and strict-disconnection blocker below are superseded.

Promotion scoring — undeployed review candidate

This enhancement is based on the canonical modular source at c8636ed. This task changed no installed agent, production configuration, fencing setting or live cluster attribute. The earlier baseline installer is now pinned to that baseline entrypoint and refuses this candidate. Do not deploy it using the old installation instructions.

Decision still required before rollout

Strict live-streaming eligibility can prevent failover when the primary is lost. A previously healthy standby can lose its WAL receiver before Pacemaker chooses a successor. This candidate then replaces its positive score with -INFINITY. With all candidates ineligible, no instance is promoted. The SQL and scheduler tests confirm this outcome; it is not a claim of preserved automatic failover. Controlled switchovers and abrupt primary-loss failovers must both be reviewed. The user's choice between this conservative first candidate and a separate disconnected-standby eligibility policy is pending.

There is no inferred grace period, persisted last-known-good eligibility, or cross-node WAL ranking. Retaining an old positive score merely because a standby once streamed would contradict the requested stale-score cleanup. Resolving disconnected-standby promotion safely requires an explicit data-loss policy and validated fencing/recovery assumptions. Do not remove the production preference or reset production metadata until this decision and lab validation are complete.

Scoring policy

Observed state	Published score	Monitor result
Original monitor reports healthy primary	1000	8 (Promoted)
Healthy standby; in recovery, replay unpaused, streaming receiver, replay LSN at least receiver's flushed LSN	100	0 (Unpromoted)
Healthy standby but eligibility SQL false, empty, malformed, failed or timed out	-INFINITY	0 (Unpromoted)
Service/container/PostgreSQL stopped	-INFINITY	Original result, normally 7
Unready, failed role query or unexpected role	-INFINITY	Original failure code

The eligibility query is local to the running container. It uses the durable flushed_lsn, not written_lsn. SQL NULL values cannot yield eligibility. A CASE expression avoids invoking recovery-only functions on a primary. Standbys receive equal scores when eligible; no preferred hostname is encoded. Requiring replay to catch up to locally flushed WAL is conservative and can make eligibility fluctuate under sustained write load. This is a real state change, not artificial periodic CIB rewriting.

A positive score is an observation of local eligibility, not proof that a node has every acknowledged transaction from the primary, is freshest among peers, or is safe to promote without fencing. Asynchronous replication can lose transactions. These scores do not implement consensus or replace PostgreSQL recovery checks, Pacemaker constraints, quorum, or fencing.

Placement and scope

All 29 original function definitions, defaults and metadata remain byte-for-byte identical. The existing nine-module layout remains. Six helpers were added:

• common.sh: mc_promotion_score_set, mc_promotion_score_from_monitor, mc_promotion_monitor, mc_promotion_action, mc_promotion_validate.
• postgres.sh: mc_pg_promotion_eligible.

The dispatcher now routes monitor/status through mc_promotion_monitor, which calls the original managed_monitor once, publishes a score and returns the original role code. Internal lifecycle monitor calls remain unchanged and do not publish scores mid-transition.

Start/promote/demote/stop/rejoin publish -INFINITY before validation or mutation. After a successful non-stop action, the original monitor is called again and eligibility is reassessed. Failure leaves the negative score until a subsequent fresh monitor observation. Successful stop leaves -INFINITY. Stop still skips validate_all and attempts shutdown even if publication fails. Notify retains its original notification persistence behavior and does not affect scores. Metadata and validate-all publish no scores. validate-all additionally checks that crm_attribute and timeout exist; deployments require Pacemaker >=2.1.0 for the modern --promotion option, enforced by the RPM dependency.

The entrypoint captures the explicitly supplied OCF_RESOURCE_INSTANCE before ocf-shellfuncs can supply a dummy standalone default. If none was supplied, publication logs a warning and performs no write. Pacemaker resolves the node from its execution context (including OCF_RESKEY_CRM_meta_on_node); there are no node-name literals or manual node-resolution rules in the RA scoring code.

The helper uses:

crm_attribute --promotion=<explicit resource instance> --lifetime=reboot --query --quiet
crm_attribute --promotion=<explicit resource instance> --lifetime=reboot --update=<score>

Only a missing or different value causes an update. No local cache or durable eligibility file is used, so reprobes restore missing transient scores. Values are replaced, never incremented. A negative value is used instead of deletion: a missing score alone is not an explicit promotion prohibition.

Pacemaker currently implements this API with an internal master- attribute name. Seeing that prefix for the NEW resource is expected. The RA never constructs, reads, deletes or migrates master-postgresql, postgresql-data-status, or postgresql-status. Stale attributes for the old primitive are not inputs to this code. Independently configured location rules that reference old attributes remain cluster configuration and require their own operator review.

Each score query/write is bounded by a two-second timeout plus a one-second kill grace. The new eligibility command has a three-second outer timeout plus a one-second kill grace and a two-second SQL statement timeout. The existing monitor's own command timeouts are unchanged. Account for these additions in lab operation timing before production rollout.

If attribute publication fails, the helper logs an error explicitly warning that the prior score may remain. It retries on the next observation. Role return codes and existing lifecycle results are preserved; shutdown is never blocked by a score-write failure. No RA can guarantee stale-score removal while the cluster attribute service is unreachable. This failure mode needs lab validation and is not hidden by reporting a false PostgreSQL role. Pacemaker normally serializes actions for a resource; parallel direct/manual calls that race score updates are not supported by this implementation.

Tests and evidence

Commands run from the source tree:

bash -n refactored/managed-container
for module in refactored/managed-container.d/*.sh; do bash -n "$module"; done
python3 tests/verify_refactor.py
python3 tests/test_promotion_scores.py
python3 tests/test_promotion_scheduler.py
python3 tests/test_promotion_cleanup.py
python3 tests/verify_promotion_sql.py

• Original 22 differential scenarios pass. All 29 original definitions and the same 12 metadata parameters / eight advertised actions are preserved.
• 25 scoring tests pass, including failures, stop/start, promote/demote/rejoin, stale positive replacement, repeated monitors, loss/recreation of transient attributes, missing explicit resource identity and score-publication errors.
• A real crm_attribute test uses CIB_file with a synthetic CIB. Repeated equal observations do not change the file; negative replacement and recreation after loss of transient attributes work.
• Six real crm_simulate tests on synthetic CIBs pass: absent and explicit Started target-role each permit one promotion; Unpromoted prevents it; larger scores win, ineligible nodes are excluded, and all-negative scores permit no promotion. Old resource attributes are seeded and do not alter this new resource's election. These are scheduler simulations, not live HA transitions. The synthetic CIB disables fencing only inside its test file; the live lab/production fencing configuration was not touched.
• Nine SQL cases pass in a separate ephemeral PostgreSQL 18 container with no host ports or data mounts, evaluating the actual production SQL against controlled state fixtures. Includes replay pause/lag, missing positions, stopped/absent receiver, and recovery-only function handling on a primary.
• Four cleanup-command planner tests pass, including ambiguous-rule rejection and preservation of unrelated rules.

The RPM %check now runs the regression, scoring and scheduler suites. PostgreSQL integration is an explicit opt-in local test needing Podman and a local postgres:18 image; it does not touch the lab postgres container.

Specific disposable two-node lab sequence (after review)

Keep maintenance-mode=true as required by the lab handoff. Use direct RA operations only after installing the reviewed version under a separate approved installation procedure. Automatic scheduling claims must first be exercised with a copied CIB and crm_simulate; do not disable maintenance mode as part of these instructions. Do not use this sequence on production.

1. Save the lab CIB, installed executable/modules, and current state on both nodes. Verify the cluster is postgres-ra-test, the roles and streaming source, current PGDATA backup/recovery baseline, and that production is not targeted. Establish timing and monitor outputs from the existing implementation.

2. Run metadata and validate-all for the candidate on each lab node. For direct actions always supply the actual primitive ID (pg-ha in saved lab context), not the clone ID. Example observation commands on each lab node:

   sudo env OCF_RESOURCE_INSTANCE=pg-ha \
     /usr/lib/ocf/resource.d/loxthelion/managed-container monitor
   # Interpret rc=8 as healthy primary, rc=0 as healthy standby.
   sudo crm_attribute --promotion=pg-ha --lifetime=reboot --query --quiet
   

   Expect primary=1000 and a caught-up streaming standby=100. Repeat monitors; verify no writes for identical scores and no unrelated attribute changes.

3. On the standby, test full stop and start: stopped service/container and PostgreSQL must yield monitor=7 and score=-INFINITY; standby start restores monitor=0 and score=100 after streaming/catch-up. Verify primary start refusal remains intact. Do not use a successful role-return code as a score assertion.

4. Separately stop only standby PostgreSQL while keeping the maintenance container alive; observe -INFINITY, then start it through the proven standby path. Verify no rejoin is attempted while PostgreSQL is running.

5. Pause replay on the standby (lab only), observe monitor=0 and -INFINITY, resume replay and await catch-up/100. Test receiver interruption, SQL errors, stopped container, and a seeded prior positive score becoming ineligible.

6. Exercise the existing controlled switchover sequence in both directions: persist the fresh matching pre-demote notification on the old primary, demote through the Corosync-resolved successor, and promote the successor. Verify -INFINITY during transitions, 100 for a successful eligible demotion, 1000 after promotion/invariant reconciliation, and persistent container identity throughout demotion. Return to DV6 primary / ELITEDESK standby.

7. Test two eligible standbys using the existing cascading-standby intermediate state. Both should have the same positive score; a copied-CIB simulation should select one with promoted-max=1. Do not promote both manually.

8. Reproduce the abrupt-primary-loss case in the isolated lab only after a recoverable baseline exists. Allow the surviving receiver to disappear, run its monitor, and inspect the score. This candidate is EXPECTED to deny promotion then. That is a rollout blocker unless the strict policy is explicitly chosen; it cannot be counted as a passed failover test.

9. Remove only the new resource's transient score in the lab, then reprobe: fresh eligibility must recreate it. Exercise score-service/query/update failure and recovery without confusing role codes. Repeated successful monitoring must replace any stale positive score after ineligibility.

10. With a saved CIB, simulate removal of only the temporary node preference. Retain target-role=Promoted, promoted-max=1, and the VIP constraints. Earlier absent/Started scheduler cases are compatibility tests, not production configuration recommendations.

11. Restore the baseline topology, streaming, test-induced settings and intended agent version. Confirm maintenance mode still true. Record both nodes' installed hashes, OCF codes, scores, receiver status and action durations.

Production cleanup and rollout: gated on lab validation

The conceptual description does not include the actual preference constraint ID, so inventing a deletion command would be unsafe. Read-only discovery is:

sudo pcs constraint config --full
sudo pcs resource config postgresql-ha-clone
sudo cibadmin --query > production-before-scoring.xml

Do not commit that CIB snapshot: it may contain sensitive deployment settings. After lab validation and policy approval, scripts/plan-promotion-cleanup.py can read the supplied snapshot and PRINT the exact commands using the real IDs:

python3 scripts/plan-promotion-cleanup.py production-before-scoring.xml \
  --clone postgresql-ha-clone --preferred-node david-eve --lab-validated

The planner never executes commands. It matches only the simple rule described in the task (Promoted, score=200, #uname eq david-eve), refuses zero/multiple matches, preserves other rules, and emits only the preference constraint/rule removal. It requires and preserves direct clone-level target-role=Promoted. Rule-based/inherited settings require manual inspection. Do not confuse preference removal with clearing failed actions: former-primary failure state must remain until rejoin succeeds and streaming standby status is verified.

There is no RA code that deletes constraints or changes clone metadata. No production installation, cleanup, or restart was executed during that original source-review session. Subsequent production validation is in ../docs/testing.md. Exact production rollout commands are intentionally deferred until the policy, real constraint ID, candidate hashes and successful lab results are known, as requested.

Sources used for API semantics

• https://clusterlabs.org/projects/pacemaker/man/crm_attribute.8.html
• https://www.postgresql.org/docs/18/monitoring-stats.html#MONITORING-PG-STAT-WAL-RECEIVER-VIEW
• https://www.postgresql.org/docs/18/functions-admin.html
• Installed pcs help: pcs constraint remove --help, pcs resource meta --help.

The source diff is provided in PROMOTION-SCORING.patch. Source and test file hashes are recorded in REFACTOR-SHA256SUMS, which is refreshed for this candidate. The frozen reference hash remains unchanged.

RPM validation of this candidate also passed in the disposable Fedora 43 build container: binary/source packages built, %check ran the regression, scoring, scheduler and cleanup tests, DNF resolved the new Pacemaker dependency, and the installed-RPM integrity/permissions/metadata checks passed. Local artifacts are under dist/scoring-review/ and are not releases or deployment approval.

A final read-only inspection found that ELITEDESK's installed executable and all nine modules match the earlier modular baseline at c8636ed exactly. Its entrypoint SHA256 is c91f09adfdeb45a3be1a72df3f5ca5b02fd4555dec4397ad944eab439ace0b89. The promotion-scoring candidate is not installed there.