ocf-managed-container/notes/PROMOTION-SCORING.md

> Current production report and required operating policy: [../docs/testing.md](../docs/testing.md)
> and [../docs/operations.md](../docs/operations.md). The historical lab plan below
> is not a production runbook.

> Historical accepted scoring baseline. The current disconnected policy, tests,
> and limitations are in [DISCONNECTED-STANDBY.md](DISCONNECTED-STANDBY.md).
> The pending decision and strict-disconnection blocker below are superseded.

# Promotion scoring — undeployed review candidate

This enhancement is based on the canonical modular source at c8636ed. This task changed no
installed agent, production configuration, fencing setting or live cluster
attribute. The earlier baseline installer is now pinned to that
baseline entrypoint and refuses this candidate. Do not deploy it using the old
installation instructions.

## Decision still required before rollout

**Strict live-streaming eligibility can prevent failover when the primary is
lost.** A previously healthy standby can lose its WAL receiver before Pacemaker
chooses a successor. This candidate then replaces its positive score with
-INFINITY. With all candidates ineligible, no instance is promoted. The SQL and
scheduler tests confirm this outcome; it is not a claim of preserved automatic
failover. Controlled switchovers and abrupt primary-loss failovers must both be
reviewed. The user's choice between this conservative first candidate and a
separate disconnected-standby eligibility policy is pending.

There is no inferred grace period, persisted last-known-good eligibility, or
cross-node WAL ranking. Retaining an old positive score merely because a
standby once streamed would contradict the requested stale-score cleanup.
Resolving disconnected-standby promotion safely requires an explicit data-loss
policy and validated fencing/recovery assumptions. Do not remove the production
preference or reset production metadata until this decision and lab validation
are complete.

## Scoring policy

| Observed state | Published score | Monitor result |
| --- | --- | --- |
| Original monitor reports healthy primary | 1000 | 8 (Promoted) |
| Healthy standby; in recovery, replay unpaused, streaming receiver, replay LSN at least receiver's flushed LSN | 100 | 0 (Unpromoted) |
| Healthy standby but eligibility SQL false, empty, malformed, failed or timed out | -INFINITY | 0 (Unpromoted) |
| Service/container/PostgreSQL stopped | -INFINITY | Original result, normally 7 |
| Unready, failed role query or unexpected role | -INFINITY | Original failure code |

The eligibility query is local to the running container. It uses the durable
`flushed_lsn`, not `written_lsn`. SQL NULL values cannot yield eligibility.
A CASE expression avoids invoking recovery-only functions on a primary.
Standbys receive equal scores when eligible; no preferred hostname is encoded.
Requiring replay to catch up to locally flushed WAL is conservative and can
make eligibility fluctuate under sustained write load. This is a real state
change, not artificial periodic CIB rewriting.

A positive score is an observation of local eligibility, not proof that a node
has every acknowledged transaction from the primary, is freshest among peers,
or is safe to promote without fencing. Asynchronous replication can lose
transactions. These scores do not implement consensus or replace PostgreSQL
recovery checks, Pacemaker constraints, quorum, or fencing.

## Placement and scope

All 29 original function definitions, defaults and metadata remain byte-for-byte
identical. The existing nine-module layout remains. Six helpers were added:

- common.sh: mc_promotion_score_set, mc_promotion_score_from_monitor,
  mc_promotion_monitor, mc_promotion_action, mc_promotion_validate.
- postgres.sh: mc_pg_promotion_eligible.

The dispatcher now routes monitor/status through mc_promotion_monitor, which
calls the original managed_monitor once, publishes a score and returns the
original role code. Internal lifecycle monitor calls remain unchanged and do
not publish scores mid-transition.

Start/promote/demote/stop/rejoin publish -INFINITY before validation or mutation.
After a successful non-stop action, the original monitor is called again and
eligibility is reassessed. Failure leaves the negative score until a subsequent
fresh monitor observation. Successful stop leaves -INFINITY. Stop still skips
validate_all and attempts shutdown even if publication fails. Notify retains
its original notification persistence behavior and does not affect scores.
Metadata and validate-all publish no scores. validate-all additionally checks
that crm_attribute and timeout exist; deployments require Pacemaker >=2.1.0
for the modern --promotion option, enforced by the RPM dependency.

The entrypoint captures the explicitly supplied OCF_RESOURCE_INSTANCE before
ocf-shellfuncs can supply a dummy standalone default. If none was supplied,
publication logs a warning and performs no write. Pacemaker resolves the node
from its execution context (including OCF_RESKEY_CRM_meta_on_node); there are no
node-name literals or manual node-resolution rules in the RA scoring code.

The helper uses:

```text
crm_attribute --promotion=<explicit resource instance> --lifetime=reboot --query --quiet
crm_attribute --promotion=<explicit resource instance> --lifetime=reboot --update=<score>
```

Only a missing or different value causes an update. No local cache or durable
eligibility file is used, so reprobes restore missing transient scores. Values
are replaced, never incremented. A negative value is used instead of deletion:
a missing score alone is not an explicit promotion prohibition.

Pacemaker currently implements this API with an internal master-<resource>
attribute name. Seeing that prefix for the NEW resource is expected. The RA
never constructs, reads, deletes or migrates master-postgresql,
postgresql-data-status, or postgresql-status. Stale attributes for the old
primitive are not inputs to this code. Independently configured location rules
that reference old attributes remain cluster configuration and require their
own operator review.

Each score query/write is bounded by a two-second timeout plus a one-second
kill grace. The new eligibility command has a three-second outer timeout plus
a one-second kill grace and a two-second SQL statement timeout. The existing
monitor's own command timeouts are unchanged. Account for these additions in
lab operation timing before production rollout.

If attribute publication fails, the helper logs an error explicitly warning
that the prior score may remain. It retries on the next observation. Role
return codes and existing lifecycle results are preserved; shutdown is never
blocked by a score-write failure. **No RA can guarantee stale-score removal
while the cluster attribute service is unreachable.** This failure mode needs
lab validation and is not hidden by reporting a false PostgreSQL role.
Pacemaker normally serializes actions for a resource; parallel direct/manual
calls that race score updates are not supported by this implementation.

## Tests and evidence

Commands run from the source tree:

```bash
bash -n refactored/managed-container
for module in refactored/managed-container.d/*.sh; do bash -n "$module"; done
python3 tests/verify_refactor.py
python3 tests/test_promotion_scores.py
python3 tests/test_promotion_scheduler.py
python3 tests/test_promotion_cleanup.py
python3 tests/verify_promotion_sql.py
```

- Original 22 differential scenarios pass. All 29 original definitions and the
  same 12 metadata parameters / eight advertised actions are preserved.
- 25 scoring tests pass, including failures, stop/start, promote/demote/rejoin,
  stale positive replacement, repeated monitors, loss/recreation of transient
  attributes, missing explicit resource identity and score-publication errors.
- A real crm_attribute test uses CIB_file with a synthetic CIB. Repeated equal
  observations do not change the file; negative replacement and recreation
  after loss of transient attributes work.
- Six real crm_simulate tests on synthetic CIBs pass: absent and explicit
  Started target-role each permit one promotion; Unpromoted prevents it;
  larger scores win, ineligible nodes are excluded, and all-negative scores
  permit no promotion. Old resource attributes are seeded and do not alter
  this new resource's election. These are scheduler simulations, not live HA
  transitions. The synthetic CIB disables fencing only inside its test file;
  the live lab/production fencing configuration was not touched.
- Nine SQL cases pass in a separate ephemeral PostgreSQL 18 container with no
  host ports or data mounts, evaluating the actual production SQL against
  controlled state fixtures. Includes replay pause/lag, missing positions,
  stopped/absent receiver, and recovery-only function handling on a primary.
- Four cleanup-command planner tests pass, including ambiguous-rule rejection
  and preservation of unrelated rules.

The RPM %check now runs the regression, scoring and scheduler suites. PostgreSQL
integration is an explicit opt-in local test needing Podman and a local
postgres:18 image; it does not touch the lab postgres container.

## Specific disposable two-node lab sequence (after review)

Keep maintenance-mode=true as required by the lab handoff. Use direct RA
operations only after installing the reviewed version under a separate approved
installation procedure. Automatic scheduling claims must first be exercised
with a copied CIB and crm_simulate; do not disable maintenance mode as part of
these instructions. Do not use this sequence on production.

1. Save the lab CIB, installed executable/modules, and current state on both
   nodes. Verify the cluster is postgres-ra-test, the roles and streaming source,
   current PGDATA backup/recovery baseline, and that production is not targeted.
   Establish timing and monitor outputs from the existing implementation.
2. Run metadata and validate-all for the candidate on each lab node. For direct
   actions always supply the actual primitive ID (pg-ha in saved lab context),
   not the clone ID. Example observation commands on each lab node:

   ```bash
   sudo env OCF_RESOURCE_INSTANCE=pg-ha \
     /usr/lib/ocf/resource.d/loxthelion/managed-container monitor
   # Interpret rc=8 as healthy primary, rc=0 as healthy standby.
   sudo crm_attribute --promotion=pg-ha --lifetime=reboot --query --quiet
   ```

   Expect primary=1000 and a caught-up streaming standby=100. Repeat monitors;
   verify no writes for identical scores and no unrelated attribute changes.
3. On the standby, test full stop and start: stopped service/container and
   PostgreSQL must yield monitor=7 and score=-INFINITY; standby start restores
   monitor=0 and score=100 after streaming/catch-up. Verify primary start refusal
   remains intact. Do not use a successful role-return code as a score assertion.
4. Separately stop only standby PostgreSQL while keeping the maintenance
   container alive; observe -INFINITY, then start it through the proven standby
   path. Verify no rejoin is attempted while PostgreSQL is running.
5. Pause replay on the standby (lab only), observe monitor=0 and -INFINITY,
   resume replay and await catch-up/100. Test receiver interruption, SQL errors,
   stopped container, and a seeded prior positive score becoming ineligible.
6. Exercise the existing controlled switchover sequence in both directions:
   persist the fresh matching pre-demote notification on the old primary,
   demote through the Corosync-resolved successor, and promote the successor.
   Verify -INFINITY during transitions, 100 for a successful eligible demotion,
   1000 after promotion/invariant reconciliation, and persistent container
   identity throughout demotion. Return to DV6 primary / ELITEDESK standby.
7. Test two eligible standbys using the existing cascading-standby intermediate
   state. Both should have the same positive score; a copied-CIB simulation
   should select one with promoted-max=1. Do not promote both manually.
8. Reproduce the abrupt-primary-loss case in the isolated lab only after a
   recoverable baseline exists. Allow the surviving receiver to disappear,
   run its monitor, and inspect the score. This candidate is EXPECTED to deny
   promotion then. That is a rollout blocker unless the strict policy is
   explicitly chosen; it cannot be counted as a passed failover test.
9. Remove only the new resource's transient score in the lab, then reprobe:
   fresh eligibility must recreate it. Exercise score-service/query/update
   failure and recovery without confusing role codes. Repeated successful
   monitoring must replace any stale positive score after ineligibility.
10. With a saved CIB, simulate removal of only the temporary node preference.
    Retain target-role=Promoted, promoted-max=1, and the VIP constraints.
    Earlier absent/Started scheduler cases are compatibility tests, not
    production configuration recommendations.
11. Restore the baseline topology, streaming, test-induced settings and intended
    agent version. Confirm maintenance mode still true. Record both nodes'
    installed hashes, OCF codes, scores, receiver status and action durations.

## Production cleanup and rollout: gated on lab validation

The conceptual description does not include the actual preference constraint
ID, so inventing a deletion command would be unsafe. Read-only discovery is:

```bash
sudo pcs constraint config --full
sudo pcs resource config postgresql-ha-clone
sudo cibadmin --query > production-before-scoring.xml
```

Do not commit that CIB snapshot: it may contain sensitive deployment settings.
After lab validation and policy approval, scripts/plan-promotion-cleanup.py can
read the supplied snapshot and PRINT the exact commands using the real IDs:

```bash
python3 scripts/plan-promotion-cleanup.py production-before-scoring.xml \
  --clone postgresql-ha-clone --preferred-node david-eve --lab-validated
```

The planner never executes commands. It matches only the simple rule described
in the task (Promoted, score=200, #uname eq david-eve), refuses zero/multiple
matches, preserves other rules, and emits only the preference constraint/rule
removal. It requires and preserves direct clone-level target-role=Promoted.
Rule-based/inherited settings require manual inspection. Do not confuse
preference removal with clearing failed actions: former-primary failure state
must remain until rejoin succeeds and streaming standby status is verified.

There is no RA code that deletes constraints or changes clone metadata. No
production installation, cleanup, or restart was executed during that original
source-review session. Subsequent production validation is in ../docs/testing.md. Exact production
rollout commands are intentionally deferred until the policy, real constraint
ID, candidate hashes and successful lab results are known, as requested.

## Sources used for API semantics

- https://clusterlabs.org/projects/pacemaker/man/crm_attribute.8.html
- https://www.postgresql.org/docs/18/monitoring-stats.html#MONITORING-PG-STAT-WAL-RECEIVER-VIEW
- https://www.postgresql.org/docs/18/functions-admin.html
- Installed pcs help: `pcs constraint remove --help`, `pcs resource meta --help`.

The source diff is provided in PROMOTION-SCORING.patch. Source and test file
hashes are recorded in REFACTOR-SHA256SUMS, which is refreshed for this candidate.
The frozen reference hash remains unchanged.

RPM validation of this candidate also passed in the disposable Fedora 43 build
container: binary/source packages built, %check ran the regression, scoring,
scheduler and cleanup tests, DNF resolved the new Pacemaker dependency, and the
installed-RPM integrity/permissions/metadata checks passed. Local artifacts are
under dist/scoring-review/ and are not releases or deployment approval.

A final read-only inspection found that ELITEDESK's installed executable and
all nine modules match the earlier modular baseline at c8636ed exactly. Its
entrypoint SHA256 is c91f09adfdeb45a3be1a72df3f5ca5b02fd4555dec4397ad944eab439ace0b89.
The promotion-scoring candidate is not installed there.