Freeze lab-tested PostgreSQL plugin engine

Extract PostgreSQL, Podman runtime and systemd lifecycle adapters while preserving existing lifecycle and scoring behavior. Update RPM layout and add guarded operator-run lab harnesses. Record successful manual stop/start, promotion, demotion and rewind-only rejoin with pinned installed hashes. The unfenced lab does not validate automatic failover, Pacemaker-managed transitions or divergent-data rewind. Validation: 103 offline tests, original-function differential checks, checksum manifest and whitespace checks passed.
Freeze HA scoring and document reported production failover
2026-09-13 01:57:04 -04:00 · 2026-09-12 18:07:28 -04:00
53 changed files with 4127 additions and 294 deletions
@@ -14,7 +14,7 @@ jobs:
    steps:
      - name: Install build tools
        run: |
-          dnf -y install rpm-build python3 resource-agents git tar gzip diffutils findutils coreutils bash nodejs
+          dnf -y install rpm-build python3 resource-agents pacemaker-cli git tar gzip diffutils findutils coreutils bash nodejs
      - name: Check out source
        uses: https://github.com/actions/checkout@v4
        with:
@@ -25,6 +25,7 @@ jobs:
        run: |
          dnf -y install ./dist/*.noarch.rpm
          python3 tests/verify_installed_rpm.py
          MC_DISPOSABLE_STATE_TEST=1 python3 tests/verify_promotion_state_cli.py /usr/lib/ocf/resource.d/loxthelion/managed-container.d/applications/postgresql/promotion_state.py
      # v3 uses the artifact protocol supported by older act_runner instances.
      - name: Save development packages
        uses: https://github.com/actions/upload-artifact@v3
@@ -17,3 +17,6 @@ secrets/
 # RPM build trees and development artifacts
 .build/
 dist/
 # Generated review diffs; Git stores the reviewed source itself
 /notes/*.patch
@@ -1,3 +1,7 @@
 > Historical refactor brief. Current production operating policy and validation
 > are documented in [docs/operations.md](docs/operations.md) and
 > [docs/testing.md](docs/testing.md).
 # PostgreSQL HA OCF Resource Agent — Codex Refactor Handoff
 ## Objective
@@ -30,7 +34,9 @@ Control path:
      -> persistent PostgreSQL maintenance container
      -> PostgreSQL
-The maintenance container PID 1 is intentionally `sleep infinity`.
+The container entrypoint runs the bootstrap wrapper, which executes
 `sleep infinity` after its bootstrap gate. Podman may place its init shim at
 PID 1; PostgreSQL is not the container lifetime process.
 PostgreSQL itself is independently started/stopped with `pg_ctl`.
@@ -1,13 +1,21 @@
 # PostgreSQL HA OCF resource agent
-A behavior-preserving modular refactor of `ocf:loxthelion:managed-container`.
+A modular PostgreSQL HA implementation of `ocf:loxthelion:managed-container`.
 Pacemaker invokes the agent to manage a rootless systemd/Quadlet maintenance
 container. PostgreSQL is started and stopped independently with `pg_ctl`.
 This branch extracts the frozen implementation into a PostgreSQL plugin, Podman
 runtime adapter and systemd lifecycle adapter. It is **installed on ELITEDESK
 in the test cluster only**. [Basic lifecycle checks](docs/lab-plugin-basic-test.md)
 and [manual promotion, demotion and rewind-only rejoin tests](docs/lab-role-tests.md)
 passed according to operator-supplied output. The lab has no fencing. See the
 [engine contract and validation plan](docs/plugin-engine.md). The production test
 report applies to the earlier freeze, not automatically to this extraction.
 ## Project layout
 - `reference/`: original agent and immutable known-good reference copy.
- `refactored/`: executable agent and nine sourced shell modules.
+- `refactored/`: executable agent, application/runtime/lifecycle modules, and a Python state helper.
 - `tests/`: offline structural and behavioral comparison checks.
 - `cluster-context/` and `runtime-context/`: saved lab configuration and state;
  these snapshots are not live status. Environment credentials are redacted.
@@ -21,21 +29,27 @@ Requires Bash, Python 3, and the OCF shell library (normally
 ```bash
 python3 tests/verify_refactor.py
 python3 tests/test_plugin_engine.py
 python3 tests/test_promotion_scores.py
 python3 tests/test_promotion_scheduler.py
 python3 tests/test_disconnected_state.py
 python3 tests/test_promotion_cleanup.py
 sha256sum -c notes/REFACTOR-SHA256SUMS
 ```
-The tests compare the refactor against the frozen reference without operating
+The tests retain all 29 original functions and verify exact equivalence after
-the live cluster. Target-node validation and controlled HA transition tests
+reversing only adapter-call substitutions. They exercise the
-remain outstanding. Existing function names and recovery configuration blocks
+added promotion-scoring layer without operating the live cluster. The operator
-are retained in this initial modular baseline.
+has reported successful production SBD fencing, automatic promotion, VIP movement,
 and former-primary rewind/rejoin in a three-node cluster.
-Read [the refactor report](notes/REFACTOR-REPORT.md) for installation commands,
+Read [operations](docs/operations.md), [testing and snapshot provenance](docs/testing.md),
-validation results, limitations, and the recommended lab comparison sequence.
+and [the review findings](docs/review.md). Historical lab snapshots and notes are
-The public entrypoint remains
+not current production configuration. Keep `target-role=Promoted`, single-primary
-`/usr/lib/ocf/resource.d/loxthelion/managed-container`.
+limits, and the VIP's Promoted-role colocation and promotion-before-start ordering.
-Keep lab Pacemaker maintenance mode enabled during this work. Do not modify
+The public entrypoint remains `/usr/lib/ocf/resource.d/loxthelion/managed-container`.
-the frozen reference or automatically run destructive HA tests.
+No production changes are part of this documentation freeze.
 ## Clone
@@ -49,4 +63,12 @@ Gitea Actions builds development RPMs on `main` pushes and pull requests using
 an `ubuntu-latest` runner with a Fedora 43 job container. See
 [packaging instructions](packaging/README.md) for local builds and artifacts.
 [License provenance remains pending](packaging/PROVENANCE.md); release publishing
-and deployment are not configured.
+and automated deployment are not configured.
 ## Promotion scoring
 Scores are 1000 for a healthy primary, 100 for a strictly verified streaming
 standby, 50 for a recently verified eligible disconnected standby, and -INFINITY
 otherwise. See [the scoring design](notes/DISCONNECTED-STANDBY.md) for identity,
 local state ownership, the 300-second default freshness policy and limitations.
 Fencing remains mandatory; scores do not authorize unfenced manual promotion.
@@ -0,0 +1,11 @@
 a4b775a0ddcabdb0558da9b34c7fe2a883c07d0ef308c1c7224526abf53f63a0  managed-container
 93191771d3cb8aefadb2a16d2f89d64a8b6c584071a473439be818237bdf7f17  managed-container.d/auth.sh
 97e93d9164e854e5f3688cf9d3cc656b001eb30ef3c2073c907450ca43588f3d  managed-container.d/common.sh
 ac6bc363a2301f1491d162ef609103dbfbceb3b099a3556614c6165952309374  managed-container.d/container.sh
 0af3bb362a303017372c631354ba94ef9e723bf14e64d2eff91b6662e54a5be0  managed-container.d/demote.sh
 6a98fcdda82cea9365db95b4ee779be5c567b3d846f48b90fbc8c254af697670  managed-container.d/notify.sh
 d2779ce6e7d1f426ffca2e41bbb8de77503005dc38d658d69f184020365796eb  managed-container.d/postgres.sh
 3085ec9b4410905ab9d4370786ed776eac3470a0415aa1b4f608b083945aeb58  managed-container.d/promote.sh
 21beea72bcb7f24a51c17ea48666ac87df9436702ad77b166b45f4647e28e8ad  managed-container.d/rejoin.sh
 6b2021887e3b4cd6a12490e7a8dff98320b2133b472ab25a22a0eead46137547  managed-container.d/replication.sh
 5166fc3370f49ab14cd975ded4b03a74b9bd7dbff5762b41a15552c42f934ea7  managed-container.d/promotion_state.py
@@ -0,0 +1,15 @@
 01eb9b161c8e3db289f09bf63953be779921e15bdf8b1a4d8ce7e60032b826e4  /usr/lib/ocf/resource.d/loxthelion/managed-container
 10932919c4339b553c7c9dbf30b45a917c51c0a08f362e12678003d58b12fa82  /usr/lib/ocf/resource.d/loxthelion/managed-container.d/applications/postgresql/auth.sh
 da4e58cadf1df3953cb97a0512e3931ee1b9d3ee61c39a1ad1a6047ef537addc  /usr/lib/ocf/resource.d/loxthelion/managed-container.d/applications/postgresql/demote.sh
 6a98fcdda82cea9365db95b4ee779be5c567b3d846f48b90fbc8c254af697670  /usr/lib/ocf/resource.d/loxthelion/managed-container.d/applications/postgresql/notify.sh
 d3a989de97ceb61a56828f006d5ccb87b7c34fc155a944585bedf928785021fc  /usr/lib/ocf/resource.d/loxthelion/managed-container.d/applications/postgresql/plugin.sh
 fa222e6767249999767eb82fac2d85e7e35404da240517a8b630571e77282053  /usr/lib/ocf/resource.d/loxthelion/managed-container.d/applications/postgresql/postgres.sh
 3085ec9b4410905ab9d4370786ed776eac3470a0415aa1b4f608b083945aeb58  /usr/lib/ocf/resource.d/loxthelion/managed-container.d/applications/postgresql/promote.sh
 5166fc3370f49ab14cd975ded4b03a74b9bd7dbff5762b41a15552c42f934ea7  /usr/lib/ocf/resource.d/loxthelion/managed-container.d/applications/postgresql/promotion_state.py
 28b0e44554a88ccb87cd16a68315c045dd37d354aceb2fa25da7c7399a659b23  /usr/lib/ocf/resource.d/loxthelion/managed-container.d/applications/postgresql/rejoin.sh
 6b2021887e3b4cd6a12490e7a8dff98320b2133b472ab25a22a0eead46137547  /usr/lib/ocf/resource.d/loxthelion/managed-container.d/applications/postgresql/replication.sh
 a482ef3cc1048e6a4f427e76dd2dce173ffd00760dbed1afa601a3fd05b0ad99  /usr/lib/ocf/resource.d/loxthelion/managed-container.d/common.sh
 3a27ac9c63c61310da8b8711fa2c4400b37ea8eae2ee2a79e28bc63180b06c7c  /usr/lib/ocf/resource.d/loxthelion/managed-container.d/container.sh
 53978e2367394695d61c6fb38f2ee46814474c5a4e0448d2692f1a3ce583af93  /usr/lib/ocf/resource.d/loxthelion/managed-container.d/engine.sh
 6c92c095ce10ac95d843fdc1c49e31c19491ca22fdac960affd8649606d76d82  /usr/lib/ocf/resource.d/loxthelion/managed-container.d/lifecycles/systemd.sh
 dc5820067a78c4a1c531426902793aae97a03fd64912b017b3b102031dcc1e04  /usr/lib/ocf/resource.d/loxthelion/managed-container.d/runtimes/podman.sh
@@ -0,0 +1,119 @@
 # ELITEDESK plugin installation and basic checks
 Status: installed on DAVID-ELITEDESK; operator-reported basic checks passed.
 All 15 installed code files were subsequently verified read-only against the
 working candidate and matched byte-for-byte.
 No production or peer-node actions are part of this procedure.
 Read-only preflight on DAVID-ELITEDESK found its PostgreSQL container running as
 an asynchronous streaming standby of 192.168.1.2 using ha_david_elitedesk.
 All 11 installed RA/module/helper files match source freeze 488a167; their exact
 hashes are captured in lab-installed-before-plugin.sha256. This is a lab snapshot,
 not the outstanding production snapshot. Current cluster maintenance state could
 not be read without sudo; the installer verifies it live before any replacement.
 Run in the operator's terminal:
 ```sh
 sudo python3 /home/david/codex-postgres-ha-refactor/scripts/test-elitedesk-plugin.py
 ```
 The command requires the existing postgres-ra-test cluster, exactly the two lab
 nodes, pg-ha under pg-ha-clone, expected local container/database settings, and
 maintenance-mode already true. It does not enable maintenance mode or modify CIB
 configuration. It copies parameters from the live resource and rejects ambiguous
 parameter sets. Native sudo prompts stay in the terminal; never share a password.
 It verifies source checksums, stages root-owned code, backs up the installed agent
 and modules under /var/backups/managed-container-plugin-lab-*, validates staged
 metadata and configuration, then replaces the complete module tree and entrypoint.
 The installation remains in place after success. A failed post-install check
 restores the old agent and module tree and retains diagnostic files.
 Basic checks are metadata (16 parameters), validate-all, monitor, status, an
 idempotent start on the already-running standby, and another monitor. Before and
 after checks require the same PostgreSQL system identifier, postmaster start time,
 recovery role, streaming sender and slot. No stop, promote, demote, rejoin or
 fencing is invoked. Normal monitors/start may refresh local promotion evidence
 and the pg-ha promotion score; those expected side effects are part of testing.
 A score of -INFINITY is recorded rather than mistaken for a failed role monitor;
 strict catch-up can fluctuate. Investigate that result before broader HA testing.
 The script prints its backup/result directory and action results. It records
 results.json and installed-SHA256SUMS there with root-only access. Return the
 terminal output before progressing to disruptive stop/start or switchover tests.
 Preparation validation: current structural/differential tests passed; two
 isolated installer tests passed, covering basic-action-only success and restoration
 of both agent and modules on an injected installed-monitor failure. These tests
 mock privilege/cluster boundaries and touch only temporary directories; they do
 not claim live installation success.
 ## Completed basic lab run
 The operator supplied successful installer output for backup/result directory
 `/var/backups/managed-container-plugin-lab-mow_t6zi`:
 - Staged metadata and validate-all: rc=0.
 - Installed validate-all, monitor, status, idempotent start, monitor: rc=0.
 - Final promotion score: 100.
 - The installer reported PostgreSQL was not restarted and maintenance remained true.
 - No stop, promote, demote, rejoin, fencing, or peer-node actions ran.
 Codex independently compared every installed entrypoint/module/helper file with
 the working candidate using read-only filesystem access; all 15 matched. Exact
 installed SHA-256 values are recorded in [lab-installed-plugin.sha256](lab-installed-plugin.sha256).
 The root-only results.json was not independently read; the action results above
 are attributed to the operator-supplied output. This validates the installation
 and non-disruptive standby path, not actual stop/start, switchover or failover.
 Installed-file comparison captured at 2026-09-13T03:01:11.114949+00:00 UTC.
 ## Controlled standby stop/start — operator-reported pass
 The operator ran:
 ```sh
 sudo python3 /home/david/codex-postgres-ha-refactor/scripts/test-elitedesk-cycle.py
 ```
 This briefly interrupts only the ELITEDESK standby. The script requires the same
 lab identity, maintenance=true, streaming sender/slot and exact installed hashes.
 It validates/monitors, runs RA stop, checks systemd inactive and the container
 stopped/absent, expects monitor=7 and score=-INFINITY, then runs RA start. It
 requires the same database identifier, a new postmaster start time, recovery and
 streaming from DV6, monitor/status=0 and score=100 within the verification window.
 Each RA action has a 60-second timeout with a 5-second kill grace. Post-start
 streaming/eligibility checks have a 60-second retry window (an in-flight bounded
 check can finish after the deadline). No source files or cluster configuration
 are replaced, and no promotion, demotion, rejoin, fencing or peer action runs.
 Failures halt without automatic recovery and may leave the standby stopped.
 Inspect the printed /var/backups/managed-container-cycle-lab-* result directory
 before proceeding. It contains root-only results.json and per-action output.
 Three mocked harness tests passed: successful cycle, stop failure without start,
 and start failure without retry. Those are preparation checks, not live results.
 ### Completed cycle result
 The operator supplied successful output with evidence directory
 `/var/backups/managed-container-cycle-lab-b1048r6q`:
 | Check | Result |
 | --- | --- |
 | Initial validate-all and monitor | rc=0 |
 | RA stop | rc=0 |
 | Monitor while stopped | rc=7 |
 | Stopped lifecycle state | systemd inactive; container stopped/absent |
 | Stopped promotion score | -INFINITY |
 | RA start, subsequent monitor and status | rc=0 |
 | Database identity | Same system identifier; new postmaster start time |
 | Replication after restart | Streaming from DV6; score 100 |
 | Cluster maintenance | Remained true |
 These results are attributed to the operator-supplied script output; Codex has
 not independently read the root-only results.json. No promotion, demotion,
 rejoin, fencing or peer-node actions ran. The basic standby lifecycle now has
 live lab coverage; this does not establish switchover/failover behavior for the
 plugin extraction or constitute production deployment validation.
@@ -0,0 +1,113 @@
 # Staged lab role tests
 **All five live stages passed according to operator-supplied terminal output.**
 The sequence followed the successful ELITEDESK stop/start cycle. Only the two-node postgres-ra-test lab is supported; maintenance
 must remain true throughout. DV6 retains the frozen RA and ELITEDESK has the
 plugin candidate. Neither implementation is installed or changed by this tool.
 The operator confirmed that this lab has no fencing; fencing is available only
 in production. This checkpoint covers controlled, manual role transitions with
 a reachable primary. Abrupt-primary-loss automatic failover is not validated
 for the plugin candidate. The earlier production fencing report applies to the
 earlier implementation, not this extraction.
 During maintenance, Pacemaker's recurring role monitors are paused. The harness
 invokes the RA directly, so its action results do not update Pacemaker's recorded
 operation history. A Cockpit cluster role display can therefore remain stale
 even while the RA explicitly updates promotion scores. Database roles were
 verified using direct SQL; systemd/container running state remains distinct
 from PostgreSQL role. Maintenance was not disabled to refresh the display.
 `scripts/test-lab-roles.py` is a portable, operator-run harness with embedded
 hashes from both lab snapshots. It needs root locally and noninteractive SSH
 as david to the peer for read-only observations. SSH host keys must already be
 verified. Passwords are entered in the operator's terminal, never in chat.
 Run `sudo python3 /path/to/test-lab-roles.py inspect` on each node first.
 Inspection checks cluster name, membership, maintenance, resource parameters,
 installed hashes, SQL readiness, roles, system identity and maintenance service.
 It changes no database or cluster state. Do not proceed if either check fails.
 Each stage requires a separate invocation and review of its successful output:
 | Order | Host | Stage argument | Expected result |
 | --- | --- | --- | --- |
 | 1 | DV6 | demote | DV6 streams from ELITEDESK; both are standbys |
 | 2 | ELITEDESK | promote | ELITEDESK primary, DV6 streaming standby |
 | 3 | ELITEDESK | demote | ELITEDESK streams from DV6; both are standbys |
 | 4 | DV6 | promote | DV6 primary, ELITEDESK streaming standby |
 | 5 | ELITEDESK | rejoin | Stop PostgreSQL only, run rewind-only rejoin to DV6, verify streaming |
 The demote stage submits a **synthetic lab pre-demote notification** through the
 RA notify action, then invokes demote. This tests the notification consumer and
 controlled demotion; it does not prove Pacemaker generates/orders notifications.
 The RA retains its own catch-up and successor checks. The harness refuses promote
 unless the peer is already in recovery and streaming from the local standby.
 Between demote and promote there is deliberately no writable primary. Arrange
 both terminals before starting; do not perform other manual database operations
 during the sequence. Scores on both standbys can fluctuate during this interval.
 The rejoin stage is restricted to the restored ELITEDESK standby. It explicitly
 sets basebackup fallback=false. It tests the stopped-PostgreSQL precondition,
 maintenance container survival, source authentication, pg_rewind invocation,
 recovery configuration, PostgreSQL restart, streaming and score restoration.
 Because it does not create divergent data, it does **not** validate divergent
 former-primary rewind repair. No fencing, data replacement fallback, VIP moves,
 Pacemaker-managed switchover or automatic failover are tested here.
 Each modifying invocation writes root-only results/action logs beneath
 `/var/backups/managed-container-roles-lab-*`. Failures halt without automatic
 retry/recovery. A failure after demotion can leave no primary; a rejoin failure
 can leave ELITEDESK stopped. Stop the sequence and inspect the evidence rather
 than running the next stage. Do not clear failed actions or change maintenance.
 The checks sample state; they cannot prevent a separate administrator from
 changing roles concurrently. Controlled demotion of a reachable primary is the
 basis for these manual lab promotions. This is not a procedure for promoting
 after an unreachable primary failure; that requires fencing and Pacemaker's
 ordering under the operating policy.
 ## Completed live sequence
 The operator supplied successful output for each stage below. Evidence directories
 are on the host named in the table, under `/var/backups/`. Codex has not read the
 root-only results.json or action logs; these are operator-reported passes, not
 independently witnessed live actions.
 | Host | Stage | Result | Evidence directory |
 | --- | --- | --- | --- |
 | DV6 | demote | rc=0; standby monitor=0; score=100 | managed-container-roles-lab-o1xjobi2 |
 | ELITEDESK | promote | rc=0; primary monitor=8; score=1000 | managed-container-roles-lab-qdz57g70 |
 | ELITEDESK | demote | rc=0; standby monitor=0; score=100 | managed-container-roles-lab-nwbzdxnl |
 | DV6 | promote | rc=0; primary monitor=8; score=1000 | managed-container-roles-lab-r69amyxg |
 | ELITEDESK | rejoin | stopped monitor=7; rejoin=0; standby monitor=0; score=100 | managed-container-roles-lab-7c4vw7_p |
 Each stage reported preserved database identity and maintenance container,
 verified roles and streaming, and maintenance=true. The final reported state is
 DV6 primary and ELITEDESK streaming standby. The rejoin harness also checks the
 stopped score is -INFINITY before invoking rejoin, with basebackup disabled.
 This establishes successful pg_rewind invocation, not repair of deliberately
 divergent timelines or a measured amount of rewound data.
 The supplied inspection after ELITEDESK demotion showed both nodes in recovery,
 streaming from each other, with replay LSN 0/A02B560 and system identifier
 7684451577990111248. An earlier inspection in the same paste showed DV6 primary;
 the later observation was used to resolve the role confusion before restoring DV6.
 Both-way SSH access was established and independently checked before the role
 sequence. The installed code was pinned to the existing lab snapshots. No RA
 installation, maintenance change, fencing, VIP move or failure cleanup was part
 of these tests. The limitations above continue to apply.
 Offline original-function comparisons and 97 tests passed before the live
 sequence (the 11 scheduler tests required execution outside the sandbox).
 Six additional role-harness safety-gate tests passed before staging the harness.
 ## Commit checkpoint verification
 On 2026-09-13, all 103 tests in `python3 -m unittest discover -s tests -p
 'test_*.py'` passed, including offline scheduler simulations.
 `python3 tests/verify_refactor.py` also passed: frozen reference hashes, all 29
 original functions after exact adapter-call reversal, syntax, metadata and
 22 differential lifecycle cases. No additional live actions were run for this
 commit checkpoint.
@@ -0,0 +1,131 @@
 # PostgreSQL HA operating policy
 This document describes the intentional production design. Historical two-node
 lab snapshots and old installation notes are not production runbooks. Keep the
 three-node production clone's **target-role=Promoted** and **promoted-max=1**.
 Promotion scores select an eligible node; the target role expresses the desired
 promoted service. This project requires that explicit configuration even though
 Pacemaker can permit promotion with other default target-role settings.
 The VIP must be colocated with the Promoted instance and ordered to start after
 promotion, with reverse shutdown ordering. Scores do not create those constraints.
 ## Ownership and lifecycles
 Pacemaker invokes the root OCF RA. Its `runuser` boundary enters the rootless
 service owner's home and user-systemd runtime, then invokes systemd/Podman with
 argument arrays. Quadlet constructs the maintenance container and supplies its
 environment, mounts and bootstrap. The bootstrap gates initialization/adoption,
 then executes `sleep infinity`; Podman may use an init shim as PID 1. PostgreSQL
 is not the container lifetime process.
 The RA independently controls PostgreSQL with `pg_ctl`. During demotion and
 rejoin, the maintenance container stays available even when PostgreSQL is
 stopped. This is required for inspection, `pg_rewind`, and optional basebackup.
 The OCF stop action stops both PostgreSQL and the maintenance service/container.
 Starting the container alone is not equivalent to starting the OCF resource.
 The RA reads PGDATA and intentional replication settings from the running
 container. It does not parse or source host PostgreSQL env/config files. Quadlet
 may intentionally consume a host env-file when constructing that container.
 The bootstrap manages its secret-file/pgpass interfaces separately; the RA's
 replication commands read `POSTGRES_REPLICATION_PASSWORD` inside the container
 and set PGPASSWORD only for the child process.
 ## Actions and role reporting
 | Action | Implementation contract |
 | --- | --- |
 | start | Starts or verifies an Unpromoted standby. Refuses an already-primary instance; when PostgreSQL is stopped, refuses PGDATA without standby.signal. |
 | monitor/status | Reports healthy standby 0, healthy primary 8, stopped service/container/PostgreSQL 7, and unhealthy/failed query 1. Promotion eligibility does not change these role codes. |
 | promote | Promotes a healthy standby and reconciles local HA slots/rewind grants; an existing primary reconciles those invariants. |
 | demote | Controlled switchover: uses fresh pre-demote notification and Corosync successor address, verifies successor standby/catch-up, stops local PostgreSQL and rejoins it as a cascading standby. |
 | stop | Stops PostgreSQL, then the maintenance service/container. |
 | notify | Persists Pacemaker transition context; not promotion eligibility or fencing proof. |
 | rejoin | Private/manual recovery action, deliberately absent from advertised OCF actions. Requires a running maintenance container, stopped PostgreSQL, source and slot. Rewinds then verifies streaming recovery. |
 | validate-all/meta-data | Validate dependencies/configuration and report metadata. Validation can return 5 for missing utilities or 6 for invalid configuration. |
 Lifecycle success is 0; monitor's 8 is a role report, not a generic lifecycle
 success. Promoted monitoring must expect the correct role. Metadata advertises
 16 parameters and eight actions; status is an alias and rejoin stays private.
 The scoring wrappers withdraw scores before lifecycle actions and reassess after
 successful actions. The engine extraction preserves the original 29 functions with explicit
 adapter-call substitutions; see plugin-engine.md.
 ## Scoring and fencing
 Healthy primary: 1000. Strictly verified streaming standby: 100. Recently verified
 eligible disconnected standby: 50. Unsafe or missing evidence: -INFINITY.
 Disconnected evidence is bounded by `disconnected_max_age` (default 300 seconds),
 bound to the PostgreSQL incarnation and local boot, and recalculated each monitor.
 See [the detailed state model](../notes/DISCONNECTED-STANDBY.md).
 An eligible score is not a fencing certificate. Pacemaker/SBD must isolate the
 failed primary and order promotion safely. Never bypass fencing by manually
 calling promote, creating standby.signal on divergent data, or deleting data.
 Asynchronous replication accepts possible loss of acknowledged transactions;
 the freshness window does not bound the number of lost transactions.
 Controlled switchover differs from abrupt failover: demote requires a reachable,
 caught-up successor and transition notification. Former-primary rejoin is a
 separate recovery procedure after fencing, not an automatic start workaround.
 ## Former-primary recovery order
 1. Establish that fencing completed and identify the surviving writable primary
   and its VIP. Coordinate cluster management so Pacemaker cannot race the manual
   repair. Do not clear the former primary's failed start yet.
 2. Bring up only its maintenance container using the existing Quadlet as the
   service owner. Verify PostgreSQL is stopped. Preserve the existing data.
 3. Confirm the running container's replication credentials are correct without
   printing them. Supply the existing resource instance/parameters and a unique
   local replication slot to the private rejoin action, using the PostgreSQL VIP
   as `rejoin_source_host`. This documentation does not invent missing live IDs.
 4. Rejoin first attempts pg_rewind. If it fails, stop and inspect the error.
   Basebackup fallback remains disabled unless explicitly authorized to replace
   divergent data. It preserves the old directory but is still a destructive
   recovery decision; a failed rewind is not guaranteed to leave usable data.
 5. Require rejoin success AND verify `pg_is_in_recovery() = true`, a streaming WAL
   receiver connected to the intended primary, and the expected slot/application
   identity. Verify the maintenance container remained available.
 6. **Only now** clear the relevant failed actions for that resource on that node,
   and allow Pacemaker to adopt it as Unpromoted. Recheck roles, scores, streaming,
   VIP placement, and failed actions. Do not use broad cleanup before recovery.
 Safe inspection commands, run on the appropriate node, include:
 ```sh
 sudo pcs status --full
 sudo pcs resource config postgresql-ha-clone
 sudo pcs constraint config --full
 ```
 Inspect output locally; raw CIB, environment dumps and logs may contain sensitive
 configuration and should not be committed. The optional preference planner reads
 an operator-supplied CIB and prints only a narrowly matched temporary preference
 removal. It preserves target-role=Promoted and all unrelated constraints. It does
 not perform resource failure cleanup or make a node safe to rejoin.
 ## Podman env-file incident
 The production test exposed a quoting mismatch. Podman's env-file parser retained
 surrounding quote characters, while the generated pgpass had the intended value.
 Streaming worked, but rejoin's direct environment password failed authentication.
 These are intentionally fake examples, not deployment values:
 ```text
 # Incorrect if the intended value contains no quotation marks:
 POSTGRES_REPLICATION_PASSWORD="EXAMPLE_ONLY_NOT_A_SECRET"
 # Correct representation of that same intended value:
 POSTGRES_REPLICATION_PASSWORD=EXAMPLE_ONLY_NOT_A_SECRET
 ```
 Podman env-file syntax is not shell assignment syntax; do not source it to
 validate container behavior. Correct the secret locally and apply it through the
 planned Quadlet/container lifecycle: editing a file alone does not alter an
 already-running container's environment. Do not blindly restart a primary to
 apply this change. Keep real values, .pgpass contents, secret files, and raw
 container environment inspection out of Git and shared logs.
 References: [OCF role semantics](https://github.com/ClusterLabs/pacemaker/blob/main/doc/sphinx/Pacemaker_Administration/agents.rst),
 [Pacemaker configuration and ordering](https://clusterlabs.org/projects/pacemaker/doc/3.0/Pacemaker_Explained/pdf/Pacemaker_Explained.pdf),
 and [Podman run options](https://docs.podman.io/en/latest/markdown/podman-run.1.html).
@@ -0,0 +1,210 @@
 # Podman plugin engine: first extraction
 Development baseline: freeze commit `488a167`. This branch extracts the existing PostgreSQL/Podman/systemd implementation.
 It is installed on DAVID-ELITEDESK in the lab only; see lab-plugin-basic-test.md. Production
 hash provenance is still pending as described in testing.md. Quadlet, bootstrap and cluster configuration remain unchanged. The lab RA files
 were replaced, and basic monitor/start checks refreshed standby scoring without
 restarting PostgreSQL. No production installation has occurred.
 ## Scope and supported configuration
 The default and only supported combination is:
 ```text
 application=postgresql
 container_runtime=podman
 lifecycle_manager=systemd
 ```
 These are advertised OCF instance parameters, not Pacemaker meta-attributes.
 Existing resource configurations need no new parameters. Missing or empty
 selectors use the defaults. Explicit unsupported values, including Docker or
 direct-runtime lifecycle, fail with OCF_ERR_CONFIGURED (6) before operations.
 The entrypoint/provider and every existing parameter default remain unchanged.
 The engine loads a fixed allowlist of packaged files beside the entrypoint.
 There is no arbitrary plugin pathname, dynamic shell evaluation, network plugin
 installation, or fallback to another runtime. Every implementation declares API
 version 1; missing files, incompatible versions and missing required hooks fail
 configuration. Packaged code is trusted/root-owned, as was the original RA.
 ## Boundaries
 ```text
 Pacemaker OCF action
  -> shared scoring/action wrappers
     -> applications/postgresql/plugin.sh
        -> existing PostgreSQL lifecycle and recovery modules
           -> runtimes/podman.sh (inspect / exec / info)
           -> lifecycles/systemd.sh (user service lifecycle)
              -> unchanged as_service_user privilege boundary
 ```
 `engine.sh` selects and checks plugins and adds selector metadata. `common.sh`
 contains the existing score-publication and action wrappers, now calling the
 application interface. `container.sh` retains service-user access plus
 compatibility health helpers that delegate to the selected adapters.
 `applications/postgresql/plugin.sh` owns the existing PostgreSQL defaults, metadata,
 validation and seven PostgreSQL/HA modules. Application modules and the state
 helper live together under
 `applications/postgresql/`; runtime and lifecycle adapters occupy their own
 `runtimes/` and `lifecycles/` directories. The loader uses fixed relative paths.
 Future applications will have their own
 metadata and dependency validation rather than inherit PostgreSQL assumptions.
 The packaged layout is:
 ```text
 managed-container.d/
  engine.sh
  common.sh
  container.sh
  applications/postgresql/
    plugin.sh
    postgres.sh, auth.sh, replication.sh, notify.sh
    promote.sh, demote.sh, rejoin.sh
    promotion_state.py
  lifecycles/systemd.sh
  runtimes/podman.sh
 ```
 The PostgreSQL plugin sets its own `MC_APPLICATION_DIR` for sibling modules and
 the Python state helper. Host evidence remains at its existing `/run` location;
 only packaged code paths changed. The public RA entrypoint is unchanged.
 ## Interface contract, API 1
 All interfaces are sourced Bash functions in a single RA invocation. Arguments
 are passed as arrays using `"$@"`; adapters do not concatenate or evaluate command
 strings. Existing PostgreSQL shell snippets remain explicit arguments to `sh -c`;
 this extraction does not redesign or broaden their trusted configuration inputs.
 | Interface | Contract |
 | --- | --- |
 | mc_runtime_exec ARGS... | Execute in the container through the service owner; preserve stdout, stderr, and exact exit status. |
 | mc_runtime_exec_bounded DURATION ARGS... | Same execution with timeout placed inside the service-user boundary; retain kill-after=1s. Current scoring callers pass 3s. |
 | mc_runtime_inspect ARGS... | Read container properties through the same runtime/user boundary. |
 | mc_runtime_info ARGS... | Validate runtime access through the service owner. |
 | mc_lifecycle_start SERVICE | Start the configured user-systemd service. |
 | mc_lifecycle_stop SERVICE | Stop that service, preserving failures. |
 | mc_lifecycle_running SERVICE | systemctl --user is-active --quiet. |
 | mc_lifecycle_cat SERVICE | Inspect service definition for existing validation. |
 | mc_application_monitor | Return the application's normalized OCF health/role code. |
 | mc_application_standby_score | Emit the eligible standby score, or fail; PostgreSQL owns evidence and freshness. |
 | mc_application_action ACTION | Dispatch only start/stop/promote/demote/rejoin to the existing application functions. |
 | mc_application_validate | Run existing application/configuration validation. |
 | mc_application_metadata | Return application metadata without runtime or CIB access. |
 | mc_application_notify | Process the existing transition notification. |
 The runtime exec argument grammar currently follows the existing callers:
 `--user USER`, optional `--env KEY=VALUE` entries, container name, command and
 arguments. Inspect retains the existing Go-template argument. This is a narrow
 Podman adapter contract, not a claim that every future runtime accepts it.
 A second runtime must explicitly implement and test those semantics.
 ## Normalization and state
 For this extraction the existing PostgreSQL monitor is the normalizer: it turns
 PostgreSQL's role/health observations into OCF codes. No extra JSON protocol or
 new independently maintained role state is introduced.
 | Observation | Monitor code |
 | --- | ---: |
 | Healthy standby | 0 (Unpromoted) |
 | Healthy primary | 8 (Promoted) |
 | Container absent/stopped, or PostgreSQL stopped | 7 (not running) |
 | PostgreSQL unready or failed/invalid role query | 1 (generic error) |
 Container availability, application health/role and promotion eligibility remain
 separate concepts. Container-running/PostgreSQL-stopped is a legitimate
 maintenance condition, but is not OCF start success. A healthy standby can have
 score -INFINITY without losing its Unpromoted role report. Core score validation
 still replaces malformed/failed score output with -INFINITY.
 All existing scores, freshness checks, persisted identity/LSNs, timeout locations,
 notification handling, fencing assumptions, standby-only start and manual fenced
 rejoin rules are preserved. Direct application-plugin calls are internal;
 Pacemaker dispatch continues through the score/lifecycle wrappers.
 ## Maintenance and lifecycle ownership
 Quadlet remains responsible for the bootstrap entrypoint, `Exec=sleep infinity`,
 `RunInit=true`, stop signal/timeout, mounts, image, environment and systemd service
 policies. `SuccessExitStatus=143` is systemd's classification of its managed
 service; the runtime exec adapter must still return 143 as an application-command
 failure. `Restart=on-abnormal` remains solely in the existing unit. No new
 restart loop, container creation, container replacement or supervision is added.
 During demote/rejoin, PostgreSQL may be stopped while its container remains
 available. Ordinary stop still stops PostgreSQL before asking systemd to stop
 the container service. These ordering rules remain in the PostgreSQL lifecycle
 functions; the adapters do not reorder them.
 ## Preservation and validation
 The original 29 functions are retained by name. Some now call adapter functions,
 so their literal hashes differ. `tests/verify_refactor.py` reverses only the
 explicit command-boundary substitutions and compares every remaining function
 byte with the immutable reference. It also retains all 22 differential lifecycle
 cases and verifies original metadata plus the four accumulated added parameters
 (freshness plus three selectors). Existing scoring tests use the actual engine
 loader and still exercise the original handlers against fake external boundaries.
 `tests/test_plugin_engine.py` checks argument fidelity (including quotes, spaces,
 newlines and literal shell syntax), lifecycle ownership, exit-code propagation,
 timeout placement, application dispatch, metadata discovery, missing/incompatible
 plugins, and unsupported/path-like selections. It also checks that PostgreSQL
 modules no longer invoke Podman/systemd or service-user execution directly.
 Run:
 ```sh
 python3 tests/verify_refactor.py
 python3 tests/test_plugin_engine.py
 python3 tests/test_promotion_scores.py
 python3 tests/test_disconnected_state.py
 python3 tests/test_promotion_scheduler.py
 python3 tests/test_promotion_cleanup.py
 python3 tests/verify_promotion_sql.py
 ```
 The last command uses only its own disposable PostgreSQL container. RPM checks
 include the plugin tests and verify discovery from the staged installation and
 an unrelated working directory. No live HA transitions are part of this branch's
 validation. Live lab authorization and actual configuration comparison remain
 necessary before deployment.
 ## Follow-on work
 After this extraction passes lab validation, a separate change can add direct
 Podman lifecycle management for pre-created maintenance containers. That work
 must define exit/restart behavior and sole lifecycle ownership. Container creation,
 RA-coordinated standby provisioning, a second runtime or another application are
 separate extensions. The current PostgreSQL plugin is intentionally still the
 only supported application; this is not yet a general multi-application framework.
 ## Results for this extraction
 Passed: 22 original differential lifecycle cases, 25 scoring tests, 16 state/monitor
 tests, 11 scheduler tests, 4 preference-planner tests, 11 engine contract tests,
 and all 9 PostgreSQL 18 SQL scenarios. Fedora 43 RPM build, disposable installation,
 15-file ownership/mode/payload checks, and the installed state-helper CLI passed.
 Folder-layout build log: `/tmp/mc-plugin-folders-rpm.log`; development artifacts:
 `dist/plugin-folders/`. Those build and offline checks did not run live HA transitions; subsequent
 operator-run lab transitions are recorded below.
 The core still retains the existing score vocabulary and freshness-parameter
 validation for this sole supported PostgreSQL profile. API 1 is an internal
 extraction interface, not a stable third-party plugin SDK. Generalizing these
 remaining policy assumptions should be driven by a second real application,
 not assumed complete by this milestone.
 Basic ELITEDESK installation checks subsequently passed with score 100; all 15
 installed files match the candidate. A controlled ELITEDESK standby stop/start subsequently passed according to
 operator-supplied output: stopped score -INFINITY, resumed streaming score 100,
 same database and new postmaster. The operator subsequently reported successful controlled switchovers in both
 directions and a rewind-only rejoin on ELITEDESK; see [role test results](lab-role-tests.md).
 Fencing, Pacemaker-managed transitions and divergent-data rewind remain untested
 with this extraction.
 See [the lab record](lab-plugin-basic-test.md) for evidence attribution.
@@ -0,0 +1,115 @@
 # Freeze review — 2026-09-12
 This review preserves the implementation reported as production-tested by the
 operator. No production systems were changed. It is a source/documentation
 freeze, not certification of every failure mode or a byte-level deployment
 attestation. Production hash collection remains outstanding in [testing.md](testing.md).
 ## Corrections and their reasons
 - Added operations.md to distinguish container lifecycle, PostgreSQL lifecycle,
  Pacemaker state, scores, fencing, controlled switchover and former-primary rejoin.
 - Added testing.md to record the operator's SBD failover/VIP/rewind/rejoin sequence
  and final three-node roles/scores, separately from local automated checks.
  Recorded exact repository RA/module/helper/bootstrap hashes without inventing
  deployed hashes. Included read-only per-node hash collection commands.
 - Required target-role=Promoted, promoted-max=1, Promoted-role VIP colocation and
  promotion-before-VIP-start ordering in the operating policy. Compatibility
  tests for absent/Started target roles are not recommendations to change production.
 - Fixed plan-promotion-cleanup.py to emit only temporary node-preference removal
  and preserve target-role=Promoted. Removed its obsolete inherited-Started
  check because it no longer clears the direct target role. Updated its exact
  expected-output test; ambiguity and unrelated-rule protections remain. Added
  an explicit Promoted-target scheduler case and used that target in fencing
  scenarios; compatibility cases remain.
 - Explicitly prohibited clearing former-primary failed actions before successful
  rejoin and verification of streaming recovery. Documented rewind before any
  explicitly authorized basebackup fallback, and that failed rewind may require
  further recovery rather than a normal restart.
 - Documented literal env-file quotes and the mismatch between generated pgpass
  and the RA's direct container environment password. All examples use obvious
  fake values; no actual password from the handoff was copied into the repository.
 - Updated README and historical handoffs/notes to point to current operations
  and evidence. Corrected the PID 1 simplification: bootstrap executes the
  maintenance command, possibly behind Podman's init shim.
 - Included current operating/testing/review documents in source/RPM documentation.
  Retained the old ELITEDESK installer as a clearly marked historical lab tool:
  its hostname/cluster/hash guards reject the scoring implementation. It is not
  a production installer and was not executed.
 - Ignored generated review patches; the commit records the actual files instead
  of duplicating obsolete proposed commands in patch attachments. Refreshed the
  checksum manifest. Existing local patch files remain available locally.
 The staged scoring, state helper, tests and RPM workflow from the preceding
 implementation work are included in this freeze commit. This review did not
 modify RA/module/scoring-helper or bootstrap code. The frozen original remains
 unchanged, and all 29 original lifecycle bodies still compare byte-for-byte.
 ## Checks completed in this review
 - Structural/default/metadata verification and all 22 differential lifecycle cases.
 - 25 promotion-scoring tests, 16 disconnected-state/monitor tests, 11 synthetic
  Pacemaker scheduler tests and 4 corrected preference-planner tests: all pass.
 - Nine SQL scenarios against a disposable PostgreSQL 18 container: all pass for
  the original strict predicate and current observation query. Container removed.
 - Bash syntax and Python AST parsing across repository code and historical installer:
  pass. ShellCheck and Gitleaks were not installed; no claim of running them is made.
 - Repository/diff review for private keys, recognizable tokens, credential-bearing
  URLs, and credential assignments, plus manual inspection of matches: no actual
  secrets found. Matches were variable references, redacted snapshots, or explicit
  fake examples. No CIB, secret file, .pgpass content or raw environment dump was added.
 - A disposable Podman test reproduced literal env-file quotes using only a fake
  value. No production credentials or environments were inspected.
 - Fedora 43 development RPM build with the new docs passed; artifacts are in
  dist/documentation-freeze/ and the log is /tmp/mc-freeze-build.log. The extra
  Promoted-target scheduler case was subsequently verified separately.
 - Checksum manifest and git diff whitespace checks are verified immediately before
  commit.
 ## Privilege and safety boundaries reviewed
 The RA does not source host-side PostgreSQL environment/configuration. Its
 root-to-service-user boundary preserves argument arrays, and container commands
 consume intentional injected settings. Authentication secrets are not placed in
 RA command arguments or host-side pgpass files. The stored bootstrap is separate
 and has a different provisioning interface.
 The scoring helper uses bounded subprocesses, a root-only runtime directory,
 strictly parsed JSON, per-resource locks and atomic replacement. Its identity
 includes postmaster start time, so a restarted/replaced data instance must stream
 successfully before receiving disconnected eligibility. Notification state uses
 a separate root-owned /run directory and relies on Pacemaker serializing actions
 for the same resource; it is not a distributed locking or fencing mechanism.
 Original SQL/shell construction in some lifecycle helpers assumes trusted,
 conventional resource configuration (identifiers, paths, connection fields).
 It is not a general escaping layer for arbitrary user-supplied text. Restrict CIB
 write access and container ownership; do not expose RA parameters to untrusted
 input. No new untrusted input interface was introduced by this review.
 ## Remaining concerns and explicit limits
 - The three production nodes' exact deployed hashes and confirmation that those
  bytes were used in the fencing test are missing. Source hashes are not a
  substitute. The runtime-context bootstrap may differ from deployed production.
 - The saved bootstrap's fresh-primary provisioning path passes a replication
  password through a psql --set argument. That can expose it to process-argument
  inspection under local /proc permissions. This is a concrete existing exposure,
  not a claim that the RA rejoin path leaks a password. It must be reviewed against
  the actual deployed bootstrap before calling provisioning secret-safe; no
  bootstrap rewrite or production secret change was made in this freeze.
 - Existing bootstrap pgpass generation assumes values needing no additional
  colon/backslash escaping. Do not treat the documented quote correction as a
  universal password-encoding solution. Inspect the deployed bootstrap before
  changing provisioning behavior.
 - Scores are observations, not fencing authorization or atomic promotion guards.
  IPC failure can leave a prior score; monitor cadence/fencing latency can exceed
  freshness. Two disconnected candidates are not ranked by cross-node WAL history.
 - Asynchronous replication can lose acknowledged transactions. Neither the test
  nor the freshness threshold proves a numerical data-loss bound.
 - The reported fenced-failure/rejoin success does not prove every concurrent-write
  controlled-switchover scenario or optional destructive basebackup fallback.
 - License provenance remains unresolved as documented in packaging/PROVENANCE.md.
  This source freeze is not a license grant or public-release authorization.
 No push or production rollout is part of this review.
@@ -0,0 +1,125 @@
 > These production reports and source hashes describe the 488a167 freeze.
 > The current undeployed extraction is described in [plugin-engine.md](plugin-engine.md).
 # Validation and freeze-point evidence
 ## Reported production fencing and recovery test
 Evidence source: operator-supplied production handoff received during the
 2026-09-12 documentation review. The operator reports a successful real
 three-node Pacemaker/Corosync test with SBD fencing. Codex did not execute or
 independently witness this production test. Its exact start/end timestamps and
 raw cluster logs were not supplied; no timestamps or measurements are invented.
 1. david-eve was primary and was fenced successfully through SBD.
 2. david-prodesk promoted automatically; the PostgreSQL VIP moved with it.
 3. david-macbookpro remained/reconnected as a streaming standby.
 4. david-eve rebooted with former-primary PGDATA. Ordinary resource start
   correctly refused it because standby.signal was absent.
 5. The operator manually started its maintenance container with PostgreSQL stopped.
 6. The private rejoin action used the PostgreSQL VIP as rewind source.
   pg_rewind detected divergence, found the common checkpoint and completed.
 7. david-eve started as a streaming standby. Only after recovery, Pacemaker
   cleanup allowed the cluster to adopt it as Unpromoted.
 Reported final state:
 | Node | Role | Score |
 | --- | --- | ---: |
 | david-prodesk | Promoted; PostgreSQL VIP local | 1000 |
 | david-macbookpro | Unpromoted; asynchronous streaming | 100 |
 | david-eve | Unpromoted; asynchronous streaming | 100 |
 No PostgreSQL failed actions remained. The successful run demonstrates the
 reported fencing/failover/VIP/rejoin sequence, not every possible partition,
 latency, concurrent-write, or recovery failure. The intermediate disconnected
 score and lost-transaction count were not supplied. The quoting/authentication
 incident and required fix are documented in [operations](operations.md).
 ## Validated production snapshot — exact deployed hashes pending
 The production outcome is operator-reported validated; **the byte-level production
 snapshot is not yet established**. Exact hashes from david-prodesk,
 david-macbookpro and david-eve have not been supplied or collected. Repository
 hashes below must not be represented as deployed hashes. A later capture also
 requires operator confirmation that those bytes are the ones used during the test.
 | Artifact on each production node | Snapshot status |
 | --- | --- |
 | /usr/lib/ocf/resource.d/loxthelion/managed-container | Pending production SHA-256 |
 | managed-container.d/*.sh (all nine modules) | Pending production SHA-256 |
 | managed-container.d/promotion_state.py | Pending production SHA-256 |
 | Running container /usr/local/sbin/postgres-ha-bootstrap | Pending production SHA-256 |
 Run the following read-only commands on **each production node**, using the
 rootless container owner for podman. They hash code, not environment or secrets:
 ```sh
 hostname
 date -u '+%Y-%m-%dT%H:%M:%SZ'
 sha256sum /usr/lib/ocf/resource.d/loxthelion/managed-container \
  /usr/lib/ocf/resource.d/loxthelion/managed-container.d/*.sh \
  /usr/lib/ocf/resource.d/loxthelion/managed-container.d/promotion_state.py
 podman exec postgres sha256sum /usr/local/sbin/postgres-ha-bootstrap
 podman inspect --format '{{.Image}}' postgres
 ```
 If permissions require it, run only the host sha256sum command with sudo. Do not
 run rootful podman instead of the actual container owner. If the installed
 bootstrap path/container name differs, record the actual path/name. Capture
 node identity, UTC capture time, all filenames/digests, immutable image ID, and
 whether any code changed since the test. Hash the running container's bootstrap;
 a host snapshot or image tag alone is not equivalent. Do not dump environment,
 secret files, or pgpass. Compare every node individually, preserving mismatches.
 ## Repository freeze snapshot (488a167)
 The exact repository code hashes appear below and in the reviewed checksum
 manifest. These identify the source freeze only, pending production comparison.
 The bootstrap is a saved runtime-context artifact, not part of the RA RPM and
 not yet confirmed identical to the production bootstrap.
 | Repository file | SHA-256 |
 | --- | --- |
 | `refactored/managed-container` | `a4b775a0ddcabdb0558da9b34c7fe2a883c07d0ef308c1c7224526abf53f63a0` |
 | `refactored/managed-container.d/auth.sh` | `93191771d3cb8aefadb2a16d2f89d64a8b6c584071a473439be818237bdf7f17` |
 | `refactored/managed-container.d/common.sh` | `97e93d9164e854e5f3688cf9d3cc656b001eb30ef3c2073c907450ca43588f3d` |
 | `refactored/managed-container.d/container.sh` | `ac6bc363a2301f1491d162ef609103dbfbceb3b099a3556614c6165952309374` |
 | `refactored/managed-container.d/demote.sh` | `0af3bb362a303017372c631354ba94ef9e723bf14e64d2eff91b6662e54a5be0` |
 | `refactored/managed-container.d/notify.sh` | `6a98fcdda82cea9365db95b4ee779be5c567b3d846f48b90fbc8c254af697670` |
 | `refactored/managed-container.d/postgres.sh` | `d2779ce6e7d1f426ffca2e41bbb8de77503005dc38d658d69f184020365796eb` |
 | `refactored/managed-container.d/promote.sh` | `3085ec9b4410905ab9d4370786ed776eac3470a0415aa1b4f608b083945aeb58` |
 | `refactored/managed-container.d/rejoin.sh` | `21beea72bcb7f24a51c17ea48666ac87df9436702ad77b166b45f4647e28e8ad` |
 | `refactored/managed-container.d/replication.sh` | `6b2021887e3b4cd6a12490e7a8dff98320b2133b472ab25a22a0eead46137547` |
 | `refactored/managed-container.d/promotion_state.py` | `5166fc3370f49ab14cd975ded4b03a74b9bd7dbff5762b41a15552c42f934ea7` |
 | `runtime-context/postgres-ha-bootstrap` | `3f1a15f7054fb69b9aa92204de30306fd6053c63560310f4231f314b148e2dcf` |
 ## Non-production checks
 The frozen source retained all 29 original lifecycle function bodies and original
 parameter defaults. Added scoring changes public dispatch and metadata; it is
 not a claim of identical total behavior to the monolithic reference.
 Run from the repository root:
 ```sh
 python3 tests/verify_refactor.py
 python3 tests/test_promotion_scores.py
 python3 tests/test_disconnected_state.py
 python3 tests/test_promotion_scheduler.py
 python3 tests/test_promotion_cleanup.py
 sha256sum -c notes/REFACTOR-SHA256SUMS
 git diff --check
 ```
 The baseline suite covers 22 differential lifecycle cases, 25 scoring tests,
 16 state/monitor tests, 11 real scheduler simulations and 4 planner tests.
 Scheduler failure simulations use synthetic CIBs, enable fencing, and assert
 fencing prerequisites for promotion. They do not operate a live cluster.
 `tests/verify_promotion_sql.py` explicitly creates/removes a disposable PostgreSQL
 18 container and checks nine strict/observation SQL scenarios. Previous candidate
 RPM build/install and root-helper CLI verification passed in disposable Fedora
 containers. They are separate from the operator's production validation.
 Current freeze review results are recorded in [review.md](review.md).
@@ -0,0 +1,198 @@
 > Production fencing/rejoin was subsequently reported successful by the operator.
 > See [../docs/testing.md](../docs/testing.md) for evidence and snapshot status,
 > and [../docs/operations.md](../docs/operations.md) for current operating policy.
 > Deployment statements below describe the original development session only.
 # Disconnected standby implementation and original candidate validation
 This revision uses the accepted promotion-scoring candidate as its baseline.
 The user explicitly accepts asynchronous replication data-loss risk after an
 unexpected primary failure, provided Pacemaker fences the failed primary before
 promotion. No installed RA, live cluster attributes, constraints, fencing
 configuration, or PostgreSQL instance was changed during this revision.
 ## State model and scores
 Every monitor first runs the original role/health monitor. Primary remains
 1000 with OCF return 8; stopped, unready, and failed role detection publish
 -INFINITY with their original return codes. Healthy standbys report OCF return
 0 regardless of eligibility. Each standby monitor collects a new local SQL
 observation and evaluates one of these states:
 | State | Evidence | Score |
 | --- | --- | --- |
 | Connected and verified | In recovery; replay unpaused; receiver streaming; flush non-NULL; replay >= flush | 100 |
 | Disconnected and verified recently | Same instance; recovery and replay healthy; no streaming receiver; valid fresh stored observation; current replay >= saved replay | 50 |
 | Ineligible | Missing, expired, invalid or incompatible evidence; replay paused/regressed; connected strict check failed; query/helper failure | -INFINITY |
 The strict connected predicate has not been relaxed. A streaming receiver with
 lag or NULL flush cannot fall back to disconnected eligibility. A missing
 receiver, or a PostgreSQL receiver in starting/waiting/restarting/stopping state,
 can use the disconnected path. A valid connected observation replaces the
 record atomically; disconnected observations never renew its timestamp.
 Reconnection must pass the strict checks to create fresh evidence.
 Score publication still uses bounded `crm_attribute --promotion` query/update
 calls. A missing transient attribute is recreated from current eligibility,
 including disconnected eligibility. This is not retention of an old +100.
 Lifecycle wrappers continue to withdraw scores before actions and re-evaluate
 after successful actions. The original 29 function bodies, defaults, and all
 original safety checks remain byte-identical. An added metadata wrapper
 advertises the new parameter without modifying the original metadata function.
 ## Exact persistence and ownership
 Host-side directory: `/run/managed-container/promotion/`.
 Both `/run/managed-container` and its `promotion` child are root-owned mode 0700.
 Each record is `<sha256>.json`, root-owned mode 0600, with a corresponding
 root-owned mode 0600 `<sha256>.lock`. The key is SHA-256 of the UTF-8 compact JSON
 array `[OCF_RESOURCE_INSTANCE, service_user, service_name, container_name]`.
 An instance suffix change conservatively selects a different record.
 The RA's root process runs `managed-container.d/promotion_state.py` with Python
 3; the rootless service user cannot read/write the host state directory. There
 are no credentials, connection strings, passwords, or upstream secrets in the
 record. Contents are schema version, resource context, host boot ID, PostgreSQL
 system identifier, actual SQL `data_directory`, postmaster start timestamp,
 verified replay and flush LSNs, wall-clock observation timestamp for audit,
 and boot-relative observation time for expiry.
 A nonblocking per-resource lock serializes evaluation and same-directory atomic
 replacement. Temporary files use mode 0600; JSON is bounded and parsed as data,
 with duplicate keys, invalid types, invalid LSNs, and extra/missing fields
 rejected. Unsafe directory/file modes, ownership or symlinks fail closed.
 Failed persistence withdraws eligibility. Runtime state survives separate RA
 invocations and Pacemaker transient score loss, but not host reboot; boot ID
 also guards against restored state from a previous boot.
 ## Freshness and identity policy
 OCF parameter `disconnected_max_age`: integer seconds 0–99999, default **300**.
 Zero disables disconnected eligibility. Ages from zero through the threshold
 are accepted; negative, future, or greater ages are rejected. Invalid parameter
 values fail validation and cannot yield a positive standby score.
 Expiry uses Linux CLOCK_BOOTTIME, which includes suspended time. Changing the
 wall clock does not extend or prematurely expire an observation. Choose the
 threshold to cover the monitoring, failure detection, fencing, and scheduling
 time needed by the cluster. Slow fencing may outlast this window and leave no
 eligible candidate; freshness is deliberately not extended in response.
 The data identity is `(system_identifier, data_directory, postmaster_start_time)`
 plus host boot/resource context. A new PostgreSQL system ID or data path
 invalidates the observation. **Any PostgreSQL restart also requires a new strict
 streaming verification**, even when restarting the same data. This is deliberate:
 a basebackup or same-cluster replacement can preserve the system identifier,
 but cannot preserve the running postmaster incarnation. Normal reinitialization,
 rewind/rejoin restart, or container restart therefore cannot inherit eligibility.
 Current replay must also be at least the stored replay LSN.
 Malformed, stale or mismatching records are deleted when observed by the
 state evaluator. A subsequent strict streaming verification creates a new
 record. A stopped PostgreSQL process is immediately scored -INFINITY; its file
 may remain until the next standby evaluation, but its replacement/restarted
 process cannot use it. A healthy primary keeps 1000 and does not consume standby
 evidence. Stale unused resource-key files disappear at host reboot.
 ## Candidate selection and fencing
 100 ranks currently verified connected evidence above recent disconnected
 proof at 50, while preserving primary 1000. Two disconnected candidates receive
 equal 50. Pacemaker resolves ties using its configured scheduling rules; no
 preferred hostname is hardcoded. Tests show repeatable selection for an
 identical synthetic CIB, not a guarantee of a particular host in every cluster.
 We intentionally do not map LSNs or local observation timestamps into unequal
 scores. A 64-bit LSN cannot be represented faithfully in Pacemaker's bounded
 score range, and comparison needs compatible timeline/history context. Local
 observation age is not a measure of WAL completeness: a more recent observation
 can contain less WAL than an older observation on another node. Cross-node
 clock timestamps introduce further uncertainty. Replay and flush LSNs are
 stored for a future explicit comparison policy; this revision has no cross-node
 shell access or invented distributed ranking mechanism.
 Fencing remains mandatory and is entirely Pacemaker's responsibility. The RA
 does not inspect fencing completion or create/modify constraints. A score of 50
 means locally eligible under the accepted asynchronous policy; it is not
 permission to run the promote action manually against an unfenced primary.
 Configured fencing, quorum, and action ordering must be validated in the lab.
 The synthetic failure tests use fencing enabled and check that every promotion
 has a fencing action as a transitive prerequisite in the scheduler graph.
 ## Verification and review artifacts
 - Original frozen hashes and all 29 original function bodies/defaults: unchanged.
 - All 22 original differential lifecycle cases pass.
 - All 25 accepted scoring/action tests pass; mock boundary and metadata handler
  were adjusted for the added helper without removing existing assertions.
 - 16 state/monitor tests cover streaming-disconnect, never verified, exact
  expiry/disabled policy, malformed and unsafe state, system/path/postmaster
  changes, reinitialization replacement, replay pause/regression, connected
  failures, reconnect, clock changes, stopped PostgreSQL, score loss/reprobe, and query/helper failures.
 - 10 real `crm_simulate` synthetic-CIB tests pass, including the original six,
  unexpected primary failure with one/two disconnected candidates, mandatory
  fencing dependencies, all-stale refusal, and connected preference.
 - All 4 cleanup-planner tests pass. No generated cleanup commands were executed.
 - PostgreSQL 18 integration: all nine original strict SQL cases pass, and the
  new JSON observation query agrees with each. Only a disposable container with
  tmpfs data and no network/host ports was used.
 - The root helper CLI also passed in a disposable Fedora container, using the
  actual fixed /run path: never-verified refusal, 100 -> 50, ownership/modes,
  unchanged disconnected timestamp, malformed-state deletion and identity change.
 - Fedora 43 RPM build and disposable installation validation passed, recorded in
  `/tmp/mc-disconnected-rpm.log`; artifacts are in `dist/disconnected-review/`.
 - Incremental diff from the accepted candidate: `DISCONNECTED-STANDBY.patch`.
  Full diff from committed baseline: `PROMOTION-SCORING.patch`.
 Run the offline suite with:
 ```sh
 python3 tests/verify_refactor.py
 python3 tests/test_promotion_scores.py
 python3 tests/test_disconnected_state.py
 python3 tests/test_promotion_scheduler.py
 python3 tests/test_promotion_cleanup.py
 ```
 The optional disposable SQL test is `python3 tests/verify_promotion_sql.py`.
 During the original candidate-development session, no live HA transitions or
 deployment were performed. Subsequent operator-reported production results are
 recorded in ../docs/testing.md. The previous scoring
 report is retained as historical context; its strict-disconnection blocker is
 superseded by this policy. Existing lab/production cleanup prerequisites remain.
 ## Remaining limitations
 Asynchronous replication does not guarantee zero data loss. The 300-second
 window bounds observation age, **not lost bytes or transactions**, and does not
 prove that a standby has the failed primary's last acknowledged transaction.
 Nor does this design guarantee selection of the most advanced of two eligible
 standbys. It does not establish common timelines across peers.
 The SQL user must be allowed to execute `pg_control_system()` and read the
 required monitoring fields. Insufficient privileges fail closed; no grants are
 changed automatically. Restarting a disconnected standby forfeits eligibility
 until it streams again. Loss of /run state likewise requires fresh verification.
 Replacing files underneath a running PostgreSQL process or deliberately restoring
 fabricated identity/state is outside supported PostgreSQL operation; root and
 the database monitoring boundary are trusted.
 The scoring SQL retains its 3-second external bound (1-second kill grace) and
 2-second statement timeout. State evaluation adds a 1-second external bound
 (1-second kill grace); crm_attribute retains each 2-second bound and 1-second
 grace. Original lifecycle/role queries and operation timeouts are unchanged;
 validate the aggregate monitor budget on actual hardware.
 Eligibility is sampled, not an atomic transaction with future promotion.
 Publication/IPC failure can leave an old Pacemaker score until the next
 successful update, as in the accepted baseline. Scheduling and fencing can
 also outlast an observation's expiry before the next monitor. This revision
 preserves the existing promote implementation rather than adding a new final
 promotion guard. Lab testing must cover detection/fencing latency, failover in
 both directions, reconnect/rejoin, monitoring load, and score updates during
 failure before a production rollout is approved.
 Sources: PostgreSQL documents the system identifier and server start timestamp
 in [system information functions](https://www.postgresql.org/docs/18/functions-info.html).
 Pacemaker's fencing, resource requirements and promotion scheduling are described
 in [Pacemaker Explained](https://clusterlabs.org/projects/pacemaker/doc/3.0/Pacemaker_Explained/pdf/Pacemaker_Explained.pdf).
@@ -0,0 +1,280 @@
 > Current production report and required operating policy: [../docs/testing.md](../docs/testing.md)
 > and [../docs/operations.md](../docs/operations.md). The historical lab plan below
 > is not a production runbook.
 > Historical accepted scoring baseline. The current disconnected policy, tests,
 > and limitations are in [DISCONNECTED-STANDBY.md](DISCONNECTED-STANDBY.md).
 > The pending decision and strict-disconnection blocker below are superseded.
 # Promotion scoring — undeployed review candidate
 This enhancement is based on the canonical modular source at c8636ed. This task changed no
 installed agent, production configuration, fencing setting or live cluster
 attribute. The earlier baseline installer is now pinned to that
 baseline entrypoint and refuses this candidate. Do not deploy it using the old
 installation instructions.
 ## Decision still required before rollout
 **Strict live-streaming eligibility can prevent failover when the primary is
 lost.** A previously healthy standby can lose its WAL receiver before Pacemaker
 chooses a successor. This candidate then replaces its positive score with
 -INFINITY. With all candidates ineligible, no instance is promoted. The SQL and
 scheduler tests confirm this outcome; it is not a claim of preserved automatic
 failover. Controlled switchovers and abrupt primary-loss failovers must both be
 reviewed. The user's choice between this conservative first candidate and a
 separate disconnected-standby eligibility policy is pending.
 There is no inferred grace period, persisted last-known-good eligibility, or
 cross-node WAL ranking. Retaining an old positive score merely because a
 standby once streamed would contradict the requested stale-score cleanup.
 Resolving disconnected-standby promotion safely requires an explicit data-loss
 policy and validated fencing/recovery assumptions. Do not remove the production
 preference or reset production metadata until this decision and lab validation
 are complete.
 ## Scoring policy
 | Observed state | Published score | Monitor result |
 | --- | --- | --- |
 | Original monitor reports healthy primary | 1000 | 8 (Promoted) |
 | Healthy standby; in recovery, replay unpaused, streaming receiver, replay LSN at least receiver's flushed LSN | 100 | 0 (Unpromoted) |
 | Healthy standby but eligibility SQL false, empty, malformed, failed or timed out | -INFINITY | 0 (Unpromoted) |
 | Service/container/PostgreSQL stopped | -INFINITY | Original result, normally 7 |
 | Unready, failed role query or unexpected role | -INFINITY | Original failure code |
 The eligibility query is local to the running container. It uses the durable
 `flushed_lsn`, not `written_lsn`. SQL NULL values cannot yield eligibility.
 A CASE expression avoids invoking recovery-only functions on a primary.
 Standbys receive equal scores when eligible; no preferred hostname is encoded.
 Requiring replay to catch up to locally flushed WAL is conservative and can
 make eligibility fluctuate under sustained write load. This is a real state
 change, not artificial periodic CIB rewriting.
 A positive score is an observation of local eligibility, not proof that a node
 has every acknowledged transaction from the primary, is freshest among peers,
 or is safe to promote without fencing. Asynchronous replication can lose
 transactions. These scores do not implement consensus or replace PostgreSQL
 recovery checks, Pacemaker constraints, quorum, or fencing.
 ## Placement and scope
 All 29 original function definitions, defaults and metadata remain byte-for-byte
 identical. The existing nine-module layout remains. Six helpers were added:
 - common.sh: mc_promotion_score_set, mc_promotion_score_from_monitor,
  mc_promotion_monitor, mc_promotion_action, mc_promotion_validate.
 - postgres.sh: mc_pg_promotion_eligible.
 The dispatcher now routes monitor/status through mc_promotion_monitor, which
 calls the original managed_monitor once, publishes a score and returns the
 original role code. Internal lifecycle monitor calls remain unchanged and do
 not publish scores mid-transition.
 Start/promote/demote/stop/rejoin publish -INFINITY before validation or mutation.
 After a successful non-stop action, the original monitor is called again and
 eligibility is reassessed. Failure leaves the negative score until a subsequent
 fresh monitor observation. Successful stop leaves -INFINITY. Stop still skips
 validate_all and attempts shutdown even if publication fails. Notify retains
 its original notification persistence behavior and does not affect scores.
 Metadata and validate-all publish no scores. validate-all additionally checks
 that crm_attribute and timeout exist; deployments require Pacemaker >=2.1.0
 for the modern --promotion option, enforced by the RPM dependency.
 The entrypoint captures the explicitly supplied OCF_RESOURCE_INSTANCE before
 ocf-shellfuncs can supply a dummy standalone default. If none was supplied,
 publication logs a warning and performs no write. Pacemaker resolves the node
 from its execution context (including OCF_RESKEY_CRM_meta_on_node); there are no
 node-name literals or manual node-resolution rules in the RA scoring code.
 The helper uses:
 ```text
 crm_attribute --promotion=<explicit resource instance> --lifetime=reboot --query --quiet
 crm_attribute --promotion=<explicit resource instance> --lifetime=reboot --update=<score>
 ```
 Only a missing or different value causes an update. No local cache or durable
 eligibility file is used, so reprobes restore missing transient scores. Values
 are replaced, never incremented. A negative value is used instead of deletion:
 a missing score alone is not an explicit promotion prohibition.
 Pacemaker currently implements this API with an internal master-<resource>
 attribute name. Seeing that prefix for the NEW resource is expected. The RA
 never constructs, reads, deletes or migrates master-postgresql,
 postgresql-data-status, or postgresql-status. Stale attributes for the old
 primitive are not inputs to this code. Independently configured location rules
 that reference old attributes remain cluster configuration and require their
 own operator review.
 Each score query/write is bounded by a two-second timeout plus a one-second
 kill grace. The new eligibility command has a three-second outer timeout plus
 a one-second kill grace and a two-second SQL statement timeout. The existing
 monitor's own command timeouts are unchanged. Account for these additions in
 lab operation timing before production rollout.
 If attribute publication fails, the helper logs an error explicitly warning
 that the prior score may remain. It retries on the next observation. Role
 return codes and existing lifecycle results are preserved; shutdown is never
 blocked by a score-write failure. **No RA can guarantee stale-score removal
 while the cluster attribute service is unreachable.** This failure mode needs
 lab validation and is not hidden by reporting a false PostgreSQL role.
 Pacemaker normally serializes actions for a resource; parallel direct/manual
 calls that race score updates are not supported by this implementation.
 ## Tests and evidence
 Commands run from the source tree:
 ```bash
 bash -n refactored/managed-container
 for module in refactored/managed-container.d/*.sh; do bash -n "$module"; done
 python3 tests/verify_refactor.py
 python3 tests/test_promotion_scores.py
 python3 tests/test_promotion_scheduler.py
 python3 tests/test_promotion_cleanup.py
 python3 tests/verify_promotion_sql.py
 ```
 - Original 22 differential scenarios pass. All 29 original definitions and the
  same 12 metadata parameters / eight advertised actions are preserved.
 - 25 scoring tests pass, including failures, stop/start, promote/demote/rejoin,
  stale positive replacement, repeated monitors, loss/recreation of transient
  attributes, missing explicit resource identity and score-publication errors.
 - A real crm_attribute test uses CIB_file with a synthetic CIB. Repeated equal
  observations do not change the file; negative replacement and recreation
  after loss of transient attributes work.
 - Six real crm_simulate tests on synthetic CIBs pass: absent and explicit
  Started target-role each permit one promotion; Unpromoted prevents it;
  larger scores win, ineligible nodes are excluded, and all-negative scores
  permit no promotion. Old resource attributes are seeded and do not alter
  this new resource's election. These are scheduler simulations, not live HA
  transitions. The synthetic CIB disables fencing only inside its test file;
  the live lab/production fencing configuration was not touched.
 - Nine SQL cases pass in a separate ephemeral PostgreSQL 18 container with no
  host ports or data mounts, evaluating the actual production SQL against
  controlled state fixtures. Includes replay pause/lag, missing positions,
  stopped/absent receiver, and recovery-only function handling on a primary.
 - Four cleanup-command planner tests pass, including ambiguous-rule rejection
  and preservation of unrelated rules.
 The RPM %check now runs the regression, scoring and scheduler suites. PostgreSQL
 integration is an explicit opt-in local test needing Podman and a local
 postgres:18 image; it does not touch the lab postgres container.
 ## Specific disposable two-node lab sequence (after review)
 Keep maintenance-mode=true as required by the lab handoff. Use direct RA
 operations only after installing the reviewed version under a separate approved
 installation procedure. Automatic scheduling claims must first be exercised
 with a copied CIB and crm_simulate; do not disable maintenance mode as part of
 these instructions. Do not use this sequence on production.
 1. Save the lab CIB, installed executable/modules, and current state on both
   nodes. Verify the cluster is postgres-ra-test, the roles and streaming source,
   current PGDATA backup/recovery baseline, and that production is not targeted.
   Establish timing and monitor outputs from the existing implementation.
 2. Run metadata and validate-all for the candidate on each lab node. For direct
   actions always supply the actual primitive ID (pg-ha in saved lab context),
   not the clone ID. Example observation commands on each lab node:
   ```bash
   sudo env OCF_RESOURCE_INSTANCE=pg-ha \
     /usr/lib/ocf/resource.d/loxthelion/managed-container monitor
   # Interpret rc=8 as healthy primary, rc=0 as healthy standby.
   sudo crm_attribute --promotion=pg-ha --lifetime=reboot --query --quiet
   ```
   Expect primary=1000 and a caught-up streaming standby=100. Repeat monitors;
   verify no writes for identical scores and no unrelated attribute changes.
 3. On the standby, test full stop and start: stopped service/container and
   PostgreSQL must yield monitor=7 and score=-INFINITY; standby start restores
   monitor=0 and score=100 after streaming/catch-up. Verify primary start refusal
   remains intact. Do not use a successful role-return code as a score assertion.
 4. Separately stop only standby PostgreSQL while keeping the maintenance
   container alive; observe -INFINITY, then start it through the proven standby
   path. Verify no rejoin is attempted while PostgreSQL is running.
 5. Pause replay on the standby (lab only), observe monitor=0 and -INFINITY,
   resume replay and await catch-up/100. Test receiver interruption, SQL errors,
   stopped container, and a seeded prior positive score becoming ineligible.
 6. Exercise the existing controlled switchover sequence in both directions:
   persist the fresh matching pre-demote notification on the old primary,
   demote through the Corosync-resolved successor, and promote the successor.
   Verify -INFINITY during transitions, 100 for a successful eligible demotion,
   1000 after promotion/invariant reconciliation, and persistent container
   identity throughout demotion. Return to DV6 primary / ELITEDESK standby.
 7. Test two eligible standbys using the existing cascading-standby intermediate
   state. Both should have the same positive score; a copied-CIB simulation
   should select one with promoted-max=1. Do not promote both manually.
 8. Reproduce the abrupt-primary-loss case in the isolated lab only after a
   recoverable baseline exists. Allow the surviving receiver to disappear,
   run its monitor, and inspect the score. This candidate is EXPECTED to deny
   promotion then. That is a rollout blocker unless the strict policy is
   explicitly chosen; it cannot be counted as a passed failover test.
 9. Remove only the new resource's transient score in the lab, then reprobe:
   fresh eligibility must recreate it. Exercise score-service/query/update
   failure and recovery without confusing role codes. Repeated successful
   monitoring must replace any stale positive score after ineligibility.
 10. With a saved CIB, simulate removal of only the temporary node preference.
    Retain target-role=Promoted, promoted-max=1, and the VIP constraints.
    Earlier absent/Started scheduler cases are compatibility tests, not
    production configuration recommendations.
 11. Restore the baseline topology, streaming, test-induced settings and intended
    agent version. Confirm maintenance mode still true. Record both nodes'
    installed hashes, OCF codes, scores, receiver status and action durations.
 ## Production cleanup and rollout: gated on lab validation
 The conceptual description does not include the actual preference constraint
 ID, so inventing a deletion command would be unsafe. Read-only discovery is:
 ```bash
 sudo pcs constraint config --full
 sudo pcs resource config postgresql-ha-clone
 sudo cibadmin --query > production-before-scoring.xml
 ```
 Do not commit that CIB snapshot: it may contain sensitive deployment settings.
 After lab validation and policy approval, scripts/plan-promotion-cleanup.py can
 read the supplied snapshot and PRINT the exact commands using the real IDs:
 ```bash
 python3 scripts/plan-promotion-cleanup.py production-before-scoring.xml \
  --clone postgresql-ha-clone --preferred-node david-eve --lab-validated
 ```
 The planner never executes commands. It matches only the simple rule described
 in the task (Promoted, score=200, #uname eq david-eve), refuses zero/multiple
 matches, preserves other rules, and emits only the preference constraint/rule
 removal. It requires and preserves direct clone-level target-role=Promoted.
 Rule-based/inherited settings require manual inspection. Do not confuse
 preference removal with clearing failed actions: former-primary failure state
 must remain until rejoin succeeds and streaming standby status is verified.
 There is no RA code that deletes constraints or changes clone metadata. No
 production installation, cleanup, or restart was executed during that original
 source-review session. Subsequent production validation is in ../docs/testing.md. Exact production
 rollout commands are intentionally deferred until the policy, real constraint
 ID, candidate hashes and successful lab results are known, as requested.
 ## Sources used for API semantics
 - https://clusterlabs.org/projects/pacemaker/man/crm_attribute.8.html
 - https://www.postgresql.org/docs/18/monitoring-stats.html#MONITORING-PG-STAT-WAL-RECEIVER-VIEW
 - https://www.postgresql.org/docs/18/functions-admin.html
 - Installed pcs help: `pcs constraint remove --help`, `pcs resource meta --help`.
 The source diff is provided in PROMOTION-SCORING.patch. Source and test file
 hashes are recorded in REFACTOR-SHA256SUMS, which is refreshed for this candidate.
 The frozen reference hash remains unchanged.
 RPM validation of this candidate also passed in the disposable Fedora 43 build
 container: binary/source packages built, %check ran the regression, scoring,
 scheduler and cleanup tests, DNF resolved the new Pacemaker dependency, and the
 installed-RPM integrity/permissions/metadata checks passed. Local artifacts are
 under dist/scoring-review/ and are not releases or deployment approval.
 A final read-only inspection found that ELITEDESK's installed executable and
 all nine modules match the earlier modular baseline at c8636ed exactly. Its
 entrypoint SHA256 is c91f09adfdeb45a3be1a72df3f5ca5b02fd4555dec4397ad944eab439ace0b89.
 The promotion-scoring candidate is not installed there.
@@ -1,5 +1,10 @@
 # PostgreSQL HA resource-agent modular refactor
 Historical baseline report (commit c8636ed and earlier). The later promotion
 scoring candidate changes dispatcher behavior and adds helpers; see
 [DISCONNECTED-STANDBY.md](DISCONNECTED-STANDBY.md). Do not use the installation
 commands below to deploy that unreviewed candidate.
 The refactored agent is in `refactored/managed-container`, with nine sourced modules beside it. Both reference copies remain unchanged at SHA256 `5eae1f3dba0586dc06105e4dd7a9b8809a5c9961452abe7654dcfce77c3c5ce2`.
 ## Final file tree
@@ -1,13 +1,56 @@
 175ccb8e4b851fe1c06e3490dbd1bb0ab12f9b1c8374dd001effd88256b9dde2  .gitea/workflows/build-rpm.yml
 bb9026b0c528721ab80038344929a8bcd0ab8c5c8e438a22978bc362572d6112  .gitignore
 4351008e41de847f0dd8323843a28ea3b0fbe100d0268faa77183cdb4f7c7da0  HANDOFF.md
 b43b3c36a6a039002b40bf655d7af06d25f240059466ed457a97f8ed841bb6a5  README.md
 6832fdee40cf85b006dfd4b542d8a2ee4f9498775f0e7cea310df765f8b00203  docs/lab-installed-before-plugin.sha256
 ab6ffa63105281fd4cb010f0b954367d8dfd4163528eb5679a1c25f499c5ed29  docs/lab-installed-plugin.sha256
 9747927dfbe8a87db803e4f3139ce2386e8cb849726741b6a595158667fefa13  docs/lab-plugin-basic-test.md
 e3224fd761f46a518a705380ea867f4b8e55086e55a47f8272d0dce2afdc3e0d  docs/operations.md
 5678ef595be40be398c51ccb630884cb1ecdb7431922600eba0b80aa1f4ab8bf  docs/plugin-engine.md
 3e5f075baa7863bee3d12dd50319466291e30e7be895dfe16e9724da9ae4f021  docs/review.md
 8b7455b45f27dda61071be549abaffa1ac033ac7f652c0831402bde647b8be4c  docs/testing.md
 bfdeecfb1791c4f1d8ec08f28774f94769fffeb51b3268342fd6c63e2322eb23  notes/DISCONNECTED-STANDBY.md
 0a6af2a5fbf1ea1d9551893578df094442ee10e032e1b2de7c92f9662b133ad3  notes/PROMOTION-SCORING.md
 6bdc385dff0e66de372f802a577f67d852f7a2a0e45610865f558ee3b93adf20  notes/REFACTOR-PLAN.md
-08135e93b355d2f8239d9abbe0297eafdd6109d025a27d13ab35ccd74811e03a  notes/REFACTOR-REPORT.md
+05b860269ba536361f7b806f415ffb6c2f6632ecf8374bfcd02e7bb7ad1069bb  notes/REFACTOR-REPORT.md
-c91f09adfdeb45a3be1a72df3f5ca5b02fd4555dec4397ad944eab439ace0b89  refactored/managed-container
+2d728176c301e24f673b8d36a9ae0fd3bd3816d37b1f945100b37a74779887ef  packaging/Containerfile
-93191771d3cb8aefadb2a16d2f89d64a8b6c584071a473439be818237bdf7f17  refactored/managed-container.d/auth.sh
+676455c64d130eb1ad36e007f023f0f30c98beedf87f18be74ae33202b646ffa  packaging/PROVENANCE.md
-e0bb13e9fda84ba7b657b2e774c08fa64988d463f45c699681e0e6ce49109440  refactored/managed-container.d/common.sh
+1a05f3153b0bd8c83dd01b925670193ca763caceee9dc07bf63ac2e71eff9d7b  packaging/README.md
-ac6bc363a2301f1491d162ef609103dbfbceb3b099a3556614c6165952309374  refactored/managed-container.d/container.sh
+e9dd8507f4bf0c6f42458e41aea833ad0bd3f6127272335eee9bf4d58541ed67  packaging/VERSION
-0af3bb362a303017372c631354ba94ef9e723bf14e64d2eff91b6662e54a5be0  refactored/managed-container.d/demote.sh
+93b5a897b8a6b8f92687c242df353fae9e6a3b24872fadd1553252fb8a227512  packaging/ocf-managed-container.spec
-6a98fcdda82cea9365db95b4ee779be5c567b3d846f48b90fbc8c254af697670  refactored/managed-container.d/notify.sh
+01eb9b161c8e3db289f09bf63953be779921e15bdf8b1a4d8ce7e60032b826e4  refactored/managed-container
-310132d8032891727076295f39548b1cc0e51a10301a0ee4d934161f4f903e3e  refactored/managed-container.d/postgres.sh
+10932919c4339b553c7c9dbf30b45a917c51c0a08f362e12678003d58b12fa82  refactored/managed-container.d/applications/postgresql/auth.sh
-3085ec9b4410905ab9d4370786ed776eac3470a0415aa1b4f608b083945aeb58  refactored/managed-container.d/promote.sh
+da4e58cadf1df3953cb97a0512e3931ee1b9d3ee61c39a1ad1a6047ef537addc  refactored/managed-container.d/applications/postgresql/demote.sh
-21beea72bcb7f24a51c17ea48666ac87df9436702ad77b166b45f4647e28e8ad  refactored/managed-container.d/rejoin.sh
+6a98fcdda82cea9365db95b4ee779be5c567b3d846f48b90fbc8c254af697670  refactored/managed-container.d/applications/postgresql/notify.sh
-6b2021887e3b4cd6a12490e7a8dff98320b2133b472ab25a22a0eead46137547  refactored/managed-container.d/replication.sh
+d3a989de97ceb61a56828f006d5ccb87b7c34fc155a944585bedf928785021fc  refactored/managed-container.d/applications/postgresql/plugin.sh
-e3bc103b61b317e91d28260cc7bb86a3cc8390854762d6c5b9f325a1a8b1c362  tests/verify_refactor.py
+fa222e6767249999767eb82fac2d85e7e35404da240517a8b630571e77282053  refactored/managed-container.d/applications/postgresql/postgres.sh
 3085ec9b4410905ab9d4370786ed776eac3470a0415aa1b4f608b083945aeb58  refactored/managed-container.d/applications/postgresql/promote.sh
 5166fc3370f49ab14cd975ded4b03a74b9bd7dbff5762b41a15552c42f934ea7  refactored/managed-container.d/applications/postgresql/promotion_state.py
 28b0e44554a88ccb87cd16a68315c045dd37d354aceb2fa25da7c7399a659b23  refactored/managed-container.d/applications/postgresql/rejoin.sh
 6b2021887e3b4cd6a12490e7a8dff98320b2133b472ab25a22a0eead46137547  refactored/managed-container.d/applications/postgresql/replication.sh
 a482ef3cc1048e6a4f427e76dd2dce173ffd00760dbed1afa601a3fd05b0ad99  refactored/managed-container.d/common.sh
 3a27ac9c63c61310da8b8711fa2c4400b37ea8eae2ee2a79e28bc63180b06c7c  refactored/managed-container.d/container.sh
 53978e2367394695d61c6fb38f2ee46814474c5a4e0448d2692f1a3ce583af93  refactored/managed-container.d/engine.sh
 6c92c095ce10ac95d843fdc1c49e31c19491ca22fdac960affd8649606d76d82  refactored/managed-container.d/lifecycles/systemd.sh
 dc5820067a78c4a1c531426902793aae97a03fd64912b017b3b102031dcc1e04  refactored/managed-container.d/runtimes/podman.sh
 3f1a15f7054fb69b9aa92204de30306fd6053c63560310f4231f314b148e2dcf  runtime-context/postgres-ha-bootstrap
 17ce35741b0ef55a6e4785a093a5097f13139edd428fd4c65e8f50d222161b65  scripts/build-rpm.sh
 42744b1b407ab4a79725bd90d7d067f46dd8b36728f104c52544d464b2e323c3  scripts/install-elitedesk-ra.sh
 470fd174e42efef876e540dfbfa42b4f36bf70b29a26ab3e5f0c10c813f3b6ae  scripts/plan-promotion-cleanup.py
 7c426f3924cdaa92252987c784844d7a93675bee3259b536fba9a9347fcdc600  scripts/test-elitedesk-cycle.py
 5d9ddb7d554508e02bd0c4ab4aa208998d6b54b489727aa901b946d2990e8460  scripts/test-elitedesk-plugin.py
 7e2ec6e9201b239607c5119674543a5d8ae53e9bb54bc5d02bb1119d1769cee1  tests/plugin_test_support.py
 9e4db508ebabc077a99299c3df1f6881948fc5766d326cb4bccb70fddbd84889  tests/test_disconnected_state.py
 b6356d055861fa014b6210a193f565fec23f08935dab80e1320df77d558d1bbd  tests/test_lab_cycle.py
 347b05d4bcd4df9e0f8fcbd0cda10d3d54113c88635e54f7d94d91c74d4cfe05  tests/test_lab_installer.py
 81ce2770d872a064d0ac0ff0c53ad3851650d9002249d79eac882558d2b077c8  tests/test_plugin_engine.py
 e44ac0e6fc322d7e314814c69d9c3b1ab582b43afbf8b66bd30a07713682ae1c  tests/test_promotion_cleanup.py
 eb88cdaffeccb00c92e1c43725112ae16aa15faf923b84db62ae5950c5c26d2e  tests/test_promotion_scheduler.py
 ac3976873c018d0c5aa6eaf496fb3356df01fb26bc1e782a5fcb72aeb3baa1c3  tests/test_promotion_scores.py
 6d865d4abaee977646b5ee51ea940ee15e06ca5fb46010ec48fb8a94d4303711  tests/verify_installed_rpm.py
 6bae43d0b5b702078bdd8a1fb6afc7ac3eb7b9f6d05c94128fdab45bd28f05c2  tests/verify_promotion_sql.py
 d51bc222bddf861dddb3b6dd7f2694666c38d3d46b4cddbe13ce86dcc2836994  tests/verify_promotion_state_cli.py
 0226e88f0c997b7a1c99c810534e24e48efa14210bb5183bb1ac6c3425398e2b  tests/verify_refactor.py
 4d6036b4b8aa45f3b297fafc868b045ab904e421b12da1fea1f57ef4785ca93f  scripts/test-lab-roles.py
 e39a786f3fb8d67b8f52faf74650b2a63cfe090567a1117b60fd338add31cadb  tests/test_lab_roles.py
 0c41d7db4094db56093efeae2dd5bdc25b5076d0c6ec86feb1ea98f922a1f924  docs/lab-role-tests.md
@@ -1,3 +1,3 @@
 FROM registry.fedoraproject.org/fedora:43
-RUN dnf -y install rpm-build python3 resource-agents git tar gzip diffutils findutils \
+RUN dnf -y install rpm-build python3 resource-agents pacemaker-cli git tar gzip diffutils findutils \
        coreutils bash nodejs && dnf clean all
@@ -5,7 +5,7 @@ Fedora 43, matching the saved lab package context. The workstation currently
 reports Fedora 44; this first workflow does not claim a Fedora 44 build test.
 It runs on main pushes, pull requests and manual dispatch.
-The build installs only the OCF executable and nine modules, plus documentation.
+The build installs only the OCF executable and 13 shell modules, one Python state helper, plus documentation.
 It includes no Quadlet, secrets, databases or cluster configuration. There are
 no RPM lifecycle scriptlets. Installation does not start, stop or restart a
 service or change maintenance mode. Host PostgreSQL packages are not required.
@@ -72,3 +72,20 @@ on Fedora 43). No package was installed on either lab host.
 The Gitea server reports 1.27.3. Its Actions API returned HTTP 401 without
 authentication, so a successful local build does not establish runner checkout
 or artifact upload success. Check the first run in the repository's Actions tab.
 The promotion-scoring review candidate additionally requires pacemaker-cli
 2.1.0 or later. Its %check includes scoring and synthetic-CIB scheduler tests.
 See notes/PROMOTION-SCORING.md in the source checkout for the unresolved
 disconnected-standby policy; do not deploy the candidate from CI artifacts.
 The disconnected candidate adds a runtime Python 3 dependency and advertises
 `disconnected_max_age` (default 300 seconds). See
 [the current policy](../notes/DISCONNECTED-STANDBY.md) before any installation.
 Current operating policy and operator-reported production validation are packaged
 from `docs/`. The source freeze is distinct from verified deployed-byte identity;
 see [testing](../docs/testing.md) for production snapshot status.
 The plugin extraction adds three selectors, defaulting to PostgreSQL/Podman/systemd.
 The package includes all API-1 plugin files and `plugin-engine.md`; Docker and
 direct-runtime lifecycle management are not implemented.
@@ -20,6 +20,8 @@ BuildRequires:  resource-agents
 BuildRequires:  hostname
 BuildRequires:  diffutils
 BuildRequires:  coreutils
 BuildRequires:  pacemaker-cli >= 2.1.0
 Requires:       python3
 Requires:       bash
 Requires:       /usr/lib/ocf/lib/heartbeat/ocf-shellfuncs
 Requires:       podman
@@ -31,6 +33,7 @@ Requires:       coreutils
 Requires:       gawk
 Requires:       sed
 Requires:       hostname
 Requires:       pacemaker-cli >= 2.1.0
 %description
 OCF agent managing PostgreSQL through a persistent rootless systemd/Quadlet
@@ -48,22 +51,40 @@ cp packaging/README.md RPM-BUILD.md
 %install
 install -d %{buildroot}%{ocf_provider_dir}/managed-container.d
 install -m 0755 refactored/managed-container %{buildroot}%{ocf_provider_dir}/managed-container
-install -m 0644 refactored/managed-container.d/*.sh %{buildroot}%{ocf_provider_dir}/managed-container.d/
+for subdir in . applications/postgresql lifecycles runtimes; do
    install -d %{buildroot}%{ocf_provider_dir}/managed-container.d/$subdir
    install -m 0644 refactored/managed-container.d/$subdir/*.sh %{buildroot}%{ocf_provider_dir}/managed-container.d/$subdir/
 done
 install -m 0644 refactored/managed-container.d/applications/postgresql/promotion_state.py %{buildroot}%{ocf_provider_dir}/managed-container.d/applications/postgresql/
 %check
 python3 tests/verify_refactor.py
 python3 tests/test_promotion_scores.py
 python3 tests/test_promotion_scheduler.py
 python3 tests/test_promotion_cleanup.py
 python3 tests/test_disconnected_state.py
 python3 tests/test_plugin_engine.py
 # Check the actual staged layout, including resolution from an unrelated cwd.
 agent="%{buildroot}%{ocf_provider_dir}/managed-container"
 "$agent" meta-data > staged-metadata.xml
-reference/managed-container meta-data > reference-metadata.xml
+refactored/managed-container meta-data > reference-metadata.xml
 cmp reference-metadata.xml staged-metadata.xml
 (cd /tmp && "$agent" meta-data) > other-cwd-metadata.xml
 cmp reference-metadata.xml other-cwd-metadata.xml
 %files
 %defattr(-,root,root,-)
-%doc README.md packaging/PROVENANCE.md RPM-BUILD.md
+%doc README.md packaging/PROVENANCE.md RPM-BUILD.md notes/PROMOTION-SCORING.md notes/DISCONNECTED-STANDBY.md
 %doc docs/operations.md docs/testing.md docs/review.md docs/plugin-engine.md
 %dir %{ocf_provider_dir}
 %attr(0755,root,root) %{ocf_provider_dir}/managed-container
 %dir %{ocf_provider_dir}/managed-container.d
 %attr(0644,root,root) %{ocf_provider_dir}/managed-container.d/*.sh
 %dir %{ocf_provider_dir}/managed-container.d/applications
 %dir %{ocf_provider_dir}/managed-container.d/applications/postgresql
 %dir %{ocf_provider_dir}/managed-container.d/lifecycles
 %dir %{ocf_provider_dir}/managed-container.d/runtimes
 %attr(0644,root,root) %{ocf_provider_dir}/managed-container.d/applications/postgresql/*.sh
 %attr(0644,root,root) %{ocf_provider_dir}/managed-container.d/applications/postgresql/*.py
 %attr(0644,root,root) %{ocf_provider_dir}/managed-container.d/lifecycles/*.sh
 %attr(0644,root,root) %{ocf_provider_dir}/managed-container.d/runtimes/*.sh
@@ -13,6 +13,9 @@
 : "${OCF_ROOT:=/usr/lib/ocf}"
 OCF_FUNCTIONS_DIR="${OCF_ROOT}/lib/heartbeat"
 # Preserve explicit resource identity before ocf-shellfuncs supplies its
 # standalone-invocation default; never publish a score for that dummy resource.
 MC_PROMOTION_RESOURCE_INSTANCE="${OCF_RESOURCE_INSTANCE:-}"
 . "${OCF_FUNCTIONS_DIR}/ocf-shellfuncs"
 # Pacemaker uses "Promoted" terminology, while many OCF shell libraries
@@ -23,61 +26,58 @@ OCF_FUNCTIONS_DIR="${OCF_ROOT}/lib/heartbeat"
 # Resolve modules relative to this entrypoint, independent of Pacemaker's cwd.
 MC_MODULE_DIR="$(CDPATH= cd -- "$(command dirname -- "${BASH_SOURCE[0]}")" && pwd)/managed-container.d"
-for MC_MODULE in common container postgres auth replication notify promote demote rejoin; do
+for MC_MODULE in common container engine; do
    if ! . "${MC_MODULE_DIR}/${MC_MODULE}.sh"; then
        ocf_exit_reason "Unable to load resource-agent module ${MC_MODULE_DIR}/${MC_MODULE}.sh"
        exit "${OCF_ERR_CONFIGURED}"
    fi
 done
 unset MC_MODULE
 mc_engine_load || exit $?
 case "$1" in
    meta-data)
-        meta_data
+        mc_engine_metadata
-        exit "${OCF_SUCCESS}"
+        exit $?
        ;;
    validate-all)
-        validate_all
+        mc_promotion_validate
        exit $?
        ;;
    monitor|status)
-        managed_monitor
+        mc_promotion_monitor
        exit $?
        ;;
    start)
-        validate_all || exit $?
+        mc_promotion_action start
        managed_start
        exit $?
        ;;
    stop)
-        managed_stop
+        mc_promotion_action stop
        exit $?
        ;;
    promote)
-        validate_all || exit $?
+        mc_promotion_action promote
        managed_promote
        exit $?
        ;;
    demote)
-        validate_all || exit $?
+        mc_promotion_action demote
        managed_demote
        exit $?
        ;;
    notify)
-        managed_notify
+        mc_application_notify
        exit $?
        ;;
    rejoin)
-        validate_all || exit $?
+        mc_promotion_action rejoin
        managed_rejoin
        exit $?
        ;;
@@ -2,8 +2,7 @@
 pg_replication_exec()
 {
-    as_service_user \
+    mc_runtime_exec \
        podman exec \
        --user "${PG_OS_USER}" \
        "${CONTAINER_NAME}" \
        sh -c '
@@ -130,8 +130,7 @@ managed_demote()
    ocf_log info \
        "Configuring ${local_node} as cascading standby of ${target_node}"
-    if ! as_service_user \
+    if ! mc_runtime_exec \
        podman exec \
        --user "${PG_OS_USER}" \
        --env "RA_PRIMARY_HOST=${target_addr}" \
        --env "RA_PRIMARY_PORT=${target_port}" \
@@ -0,0 +1,281 @@
 # Sourced by managed-container; uses the shared OCF and parameter environment.
 OCF_RESKEY_service_user_default="david"
 OCF_RESKEY_service_name_default="postgres.service"
 OCF_RESKEY_container_name_default="postgres"
 OCF_RESKEY_pg_os_user_default="postgres"
 OCF_RESKEY_db_user_default="david"
 OCF_RESKEY_db_name_default="david"
 OCF_RESKEY_rejoin_source_host_default=""
 OCF_RESKEY_rejoin_source_port_default="5432"
 OCF_RESKEY_replication_user_default="replicator"
 OCF_RESKEY_replication_slot_default=""
 OCF_RESKEY_application_name_default=""
 OCF_RESKEY_rejoin_allow_basebackup_default="false"
 : "${OCF_RESKEY_service_user:=${OCF_RESKEY_service_user_default}}"
 : "${OCF_RESKEY_service_name:=${OCF_RESKEY_service_name_default}}"
 : "${OCF_RESKEY_container_name:=${OCF_RESKEY_container_name_default}}"
 : "${OCF_RESKEY_pg_os_user:=${OCF_RESKEY_pg_os_user_default}}"
 : "${OCF_RESKEY_db_user:=${OCF_RESKEY_db_user_default}}"
 : "${OCF_RESKEY_db_name:=${OCF_RESKEY_db_name_default}}"
 : "${OCF_RESKEY_rejoin_source_host:=${OCF_RESKEY_rejoin_source_host_default}}"
 : "${OCF_RESKEY_rejoin_source_port:=${OCF_RESKEY_rejoin_source_port_default}}"
 : "${OCF_RESKEY_replication_user:=${OCF_RESKEY_replication_user_default}}"
 : "${OCF_RESKEY_replication_slot:=${OCF_RESKEY_replication_slot_default}}"
 : "${OCF_RESKEY_application_name:=${OCF_RESKEY_application_name_default}}"
 : "${OCF_RESKEY_rejoin_allow_basebackup:=${OCF_RESKEY_rejoin_allow_basebackup_default}}"
 SERVICE_USER="${OCF_RESKEY_service_user}"
 SERVICE_NAME="${OCF_RESKEY_service_name}"
 CONTAINER_NAME="${OCF_RESKEY_container_name}"
 PG_OS_USER="${OCF_RESKEY_pg_os_user}"
 DB_USER="${OCF_RESKEY_db_user}"
 DB_NAME="${OCF_RESKEY_db_name}"
 REJOIN_SOURCE_HOST="${OCF_RESKEY_rejoin_source_host}"
 REJOIN_SOURCE_PORT="${OCF_RESKEY_rejoin_source_port}"
 REPLICATION_USER="${OCF_RESKEY_replication_user}"
 REPLICATION_SLOT="${OCF_RESKEY_replication_slot}"
 APPLICATION_NAME="${OCF_RESKEY_application_name}"
 REJOIN_ALLOW_BASEBACKUP="${OCF_RESKEY_rejoin_allow_basebackup}"
 if [ -z "${APPLICATION_NAME}" ]; then
    APPLICATION_NAME="$(hostname -s)"
 fi
 meta_data()
 {
    cat <<EOF_META
 <?xml version="1.0"?>
 <!DOCTYPE resource-agent SYSTEM "ra-api-1.dtd">
 <resource-agent name="managed-container" version="0.4">
  <version>1.0</version>
  <longdesc lang="en">
    Manage an application inside a persistent rootless container.
    Version 0.4 implements role-aware monitoring, promotion, and demotion.
    The systemd user Quadlet owns the container lifecycle while pg_ctl owns
    the PostgreSQL lifecycle inside the container.
  </longdesc>
  <shortdesc lang="en">
    Manage PostgreSQL independently inside a persistent container
  </shortdesc>
  <parameters>
    <parameter name="service_user" unique="0" required="1">
      <longdesc lang="en">
        Linux user owning the rootless systemd/Podman service.
      </longdesc>
      <shortdesc lang="en">Rootless service owner</shortdesc>
      <content type="string" default="${OCF_RESKEY_service_user_default}"/>
    </parameter>
    <parameter name="service_name" unique="0" required="1">
      <longdesc lang="en">
        systemd user service generated by Quadlet.
      </longdesc>
      <shortdesc lang="en">Quadlet service name</shortdesc>
      <content type="string" default="${OCF_RESKEY_service_name_default}"/>
    </parameter>
    <parameter name="container_name" unique="0" required="1">
      <longdesc lang="en">
        Podman container containing PostgreSQL.
      </longdesc>
      <shortdesc lang="en">Container name</shortdesc>
      <content type="string" default="${OCF_RESKEY_container_name_default}"/>
    </parameter>
    <parameter name="pg_os_user" unique="0" required="1">
      <longdesc lang="en">
        Operating-system user running PostgreSQL inside the container.
      </longdesc>
      <shortdesc lang="en">PostgreSQL OS user</shortdesc>
      <content type="string" default="${OCF_RESKEY_pg_os_user_default}"/>
    </parameter>
    <parameter name="db_user" unique="0" required="1">
      <longdesc lang="en">
        PostgreSQL database user used for health queries.
      </longdesc>
      <shortdesc lang="en">Database user</shortdesc>
      <content type="string" default="${OCF_RESKEY_db_user_default}"/>
    </parameter>
    <parameter name="db_name" unique="0" required="1">
      <longdesc lang="en">
        PostgreSQL database used for health queries.
      </longdesc>
      <shortdesc lang="en">Database name</shortdesc>
      <content type="string" default="${OCF_RESKEY_db_name_default}"/>
    </parameter>
    <parameter name="rejoin_source_host" unique="0" required="0">
      <longdesc lang="en">
        PostgreSQL primary host used as the source for rejoin operations.
      </longdesc>
      <shortdesc lang="en">Rejoin source host</shortdesc>
      <content type="string" default="${OCF_RESKEY_rejoin_source_host_default}"/>
    </parameter>
    <parameter name="rejoin_source_port" unique="0" required="0">
      <longdesc lang="en">
        PostgreSQL port on the rejoin source.
      </longdesc>
      <shortdesc lang="en">Rejoin source port</shortdesc>
      <content type="integer" default="${OCF_RESKEY_rejoin_source_port_default}"/>
    </parameter>
    <parameter name="replication_user" unique="0" required="0">
      <longdesc lang="en">
        PostgreSQL role used for replication and pg_rewind.
      </longdesc>
      <shortdesc lang="en">Replication user</shortdesc>
      <content type="string" default="${OCF_RESKEY_replication_user_default}"/>
    </parameter>
    <parameter name="replication_slot" unique="0" required="0">
      <longdesc lang="en">
        Physical replication slot used by this standby.
      </longdesc>
      <shortdesc lang="en">Physical replication slot</shortdesc>
      <content type="string" default="${OCF_RESKEY_replication_slot_default}"/>
    </parameter>
    <parameter name="application_name" unique="0" required="0">
      <longdesc lang="en">
        PostgreSQL replication application_name. Defaults to the local hostname.
      </longdesc>
      <shortdesc lang="en">Replication application name</shortdesc>
      <content type="string" default="${OCF_RESKEY_application_name_default}"/>
    </parameter>
    <parameter name="rejoin_allow_basebackup" unique="0" required="0">
      <longdesc lang="en">
        Allow rejoin to fall back to pg_basebackup when pg_rewind cannot
        repair the local PostgreSQL data directory. The old PGDATA is
        preserved before replacement. This must only be enabled when the
        operator has established that this node may safely discard its
        divergent PostgreSQL history.
      </longdesc>
      <shortdesc lang="en">Allow pg_basebackup rejoin fallback</shortdesc>
      <content type="boolean" default="${OCF_RESKEY_rejoin_allow_basebackup_default}"/>
    </parameter>
  </parameters>
  <actions>
    <action name="start" timeout="60s"/>
    <action name="stop" timeout="60s"/>
    <action name="promote" timeout="60s"/>
    <action name="demote" timeout="60s"/>
    <action name="notify" timeout="20s"/>
    <action name="monitor" timeout="20s" interval="10s"/>
    <action name="validate-all" timeout="20s"/>
    <action name="meta-data" timeout="5s"/>
  </actions>
 </resource-agent>
 EOF_META
 }
 validate_all()
 {
    if ! id "${SERVICE_USER}" >/dev/null 2>&1; then
        ocf_exit_reason "Service user '${SERVICE_USER}' does not exist"
        return "${OCF_ERR_CONFIGURED}"
    fi
    for cmd in runuser systemctl podman getent; do
        if ! command -v "${cmd}" >/dev/null 2>&1; then
            ocf_exit_reason "Required command '${cmd}' is not installed"
            return "${OCF_ERR_INSTALLED}"
        fi
    done
    if [ ! -d "/run/user/$(service_uid)" ]; then
        ocf_exit_reason \
            "Runtime directory /run/user/$(service_uid) does not exist"
        return "${OCF_ERR_GENERIC}"
    fi
    if ! mc_lifecycle_cat "${SERVICE_NAME}" >/dev/null 2>&1
    then
        ocf_exit_reason \
            "User service '${SERVICE_NAME}' cannot be found for ${SERVICE_USER}"
        return "${OCF_ERR_CONFIGURED}"
    fi
    local podman_error
    podman_error="$(
        mc_runtime_info 2>&1 >/dev/null
    )"
    local podman_rc=$?
    if [ "${podman_rc}" -ne 0 ]; then
        ocf_exit_reason \
            "Rootless Podman failed for ${SERVICE_USER}: ${podman_error}"
        return "${OCF_ERR_GENERIC}"
    fi
    return "${OCF_SUCCESS}"
 }
 # Keep the original metadata function intact; advertise the added policy here.
 mc_promotion_metadata()
 {
    meta_data | sed '/<\/parameters>/i\
    <parameter name="disconnected_max_age" unique="0" required="0">\
      <longdesc lang="en">Maximum age in seconds of a verified streaming observation for disconnected promotion eligibility. Default 300; 0 disables disconnected eligibility. Requires fencing managed by Pacemaker. PostgreSQL restart requires fresh streaming verification.</longdesc>\
      <shortdesc lang="en">Disconnected standby observation lifetime</shortdesc>\
      <content type="integer" default="300"/>\
    </parameter>'
 }
 MC_APPLICATION_API=1
 MC_APPLICATION_DIR="${MC_MODULE_DIR}/applications/postgresql"
 for MC_PG_MODULE in postgres auth replication notify promote demote rejoin; do
    if ! . "${MC_APPLICATION_DIR}/${MC_PG_MODULE}.sh"; then
        ocf_exit_reason "Unable to load resource-agent module ${MC_APPLICATION_DIR}/${MC_PG_MODULE}.sh"
        return "${OCF_ERR_CONFIGURED}"
    fi
 done
 unset MC_PG_MODULE
 mc_application_monitor()
 {
    managed_monitor
 }
 mc_application_standby_score()
 {
    mc_pg_standby_score
 }
 mc_application_validate()
 {
    validate_all
 }
 mc_application_metadata()
 {
    mc_promotion_metadata
 }
 mc_application_notify()
 {
    managed_notify
 }
 mc_application_action()
 {
    case "$1" in
        start|stop|promote|demote|rejoin) "managed_$1" ;;
        *) return "${OCF_ERR_UNIMPLEMENTED}" ;;
    esac
 }
@@ -4,8 +4,7 @@ pg_command()
 {
    local command="$1"
-    as_service_user \
+    mc_runtime_exec \
        podman exec \
        --user "${PG_OS_USER}" \
        "${CONTAINER_NAME}" \
        sh -c "${command}"
@@ -32,8 +31,7 @@ pg_running()
 pg_ready()
 {
-    as_service_user \
+    mc_runtime_exec \
        podman exec \
        --user "${PG_OS_USER}" \
        "${CONTAINER_NAME}" \
        pg_isready -q \
@@ -43,8 +41,7 @@ pg_ready()
 pg_role()
 {
-    as_service_user \
+    mc_runtime_exec \
        podman exec \
        --user "${PG_OS_USER}" \
        "${CONTAINER_NAME}" \
        psql \
@@ -55,8 +52,7 @@ pg_role()
 pg_wal_receiver_status()
 {
-    as_service_user \
+    mc_runtime_exec \
        podman exec \
        --user "${PG_OS_USER}" \
        "${CONTAINER_NAME}" \
        psql \
@@ -153,8 +149,7 @@ managed_start()
        "Ensuring container service ${SERVICE_NAME} is running"
    if ! service_running; then
-        if ! as_service_user \
+        if ! mc_lifecycle_start "${SERVICE_NAME}"
            systemctl --user start "${SERVICE_NAME}"
        then
            ocf_exit_reason \
                "Failed to start ${SERVICE_NAME}"
@@ -260,8 +255,7 @@ managed_stop()
        ocf_log info \
            "Stopping container service ${SERVICE_NAME}"
-        if ! as_service_user \
+        if ! mc_lifecycle_stop "${SERVICE_NAME}"
            systemctl --user stop "${SERVICE_NAME}"
        then
            ocf_exit_reason \
                "Failed to stop ${SERVICE_NAME}"
@@ -290,3 +284,74 @@ managed_stop()
    return "${OCF_SUCCESS}"
 }
 # Conservative scoring eligibility only; this does not replace role detection,
 # fencing, the promotion action, or the existing controlled-demote WAL checks.
 # This strict predicate remains unchanged; disconnected policy is separate below.
 mc_pg_promotion_eligible()
 {
    local eligible
    eligible="$(
        mc_runtime_exec_bounded 3s --user "${PG_OS_USER}" \
            --env 'PGOPTIONS=-c statement_timeout=2000' \
            "${CONTAINER_NAME}" \
            psql -X -v ON_ERROR_STOP=1 -U "${DB_USER}" -d "${DB_NAME}" -Atqc "
                SELECT CASE WHEN pg_is_in_recovery() THEN
                    NOT pg_is_wal_replay_paused()
                    AND EXISTS (
                        SELECT 1 FROM pg_stat_wal_receiver
                        WHERE status = 'streaming'
                          AND flushed_lsn IS NOT NULL
                          AND pg_last_wal_replay_lsn() >= flushed_lsn
                    )
                ELSE false END;
            " 2>/dev/null
    )" || return 1
    [ "${eligible}" = t ]
 }
 # One observation supplies both strict connected eligibility and disconnected
 # evidence. PostgreSQL restart time deliberately invalidates prior observations,
 # including a new basebackup sharing the same PostgreSQL system identifier.
 mc_pg_standby_observation()
 {
    mc_runtime_exec_bounded 3s --user "${PG_OS_USER}" \
        --env 'PGOPTIONS=-c statement_timeout=2000' \
        "${CONTAINER_NAME}" \
        psql -X -v ON_ERROR_STOP=1 -U "${DB_USER}" -d "${DB_NAME}" -Atqc "
            WITH s AS MATERIALIZED (
                SELECT pg_is_in_recovery() AS recovery,
                       pg_last_wal_replay_lsn() AS replay,
                       r.status AS receiver, r.flushed_lsn AS flush
                FROM (SELECT 1) AS singleton
                LEFT JOIN pg_stat_wal_receiver r ON true
            )
            SELECT json_build_object(
                'identity', json_build_array(
                    (SELECT system_identifier::text FROM pg_control_system()),
                    current_setting('data_directory'),
                    extract(epoch FROM pg_postmaster_start_time())::text),
                'recovery', recovery,
                'paused', CASE WHEN recovery THEN pg_is_wal_replay_paused() ELSE true END,
                'receiver', receiver, 'replay', replay::text, 'flush', flush::text,
                'strict', CASE WHEN recovery THEN
                    NOT pg_is_wal_replay_paused() AND receiver = 'streaming'
                    AND flush IS NOT NULL AND replay >= flush
                    ELSE false END)
            FROM s;
        " 2>/dev/null
 }
 mc_pg_standby_score()
 {
    local observation
    observation="$(mc_pg_standby_observation)" || return 1
    # The helper is root-owned alongside the modules; state never enters the
    # rootless container. A failed/slow helper always withdraws eligibility.
    printf '%s' "${observation}" | timeout --kill-after=1s 1s python3 \
        "${MC_APPLICATION_DIR}/promotion_state.py" \
        "${MC_PROMOTION_RESOURCE_INSTANCE:-}" \
        "${SERVICE_USER}" "${SERVICE_NAME}" "${CONTAINER_NAME}" \
        "${OCF_RESKEY_disconnected_max_age:-300}"
 }
@@ -0,0 +1,173 @@
 """Local promotion evidence. Invoked by the root RA; never invokes cluster tools.
 State is a bounded JSON record, never executable input. The public CLI uses a
 fixed root-owned /run directory and CLOCK_BOOTTIME, including suspend duration.
 """
 import fcntl
 import hashlib
 import json
 import math
 import os
 from pathlib import Path
 import re
 import stat
 import sys
 import tempfile
 import time
 STATE_DIR = Path('/run/managed-container/promotion')
 def unique_object(pairs):
    result = {}
    for key, value in pairs:
        if key in result:
            raise ValueError('duplicate JSON key')
        result[key] = value
    return result
 def lsn(value):
    if not isinstance(value, str) or not re.fullmatch(r'[0-9A-F]{1,8}/[0-9A-F]{1,8}', value):
        raise ValueError('invalid LSN')
    high, low = value.split('/')
    return (int(high, 16) << 32) | int(low, 16)
 def number(value):
    return type(value) in (int, float) and math.isfinite(value) and value >= 0
 def validate_observation(obs):
    if set(obs) != {'identity', 'recovery', 'paused', 'receiver', 'replay', 'flush', 'strict'}:
        raise ValueError('invalid observation fields')
    identity = obs['identity']
    if not isinstance(identity, list) or len(identity) != 3:
        raise ValueError('invalid identity')
    system, path, started = identity
    if (not isinstance(system, str) or not re.fullmatch(r'[0-9]{1,20}', system)
            or not isinstance(path, str) or not path.startswith('/') or len(path) > 2048
            or not isinstance(started, str) or not re.fullmatch(r'[0-9]{1,12}\.[0-9]{1,6}', started)):
        raise ValueError('invalid identity values')
    if type(obs['recovery']) is not bool or type(obs['paused']) is not bool:
        raise ValueError('invalid role')
    if obs['strict'] is not None and type(obs['strict']) is not bool:
        raise ValueError('invalid strict flag')
    if obs['receiver'] is not None and obs['receiver'] not in (
            'starting', 'streaming', 'waiting', 'restarting', 'stopping'):
        raise ValueError('invalid receiver status')
    replay = lsn(obs['replay']) if obs['replay'] is not None else None
    flush = lsn(obs['flush']) if obs['flush'] is not None else None
    strict = (obs['recovery'] and not obs['paused'] and obs['receiver'] == 'streaming'
              and replay is not None and flush is not None and replay >= flush)
    if (obs['strict'] is True) != strict:
        raise ValueError('inconsistent strict observation')
    return strict
 class Evidence:
    def __init__(self, directory, context, boot, now, wall, owner=0):
        self.directory, self.context = Path(directory), context
        self.boot, self.now, self.wall, self.owner = boot, now, wall, owner
        key = hashlib.sha256(json.dumps(context, separators=(',', ':')).encode()).hexdigest()
        self.path = self.directory / (key + '.json')
        self.lock = self.directory / (key + '.lock')
    def secure(self, path, mode):
        info = path.lstat()
        if (info.st_uid != self.owner or stat.S_IMODE(info.st_mode) != mode
                or (not stat.S_ISREG(info.st_mode) if mode == 0o600 else not stat.S_ISDIR(info.st_mode))):
            raise ValueError('unsafe state ownership/type/mode')
    def discard(self):
        self.path.unlink(missing_ok=True)
    def evaluate(self, obs, max_age):
        if type(max_age) is not int or not 0 <= max_age <= 99999:
            raise ValueError('invalid freshness threshold')
        self.directory.mkdir(mode=0o700, exist_ok=True)
        self.secure(self.directory, 0o700)
        fd = os.open(self.lock, os.O_CREAT | os.O_NOFOLLOW | os.O_RDWR, 0o600)
        with os.fdopen(fd, 'r+') as lock:
            self.secure(self.lock, 0o600)
            fcntl.flock(lock, fcntl.LOCK_EX | fcntl.LOCK_NB)
            try:
                return self._evaluate(obs, max_age)
            except (ValueError, TypeError, KeyError, OverflowError):
                self.discard()
                return '-INFINITY'
    def _evaluate(self, obs, max_age):
        strict = validate_observation(obs)
        if strict:
            record = dict(version=1, context=self.context, boot=self.boot,
                          identity=obs['identity'], observed_boottime=self.now,
                          observed_unix=self.wall, replay=obs['replay'], flush=obs['flush'])
            # Same-directory atomic replace; failed persistence cannot authorize
            # subsequent disconnected promotion. No secrets are recorded.
            fd, temporary = tempfile.mkstemp(prefix='.observation-', dir=self.directory)
            try:
                with os.fdopen(fd, 'w') as out:
                    json.dump(record, out, separators=(',', ':'))
                    out.flush()
                    os.fsync(out.fileno())
                os.replace(temporary, self.path)
            finally:
                Path(temporary).unlink(missing_ok=True)
            return '100'
        if (not obs['recovery'] or obs['paused'] or obs['receiver'] == 'streaming'
                or obs['replay'] is None or max_age == 0):
            self.discard()
            return '-INFINITY'
        if not self.path.exists():
            return '-INFINITY'
        self.secure(self.path, 0o600)
        if self.path.stat().st_size > 4096:
            raise ValueError('oversized state')
        with self.path.open() as source:
            record = json.load(source, object_pairs_hook=unique_object)
        if set(record) != {'version', 'context', 'boot', 'identity', 'observed_boottime',
                           'observed_unix', 'replay', 'flush'}:
            raise ValueError('invalid record fields')
        if (type(record['version']) is not int or record['version'] != 1
                or record['context'] != self.context or record['boot'] != self.boot
                or record['identity'] != obs['identity']
                or not number(record['observed_boottime']) or not number(record['observed_unix'])
                or not 0 <= self.now - record['observed_boottime'] <= max_age
                or lsn(record['replay']) < lsn(record['flush'])
                or lsn(obs['replay']) < lsn(record['replay'])):
            raise ValueError('expired or incompatible observation')
        return '50'
 def main():
    if os.geteuid() != 0 or len(sys.argv) != 6 or not all(sys.argv[1:5]):
        raise ValueError('root and explicit resource context required')
    age = sys.argv[5]
    if not re.fullmatch(r'(0|[1-9][0-9]{0,4})', age):
        raise ValueError('invalid freshness threshold')
    # /run is trusted root-owned; validate our parent before using a child.
    parent = STATE_DIR.parent
    parent.mkdir(mode=0o700, exist_ok=True)
    info = parent.lstat()
    if info.st_uid != 0 or not stat.S_ISDIR(info.st_mode) or stat.S_IMODE(info.st_mode) != 0o700:
        raise ValueError('unsafe state parent')
    raw = sys.stdin.read(8193)
    if len(raw) > 8192:
        raise ValueError('oversized observation')
    evidence = Evidence(STATE_DIR, sys.argv[1:5],
                        Path('/proc/sys/kernel/random/boot_id').read_text().strip(),
                        time.clock_gettime(time.CLOCK_BOOTTIME), time.time())
    try:
        obs = json.loads(raw, object_pairs_hook=unique_object)
    except ValueError:
        obs = {}  # Invalid query output withdraws and removes prior evidence.
    print(evidence.evaluate(obs, int(age)))
 if __name__ == '__main__':
    try:
        main()
    except (OSError, ValueError, TypeError, KeyError, OverflowError):
        print('-INFINITY')
        sys.exit(1)
@@ -268,8 +268,7 @@ managed_rejoin()
    ocf_log info \
        "Configuring PostgreSQL as standby of ${REJOIN_SOURCE_HOST}"
-    if ! as_service_user \
+    if ! mc_runtime_exec \
        podman exec \
        --user "${PG_OS_USER}" \
        --env "RA_PRIMARY_HOST=${REJOIN_SOURCE_HOST}" \
        --env "RA_PRIMARY_PORT=${REJOIN_SOURCE_PORT}" \
@@ -1,227 +1,113 @@
-# Sourced by managed-container; uses the shared OCF and parameter environment.
+# Shared OCF scoring wrappers; application policy is supplied by the plugin.
-OCF_RESKEY_service_user_default="david"
+# Promotion scores are scoped by Pacemaker to OCF_RESOURCE_INSTANCE and the
-OCF_RESKEY_service_name_default="postgres.service"
+# executing node. Do not construct a legacy attribute name or cache locally:
-OCF_RESKEY_container_name_default="postgres"
+# querying attrd on every observation also repairs a lost transient attribute.
-OCF_RESKEY_pg_os_user_default="postgres"
+mc_promotion_score_set()
 OCF_RESKEY_db_user_default="david"
 OCF_RESKEY_db_name_default="david"
 OCF_RESKEY_rejoin_source_host_default=""
 OCF_RESKEY_rejoin_source_port_default="5432"
 OCF_RESKEY_replication_user_default="replicator"
 OCF_RESKEY_replication_slot_default=""
 OCF_RESKEY_application_name_default=""
 OCF_RESKEY_rejoin_allow_basebackup_default="false"
 : "${OCF_RESKEY_service_user:=${OCF_RESKEY_service_user_default}}"
 : "${OCF_RESKEY_service_name:=${OCF_RESKEY_service_name_default}}"
 : "${OCF_RESKEY_container_name:=${OCF_RESKEY_container_name_default}}"
 : "${OCF_RESKEY_pg_os_user:=${OCF_RESKEY_pg_os_user_default}}"
 : "${OCF_RESKEY_db_user:=${OCF_RESKEY_db_user_default}}"
 : "${OCF_RESKEY_db_name:=${OCF_RESKEY_db_name_default}}"
 : "${OCF_RESKEY_rejoin_source_host:=${OCF_RESKEY_rejoin_source_host_default}}"
 : "${OCF_RESKEY_rejoin_source_port:=${OCF_RESKEY_rejoin_source_port_default}}"
 : "${OCF_RESKEY_replication_user:=${OCF_RESKEY_replication_user_default}}"
 : "${OCF_RESKEY_replication_slot:=${OCF_RESKEY_replication_slot_default}}"
 : "${OCF_RESKEY_application_name:=${OCF_RESKEY_application_name_default}}"
 : "${OCF_RESKEY_rejoin_allow_basebackup:=${OCF_RESKEY_rejoin_allow_basebackup_default}}"
 SERVICE_USER="${OCF_RESKEY_service_user}"
 SERVICE_NAME="${OCF_RESKEY_service_name}"
 CONTAINER_NAME="${OCF_RESKEY_container_name}"
 PG_OS_USER="${OCF_RESKEY_pg_os_user}"
 DB_USER="${OCF_RESKEY_db_user}"
 DB_NAME="${OCF_RESKEY_db_name}"
 REJOIN_SOURCE_HOST="${OCF_RESKEY_rejoin_source_host}"
 REJOIN_SOURCE_PORT="${OCF_RESKEY_rejoin_source_port}"
 REPLICATION_USER="${OCF_RESKEY_replication_user}"
 REPLICATION_SLOT="${OCF_RESKEY_replication_slot}"
 APPLICATION_NAME="${OCF_RESKEY_application_name}"
 REJOIN_ALLOW_BASEBACKUP="${OCF_RESKEY_rejoin_allow_basebackup}"
 if [ -z "${APPLICATION_NAME}" ]; then
    APPLICATION_NAME="$(hostname -s)"
 fi
 meta_data()
 {
-    cat <<EOF_META
+    local score="$1" current
-<?xml version="1.0"?>
+    local resource="${MC_PROMOTION_RESOURCE_INSTANCE:-}"
 <!DOCTYPE resource-agent SYSTEM "ra-api-1.dtd">
 <resource-agent name="managed-container" version="0.4">
  <version>1.0</version>
-  <longdesc lang="en">
+    if [ -z "${resource}" ]; then
-    Manage an application inside a persistent rootless container.
+        ocf_log warn "Cannot publish promotion score without OCF_RESOURCE_INSTANCE"
        return 1
    fi
-    Version 0.4 implements role-aware monitoring, promotion, and demotion.
+    case "${score}" in
-    The systemd user Quadlet owns the container lifecycle while pg_ctl owns
+        1000|100|50|-INFINITY) ;;
-    the PostgreSQL lifecycle inside the container.
+        *) return 1 ;;
-  </longdesc>
+    esac
-  <shortdesc lang="en">
+    current="$(
-    Manage PostgreSQL independently inside a persistent container
+        timeout --kill-after=1s 2s crm_attribute \
-  </shortdesc>
+            --promotion="${resource}" --lifetime=reboot \
            --query --quiet 2>/dev/null
    )" || current=""
-  <parameters>
+    [ "${current}" = "${score}" ] && return 0
-    <parameter name="service_user" unique="0" required="1">
+    if ! timeout --kill-after=1s 2s crm_attribute \
-      <longdesc lang="en">
+        --promotion="${resource}" --lifetime=reboot \
-        Linux user owning the rootless systemd/Podman service.
+        --update="${score}" >/dev/null 2>&1
-      </longdesc>
+    then
-      <shortdesc lang="en">Rootless service owner</shortdesc>
+        ocf_log err "Unable to publish promotion score ${score} for ${resource}; prior score may remain"
-      <content type="string" default="${OCF_RESKEY_service_user_default}"/>
+        return 1
-    </parameter>
+    fi
-
+    return 0
    <parameter name="service_name" unique="0" required="1">
      <longdesc lang="en">
        systemd user service generated by Quadlet.
      </longdesc>
      <shortdesc lang="en">Quadlet service name</shortdesc>
      <content type="string" default="${OCF_RESKEY_service_name_default}"/>
    </parameter>
    <parameter name="container_name" unique="0" required="1">
      <longdesc lang="en">
        Podman container containing PostgreSQL.
      </longdesc>
      <shortdesc lang="en">Container name</shortdesc>
      <content type="string" default="${OCF_RESKEY_container_name_default}"/>
    </parameter>
    <parameter name="pg_os_user" unique="0" required="1">
      <longdesc lang="en">
        Operating-system user running PostgreSQL inside the container.
      </longdesc>
      <shortdesc lang="en">PostgreSQL OS user</shortdesc>
      <content type="string" default="${OCF_RESKEY_pg_os_user_default}"/>
    </parameter>
    <parameter name="db_user" unique="0" required="1">
      <longdesc lang="en">
        PostgreSQL database user used for health queries.
      </longdesc>
      <shortdesc lang="en">Database user</shortdesc>
      <content type="string" default="${OCF_RESKEY_db_user_default}"/>
    </parameter>
    <parameter name="db_name" unique="0" required="1">
      <longdesc lang="en">
        PostgreSQL database used for health queries.
      </longdesc>
      <shortdesc lang="en">Database name</shortdesc>
      <content type="string" default="${OCF_RESKEY_db_name_default}"/>
    </parameter>
    <parameter name="rejoin_source_host" unique="0" required="0">
      <longdesc lang="en">
        PostgreSQL primary host used as the source for rejoin operations.
      </longdesc>
      <shortdesc lang="en">Rejoin source host</shortdesc>
      <content type="string" default="${OCF_RESKEY_rejoin_source_host_default}"/>
    </parameter>
    <parameter name="rejoin_source_port" unique="0" required="0">
      <longdesc lang="en">
        PostgreSQL port on the rejoin source.
      </longdesc>
      <shortdesc lang="en">Rejoin source port</shortdesc>
      <content type="integer" default="${OCF_RESKEY_rejoin_source_port_default}"/>
    </parameter>
    <parameter name="replication_user" unique="0" required="0">
      <longdesc lang="en">
        PostgreSQL role used for replication and pg_rewind.
      </longdesc>
      <shortdesc lang="en">Replication user</shortdesc>
      <content type="string" default="${OCF_RESKEY_replication_user_default}"/>
    </parameter>
    <parameter name="replication_slot" unique="0" required="0">
      <longdesc lang="en">
        Physical replication slot used by this standby.
      </longdesc>
      <shortdesc lang="en">Physical replication slot</shortdesc>
      <content type="string" default="${OCF_RESKEY_replication_slot_default}"/>
    </parameter>
    <parameter name="application_name" unique="0" required="0">
      <longdesc lang="en">
        PostgreSQL replication application_name. Defaults to the local hostname.
      </longdesc>
      <shortdesc lang="en">Replication application name</shortdesc>
      <content type="string" default="${OCF_RESKEY_application_name_default}"/>
    </parameter>
    <parameter name="rejoin_allow_basebackup" unique="0" required="0">
      <longdesc lang="en">
        Allow rejoin to fall back to pg_basebackup when pg_rewind cannot
        repair the local PostgreSQL data directory. The old PGDATA is
        preserved before replacement. This must only be enabled when the
        operator has established that this node may safely discard its
        divergent PostgreSQL history.
      </longdesc>
      <shortdesc lang="en">Allow pg_basebackup rejoin fallback</shortdesc>
      <content type="boolean" default="${OCF_RESKEY_rejoin_allow_basebackup_default}"/>
    </parameter>
  </parameters>
  <actions>
    <action name="start" timeout="60s"/>
    <action name="stop" timeout="60s"/>
    <action name="promote" timeout="60s"/>
    <action name="demote" timeout="60s"/>
    <action name="notify" timeout="20s"/>
    <action name="monitor" timeout="20s" interval="10s"/>
    <action name="validate-all" timeout="20s"/>
    <action name="meta-data" timeout="5s"/>
  </actions>
 </resource-agent>
 EOF_META
 }
-validate_all()
+mc_promotion_score_from_monitor()
 {
-    if ! id "${SERVICE_USER}" >/dev/null 2>&1; then
+    local monitor_rc="$1" score=-INFINITY
-        ocf_exit_reason "Service user '${SERVICE_USER}' does not exist"
+
    case "${monitor_rc}" in
        "${OCF_RUNNING_PROMOTED}") score=1000 ;;
        "${OCF_SUCCESS}")
            score="$(mc_application_standby_score)" || score=-INFINITY
            ;;
    esac
    case "${score}" in
        1000|100|50|-INFINITY) ;;
        *) score=-INFINITY ;;
    esac
    mc_promotion_score_set "${score}"
 }
 # Score publication must not change OCF role detection or its return code.
 mc_promotion_monitor()
 {
    local monitor_rc
    mc_application_monitor
    monitor_rc=$?
    mc_promotion_score_from_monitor "${monitor_rc}" || true
    return "${monitor_rc}"
 }
 # Lifecycle functions retain their original checks, commands and return codes.
 # Withdraw the previous score before any transition (including validation
 # failure), then reassess only after success. Their internal monitor calls do
 # not publish scores, so they cannot restore a positive score mid-transition.
 mc_promotion_action()
 {
    local action="$1" action_rc monitor_rc
    case "${action}" in
        start|stop|promote|demote|rejoin) ;;
        *) return "${OCF_ERR_UNIMPLEMENTED}" ;;
    esac
    mc_promotion_score_set -INFINITY || true
    if [ "${action}" != stop ]; then
        mc_promotion_validate
        action_rc=$?
        [ "${action_rc}" -eq "${OCF_SUCCESS}" ] || return "${action_rc}"
    fi
    mc_application_action "${action}"
    action_rc=$?
    if [ "${action_rc}" -eq "${OCF_SUCCESS}" ] && [ "${action}" != stop ]; then
        mc_application_monitor
        monitor_rc=$?
        mc_promotion_score_from_monitor "${monitor_rc}" || true
    fi
    return "${action_rc}"
 }
 mc_promotion_validate()
 {
    local utility
    if ! [[ "${OCF_RESKEY_disconnected_max_age:-300}" =~ ^(0|[1-9][0-9]{0,4})$ ]]; then
        ocf_exit_reason "disconnected_max_age must be an integer from 0 to 99999 seconds"
        return "${OCF_ERR_CONFIGURED}"
    fi
-    for cmd in runuser systemctl podman getent; do
+    for utility in crm_attribute timeout python3; do
-        if ! command -v "${cmd}" >/dev/null 2>&1; then
+        if ! command -v "${utility}" >/dev/null 2>&1; then
-            ocf_exit_reason "Required command '${cmd}' is not installed"
+            ocf_exit_reason "Required promotion scoring command '${utility}' is not installed"
            return "${OCF_ERR_INSTALLED}"
        fi
    done
-
+    mc_application_validate
    if [ ! -d "/run/user/$(service_uid)" ]; then
        ocf_exit_reason \
            "Runtime directory /run/user/$(service_uid) does not exist"
        return "${OCF_ERR_GENERIC}"
    fi
    if ! as_service_user \
        systemctl --user cat "${SERVICE_NAME}" >/dev/null 2>&1
    then
        ocf_exit_reason \
            "User service '${SERVICE_NAME}' cannot be found for ${SERVICE_USER}"
        return "${OCF_ERR_CONFIGURED}"
    fi
    local podman_error
    podman_error="$(
        as_service_user podman info 2>&1 >/dev/null
    )"
    local podman_rc=$?
    if [ "${podman_rc}" -ne 0 ]; then
        ocf_exit_reason \
            "Rootless Podman failed for ${SERVICE_USER}: ${podman_error}"
        return "${OCF_ERR_GENERIC}"
    fi
    return "${OCF_SUCCESS}"
 }
@@ -37,8 +37,7 @@ as_service_user()
 service_running()
 {
-    as_service_user \
+    mc_lifecycle_running "${SERVICE_NAME}"
        systemctl --user is-active --quiet "${SERVICE_NAME}"
 }
 container_running()
@@ -46,7 +45,7 @@ container_running()
    local state
    state="$(
-        as_service_user podman inspect \
+        mc_runtime_inspect \
            --format '{{.State.Running}}' \
            "${CONTAINER_NAME}" 2>/dev/null
    )" || return 1
@@ -0,0 +1,59 @@
 # Engine API 1. Selection is an allowlist of packaged implementations, never a
 # filename, shell fragment or a plugin downloaded from cluster configuration.
 mc_engine_load()
 {
    local plugin hook
    if [ "${OCF_RESKEY_application:-postgresql}" != postgresql ] ||
       [ "${OCF_RESKEY_container_runtime:-podman}" != podman ] ||
       [ "${OCF_RESKEY_lifecycle_manager:-systemd}" != systemd ]; then
        ocf_exit_reason "Supported plugin combination: application=postgresql container_runtime=podman lifecycle_manager=systemd"
        return "${OCF_ERR_CONFIGURED}"
    fi
    unset MC_RUNTIME_API MC_LIFECYCLE_API MC_APPLICATION_API
    for plugin in runtimes/podman lifecycles/systemd applications/postgresql/plugin; do
        if ! . "${MC_MODULE_DIR}/${plugin}.sh"; then
            ocf_exit_reason "Unable to load resource-agent module ${MC_MODULE_DIR}/${plugin}.sh"
            return "${OCF_ERR_CONFIGURED}"
        fi
    done
    if [ "${MC_RUNTIME_API:-}" != 1 ] || [ "${MC_LIFECYCLE_API:-}" != 1 ] ||
       [ "${MC_APPLICATION_API:-}" != 1 ]; then
        ocf_exit_reason "Incompatible plugin interface; API 1 required"
        return "${OCF_ERR_CONFIGURED}"
    fi
    for hook in mc_runtime_exec mc_runtime_exec_bounded mc_runtime_inspect mc_runtime_info \
                mc_lifecycle_start mc_lifecycle_stop mc_lifecycle_running mc_lifecycle_cat \
                mc_application_monitor mc_application_standby_score mc_application_validate \
                mc_application_metadata mc_application_notify mc_application_action; do
        if ! declare -F "${hook}" >/dev/null; then
            ocf_exit_reason "Required plugin hook is missing: ${hook}"
            return "${OCF_ERR_CONFIGURED}"
        fi
    done
 }
 mc_engine_metadata()
 {
    # Metadata discovery works without configured resource instance attributes.
    # Capture the plugin's status before transforming XML; avoid a pipeline
    # masking a metadata failure. No runtime/service/cluster access is required.
    local metadata
    metadata="$(mc_application_metadata)" || return "$?"
    printf '%s\n' "${metadata}" | sed '/<\/parameters>/i\
    <parameter name="application" unique="0" required="0">\
      <longdesc lang="en">Packaged application plugin. Only postgresql is supported.</longdesc>\
      <shortdesc lang="en">Application plugin</shortdesc>\
      <content type="string" default="postgresql"/>\
    </parameter>\
    <parameter name="container_runtime" unique="0" required="0">\
      <longdesc lang="en">Packaged runtime adapter. Only podman is supported.</longdesc>\
      <shortdesc lang="en">Container runtime</shortdesc>\
      <content type="string" default="podman"/>\
    </parameter>\
    <parameter name="lifecycle_manager" unique="0" required="0">\
      <longdesc lang="en">Container lifecycle owner. Only systemd with the existing user service is supported.</longdesc>\
      <shortdesc lang="en">Lifecycle manager</shortdesc>\
      <content type="string" default="systemd"/>\
    </parameter>'
 }
@@ -0,0 +1,23 @@
 # Lifecycle adapter API 1. Quadlet remains the owner of container construction,
 # init/maintenance command, signal settings and service restart/exit policies.
 MC_LIFECYCLE_API=1
 mc_lifecycle_start()
 {
    as_service_user systemctl --user start "$1"
 }
 mc_lifecycle_stop()
 {
    as_service_user systemctl --user stop "$1"
 }
 mc_lifecycle_running()
 {
    as_service_user systemctl --user is-active --quiet "$1"
 }
 mc_lifecycle_cat()
 {
    as_service_user systemctl --user cat "$1"
 }
@@ -0,0 +1,24 @@
 # Runtime adapter API 1. The existing rootless privilege boundary is unchanged.
 MC_RUNTIME_API=1
 mc_runtime_exec()
 {
    as_service_user podman exec "$@"
 }
 mc_runtime_exec_bounded()
 {
    local duration="$1"
    shift
    as_service_user timeout --kill-after=1s "${duration}" podman exec "$@"
 }
 mc_runtime_inspect()
 {
    as_service_user podman inspect "$@"
 }
 mc_runtime_info()
 {
    as_service_user podman info "$@"
 }
@@ -22,7 +22,7 @@ tar --sort=name --mtime="@$SOURCE_DATE_EPOCH" --owner=0 --group=0 --numeric-owne
    --exclude='__pycache__' --exclude='*.pyc' \
    --transform="s,^,ocf-managed-container-$package_version/," \
    -czf "$build_dir/SOURCES/ocf-managed-container-$package_version.tar.gz" \
-    README.md HANDOFF.md refactored reference tests packaging scripts
+    README.md HANDOFF.md docs refactored reference tests packaging scripts notes/PROMOTION-SCORING.md notes/DISCONNECTED-STANDBY.md
 # Freeze identity into the SRPM spec so rpmbuild --rebuild needs no Git metadata
 # or external macro definitions and retains the same version and release.
 sed -e "s/^Version:.*/Version:        $package_version/" \
@@ -0,0 +1,91 @@
 #!/usr/bin/env bash
 # HISTORICAL LAB TOOL: not a production/scoring installer; see docs/operations.md.
 # One-time installation of the modular baseline on the confirmed lab standby.
 # Run with sudo in your terminal. No service or PostgreSQL lifecycle actions.
 set -euo pipefail
 export PATH=/usr/sbin:/usr/bin:/sbin:/bin
 project_dir="$(CDPATH= cd -- "$(dirname -- "${BASH_SOURCE[0]}")/.." && pwd)"
 provider_dir=/usr/lib/ocf/resource.d/loxthelion
 agent="$provider_dir/managed-container"
 expected_hash=5eae1f3dba0586dc06105e4dd7a9b8809a5c9961452abe7654dcfce77c3c5ce2
 fail() { printf 'STOP: %s\n' "$*" >&2; exit 1; }
 [[ "$EUID" == 0 ]] || fail 'Run this installer with sudo in your terminal.'
 [[ "$(hostname -s)" == DAVID-ELITEDESK ]] || fail 'This installer is only for DAVID-ELITEDESK.'
 [[ "$(crm_attribute --type crm_config --name cluster-name --query --quiet)" == postgres-ra-test ]] || fail 'Unexpected cluster name.'
 [[ "$(crm_attribute --type crm_config --name maintenance-mode --query --quiet)" == true ]] || fail 'Pacemaker maintenance-mode must already be true.'
 [[ "$(sha256sum "$agent" | cut -d ' ' -f1)" == "$expected_hash" ]] || fail 'Installed agent differs from the confirmed reference.'
 [[ ! -e "$provider_dir/managed-container.d" ]] || fail 'A module directory already exists; review it before installing.'
 cd "$project_dir"
 [[ "$(sha256sum refactored/managed-container | cut -d ' ' -f1)" == "c91f09adfdeb45a3be1a72df3f5ca5b02fd4555dec4397ad944eab439ace0b89" ]] || fail 'This installer is for the original modular baseline only; the scoring enhancement requires separate review.'
 sha256sum -c notes/REFACTOR-SHA256SUMS
 for file in refactored/managed-container refactored/managed-container.d/*.sh; do
    bash -n "$file"
 done
 # Match the recorded pg-ha resource parameters and avoid inherited test overrides.
 run_ra() {
    env -i PATH="$PATH" OCF_ROOT=/usr/lib/ocf OCF_RESOURCE_INSTANCE=pg-ha \
        OCF_RESKEY_service_user=david OCF_RESKEY_service_name=postgres.service \
        OCF_RESKEY_container_name=postgres OCF_RESKEY_pg_os_user=postgres \
        OCF_RESKEY_db_user=david OCF_RESKEY_db_name=david \
        "$1" "$2"
 }
 run_ra "$agent" validate-all
 run_ra "$agent" monitor || fail 'The original agent must report a healthy unpromoted standby (rc=0).'
 mkdir -p /var/backups
 backup_dir="$(mktemp -d /var/backups/managed-container-ELITEDESK.XXXXXXXX)"
 cp -a "$agent" "$backup_dir/managed-container"
 printf 'Original agent backed up to: %s\n' "$backup_dir"
 stage_dir="$(mktemp -d "$provider_dir/.managed-container-install.XXXXXXXX")"
 install -d -o root -g root -m 0755 "$stage_dir/managed-container.d"
 install -o root -g root -m 0755 refactored/managed-container "$stage_dir/managed-container"
 install -o root -g root -m 0644 refactored/managed-container.d/*.sh "$stage_dir/managed-container.d/"
 run_ra "$agent" meta-data > "$backup_dir/original-metadata.xml"
 run_ra "$stage_dir/managed-container" meta-data > "$backup_dir/staged-metadata.xml"
 cmp "$backup_dir/original-metadata.xml" "$backup_dir/staged-metadata.xml"
 run_ra "$stage_dir/managed-container" validate-all
 run_ra "$stage_dir/managed-container" monitor || fail 'Staged agent must report a healthy standby.'
 published=false
 rollback_on_error() {
    local rc=$?
    trap - EXIT
    if [[ "$rc" != 0 && "$published" == true ]]; then
        printf 'Installation/check failed; restoring the original entrypoint.\n' >&2
        if install -o root -g root -m 0755 "$backup_dir/managed-container" "$stage_dir/rollback-agent" &&
           mv -f "$stage_dir/rollback-agent" "$agent" && restorecon "$agent"; then
            printf 'Original restored. New modules remain unused beside it for inspection.\n' >&2
        else
            printf 'Automatic restoration FAILED. Original backup: %s/managed-container\n' "$backup_dir" >&2
        fi
    fi
    exit "$rc"
 }
 trap rollback_on_error EXIT
 # Recheck immediately before changing the installed files.
 [[ "$(crm_attribute --type crm_config --name maintenance-mode --query --quiet)" == true ]] || fail 'Maintenance mode changed; refusing installation.'
 published=true
 mv "$stage_dir/managed-container.d" "$provider_dir/managed-container.d"
 mv -f "$stage_dir/managed-container" "$agent"
 restorecon -RF "$agent" "$provider_dir/managed-container.d"
 run_ra "$agent" meta-data > "$backup_dir/installed-metadata.xml"
 cmp "$backup_dir/original-metadata.xml" "$backup_dir/installed-metadata.xml"
 run_ra "$agent" validate-all
 run_ra "$agent" monitor || fail 'Installed agent did not report a healthy standby.'
 [[ "$(crm_attribute --type crm_config --name maintenance-mode --query --quiet)" == true ]] || fail 'Maintenance mode is no longer true.'
 for file in refactored/managed-container refactored/managed-container.d/*.sh; do
    cmp "$file" "$provider_dir/${file#refactored/}"
 done
 sha256sum "$agent" "$provider_dir/managed-container.d/"*.sh > "$backup_dir/installed-SHA256SUMS"
 cat "$backup_dir/installed-SHA256SUMS"
 published=false
 trap - EXIT
 rmdir "$stage_dir"
 printf '\nPASS: refactored RA installed; metadata identical, validate-all=0, monitor=0.\n'
 printf 'Maintenance mode remains true. No service restart or HA transition was requested.\n'
 printf 'Backup and validation evidence: %s\n' "$backup_dir"
@@ -0,0 +1,65 @@
 #!/usr/bin/env python3
 """Print (never execute) exact cleanup commands from a supplied CIB snapshot."""
 import argparse
 from pathlib import Path
 import shlex
 import sys
 import xml.etree.ElementTree as ET
 def plan(path, clone_id, preferred_node):
    root = ET.parse(path).getroot()
    clones = [c for c in root.findall('./configuration/resources/clone') if c.get('id') == clone_id]
    if len(clones) != 1:
        raise ValueError('Expected exactly one clone with the supplied ID')
    for meta in clones[0].findall('meta_attributes'):
        if meta.get('id-ref') or meta.find('rule') is not None:
            raise ValueError('Clone metadata uses references/rules; inspect manually')
    matches = []
    for location in root.findall('./configuration/constraints/rsc_location'):
        if location.get('rsc') != clone_id:
            continue
        for rule in location.findall('rule'):
            children = list(rule)
            if (rule.get('role') == 'Promoted' and rule.get('score') == '200'
                    and len(children) == 1 and children[0].tag == 'expression'
                    and children[0].get('attribute') == '#uname'
                    and children[0].get('operation') == 'eq'
                    and children[0].get('value') == preferred_node):
                # Preserve unrelated rules if the location has more than one.
                target = location.get('id') if len(list(location)) == 1 else rule.get('id')
                if not target:
                    raise ValueError('Matching constraint/rule lacks an ID')
                matches.append(target)
    if len(matches) != 1:
        raise ValueError('Expected exactly one simple Promoted/200/#uname preference; inspect the full CIB manually')
    target_roles = [p.get('value') for m in clones[0].findall('meta_attributes')
                    for p in m.findall('nvpair') if p.get('name') == 'target-role']
    if target_roles != ['Promoted']:
        raise ValueError('Expected one direct clone target-role=Promoted; inspect inheritance/rules manually')
    # target-role=Promoted is an intentional production invariant, not a
    # temporary node preference. Never remove or rewrite it here.
    return [shlex.join(['sudo', 'pcs', 'constraint', 'remove', matches[0]])]
 def main():
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument('cib', type=Path)
    parser.add_argument('--clone', required=True)
    parser.add_argument('--preferred-node', required=True)
    parser.add_argument('--lab-validated', action='store_true', help='operator acknowledgement that this candidate passed lab validation')
    args = parser.parse_args()
    if not args.lab_validated:
        parser.error('No rollout commands emitted: lab validation must be completed and acknowledged first')
    try:
        commands = plan(args.cib, args.clone, args.preferred_node)
    except (ValueError, ET.ParseError, OSError) as error:
        print('No commands emitted:', error, file=sys.stderr)
        return 1
    print('# Review against the current CIB before running. No commands have been executed.')
    print('\n'.join(commands))
    return 0
 if __name__ == '__main__':
    sys.exit(main())
@@ -0,0 +1,134 @@
 #!/usr/bin/env python3
 """One guarded RA stop/start cycle on the ELITEDESK lab standby only."""
 import importlib.util
 import json
 import os
 from pathlib import Path
 import pwd
 import socket
 import subprocess
 import sys
 import tempfile
 import time
 import hashlib
 spec = importlib.util.spec_from_file_location('lab', Path(__file__).with_name('test-elitedesk-plugin.py'))
 lab = importlib.util.module_from_spec(spec)
 spec.loader.exec_module(lab)
 def owner_command(*args):
    account = pwd.getpwnam('david')
    return lab.run(['runuser', '-u', 'david', '--', 'env', 'HOME=' + account.pw_dir,
                    'XDG_RUNTIME_DIR=/run/user/' + str(account.pw_uid),
                    'timeout', '--kill-after=1s', '5s', *args], cwd=account.pw_dir)
 def score():
    return lab.run(['crm_attribute', '--promotion=pg-ha', '--node', 'DAVID-ELITEDESK',
                    '--lifetime=reboot', '--query', '--quiet'])
 def verify_payload():
    entries = lab.hashes(lab.ROOT / 'docs/lab-installed-plugin.sha256')
    lab.require(len(entries) == 15, 'Expected the 15-file installed plugin snapshot')
    for name, digest in entries.items():
        path = Path(name)
        lab.require(lab.PROVIDER in path.parents and not path.is_symlink()
                    and hashlib.sha256(path.read_bytes()).hexdigest() == digest,
                    'Installed code changed; inspect before testing: ' + name)
 def main():
    lab.require(os.geteuid() == 0, 'Run with sudo in your terminal; do not send passwords in chat')
    lab.require(socket.gethostname() == 'DAVID-ELITEDESK', 'Only the ELITEDESK lab standby is supported')
    os.umask(0o077)
    attributes = lab.cluster()  # Exact lab membership/resource; maintenance=true.
    before = lab.observe()     # Running standby streaming from DV6 with its own slot.
    verify_payload()
    evidence = Path(tempfile.mkdtemp(prefix='managed-container-cycle-lab-', dir=lab.BACKUP_ROOT))
    print('Results:', evidence, flush=True)
    env = dict(PATH='/usr/sbin:/usr/bin:/sbin:/bin', OCF_ROOT='/usr/lib/ocf',
               OCF_RESOURCE_INSTANCE='pg-ha', OCF_RESKEY_CRM_meta_on_node='DAVID-ELITEDESK',
               **{'OCF_RESKEY_' + k: v for k, v in attributes.items()})
    results = dict(before=before, actions=[], status='running')
    def save():
        (evidence / 'results.json').write_text(json.dumps(results, indent=2) + '\n')
    def unchanged_cluster():
        lab.require(lab.cluster() == attributes, 'Lab configuration or maintenance state changed; halt')
    def action(name, expected=0):
        unchanged_cluster()
        # GNU timeout bounds the RA process group, not just its parent process.
        result = subprocess.run(['timeout', '--kill-after=5s', '60s', str(lab.AGENT), name],
                                env=env, cwd='/', text=True, capture_output=True)
        results['actions'].append(dict(action=name, rc=result.returncode, expected=expected))
        (evidence / f'{len(results["actions"]):02d}-{name}.log').write_text(result.stdout + result.stderr)
        save()
        lab.require(result.returncode == expected,
                    f'{name}: expected rc={expected}, got {result.returncode}; see {evidence}')
        print(f'PASS {name}: rc={result.returncode}', flush=True)
    save()
    try:
        action('validate-all')
        action('monitor')
        lab.require(lab.observe() == before, 'Standby changed before stop')
        action('stop')
        active = owner_command('systemctl', '--user', 'show', 'postgres.service', '--property=ActiveState', '--value')
        containers = owner_command('podman', 'ps', '--all', '--filter', 'name=^postgres$', '--format', '{{.State}}')
        results['stopped'] = dict(service=active, container=containers or 'absent')
        lab.require(active == 'inactive' and containers in ('', 'exited', 'stopped'),
                    'Service/container did not reach the expected stopped state')
        action('monitor', 7)
        results['stopped']['score'] = score()
        lab.require(results['stopped']['score'] == '-INFINITY', 'Stopped standby retained an eligible score')
        save()
        print('PASS stopped state: service inactive, container stopped/absent, score=-INFINITY', flush=True)
        verify_payload()
        action('start')  # Existing standby.signal guard remains in the RA.
        deadline = time.monotonic() + 60
        last_error = 'No streaming observation yet'
        while time.monotonic() < deadline:
            unchanged_cluster()
            try:
                after = lab.observe()
            except RuntimeError as error:
                last_error = str(error)
                time.sleep(2)
                continue
            lab.require(after['system_id'] == before['system_id'], 'PostgreSQL system identifier changed')
            lab.require(after['started'] != before['started'], 'Postmaster start time did not change')
            results['after'] = after
            action('monitor')
            results['score'] = score()
            save()
            if results['score'] == '100':
                break
            last_error = 'Streaming resumed but strict standby score has not reached 100'
            time.sleep(2)
        else:
            raise RuntimeError('Post-start verification deadline expired: ' + last_error)
        action('status')
        unchanged_cluster()
        verify_payload()
        results['status'] = 'passed'
        save()
        print('PASS cycle: same database, new postmaster, streaming from DV6, score=100.', flush=True)
        print('Maintenance remains true. No promotion, demotion, rejoin, fencing, or peer actions ran.', flush=True)
    except BaseException as error:
        results['status'] = 'failed'
        results['error'] = str(error)
        save()
        print('HALTED: no automatic retry, recovery, or data replacement. Inspect results:', evidence, flush=True)
        raise
 if __name__ == '__main__':
    try:
        main()
    except Exception as error:
        print('STOP:', error, file=sys.stderr)
        sys.exit(1)
@@ -0,0 +1,203 @@
 #!/usr/bin/env python3
 """Install the plugin candidate and run non-disruptive checks on the lab standby.
 Run with sudo from the operator's terminal. Never supplies a password, changes
 maintenance mode, or runs stop/promote/demote/rejoin. Logs are root-only.
 """
 import hashlib
 import json
 import os
 from pathlib import Path
 import pwd
 import shutil
 import socket
 import signal
 import subprocess
 import sys
 import tempfile
 import xml.etree.ElementTree as ET
 ROOT = Path(__file__).resolve().parents[1]
 PROVIDER = Path('/usr/lib/ocf/resource.d/loxthelion')
 AGENT = PROVIDER / 'managed-container'
 MODULES = PROVIDER / 'managed-container.d'
 BASELINE = ROOT / 'docs/lab-installed-before-plugin.sha256'
 BACKUP_ROOT = Path('/var/backups')
 def run(args, env=None, cwd=None):
    clean_env = dict(PATH='/usr/sbin:/usr/bin:/sbin:/bin', HOME='/root', LC_ALL='C')
    result = subprocess.run(args, env=env if env is not None else clean_env, cwd=cwd,
                            text=True, capture_output=True, timeout=90)
    if result.returncode:
        raise RuntimeError(f'{Path(args[0]).name} failed (rc={result.returncode}): {result.stderr.strip()}')
    return result.stdout.strip()
 def require(condition, message):
    if not condition:
        raise RuntimeError(message)
 def hashes(manifest):
    return dict((name, digest) for digest, name in
                (line.split('  ', 1) for line in manifest.read_text().splitlines() if line))
 def cluster():
    cib = ET.fromstring(run(['cibadmin', '--query']))
    require(run(['crm_attribute', '--type', 'crm_config', '--name', 'cluster-name', '--query', '--quiet'])
            == 'postgres-ra-test', 'This is not the postgres-ra-test lab')
    require(run(['crm_attribute', '--type', 'crm_config', '--name', 'maintenance-mode', '--query', '--quiet'])
            == 'true', 'Lab maintenance-mode must already be true; nothing changed')
    require({n.get('uname') for n in cib.findall('./configuration/nodes/node')} ==
            {'DAVID-ELITEDESK', 'DAVID-DV6'}, 'Unexpected cluster membership; inspect before proceeding')
    resources = cib.findall("./configuration/resources/clone[@id='pg-ha-clone']/primitive[@id='pg-ha']")
    require(len(resources) == 1, 'Expected pg-ha under pg-ha-clone')
    resource = resources[0]
    require((resource.get('class'), resource.get('provider'), resource.get('type')) ==
            ('ocf', 'loxthelion', 'managed-container'), 'Unexpected resource agent')
    attributes = {}
    for group in resource.findall('instance_attributes'):
        require(not group.get('id-ref') and group.find('rule') is None,
                'Referenced/rule-based parameters require manual review')
        for pair in group.findall('nvpair'):
            name, value = pair.get('name'), pair.get('value')
            require(name not in attributes and name and value is not None, 'Ambiguous resource parameters')
            attributes[name] = value
    expected = dict(service_user='david', service_name='postgres.service', container_name='postgres',
                    pg_os_user='postgres', db_user='david', db_name='david')
    require(all(attributes.get(k, v) == v for k, v in expected.items()), 'Lab service/database parameters changed')
    return dict(expected, **{k: v for k, v in attributes.items() if k not in expected})
 def observe():
    account = pwd.getpwnam('david')
    query = """SELECT json_build_object('recovery',pg_is_in_recovery(),
        'started',pg_postmaster_start_time()::text,
        'system_id',(SELECT system_identifier::text FROM pg_control_system()),
        'receiver',(SELECT status FROM pg_stat_wal_receiver),
        'sender',(SELECT sender_host FROM pg_stat_wal_receiver),
        'slot',(SELECT slot_name FROM pg_stat_wal_receiver));"""
    output = run(['runuser', '-u', 'david', '--', 'env', 'HOME=' + account.pw_dir,
                  'XDG_RUNTIME_DIR=/run/user/' + str(account.pw_uid),
                  'timeout', '--kill-after=1s', '5s', 'podman', 'exec', '--user', 'postgres',
                  '--env', 'PGOPTIONS=-c statement_timeout=2000', 'postgres',
                  'psql', '-X', '-v', 'ON_ERROR_STOP=1', '-U', 'david', '-d', 'david', '-Atqc', query],
                 cwd=account.pw_dir)
    state = json.loads(output)
    require(state['recovery'] is True and state['receiver'] == 'streaming'
            and state['sender'] == '192.168.1.2' and state['slot'] == 'ha_david_elitedesk',
            'Expected a streaming ELITEDESK standby of lab DV6; refusing installation/start')
    return state
 def main():
    require(os.geteuid() == 0, 'Run this script with sudo in your terminal; do not send passwords in chat')
    require(socket.gethostname() == 'DAVID-ELITEDESK', 'This script is only for DAVID-ELITEDESK')
    def interrupted(signum, frame):
        raise RuntimeError(f'Interrupted by signal {signum}')
    signal.signal(signal.SIGTERM, interrupted)
    signal.signal(signal.SIGINT, interrupted)
    os.umask(0o077)
    os.environ['PATH'] = '/usr/sbin:/usr/bin:/sbin:/bin'
    attributes = cluster()
    before = observe()
    for name, digest in hashes(BASELINE).items():
        path = PROVIDER / name
        require(not path.is_symlink() and hashlib.sha256(path.read_bytes()).hexdigest() == digest,
                f'Installed baseline changed: {name}; review before replacing')
    manifest = hashes(ROOT / 'notes/REFACTOR-SHA256SUMS')
    candidates = [ROOT / 'refactored/managed-container'] + sorted(
        p for p in (ROOT / 'refactored/managed-container.d').rglob('*') if p.suffix in ('.sh', '.py') and p.is_file())
    require(len(candidates) == 15, 'Unexpected candidate file layout')
    for path in candidates:
        require(not path.is_symlink() and hashlib.sha256(path.read_bytes()).hexdigest()
                == manifest.get(str(path.relative_to(ROOT))), f'Candidate checksum mismatch: {path.name}')
    # Copy verified code into a root-owned staging directory before executing it.
    backup = Path(tempfile.mkdtemp(prefix='managed-container-plugin-lab-', dir=BACKUP_ROOT))
    print('Backup and results:', backup, flush=True)
    shutil.copy2(AGENT, backup / 'managed-container')
    shutil.copytree(MODULES, backup / 'managed-container.d', symlinks=True)
    stage = Path(tempfile.mkdtemp(prefix='.mc-plugin-stage-', dir=PROVIDER))
    for source in candidates:
        destination = stage / source.relative_to(ROOT / 'refactored')
        destination.parent.mkdir(parents=True, exist_ok=True)
        shutil.copyfile(source, destination)
        destination.chmod(0o755 if destination.name == 'managed-container' else 0o644)
        require(hashlib.sha256(destination.read_bytes()).hexdigest() ==
                manifest[str(source.relative_to(ROOT))], 'Source changed while staging')
    for directory in (stage / 'managed-container.d').rglob('*'):
        if directory.is_dir(): directory.chmod(0o755)
    (stage / 'managed-container.d').chmod(0o755)
    env = dict(PATH=os.environ['PATH'], OCF_ROOT='/usr/lib/ocf', OCF_RESOURCE_INSTANCE='pg-ha',
               OCF_RESKEY_CRM_meta_on_node='DAVID-ELITEDESK',
               **{'OCF_RESKEY_' + k: v for k, v in attributes.items()})
    results = dict(before=before, actions=[])
    def action(executable, name):
        output = run([str(executable), name], env=env, cwd='/')
        results['actions'].append(dict(agent=str(executable), action=name, rc=0))
        print(f'PASS {executable.parent.name}/{name}: rc=0', flush=True)
        return output
    staged = stage / 'managed-container'
    metadata = action(staged, 'meta-data')
    require(len(ET.fromstring(metadata).findall('./parameters/parameter')) == 16, 'Unexpected plugin metadata')
    (backup / 'candidate-metadata.xml').write_text(metadata + '\n')
    action(staged, 'validate-all')
    require(cluster() == attributes, 'Cluster configuration changed before installation')
    require(observe() == before, 'PostgreSQL state changed before installation')
    retired = stage / 'previous-modules'
    modules_moved = False
    try:
        os.replace(MODULES, retired)
        modules_moved = True
        os.replace(stage / 'managed-container.d', MODULES)
        os.replace(staged, AGENT)
        run(['restorecon', '-RF', str(AGENT), str(MODULES)])
        action(AGENT, 'validate-all')
        action(AGENT, 'monitor')
        action(AGENT, 'status')
        require(cluster() == attributes and observe() == before, 'Standby/cluster changed before idempotent start')
        action(AGENT, 'start')  # Already healthy standby: exercises the no-op path.
        action(AGENT, 'monitor')
        require(cluster() == attributes, 'Cluster configuration changed during checks')
        results['after'] = observe()
        require(results['after'] == before, 'Database identity/start time or replication changed during checks')
        results['score'] = run(['crm_attribute', '--promotion=pg-ha', '--node', 'DAVID-ELITEDESK',
                                '--lifetime=reboot', '--query', '--quiet'])
        require(results['score'] in ('100', '-INFINITY'), 'Unexpected standby score')
        # Under write load a connected standby can legitimately fail strict catch-up.
        for source in candidates:
            installed = PROVIDER / source.relative_to(ROOT / 'refactored')
            require(hashlib.sha256(installed.read_bytes()).hexdigest() ==
                    manifest[str(source.relative_to(ROOT))], 'Installed payload mismatch')
        (backup / 'results.json').write_text(json.dumps(results, indent=2) + '\n')
        (backup / 'installed-SHA256SUMS').write_text(''.join(
            manifest[str(p.relative_to(ROOT))] + '  ' + str(PROVIDER / p.relative_to(ROOT / 'refactored')) + '\n'
            for p in candidates))
    except BaseException:
        if modules_moved:
            print('Check failed: restoring previous agent and modules; maintenance mode is not changed.', flush=True)
            if MODULES.exists(): os.replace(MODULES, stage / 'failed-modules')
            os.replace(retired, MODULES)
            shutil.copyfile(backup / 'managed-container', stage / 'restore-agent')
            (stage / 'restore-agent').chmod(0o755)
            os.replace(stage / 'restore-agent', AGENT)
            run(['restorecon', '-RF', str(AGENT), str(MODULES)])
        (backup / 'results.json').write_text(json.dumps(results, indent=2) + '\n')
        print('Staging/evidence retained:', stage, flush=True)
        raise
    shutil.rmtree(stage)
    print('PASS: installed; PostgreSQL was not restarted; maintenance remains true.', flush=True)
    print('Current score:', results['score'], '(strict standby catch-up can fluctuate)', flush=True)
    print('No stop, promote, demote, rejoin, fencing, or peer-node actions were run.', flush=True)
 if __name__ == '__main__':
    try:
        main()
    except Exception as error:
        print('STOP:', error, file=sys.stderr)
        sys.exit(1)
@@ -0,0 +1,250 @@
 #!/usr/bin/env python3
 """Operator-run, one-stage-at-a-time role tests for the two-node lab.
 No installation, automatic recovery, fencing, or maintenance-mode changes.
 See docs/lab-role-tests.md before running a mutating stage.
 """
 import argparse
 import hashlib
 import json
 import os
 from pathlib import Path
 import pwd
 import shlex
 import socket
 import subprocess
 import tempfile
 import time
 import xml.etree.ElementTree as ET
 PROVIDER = Path('/usr/lib/ocf/resource.d/loxthelion')
 AGENT = PROVIDER / 'managed-container'
 NODES = {'DAVID-ELITEDESK': '192.168.1.6', 'DAVID-DV6': '192.168.1.2'}
 PINNED = {'DAVID-DV6': {'managed-container': 'a4b775a0ddcabdb0558da9b34c7fe2a883c07d0ef308c1c7224526abf53f63a0',
               'managed-container.d/auth.sh': '93191771d3cb8aefadb2a16d2f89d64a8b6c584071a473439be818237bdf7f17',
               'managed-container.d/common.sh': '97e93d9164e854e5f3688cf9d3cc656b001eb30ef3c2073c907450ca43588f3d',
               'managed-container.d/container.sh': 'ac6bc363a2301f1491d162ef609103dbfbceb3b099a3556614c6165952309374',
               'managed-container.d/demote.sh': '0af3bb362a303017372c631354ba94ef9e723bf14e64d2eff91b6662e54a5be0',
               'managed-container.d/notify.sh': '6a98fcdda82cea9365db95b4ee779be5c567b3d846f48b90fbc8c254af697670',
               'managed-container.d/postgres.sh': 'd2779ce6e7d1f426ffca2e41bbb8de77503005dc38d658d69f184020365796eb',
               'managed-container.d/promote.sh': '3085ec9b4410905ab9d4370786ed776eac3470a0415aa1b4f608b083945aeb58',
               'managed-container.d/promotion_state.py': '5166fc3370f49ab14cd975ded4b03a74b9bd7dbff5762b41a15552c42f934ea7',
               'managed-container.d/rejoin.sh': '21beea72bcb7f24a51c17ea48666ac87df9436702ad77b166b45f4647e28e8ad',
               'managed-container.d/replication.sh': '6b2021887e3b4cd6a12490e7a8dff98320b2133b472ab25a22a0eead46137547'},
 'DAVID-ELITEDESK': {'managed-container': '01eb9b161c8e3db289f09bf63953be779921e15bdf8b1a4d8ce7e60032b826e4',
                     'managed-container.d/applications/postgresql/auth.sh': '10932919c4339b553c7c9dbf30b45a917c51c0a08f362e12678003d58b12fa82',
                     'managed-container.d/applications/postgresql/demote.sh': 'da4e58cadf1df3953cb97a0512e3931ee1b9d3ee61c39a1ad1a6047ef537addc',
                     'managed-container.d/applications/postgresql/notify.sh': '6a98fcdda82cea9365db95b4ee779be5c567b3d846f48b90fbc8c254af697670',
                     'managed-container.d/applications/postgresql/plugin.sh': 'd3a989de97ceb61a56828f006d5ccb87b7c34fc155a944585bedf928785021fc',
                     'managed-container.d/applications/postgresql/postgres.sh': 'fa222e6767249999767eb82fac2d85e7e35404da240517a8b630571e77282053',
                     'managed-container.d/applications/postgresql/promote.sh': '3085ec9b4410905ab9d4370786ed776eac3470a0415aa1b4f608b083945aeb58',
                     'managed-container.d/applications/postgresql/promotion_state.py': '5166fc3370f49ab14cd975ded4b03a74b9bd7dbff5762b41a15552c42f934ea7',
                     'managed-container.d/applications/postgresql/rejoin.sh': '28b0e44554a88ccb87cd16a68315c045dd37d354aceb2fa25da7c7399a659b23',
                     'managed-container.d/applications/postgresql/replication.sh': '6b2021887e3b4cd6a12490e7a8dff98320b2133b472ab25a22a0eead46137547',
                     'managed-container.d/common.sh': 'a482ef3cc1048e6a4f427e76dd2dce173ffd00760dbed1afa601a3fd05b0ad99',
                     'managed-container.d/container.sh': '3a27ac9c63c61310da8b8711fa2c4400b37ea8eae2ee2a79e28bc63180b06c7c',
                     'managed-container.d/engine.sh': '53978e2367394695d61c6fb38f2ee46814474c5a4e0448d2692f1a3ce583af93',
                     'managed-container.d/lifecycles/systemd.sh': '6c92c095ce10ac95d843fdc1c49e31c19491ca22fdac960affd8649606d76d82',
                     'managed-container.d/runtimes/podman.sh': 'dc5820067a78c4a1c531426902793aae97a03fd64912b017b3b102031dcc1e04'}}
 QUERY = """SELECT json_build_object(
 'recovery',pg_is_in_recovery(), 'paused',pg_is_wal_replay_paused(),
 'system_id',(SELECT system_identifier::text FROM pg_control_system()),
 'started',pg_postmaster_start_time()::text,
 'receiver',(SELECT status FROM pg_stat_wal_receiver),
 'sender',(SELECT sender_host FROM pg_stat_wal_receiver),
 'slot',(SELECT slot_name FROM pg_stat_wal_receiver),
 'replay',pg_last_wal_replay_lsn()::text);"""
 # pg_is_wal_replay_paused() is only callable during recovery.
 QUERY = QUERY.replace('pg_is_wal_replay_paused()',
                      'CASE WHEN pg_is_in_recovery() THEN pg_is_wal_replay_paused() ELSE false END')
 def require(ok, message):
    if not ok:
        raise RuntimeError(message)
 def run(args, **kwargs):
    result = subprocess.run(args, text=True, capture_output=True, timeout=90,
                            env=dict(PATH='/usr/sbin:/usr/bin:/sbin:/bin', LC_ALL='C'), **kwargs)
    require(result.returncode == 0, f'{args[0]} failed: {result.stderr.strip()}')
    return result.stdout.strip()
 def owner_args(*args):
    account = pwd.getpwnam('david')
    return ['runuser', '-u', 'david', '--', 'env', 'HOME=' + account.pw_dir,
            'XDG_RUNTIME_DIR=/run/user/' + str(account.pw_uid), *args]
 def cluster():
    cib = ET.fromstring(run(['cibadmin', '--query']))
    def prop(name):
        return run(['crm_attribute', '--type', 'crm_config', '--name', name, '--query', '--quiet'])
    require(prop('cluster-name') == 'postgres-ra-test' and prop('maintenance-mode') == 'true',
            'Expected postgres-ra-test with maintenance=true; nothing changed')
    require({n.get('uname') for n in cib.findall('./configuration/nodes/node')} == set(NODES),
            'Unexpected cluster membership')
    resource = cib.find("./configuration/resources/clone[@id='pg-ha-clone']/primitive[@id='pg-ha']")
    require(resource is not None and (resource.get('class'), resource.get('provider'), resource.get('type'))
            == ('ocf', 'loxthelion', 'managed-container'), 'Unexpected resource')
    attrs = {}
    for group in resource.findall('instance_attributes'):
        require(not group.get('id-ref') and group.find('rule') is None, 'Unsupported parameter rules')
        for pair in group.findall('nvpair'):
            key, value = pair.get('name'), pair.get('value')
            require(key and value is not None and key not in attrs, 'Ambiguous parameters')
            attrs[key] = value
    defaults = dict(service_user='david', service_name='postgres.service', container_name='postgres',
                    pg_os_user='postgres', db_user='david', db_name='david',
                    application='postgresql', container_runtime='podman', lifecycle_manager='systemd')
    require(all(attrs.get(k, v) == v for k, v in defaults.items()), 'Unexpected service/application settings')
    return dict(defaults, **attrs)
 def snapshot(node):
    # Executes only read operations as david on either host. Never inspects container environment.
    code = """import hashlib,json,pathlib,socket,subprocess
 def cmd(args):
 p=subprocess.run(['timeout','--kill-after=1s','5s']+args,text=True,capture_output=True)
 if p.returncode: raise RuntimeError('Read-only observation failed: '+args[0])
 return p.stdout.strip()
 files=FILES
 base=pathlib.Path('/usr/lib/ocf/resource.d/loxthelion')
 assert all(not (base/n).is_symlink() for n in files)
 print(json.dumps(dict(node=socket.gethostname(),
 hashes={n:hashlib.sha256((base/n).read_bytes()).hexdigest() for n in files},
 service=cmd(['systemctl','--user','show','postgres.service','--property=ActiveState','--value']),
 container=cmd(['podman','inspect','--format','{{.Id}}','postgres']),
 db=json.loads(cmd(['podman','exec','--user','postgres','--env','PGOPTIONS=-c statement_timeout=2000',
 'postgres','psql','-X','-v','ON_ERROR_STOP=1','-U','david','-d','david','-Atqc',QUERY])))))
 """.replace('FILES', repr(list(PINNED[node]))).replace('QUERY', repr(QUERY))
    if node == socket.gethostname():
        args = owner_args('python3', '-c', code)
    else:
        args = owner_args('ssh', '-F', '/dev/null', '-o', 'BatchMode=yes', '-o', 'ConnectTimeout=5',
                          '-o', 'StrictHostKeyChecking=yes', 'david@' + NODES[node],
                          'python3 -c ' + shlex.quote(code))
    value = json.loads(run(args, cwd=pwd.getpwnam('david').pw_dir))
    require(value['node'] == node and value['hashes'] == PINNED[node], 'Node or installed code differs: ' + node)
    require(value['service'] == 'active', 'Maintenance service not active: ' + node)
    return value
 def standby(db, source):
    return (db['recovery'] is True and db['paused'] is False and db['receiver'] == 'streaming'
            and db['sender'] == NODES[source])
 def guard(stage, node, local, peer):
    require(local['system_id'] == peer['system_id'], 'Database system identifiers differ')
    other = next(n for n in NODES if n != node)
    if stage == 'demote':
        require(local['recovery'] is False and standby(peer, node), 'Need local primary and streaming peer standby')
    elif stage == 'promote':
        require(local['recovery'] is True and local['paused'] is False and standby(peer, node),
                'Both nodes must be in recovery and the demoted peer must stream from this node')
    elif stage == 'rejoin':
        require(node == 'DAVID-ELITEDESK' and standby(local, other) and peer['recovery'] is False,
                'Rejoin test requires ELITEDESK streaming from primary DV6')
 def main(stage):
    node = socket.gethostname()
    require(os.geteuid() == 0 and node in NODES, 'Run with sudo on an expected lab node')
    os.umask(0o077)
    peer = next(n for n in NODES if n != node)
    attrs = cluster()
    before, remote = snapshot(node), snapshot(peer)
    guard(stage, node, before['db'], remote['db'])
    if stage == 'inspect':
        print(json.dumps(dict(local=before, peer=remote, maintenance=True), indent=2))
        return
    evidence = Path(tempfile.mkdtemp(prefix='managed-container-roles-lab-', dir='/var/backups'))
    print('Results:', evidence, flush=True)
    result = dict(stage=stage, node=node, before=before, peer_before=remote, actions=[], status='running')
    env = dict(PATH='/usr/sbin:/usr/bin:/sbin:/bin', OCF_ROOT='/usr/lib/ocf', OCF_RESOURCE_INSTANCE='pg-ha',
               OCF_RESKEY_CRM_meta_on_node=node, **{'OCF_RESKEY_' + k: v for k, v in attrs.items()})
    def save():
        (evidence / 'results.json').write_text(json.dumps(result, indent=2) + '\n')
    def action(name, expected=0, extra=None):
        require(cluster() == attrs, 'Cluster configuration or maintenance changed')
        response = subprocess.run(['timeout', '--kill-after=5s', '120s', str(AGENT), name],
                                  env=dict(env, **(extra or {})), cwd='/', text=True, capture_output=True)
        result['actions'].append(dict(action=name, rc=response.returncode, expected=expected))
        (evidence / f'{len(result["actions"]):02d}-{name}.log').write_text(response.stdout + response.stderr)
        save()
        require(response.returncode == expected, f'{name} returned {response.returncode}; expected {expected}')
        print(f'PASS {name}: rc={response.returncode}', flush=True)
    save()
    try:
        action('validate-all')
        action('monitor', 8 if before['db']['recovery'] is False else 0)
        # Reobserve immediately before mutation. No action retry or inferred fencing certificate.
        current, remote = snapshot(node), snapshot(peer)
        require(current['container'] == before['container'] and
                all(current['db'][k] == before['db'][k] for k in ('system_id', 'started', 'recovery')),
                'Local incarnation/role changed during preflight')
        guard(stage, node, current['db'], remote['db'])
        if stage == 'demote':
            action('notify', extra=dict(OCF_RESKEY_CRM_meta_notify_type='pre',
                                       OCF_RESKEY_CRM_meta_notify_operation='demote',
                                       OCF_RESKEY_CRM_meta_notify_demote_uname=node,
                                       OCF_RESKEY_CRM_meta_notify_promote_uname=peer))
            action('demote')
        elif stage == 'promote':
            action('promote')
        elif stage == 'rejoin':
            # This stage tests rewind/recovery of a healthy standby, not divergent former-primary repair.
            run(owner_args('timeout', '--kill-after=5s', '60s', 'podman', 'exec', '--user', 'postgres',
                           'postgres', 'sh', '-c',
                           'exec "$(pg_config --bindir)/pg_ctl" -D "$PGDATA" stop -m fast -w'))
            action('monitor', 7)
            require(run(['crm_attribute', '--promotion=pg-ha', '--node', node,
                         '--lifetime=reboot', '--query', '--quiet']) == '-INFINITY',
                    'Stopped PostgreSQL retained an eligible score')
            require(snapshot(peer)['db']['recovery'] is False, 'DV6 is no longer primary')
            action('rejoin', extra=dict(OCF_RESKEY_rejoin_source_host=NODES[peer],
                                       OCF_RESKEY_replication_slot='ha_david_elitedesk',
                                       OCF_RESKEY_application_name=node,
                                       OCF_RESKEY_rejoin_allow_basebackup='false'))
        deadline = time.monotonic() + 60
        while True:
            after, remote = snapshot(node), snapshot(peer)
            require(after['db']['system_id'] == before['db']['system_id'], 'Local database identity changed')
            require(after['container'] == before['container'], 'Maintenance container was replaced')
            require(remote['db']['system_id'] == before['db']['system_id'], 'Peer database identity changed')
            if stage == 'promote':
                require(after['db']['recovery'] is False and standby(remote['db'], node), 'Unexpected roles after promotion')
                expected_score = '1000'
            else:
                require(standby(after['db'], peer), 'Local node did not return to streaming recovery')
                require(after['db']['started'] != before['db']['started'], 'Expected a new postmaster')
                require(remote['db']['recovery'] is (stage == 'demote'), 'Unexpected peer role')
                expected_score = '100'
            action('monitor', 8 if stage == 'promote' else 0)
            value = run(['crm_attribute', '--promotion=pg-ha', '--node', node, '--lifetime=reboot', '--query', '--quiet'])
            if value == expected_score:
                break
            require(time.monotonic() < deadline, 'Expected promotion score did not converge')
            time.sleep(2)
        result.update(after=after, peer_after=remote, score=value, status='passed')
        require(cluster() == attrs, 'Cluster configuration changed')
        save()
        print(f'PASS {stage}: verified roles, streaming, same database/container, score={value}.', flush=True)
        print('Maintenance remains true. No fencing or automatic recovery ran.', flush=True)
    except BaseException as error:
        result.update(status='failed', error=str(error))
        save()
        print('HALTED. Do not continue to the next stage. Inspect:', evidence, flush=True)
        raise
 if __name__ == '__main__':
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument('stage', choices=['inspect', 'demote', 'promote', 'rejoin'])
    try:
        main(parser.parse_args().stage)
    except Exception as error:
        raise SystemExit('STOP: ' + str(error))
@@ -0,0 +1,33 @@
 """Shared test loader and narrowly reversible command-boundary substitutions."""
 from pathlib import Path
 import re
 import shlex
 ROOT = Path(__file__).resolve().parents[1]
 MODULE_DIR = ROOT / 'refactored/managed-container.d'
 LOAD = '''MC_PROMOTION_RESOURCE_INSTANCE="${OCF_RESOURCE_INSTANCE:-}"
 : "${OCF_ROOT:=/usr/lib/ocf}"
 . "$OCF_ROOT/lib/heartbeat/ocf-shellfuncs"
 : "${OCF_RUNNING_PROMOTED:=${OCF_RUNNING_MASTER:-8}}"
 MC_MODULE_DIR=''' + shlex.quote(str(MODULE_DIR)) + '''
 . "$MC_MODULE_DIR/common.sh"
 . "$MC_MODULE_DIR/container.sh"
 . "$MC_MODULE_DIR/engine.sh"
 mc_engine_load || exit $?
 '''
 def restore_boundaries(source):
    """Project only adapter calls back to their frozen original spelling.
    This is not whitespace normalization or a general semantic equivalence
    claim. All remaining function bytes must match the immutable reference.
    """
    source = source.replace('mc_runtime_inspect', 'as_service_user podman inspect')
    source = source.replace('mc_runtime_info', 'as_service_user podman info')
    pattern = r'^([ \t]*)(.*?)mc_runtime_exec \\\n'
    source = re.sub(pattern, lambda m: m[1] + m[2] + 'as_service_user \\\n' + m[1] + '    podman exec \\\n', source, flags=re.M)
    pattern = r'^([ \t]*)(.*?)mc_lifecycle_(start|stop|cat|running) '
    source = re.sub(pattern, lambda m: m[1] + m[2] + 'as_service_user \\\n' + m[1] + '    systemctl --user ' +
                    ('is-active --quiet' if m[3] == 'running' else m[3]) + ' ', source, flags=re.M)
    return source
@@ -0,0 +1,199 @@
 #!/usr/bin/env python3
 """State and monitor integration tests: temporary files and fake boundaries only."""
 import copy
 import importlib.util
 import json
 import os
 from pathlib import Path
 import shlex
 import tempfile
 import unittest
 from test_promotion_scores import Scores, LOAD, FIXTURE
 ROOT = Path(__file__).resolve().parents[1]
 HELPER = ROOT / 'refactored/managed-container.d/applications/postgresql/promotion_state.py'
 spec = importlib.util.spec_from_file_location('promotion_state', HELPER)
 policy = importlib.util.module_from_spec(spec)
 spec.loader.exec_module(policy)
 CONNECTED = dict(identity=['7684451577990111248', '/data/pg18', '1789200000.123456'],
                 recovery=True, paused=False, receiver='streaming', replay='0/30',
                 flush='0/20', strict=True)
 DISCONNECTED = dict(CONNECTED, receiver=None, flush=None, strict=None)
 class State(unittest.TestCase):
    def setUp(self):
        self.tmp = tempfile.TemporaryDirectory(prefix='mc-evidence-')
        self.addCleanup(self.tmp.cleanup)
        self.evidence = policy.Evidence(Path(self.tmp.name) / 'promotion',
                                       ['pg-ha:0', 'owner', 'postgres.service', 'postgres'],
                                       'test-boot', 1000.0, 1789201000.0, os.getuid())
    def check(self, obs=DISCONNECTED, age=300):
        return self.evidence.evaluate(copy.deepcopy(obs), age)
    def seed(self):
        self.assertEqual(self.check(CONNECTED), '100')
        return json.loads(self.evidence.path.read_text())
    def test_streaming_disconnected_eligible_and_persisted_lsn(self):
        record = self.seed()
        self.assertEqual(record['replay'], '0/30')
        self.assertEqual(record['flush'], '0/20')
        self.evidence.now += 50
        self.assertEqual(self.check(), '50')
        self.assertEqual(json.loads(self.evidence.path.read_text()), record)
        self.assertEqual(self.evidence.path.stat().st_mode & 0o777, 0o600)
        self.assertEqual(self.evidence.directory.stat().st_mode & 0o777, 0o700)
    def test_never_verified(self):
        self.assertEqual(self.check(), '-INFINITY')
    def test_stale_and_exact_boundary(self):
        self.seed(); self.evidence.now += 300
        self.assertEqual(self.check(), '50')
        self.evidence.now += .001
        self.assertEqual(self.check(), '-INFINITY')
        self.assertFalse(self.evidence.path.exists())
    def test_disabled_and_invalid_age(self):
        self.seed()
        self.assertEqual(self.check(age=0), '-INFINITY')
        for age in (-1, True, 100000, '300'):
            with self.assertRaises(ValueError): self.check(age=age)
    def test_malformed_state(self):
        for value in ('{', '[]', '{}', 'null', '{"version":true}', '{"version":1,"version":1}', 'x' * 4097):
            self.seed(); self.evidence.path.write_text(value)
            self.assertEqual(self.check(), '-INFINITY', value[:40])
            self.assertFalse(self.evidence.path.exists())
    def test_invalid_record_fields(self):
        for key, value in [('replay', '0/ZZ'), ('flush', '1/0'), ('boot', 'other'),
                           ('observed_boottime', float('nan')), ('observed_boottime', 1001),
                           ('observed_unix', True), ('context', ['other']), ('version', True)]:
            record = self.seed(); record[key] = value
            self.evidence.path.write_text(json.dumps(record))
            self.assertEqual(self.check(), '-INFINITY', key)
    def test_system_data_path_and_postmaster_identity_changes(self):
        for index, value in enumerate(['999', '/new/pgdata', '1789200100.123456']):
            self.seed(); obs = copy.deepcopy(DISCONNECTED); obs['identity'][index] = value
            self.assertEqual(self.check(obs), '-INFINITY')
            self.assertFalse(self.evidence.path.exists())
    def test_reinitialization_requires_reverification_then_replaces_state(self):
        self.seed()
        disconnected = copy.deepcopy(DISCONNECTED)
        disconnected['identity'][2] = '1789200100.123456'  # Same system ID/basebackup.
        self.assertEqual(self.check(disconnected), '-INFINITY')
        connected = dict(CONNECTED, identity=disconnected['identity'])
        self.assertEqual(self.check(connected), '100')
        self.assertEqual(self.check(disconnected), '50')
        self.assertEqual(json.loads(self.evidence.path.read_text())['identity'], connected['identity'])
    def test_paused_primary_and_replay_regression_invalidate(self):
        for changes in ({'paused': True}, {'recovery': False, 'paused': True},
                        {'replay': '0/10'}, {'replay': None}):
            self.seed()
            self.assertEqual(self.check(dict(DISCONNECTED, **changes)), '-INFINITY')
            self.assertFalse(self.evidence.path.exists())
    def test_connected_failure_cannot_use_disconnected_fallback(self):
        for changes in ({'flush': '0/40'}, {'flush': None}, {'paused': True}):
            self.seed()
            self.assertEqual(self.check(dict(CONNECTED, **changes, strict=False)), '-INFINITY')
            self.assertFalse(self.evidence.path.exists())
    def test_reconnect_refreshes_timestamp_and_lsns(self):
        self.seed(); self.evidence.now += 100
        self.assertEqual(self.check(), '50')
        self.assertEqual(self.check(dict(CONNECTED, replay='0/50', flush='0/40')), '100')
        record = json.loads(self.evidence.path.read_text())
        self.assertEqual(record['observed_boottime'], 1100)
        self.assertEqual(record['replay'], '0/50')
        self.assertEqual(self.check(dict(DISCONNECTED, replay='0/50')), '50')
    def test_wall_clock_steps_do_not_extend_or_expire_observation(self):
        self.seed(); self.evidence.wall -= 100000; self.evidence.now += 100
        self.assertEqual(self.check(), '50')
        self.evidence.wall += 200000; self.evidence.now += 201
        self.assertEqual(self.check(), '-INFINITY')
    def test_symlink_state_rejected_without_reading_target(self):
        self.seed(); self.evidence.path.unlink()
        target = Path(self.tmp.name) / 'target'; target.write_text('unchanged')
        self.evidence.path.symlink_to(target)
        self.assertEqual(self.check(), '-INFINITY')
        self.assertEqual(target.read_text(), 'unchanged')
    def test_unsafe_directory_rejected(self):
        self.seed(); self.evidence.directory.chmod(0o777)
        with self.assertRaises(ValueError): self.check()
 # Reuse fake crm_attribute and the real monitor wrapper. The real state evaluator
 # runs as this test user against a private directory; never writes /run or CIB.
 class Monitor(unittest.TestCase):
    setUp = Scores.setUp
    run_shell = Scores.run_shell
    state = Scores.state
    score = Scores.score
    def test_invalid_observation_query_and_helper_failure_replace_score(self):
        fixture = monitor_fixture(self)
        self.env['OBSERVATION'] = json.dumps(CONNECTED)
        for failure in ('mc_pg_standby_observation() { echo malformed; }',
                        'mc_pg_standby_observation() { return 1; }',
                        'python3() { return 124; }'):
            self.run_shell('mc_promotion_monitor', fixture=fixture)
            self.score('100')
            self.run_shell(failure + '; mc_promotion_monitor', fixture=fixture)
            self.score('-INFINITY')
 def monitor_fixture(test):
    driver = test.path / 'driver.py'
    driver.write_text('''import importlib.util,json,os,sys,time
 from pathlib import Path
 spec=importlib.util.spec_from_file_location('policy',sys.argv[1])
 p=importlib.util.module_from_spec(spec);spec.loader.exec_module(p)
 e=p.Evidence(Path(sys.argv[2]),['pg-ha:0','owner','service','container'],'boot',float(os.environ.get('TEST_NOW','1000')),1789201000,os.getuid())
 print(e.evaluate(json.load(sys.stdin),300))
 ''')
    # Restore the production score helper after the original tests' mock.
    return FIXTURE + '\n. ' + shlex.quote(str(HELPER.with_name('postgres.sh'))) + '''
 service_running() { [ "$SERVICE" = 1 ]; }
 container_running() { [ "$CONTAINER" = 1 ]; }
 pg_running() { [ "$RUNNING" = 1 ]; }
 pg_ready() { return 0; }
 pg_role() { echo "$ROLE"; }
 mc_pg_standby_observation() { printf '%s' "$OBSERVATION"; }
 python3() { command python3 ''' + ' '.join(map(shlex.quote, [str(driver), str(HELPER), str(test.path / 'evidence')])) + '''; }
 timeout() { shift 2; "$@"; }
 '''
 def test_monitor_disconnect_reprobe_stopped_reconnect(self):
    fixture = monitor_fixture(self)
    self.env['OBSERVATION'] = json.dumps(CONNECTED)
    self.run_shell('mc_promotion_monitor', fixture=fixture); self.score('100')
    self.env['OBSERVATION'] = json.dumps(DISCONNECTED)
    self.run_shell('mc_promotion_monitor', fixture=fixture); self.score('50')
    data = self.state(); data['scores'] = {}; self.db.write_text(json.dumps(data))
    self.run_shell('mc_promotion_monitor', fixture=fixture); self.score('50')
    self.run_shell('RUNNING=0; mc_promotion_monitor', 7, fixture=fixture); self.score('-INFINITY')
    self.env['OBSERVATION'] = json.dumps(CONNECTED)
    self.run_shell('mc_promotion_monitor', fixture=fixture); self.score('100')
    self.env['OBSERVATION'] = json.dumps(DISCONNECTED); self.env['TEST_NOW'] = '1301'
    self.run_shell('mc_promotion_monitor', fixture=fixture); self.score('-INFINITY')
 Monitor.test_monitor_disconnect_reprobe_stopped_reconnect = test_monitor_disconnect_reprobe_stopped_reconnect
 if __name__ == '__main__':
    # Explicit suite excludes imported Scores from duplicate discovery.
    unittest.TextTestRunner(verbosity=2).run(unittest.TestSuite(
        unittest.defaultTestLoader.loadTestsFromTestCase(c) for c in (State, Monitor)
    )).wasSuccessful() or __import__('sys').exit(1)
@@ -0,0 +1,65 @@
 #!/usr/bin/env python3
 """Stop/start harness tests with mocked commands and temporary evidence only."""
 import contextlib
 import importlib.util
 import io
 import os
 from pathlib import Path
 import tempfile
 from types import SimpleNamespace
 import unittest
 from unittest.mock import patch
 ROOT = Path(__file__).resolve().parents[1]
 spec = importlib.util.spec_from_file_location('cycle', ROOT / 'scripts/test-elitedesk-cycle.py')
 cycle = importlib.util.module_from_spec(spec)
 spec.loader.exec_module(cycle)
 class Cycle(unittest.TestCase):
    def scenario(self, failure=None):
        with tempfile.TemporaryDirectory() as directory, contextlib.ExitStack() as stack:
            state = {'running': True, 'started': 'old'}
            calls = []
            def command(args, **kwargs):
                action = args[-1]; calls.append(action)
                self.assertIn(action, ['validate-all', 'monitor', 'stop', 'start', 'status'])
                if action == failure:
                    return SimpleNamespace(returncode=1, stdout='', stderr='injected failure')
                if action == 'stop': state['running'] = False
                if action == 'start': state.update(running=True, started='new')
                rc = 7 if action == 'monitor' and not state['running'] else 0
                return SimpleNamespace(returncode=rc, stdout='', stderr='')
            stack.enter_context(patch.object(cycle.os, 'geteuid', return_value=0))
            stack.enter_context(patch.object(cycle.socket, 'gethostname', return_value='DAVID-ELITEDESK'))
            stack.enter_context(patch.object(cycle.lab, 'BACKUP_ROOT', Path(directory)))
            stack.enter_context(patch.object(cycle.lab, 'cluster', return_value={}))
            stack.enter_context(patch.object(cycle.lab, 'observe', side_effect=lambda:
                                            dict(system_id='same', started=state['started'])))
            stack.enter_context(patch.object(cycle, 'verify_payload'))
            stack.enter_context(patch.object(cycle, 'owner_command', side_effect=lambda *args:
                                            'inactive' if args[0] == 'systemctl' else ''))
            stack.enter_context(patch.object(cycle, 'score', side_effect=lambda:
                                            '100' if state['running'] else '-INFINITY'))
            stack.enter_context(patch.object(cycle.subprocess, 'run', side_effect=command))
            old_mask = os.umask(0o022)
            try:
                with contextlib.redirect_stdout(io.StringIO()):
                    if failure:
                        with self.assertRaises(RuntimeError): cycle.main()
                        self.assertEqual(calls[-1], failure)
                        self.assertEqual(calls.count(failure), 1)
                    else:
                        cycle.main()
                        self.assertEqual(calls, ['validate-all','monitor','stop','monitor','start','monitor','status'])
            finally:
                os.umask(old_mask)
            self.assertEqual(len(list(Path(directory).glob('*/results.json'))), 1)
    def test_successful_cycle(self): self.scenario()
    def test_stop_failure_halts_without_start(self): self.scenario('stop')
    def test_start_failure_halts_without_retry(self): self.scenario('start')
 if __name__ == '__main__':
    unittest.main(verbosity=2)
@@ -0,0 +1,78 @@
 #!/usr/bin/env python3
 """Exercise publication and rollback using temp files and mocked lab boundaries."""
 import contextlib
 import hashlib
 import importlib.util
 import io
 import os
 from pathlib import Path
 import tempfile
 import unittest
 from unittest.mock import patch
 ROOT = Path(__file__).resolve().parents[1]
 spec = importlib.util.spec_from_file_location('installer', ROOT / 'scripts/test-elitedesk-plugin.py')
 installer = importlib.util.module_from_spec(spec)
 spec.loader.exec_module(installer)
 class Installer(unittest.TestCase):
    def scenario(self, fail_monitor=False):
        with tempfile.TemporaryDirectory() as directory, contextlib.ExitStack() as stack:
            directory = Path(directory)
            provider = directory / 'provider'; provider.mkdir()
            agent = provider / 'managed-container'; agent.write_text('previous agent')
            modules = provider / 'managed-container.d'; modules.mkdir()
            old_module = modules / 'old.sh'; old_module.write_text('previous module')
            baseline = directory / 'baseline'
            baseline.write_text(''.join(hashlib.sha256(p.read_bytes()).hexdigest() + '  ' + str(p.relative_to(provider)) + '\n'
                                        for p in [agent, old_module]))
            backup = directory / 'backups'; backup.mkdir()
            for name, value in dict(PROVIDER=provider, AGENT=agent, MODULES=modules,
                                    BASELINE=baseline, BACKUP_ROOT=backup).items():
                stack.enter_context(patch.object(installer, name, value))
            stack.enter_context(patch.object(installer.os, 'geteuid', return_value=0))
            stack.enter_context(patch.object(installer.socket, 'gethostname', return_value='DAVID-ELITEDESK'))
            stack.enter_context(patch.object(installer.signal, 'signal'))
            stack.enter_context(patch.object(installer, 'cluster', return_value={}))
            stack.enter_context(patch.object(installer, 'observe', return_value={'unchanged': True}))
            stack.enter_context(patch.dict(os.environ))
            calls = []
            def fake_run(args, **kwargs):
                calls.append(args)
                if args[0] == 'restorecon': return ''
                if args[0] == 'crm_attribute': return '100'
                require_action = args[1]
                self.assertIn(require_action, ['meta-data', 'validate-all', 'monitor', 'status', 'start'])
                if require_action == 'meta-data':
                    return '<resource-agent><parameters>' + '<parameter/>' * 16 + '</parameters></resource-agent>'
                if fail_monitor and Path(args[0]) == agent and require_action == 'monitor':
                    raise RuntimeError('injected installed monitor failure')
                return ''
            stack.enter_context(patch.object(installer, 'run', side_effect=fake_run))
            old_mask = os.umask(0o022)
            try:
                with contextlib.redirect_stdout(io.StringIO()):
                    if fail_monitor:
                        with self.assertRaisesRegex(RuntimeError, 'injected'):
                            installer.main()
                        self.assertEqual(agent.read_text(), 'previous agent')
                        self.assertEqual(old_module.read_text(), 'previous module')
                    else:
                        installer.main()
                        self.assertEqual(agent.read_bytes(), (ROOT / 'refactored/managed-container').read_bytes())
                        self.assertTrue((modules / 'applications/postgresql/plugin.sh').is_file())
                        self.assertFalse(old_module.exists())
            finally:
                os.umask(old_mask)
            self.assertEqual(len(list(backup.glob('*/results.json'))), 1)
    def test_success_only_runs_basic_actions(self):
        self.scenario()
    def test_failed_installed_monitor_restores_agent_and_modules(self):
        self.scenario(fail_monitor=True)
 if __name__ == '__main__':
    unittest.main(verbosity=2)
@@ -0,0 +1,55 @@
 """Exercise the safety gates without running cluster or PostgreSQL commands."""
 import importlib.util
 from pathlib import Path
 import unittest
 spec = importlib.util.spec_from_file_location('roles', Path(__file__).resolve().parents[1] / 'scripts/test-lab-roles.py')
 roles = importlib.util.module_from_spec(spec)
 spec.loader.exec_module(roles)
 def db(recovery, source=None, **changes):
    return dict(dict(system_id='same', recovery=recovery, paused=False,
                     receiver='streaming' if source else None,
                     sender=roles.NODES.get(source)), **changes)
 class Gates(unittest.TestCase):
    def test_demote_requires_streaming_successor(self):
        roles.guard('demote', 'DAVID-DV6', db(False), db(True, 'DAVID-DV6'))
        for peer in [db(False), db(True), db(True, 'DAVID-DV6', paused=True)]:
            with self.assertRaises(RuntimeError):
                roles.guard('demote', 'DAVID-DV6', db(False), peer)
    def test_promote_refuses_live_peer_primary(self):
        with self.assertRaises(RuntimeError):
            roles.guard('promote', 'DAVID-ELITEDESK', db(True, 'DAVID-DV6'), db(False))
    def test_promote_requires_demoted_peer_streaming_back(self):
        roles.guard('promote', 'DAVID-ELITEDESK', db(True), db(True, 'DAVID-ELITEDESK'))
        with self.assertRaises(RuntimeError):
            roles.guard('promote', 'DAVID-ELITEDESK', db(True), db(True))
    def test_identity_mismatch_refused(self):
        with self.assertRaises(RuntimeError):
            roles.guard('demote', 'DAVID-DV6', db(False), db(True, 'DAVID-DV6', system_id='other'))
    def test_rejoin_only_healthy_elitedesk_standby(self):
        roles.guard('rejoin', 'DAVID-ELITEDESK', db(True, 'DAVID-DV6'), db(False))
        for node, local, peer in [('DAVID-DV6', db(True, 'DAVID-ELITEDESK'), db(False)),
                                  ('DAVID-ELITEDESK', db(False), db(True)),
                                  ('DAVID-ELITEDESK', db(True, 'DAVID-DV6'), db(True))]:
            with self.assertRaises(RuntimeError):
                roles.guard('rejoin', node, local, peer)
    def test_pinned_versions_match_lab_snapshots(self):
        root = Path(__file__).resolve().parents[1]
        for node, name in [('DAVID-DV6', 'lab-installed-before-plugin.sha256'),
                           ('DAVID-ELITEDESK', 'lab-installed-plugin.sha256')]:
            expected = {p.removeprefix(str(roles.PROVIDER) + '/'): h for h, p in
                        (line.split('  ', 1) for line in (root / 'docs' / name).read_text().splitlines())}
            self.assertEqual(roles.PINNED[node], expected)
 if __name__ == '__main__':
    unittest.main()
@@ -0,0 +1,121 @@
 #!/usr/bin/env python3
 """Plugin contract tests: no real runtime, systemd, or cluster operations."""
 import json
 import os
 from pathlib import Path
 import shlex
 import shutil
 import subprocess
 import tempfile
 import unittest
 import xml.etree.ElementTree as ET
 from plugin_test_support import LOAD, ROOT
 AGENT = ROOT / 'refactored/managed-container'
 class Engine(unittest.TestCase):
    def run_shell(self, code, rc=0, env=None):
        result = subprocess.run(['bash', '-c', LOAD + '\n' + code], text=True,
                                capture_output=True, timeout=10, env=env)
        self.assertEqual(result.returncode, rc, result.stderr)
        return result
    def trace(self, command, arguments, expected, rc=0):
        args = ' '.join(shlex.quote(x) for x in arguments)
        fixture = '''
 as_service_user() {
    python3 -c 'import json,sys; print(json.dumps(sys.argv[1:]))' "$@"
    return ''' + str(rc) + '''
 }
 '''
        result = self.run_shell(fixture + command + ' ' + args, rc)
        self.assertEqual(json.loads(result.stdout), expected)
    def test_exec_preserves_argument_boundaries(self):
        args = ['--user', 'postgres', '--env', 'VALUE=a b\n$(touch /UNEXPECTED)',
                'container', 'sh', '-c', 'printf "%s" "$1"', 'sh', "quotes'\" $HOME ;"]
        self.trace('mc_runtime_exec', args, ['podman', 'exec', *args])
    def test_exec_error_is_not_systemd_success_exit_status(self):
        self.trace('mc_runtime_exec', ['container', 'false'], ['podman', 'exec', 'container', 'false'], 143)
    def test_bounded_exec_preserves_timeout_location_and_status(self):
        self.trace('mc_runtime_exec_bounded', ['3s', '--user', 'postgres', 'container', 'psql'],
                   ['timeout', '--kill-after=1s', '3s', 'podman', 'exec', '--user', 'postgres', 'container', 'psql'], 124)
    def test_inspect_and_info(self):
        self.trace('mc_runtime_inspect', ['--format', '{{.State.Running}}', 'container'],
                   ['podman', 'inspect', '--format', '{{.State.Running}}', 'container'])
        self.trace('mc_runtime_info', [], ['podman', 'info'], 125)
    def test_lifecycle_uses_systemd_only(self):
        for action, args in [('start', ['start']), ('stop', ['stop']),
                             ('running', ['is-active', '--quiet']), ('cat', ['cat'])]:
            self.trace('mc_lifecycle_' + action, ['postgres.service'],
                       ['systemctl', '--user', *args, 'postgres.service'])
        self.trace('mc_lifecycle_stop', ['postgres.service'],
                   ['systemctl', '--user', 'stop', 'postgres.service'], 1)
    def test_application_dispatch_and_return_codes(self):
        for action in ['start', 'stop', 'promote', 'demote', 'rejoin']:
            result = self.run_shell('managed_' + action + '() { echo ' + action + '; return 6; }; '
                                    'mc_application_action ' + action, 6)
            self.assertEqual(result.stdout.strip(), action)
        self.run_shell('mc_application_action invalid', 3)
        self.run_shell('managed_monitor() { return 8; }; mc_application_monitor', 8)
        self.run_shell('managed_notify() { return 1; }; mc_application_notify', 1)
    def test_unsupported_or_path_like_selection_rejected_before_commands(self):
        for key, value in [('application', 'mysql'), ('application', '../postgresql'),
                           ('container_runtime', 'docker'), ('lifecycle_manager', 'runtime'),
                           ('application', '$(touch /UNEXPECTED)')]:
            env = dict(os.environ, **{'OCF_RESKEY_' + key: value})
            result = subprocess.run([str(AGENT), 'monitor'], env=env, text=True, capture_output=True)
            self.assertEqual(result.returncode, 6)
            self.assertIn('Supported plugin combination', result.stderr)
    def test_missing_and_incompatible_plugins(self):
        for kind in ['missing', 'version', 'hook']:
            with tempfile.TemporaryDirectory(prefix='mc-plugin-loader-') as directory:
                target = Path(directory) / 'refactored'
                shutil.copytree(AGENT.parent, target, ignore=shutil.ignore_patterns('__pycache__'))
                plugin = target / 'managed-container.d/runtimes/podman.sh'
                if kind == 'missing':
                    plugin.unlink()
                elif kind == 'version':
                    plugin.write_text(plugin.read_text().replace('MC_RUNTIME_API=1', 'MC_RUNTIME_API=2'))
                else:
                    plugin.write_text(plugin.read_text().replace('mc_runtime_exec()', 'missing_hook()'))
                result = subprocess.run([str(target / 'managed-container'), 'meta-data'],
                                        text=True, capture_output=True, cwd='/tmp')
                self.assertEqual(result.returncode, 6, result.stderr)
                self.assertEqual(result.stdout, '')
    def test_metadata_without_instance_and_with_explicit_defaults(self):
        env = {k: v for k, v in os.environ.items() if not k.startswith('OCF_')}
        baseline = subprocess.check_output([str(AGENT), 'meta-data'], env=env, cwd='/tmp', text=True)
        tree = ET.fromstring(baseline)
        params = {p.get('name'): p.find('content').get('default') for p in tree.findall('./parameters/parameter')}
        self.assertEqual(len(params), 16)
        for key, value in [('application', 'postgresql'), ('container_runtime', 'podman'), ('lifecycle_manager', 'systemd')]:
            self.assertEqual(params[key], value)
            env['OCF_RESKEY_' + key] = value
        self.assertEqual(subprocess.check_output([str(AGENT), 'meta-data'], env=env, text=True), baseline)
    def test_metadata_failure_is_not_hidden(self):
        self.run_shell('mc_application_metadata() { return 1; }; mc_engine_metadata', 1)
    def test_application_cannot_call_runtime_or_systemd_directly(self):
        # All execution crosses an explicit adapter; command dependency names
        # remain part of PostgreSQL's compatibility validation in this revision.
        for name in ['postgres', 'auth', 'replication', 'notify', 'promote', 'demote', 'rejoin']:
            text = (AGENT.parent / 'managed-container.d/applications/postgresql' / (name + '.sh')).read_text()
            self.assertNotIn('as_service_user', text, name)
            self.assertNotIn('systemctl --user', text, name)
            self.assertNotIn('podman exec', text, name)
 if __name__ == '__main__':
    unittest.main(verbosity=2)
@@ -0,0 +1,40 @@
 #!/usr/bin/env python3
 import importlib.util
 from pathlib import Path
 import tempfile
 import unittest
 ROOT = Path(__file__).resolve().parents[1]
 spec = importlib.util.spec_from_file_location('cleanup', ROOT / 'scripts/plan-promotion-cleanup.py')
 cleanup = importlib.util.module_from_spec(spec); spec.loader.exec_module(cleanup)
 class Cleanup(unittest.TestCase):
    def plan(self, extra='', role='Promoted'):
        with tempfile.TemporaryDirectory() as directory:
            p = Path(directory) / 'cib.xml'
            p.write_text(f'''<cib><configuration><resources><clone id="postgresql-ha-clone">
 <meta_attributes id="meta"><nvpair name="target-role" value="{role}"/></meta_attributes>
 </clone></resources><constraints><rsc_location id="actual-eve-preference" rsc="postgresql-ha-clone">
 <rule id="actual-rule" role="Promoted" score="200"><expression attribute="#uname" operation="eq" value="david-eve"/></rule>
 {extra}</rsc_location></constraints></configuration></cib>''')
            return cleanup.plan(p, 'postgresql-ha-clone', 'david-eve')
    def test_exact_commands_from_real_ids(self):
        self.assertEqual(self.plan(), ['sudo pcs constraint remove actual-eve-preference'])
    def test_preserve_unrelated_rule(self):
        self.assertEqual(self.plan('<rule id="unrelated" role="Unpromoted" score="10"/>')[0],
                         'sudo pcs constraint remove actual-rule')
    def test_ambiguous_match_refused(self):
        with self.assertRaises(ValueError):
            self.plan('<rule id="duplicate" role="Promoted" score="200"><expression attribute="#uname" operation="eq" value="david-eve"/></rule>')
    def test_unexpected_target_role_refused(self):
        with self.assertRaises(ValueError):
            self.plan(role='Started')
 if __name__ == '__main__':
    unittest.main(verbosity=2)
@@ -0,0 +1,114 @@
 #!/usr/bin/env python3
 """Real Pacemaker scheduler tests using synthetic CIB files only."""
 from pathlib import Path
 import shutil
 import subprocess
 import tempfile
 import unittest
 import xml.etree.ElementTree as ET
@unittest.skipUnless(shutil.which('crm_simulate') and shutil.which('cibadmin'), 'Pacemaker CLI not installed')
 class Scheduler(unittest.TestCase):
    def simulate(self, scores, target=None, fail_primary=False):
        if fail_primary and target is None:
            target = 'Promoted'  # Required production policy.
        with tempfile.TemporaryDirectory(prefix='mc-scheduler-') as directory:
            root = ET.fromstring(subprocess.check_output(['cibadmin', '--empty'], text=True))
            root.set('have-quorum', '1'); root.set('dc-uuid', '1')
            config = root.find('configuration')
            props = ET.SubElement(config.find('crm_config'), 'cluster_property_set', id='test-props')
            # ONLY this synthetic file: no fencing device is required for an offline simulation.
            ET.SubElement(props, 'nvpair', id='test-stonith', name='stonith-enabled', value='true' if fail_primary else 'false')
            for i, score in enumerate(scores, 1):
                node = 'node-' + str(i)
                ET.SubElement(config.find('nodes'), 'node', id=str(i), uname=node)
                state = ET.SubElement(root.find('status'), 'node_state', id=str(i), uname=node,
                                      in_ccm='true', crmd='online', join='member', expected='member')
                transient = ET.SubElement(state, 'transient_attributes', id=str(i))
                attrs = ET.SubElement(transient, 'instance_attributes', id='attrs-' + str(i))
                # Names generated by crm_attribute --promotion=pg-ha, not old pgsql state.
                ET.SubElement(attrs, 'nvpair', id='score-' + str(i), name='master-pg-ha', value=score)
                ET.SubElement(attrs, 'nvpair', id='legacy-' + str(i), name='master-postgresql', value='-INFINITY')
            if fail_primary:
                ET.SubElement(config.find('resources'), 'primitive',
                              {'id': 'test-fence', 'class': 'stonith', 'type': 'fence_dummy'})
            clone = ET.SubElement(config.find('resources'), 'clone', id='pg-ha-clone')
            meta = ET.SubElement(clone, 'meta_attributes', id='clone-meta')
            values = {'promotable': 'true', 'clone-max': str(len(scores)), 'clone-node-max': '1',
                      'promoted-max': '1', 'promoted-node-max': '1', 'notify': 'true'}
            if target is not None:
                values['target-role'] = target
            for name, value in values.items():
                ET.SubElement(meta, 'nvpair', id='clone-' + name, name=name, value=value)
            ET.SubElement(clone, 'primitive', {'id': 'pg-ha', 'class': 'ocf', 'provider': 'loxthelion', 'type': 'managed-container'})
            cib = Path(directory) / 'cib.xml'; graph = Path(directory) / 'graph.xml'
            ET.ElementTree(root).write(cib)
            if fail_primary:
                # Establish running primary/standbys in a synthetic simulation;
                # then inject an unexpected failure. No agents/devices execute.
                subprocess.run(['crm_simulate', '--xml-file', str(cib), '--in-place'],
                               check=True, capture_output=True, text=True)
            subprocess.run(['crm_simulate', '--xml-file', str(cib), '--run', '--save-graph', str(graph)]
                           + (['--node-fail', 'node-1'] if fail_primary else []),
                           check=True, capture_output=True, text=True)
            parsed = ET.parse(graph)
            if fail_primary:
                # Verify fencing is a transitive prerequisite of every promote.
                actions = {op.get('id'): op for op in parsed.findall('./synapse/action_set/*')}
                dependencies = {}
                for synapse in parsed.findall('./synapse'):
                    for op in synapse.findall('./action_set/*'):
                        dependencies[op.get('id')] = {x.get('id') for x in synapse.findall('./inputs/trigger/*')}
                fences = {key for key, op in actions.items() if op.get('operation') == 'stonith'}
                self.assertTrue(fences, ET.tostring(parsed.getroot()).decode())
                def ancestors(key, seen=None):
                    seen = set() if seen is None else seen
                    for parent in dependencies.get(key, set()) - seen:
                        seen.add(parent); ancestors(parent, seen)
                    return seen
                for key, op in actions.items():
                    if op.get('operation') == 'promote':
                        self.assertTrue(ancestors(key) & fences, 'promotion must depend on fencing')
            return [op.get('on_node') for op in ET.parse(graph).findall('./synapse/action_set/rsc_op')
                    if op.get('operation') == 'promote']
    def test_default_started_allows_one_promotion(self):
        self.assertEqual(len(self.simulate(['100', '100'])), 1)
    def test_explicit_started_allows_one_promotion(self):
        self.assertEqual(len(self.simulate(['100', '100'], 'Started')), 1)
    def test_explicit_promoted_allows_one_promotion(self):
        self.assertEqual(len(self.simulate(['100', '100'], 'Promoted')), 1)
    def test_unpromoted_target_prevents_promotion(self):
        self.assertEqual(self.simulate(['100', '100'], 'Unpromoted'), [])
    def test_higher_score_wins_without_host_preference(self):
        self.assertEqual(self.simulate(['100', '1000']), ['node-2'])
    def test_ineligible_node_not_selected(self):
        self.assertEqual(self.simulate(['-INFINITY', '100']), ['node-2'])
    def test_all_ineligible_no_promotion(self):
        self.assertEqual(self.simulate(['-INFINITY', '-INFINITY']), [])
    def test_failed_primary_one_disconnected_candidate(self):
        self.assertEqual(self.simulate(['1000', '50'], fail_primary=True), ['node-2'])
    def test_failed_primary_two_disconnected_candidates(self):
        first = self.simulate(['1000', '50', '50'], fail_primary=True)
        self.assertEqual(len(first), 1)
        self.assertIn(first[0], ['node-2', 'node-3'])
        self.assertEqual(first, self.simulate(['1000', '50', '50'], fail_primary=True))
    def test_failed_primary_all_disconnected_stale(self):
        self.assertEqual(self.simulate(['1000', '-INFINITY', '-INFINITY'], fail_primary=True), [])
    def test_connected_evidence_preferred(self):
        self.assertEqual(self.simulate(['50', '100']), ['node-2'])
 if __name__ == '__main__':
    unittest.main(verbosity=2)
@@ -0,0 +1,290 @@
 #!/usr/bin/env python3
 """Offline scoring/action tests. All cluster and PostgreSQL boundaries are fake."""
 import json
 import os
 from pathlib import Path
 import shlex
 import shutil
 import xml.etree.ElementTree as ET
 import subprocess
 import tempfile
 import unittest
 ROOT = Path(__file__).resolve().parents[1]
 AGENT = ROOT / 'refactored/managed-container'
 from plugin_test_support import LOAD
 CRM = '''#!/usr/bin/env python3
 import json, os, sys
 from pathlib import Path
 args=sys.argv[1:]
 p=Path(os.environ['SCORE_DB'])
 d=json.loads(p.read_text())
 assert '--lifetime=reboot' in args, args
 resource=next(x.split('=',1)[1] for x in args if x.startswith('--promotion='))
 assert not any(x.startswith('--name') for x in args), args
 node=os.environ.get('OCF_RESKEY_CRM_meta_on_node','node-a')
 key=resource+'|'+node
 d['calls'].append(args)
 if '--query' in args:
    p.write_text(json.dumps(d))
    if os.environ.get('QUERY_FAIL')=='1': sys.exit(1)
    if key not in d['scores']: sys.exit(6)
    print(d['scores'][key]); sys.exit(0)
 value=next(x.split('=',1)[1] for x in args if x.startswith('--update='))
 if os.environ.get('UPDATE_FAIL')=='1':
    p.write_text(json.dumps(d)); sys.exit(1)
 d['scores'][key]=value
 d['writes'].append([key,value])
 p.write_text(json.dumps(d))
 '''
 # Original action implementations execute against these fake external boundaries.
 FIXTURE = r'''
 ROLE=t
 RUNNING=1
 SERVICE=1
 CONTAINER=1
 ELIGIBLE=1
 SIGNAL=1
 ocf_log() { :; }
 ocf_exit_reason() { printf '%s\n' "$*" >&2; }
 service_running() { [ "$SERVICE" = 1 ]; }
 container_running() { [ "$CONTAINER" = 1 ]; }
 pg_running() { [ "$RUNNING" = 1 ]; }
 pg_ready() { return 0; }
 pg_role() { echo "$ROLE"; }
 pg_wal_receiver_status() { echo streaming; }
 mc_pg_promotion_eligible() { [ "$ELIGIBLE" = 1 ]; }
 mc_pg_standby_score() { mc_pg_promotion_eligible && echo 100; }
 validate_all() { return 0; }
 ensure_local_ha_slots() { return 0; }
 ensure_rewind_privileges() { return 0; }
 demote_target() { echo node-b; }
 node_to_addr() { echo 192.0.2.2; }
 candidate_is_standby() { return 0; }
 candidate_caught_up() { return 0; }
 ensure_remote_slot() { return 0; }
 pgdata_path() { echo /test/pgdata; }
 pg_remote_query() { echo f; }
 pg_replication_exec() { return 0; }
 pg_command() {
    case "$*" in
        *'test -f'*) [ "$SIGNAL" = 1 ] ;;
        *'promote -w'*) ROLE=f ;;
        *'stop -m fast -w'*) RUNNING=0 ;;
        *'start -w'*) RUNNING=1 ;;
        *) echo "Unexpected pg_command: $*" >&2; return 99 ;;
    esac
 }
 as_service_user() {
    case "$*" in
        'systemctl --user stop '*) SERVICE=0; CONTAINER=0 ;;
        'systemctl --user start '*) SERVICE=1; CONTAINER=1 ;;
        *RA_PRIMARY_HOST=*) ROLE=t ;;
        *) echo "Unexpected external access: $*" >&2; return 99 ;;
    esac
 }
 sleep() { :; }
 '''
 class Scores(unittest.TestCase):
    def setUp(self):
        self.tmp = tempfile.TemporaryDirectory(prefix='mc-score-test-')
        self.addCleanup(self.tmp.cleanup)
        self.path = Path(self.tmp.name)
        self.db = self.path / 'scores.json'
        self.db.write_text(json.dumps({'scores': {}, 'calls': [], 'writes': []}))
        (self.path / 'crm_attribute').write_text(CRM)
        (self.path / 'crm_attribute').chmod(0o755)
        self.env = dict(os.environ, PATH=str(self.path) + ':' + os.environ['PATH'],
                        SCORE_DB=str(self.db), OCF_RESOURCE_INSTANCE='pg-ha:0',
                        OCF_RESKEY_CRM_meta_on_node='node-a')
    def run_shell(self, code, rc=0, fixture=FIXTURE, env=None):
        result = subprocess.run(['bash', '-c', LOAD + fixture + '\n' + code],
                                env=env or self.env, text=True, capture_output=True, timeout=15)
        self.assertEqual(result.returncode, rc, result.stderr)
        return result
    def state(self):
        return json.loads(self.db.read_text())
    def score(self, value, node='node-a'):
        self.assertEqual(self.state()['scores']['pg-ha:0|' + node], value)
    def test_primary_and_standby(self):
        self.run_shell('ROLE=f; mc_promotion_monitor', 8)
        self.score('1000')
        self.env['OCF_RESKEY_CRM_meta_on_node'] = 'node-b'
        self.run_shell('mc_promotion_monitor')
        self.score('100', 'node-b')
    def test_two_standbys_equal(self):
        for node in ('node-a', 'node-b'):
            self.env['OCF_RESKEY_CRM_meta_on_node'] = node
            self.run_shell('mc_promotion_monitor')
            self.score('100', node)
    def test_ineligible_replaces_stale_positive(self):
        cases = [('RUNNING=0', 7), ('CONTAINER=0', 7), ('SERVICE=0', 7),
                 ('ELIGIBLE=0', 0), ('pg_ready() { return 1; }', 1),
                 ('pg_role() { return 1; }', 1), ('ROLE=invalid', 1)]
        for setup, rc in cases:
            with self.subTest(setup=setup):
                self.run_shell('mc_promotion_score_set 1000')
                self.run_shell(setup + '; mc_promotion_monitor', rc)
                self.score('-INFINITY')
    def test_repeat_monitors_write_once(self):
        for _ in range(3):
            self.run_shell('mc_promotion_monitor')
        self.assertEqual(len(self.state()['writes']), 1)
        self.assertEqual(len(self.state()['calls']), 4)  # Three queries, one update.
    def test_restart_reprobe_recreates_attribute(self):
        self.run_shell('mc_promotion_monitor')
        data = self.state(); data['scores'] = {}; self.db.write_text(json.dumps(data))
        self.run_shell('mc_promotion_monitor')
        self.score('100')
        self.assertEqual(len(self.state()['writes']), 2)
    def test_stop_start_original_actions(self):
        self.run_shell('mc_promotion_monitor; mc_promotion_action stop; '
                       '[ "$RUNNING$SERVICE$CONTAINER" = 000 ]; '
                       'mc_promotion_action start')
        self.score('100')
        self.assertEqual([x[1] for x in self.state()['writes']], ['100', '-INFINITY', '100'])
    def test_primary_start_refusal(self):
        self.run_shell('ROLE=f; mc_promotion_action start', 1)
        self.score('-INFINITY')
    def test_missing_standby_signal(self):
        self.run_shell('RUNNING=0; SIGNAL=0; mc_promotion_action start', 1)
        self.score('-INFINITY')
    def test_promote_original_action(self):
        self.run_shell('mc_promotion_monitor; mc_promotion_action promote')
        self.score('1000')
        self.assertEqual([x[1] for x in self.state()['writes']], ['100', '-INFINITY', '1000'])
    def test_demote_original_action(self):
        self.run_shell('ROLE=f; mc_promotion_monitor; mc_promotion_action demote; '
                       '[ "$ROLE$SERVICE$CONTAINER" = t11 ]')
        self.score('100')
        self.assertEqual([x[1] for x in self.state()['writes']], ['1000', '-INFINITY', '100'])
    def test_failed_demote_stays_negative_until_next_observation(self):
        self.run_shell('ROLE=f; demote_target() { return 1; }; mc_promotion_action demote', 1)
        self.score('-INFINITY')
        self.run_shell('ROLE=f; mc_promotion_monitor', 8)
        self.score('1000')
    def test_failed_promote_invariants_withdraw_score(self):
        self.run_shell('ensure_local_ha_slots() { return 1; }; mc_promotion_action promote', 1)
        self.score('-INFINITY')
    def test_validation_failure_withdraws_score(self):
        self.run_shell('mc_promotion_monitor; validate_all() { return 6; }; mc_promotion_action start', 6)
        self.score('-INFINITY')
    def test_stop_does_not_require_validation(self):
        self.run_shell('validate_all() { return 6; }; mc_promotion_action stop')
        self.score('-INFINITY')
    def test_rejoin_original_action(self):
        self.run_shell('RUNNING=0; REJOIN_SOURCE_HOST=192.0.2.2; REPLICATION_SLOT=ha_node; '
                       'mc_promotion_action rejoin')
        self.score('100')
    def test_update_failure_preserves_primary_role_rc(self):
        self.env['UPDATE_FAIL'] = '1'
        result = self.run_shell('ocf_log() { echo "$*" >&2; }; ROLE=f; mc_promotion_monitor', 8)
        self.assertIn('prior score may remain', result.stderr)
        self.assertFalse(self.state()['scores'])
    def test_update_failure_does_not_block_stop(self):
        self.env['UPDATE_FAIL'] = '1'
        self.run_shell('mc_promotion_action stop; [ "$RUNNING" = 0 ]')
    def test_query_failure_attempts_replace(self):
        self.env['QUERY_FAIL'] = '1'
        self.run_shell('ELIGIBLE=0; mc_promotion_monitor')
        self.score('-INFINITY')
    def test_scoring_dependency_validation(self):
        self.run_shell('command() { return 1; }; mc_promotion_validate', 5)
    def test_missing_instance_never_uses_default_attribute(self):
        env = dict(self.env)
        env.pop('OCF_RESOURCE_INSTANCE')
        self.run_shell('mc_promotion_monitor', env=env)
        self.assertFalse(self.state()['calls'])
    def test_target_role_does_not_affect_scoring(self):
        for value in ('', 'Started', 'Promoted'):
            self.env['OCF_RESKEY_CRM_meta_target_role'] = value
            self.run_shell('mc_promotion_monitor')
            self.score('100')
    def test_unrelated_legacy_attributes_untouched(self):
        d = self.state()
        d['scores']['master-postgresql'] = '-INFINITY'
        d['scores']['postgresql-data-status'] = 'DISCONNECT'
        d['scores']['postgresql-status'] = 'STOP'
        self.db.write_text(json.dumps(d))
        self.run_shell('mc_promotion_monitor')
        for key, value in d['scores'].items():
            self.assertEqual(self.state()['scores'][key], value)
    @unittest.skipUnless(shutil.which('crm_attribute') and shutil.which('cibadmin'), 'Pacemaker CLI not installed')
    def test_real_crm_attribute_against_offline_cib(self):
        # CIB_file forces Pacemaker's file backend: no live daemon is contacted.
        cib = self.path / 'offline.xml'
        tree = ET.fromstring(subprocess.check_output(['cibadmin', '--empty'], text=True))
        ET.SubElement(tree.find('./configuration/nodes'), 'node', id='1', uname='node-a')
        ET.SubElement(tree.find('./status'), 'node_state', id='1', uname='node-a')
        cib.write_text(ET.tostring(tree, encoding='unicode'))
        env = dict(self.env, PATH=os.environ['PATH'], CIB_file=str(cib))
        self.run_shell('mc_promotion_score_set 100', env=env)
        before = cib.read_bytes()
        self.run_shell('mc_promotion_score_set 100', env=env)
        self.assertEqual(cib.read_bytes(), before, 'Identical observations must not rewrite CIB')
        self.run_shell('mc_promotion_score_set -INFINITY', env=env)
        pairs = ET.parse(cib).findall('./status/node_state/transient_attributes/instance_attributes/nvpair')
        self.assertEqual([(p.get('name'), p.get('value')) for p in pairs], [('master-pg-ha:0', '-INFINITY')])
        # Reprobe after loss of transient state must recreate the correct score.
        tree = ET.parse(cib)
        state = tree.find('./status/node_state')
        state.remove(state.find('./transient_attributes'))
        tree.write(cib)
        self.run_shell('mc_promotion_score_set 1000', env=env)
        self.assertEqual(ET.parse(cib).find('.//transient_attributes//nvpair').get('value'), '1000')
    def test_actual_eligibility_helper_rejects_non_true_or_failed_sql(self):
        # Exercise the production shell helper's output and error handling.
        for answer, sql_rc, expected in [('t', 0, 0), ('f', 0, 1), ('', 0, 1),
                                         ('t\nf', 0, 1), ('t', 1, 1)]:
            fixture = '\nas_service_user() { printf "%s" ' + shlex.quote(answer) + '; return ' + str(sql_rc) + '; }\n'
            self.run_shell('mc_pg_promotion_eligible', expected, fixture=fixture)
    def test_dispatch_routes_only_operational_actions_through_scoring(self):
        # Use the real dispatcher with fake handlers; no cluster calls occur.
        dispatch = AGENT.read_text().split('case "$1" in', 1)[1]
        for action in ('start', 'stop', 'promote', 'demote', 'rejoin'):
            code = 'mc_promotion_action() { echo "scored:$1"; return 0; }; set -- ' + action + '\ncase "$1" in' + dispatch
            result = self.run_shell(code)
            self.assertEqual(result.stdout.strip(), 'scored:' + action)
        for action in ('monitor', 'status'):
            code = 'mc_promotion_monitor() { echo scored-monitor; return 8; }; set -- ' + action + '\ncase "$1" in' + dispatch
            self.assertEqual(self.run_shell(code, 8).stdout.strip(), 'scored-monitor')
        for action, handler in [('meta-data', 'mc_engine_metadata'), ('validate-all', 'mc_promotion_validate'), ('notify', 'mc_application_notify')]:
            code = handler + '() { echo plain; return 0; }; set -- ' + action + '\ncase "$1" in' + dispatch
            self.assertEqual(self.run_shell(code).stdout.strip(), 'plain')
        self.assertFalse(self.state()['calls'])
 if __name__ == '__main__':
    unittest.main(verbosity=2)
@@ -18,9 +18,11 @@ assert output('rpm', '-q', '--qf', '%{ARCH}', PACKAGE) == 'noarch'
 assert output('rpm', '-q', '--scripts', PACKAGE) == ''
 subprocess.run(['rpm', '-V', PACKAGE], check=True)
 expected = {AGENT: ROOT / 'refactored/managed-container'}
-for path in (ROOT / 'refactored/managed-container.d').glob('*.sh'):
+for path in (ROOT / 'refactored/managed-container.d').rglob('*'):
-    expected[PROVIDER / 'managed-container.d' / path.name] = path
+    if not path.is_file() or path.suffix not in ('.sh', '.py'):
-assert len(expected) == 10
+        continue
    expected[PROVIDER / path.relative_to(ROOT / 'refactored')] = path
 assert len(expected) == 15
 for path, source in expected.items():
    assert path.read_bytes() == source.read_bytes(), path
    info = path.stat()
@@ -29,10 +31,15 @@ for path, source in expected.items():
 owned = set(output('rpm', '-ql', PACKAGE).splitlines())
 allowed = {str(path) for path in expected} | {str(PROVIDER), str(PROVIDER / 'managed-container.d')}
 for path in expected:
    allowed.update(str(parent) for parent in path.parents if parent == PROVIDER or PROVIDER in parent.parents)
 for directory in allowed - {str(path) for path in expected}:
    info = Path(directory).stat()
    assert (info.st_uid, info.st_gid, stat.S_IMODE(info.st_mode)) == (0, 0, 0o755), directory
 unexpected = {path for path in owned - allowed if not path.startswith('/usr/share/doc/' + PACKAGE)}
 assert not unexpected, unexpected
-reference = output(str(ROOT / 'reference/managed-container'), 'meta-data')
+reference = output(str(ROOT / 'refactored/managed-container'), 'meta-data')
 installed = subprocess.check_output([str(AGENT), 'meta-data'], cwd='/tmp', text=True)
 assert installed == reference
 print('PASS installed RPM: noarch, file integrity, root ownership, modes, payload, no scriptlets, metadata/module loading')
@@ -0,0 +1,94 @@
 #!/usr/bin/env python3
 """Run the production eligibility SQL against controlled PostgreSQL 18 fixtures.
 Explicit opt-in integration test: creates/removes only its own disposable Podman
 container. No host ports, volumes, cluster credentials or live databases used.
 The postgres:18 image must already be present locally.
 """
 from pathlib import Path
 import os
 import json
 import shlex
 import subprocess
 import tempfile
 import time
 import uuid
 ROOT = Path(__file__).resolve().parents[1]
 with tempfile.TemporaryDirectory(prefix='mc-score-sql-') as directory:
    query_file = Path(directory) / 'query.sql'
    env = dict(os.environ, QUERY_FILE=str(query_file))
    extract = '. ' + shlex.quote(str(ROOT / 'refactored/managed-container.d/runtimes/podman.sh')) + '\n. ' + shlex.quote(str(ROOT / 'refactored/managed-container.d/applications/postgresql/postgres.sh')) + '''
 as_service_user() { printf '%s' "${@: -1}" > "$QUERY_FILE"; echo t; }
 mc_pg_promotion_eligible
 '''
    subprocess.run(['bash', '-c', extract], env=env, check=True)
    query = query_file.read_text()
    subprocess.run(['bash', '-c', extract.replace('mc_pg_promotion_eligible', 'mc_pg_standby_observation')], env=env, check=True)
    observation_query = query_file.read_text()
    name = 'mc-score-sql-' + uuid.uuid4().hex[:12]
    subprocess.run(['podman', 'run', '-d', '--pull=never', '--name', name,
                    '--network=none', '--tmpfs', '/var/lib/postgresql',
                    '-e', 'POSTGRES_HOST_AUTH_METHOD=trust',
                    'docker.io/library/postgres:18'], check=True, capture_output=True)
    try:
        for _ in range(40):
            ready = subprocess.run(['podman', 'exec', name, 'pg_isready', '-U', 'postgres'],
                                   capture_output=True)
            if ready.returncode == 0:
                break
            time.sleep(0.5)
        else:
            raise RuntimeError('Disposable PostgreSQL did not become ready')
        def sql(text):
            return subprocess.check_output(['podman', 'exec', '-i', name, 'psql',
                                            '-X', '-U', 'postgres', '-v', 'ON_ERROR_STOP=1',
                                            '-Atq'], input=text, text=True).strip()
        assert sql('SHOW server_version_num;').startswith('18')
        sql('''
 CREATE SCHEMA score_test;
 CREATE TABLE score_test.state (recovery bool, paused bool, replay pg_lsn);
 INSERT INTO score_test.state VALUES (true,false,'0/20');
 CREATE TABLE score_test.pg_stat_wal_receiver (status text, flushed_lsn pg_lsn);
 CREATE FUNCTION score_test.pg_is_in_recovery() RETURNS bool LANGUAGE sql
 AS 'SELECT recovery FROM score_test.state';
 CREATE FUNCTION score_test.pg_is_wal_replay_paused() RETURNS bool LANGUAGE plpgsql AS $$
 BEGIN
  IF NOT (SELECT recovery FROM score_test.state) THEN
    RAISE EXCEPTION 'replay pause is undefined on a primary';
  END IF;
  RETURN (SELECT paused FROM score_test.state);
 END $$;
 CREATE FUNCTION score_test.pg_last_wal_replay_lsn() RETURNS pg_lsn LANGUAGE sql
 AS 'SELECT replay FROM score_test.state';
 ''')
        cases = [
            ('caught-up streaming standby', True, False, '0/20', 'streaming', '0/20', 't'),
            ('replay ahead of receiver snapshot', True, False, '0/30', 'streaming', '0/20', 't'),
            ('replay lag', True, False, '0/10', 'streaming', '0/20', 'f'),
            ('paused replay', True, True, '0/20', 'streaming', '0/20', 'f'),
            ('no replay position', True, False, None, 'streaming', '0/20', 'f'),
            ('unknown flushed position', True, False, '0/20', 'streaming', None, 'f'),
            ('receiver stopped', True, False, '0/20', 'stopping', '0/20', 'f'),
            ('no receiver after primary failure', True, False, '0/20', None, None, 'f'),
            ('primary skips recovery-only function', False, False, '0/20', None, None, 'f'),
        ]
        def literal(value):
            return 'NULL' if value is None else "'" + value + "'"
        for label, recovery, paused, replay, status, flushed, expected in cases:
            setup = f'''SET search_path=score_test,pg_catalog;
 UPDATE score_test.state SET recovery={str(recovery).lower()},paused={str(paused).lower()},replay={literal(replay)};
 TRUNCATE score_test.pg_stat_wal_receiver;
 '''
            if status is not None:
                setup += f'INSERT INTO score_test.pg_stat_wal_receiver VALUES ({literal(status)},{literal(flushed)});\n'
            result = sql(setup + query)
            assert result == expected, (label, result, expected)
            observation = json.loads(sql(setup + observation_query))
            assert (observation['strict'] is True) == (expected == 't'), (label, observation)
            assert observation['identity'][0].isdigit() and observation['identity'][1].startswith('/')
            assert observation['recovery'] == recovery
            assert observation['receiver'] == status
            print('PASS strict and observation SQL:', label)
    finally:
        subprocess.run(['podman', 'rm', '-f', name], check=True, capture_output=True)
@@ -0,0 +1,40 @@
 #!/usr/bin/env python3
 """Explicit disposable-container-only test of the root helper CLI and /run state."""
 import json
 import os
 from pathlib import Path
 import subprocess
 import sys
 import uuid
 from test_disconnected_state import CONNECTED, DISCONNECTED
 assert os.geteuid() == 0 and os.environ.get('MC_DISPOSABLE_STATE_TEST') == '1', (
    'Run only in a disposable container with MC_DISPOSABLE_STATE_TEST=1')
 helper = (Path(sys.argv[1]) if len(sys.argv) > 1 else
          Path(__file__).resolve().parents[1] / 'refactored/managed-container.d/applications/postgresql/promotion_state.py')
 resource = 'mc-disposable-' + uuid.uuid4().hex
 args = ['python3', str(helper), resource, 'test-owner', 'test.service', 'test-container', '300']
 def run(obs, expected):
    result = subprocess.run(args, input=json.dumps(obs), text=True, capture_output=True, timeout=3)
    assert result.returncode == 0 and result.stdout.strip() == expected, result
 run(DISCONNECTED, '-INFINITY')
 run(CONNECTED, '100')
 files = list(Path('/run/managed-container/promotion').glob('*.json'))
 record_path = next(p for p in files if json.loads(p.read_text())['context'][0] == resource)
 record = json.loads(record_path.read_text())
 assert record['replay'] == CONNECTED['replay'] and record['flush'] == CONNECTED['flush']
 for path, mode in [(record_path, 0o600), (record_path.parent, 0o700), (record_path.parent.parent, 0o700)]:
    info = path.stat()
    assert (info.st_uid, info.st_gid, info.st_mode & 0o777) == (0, 0, mode), path
 run(DISCONNECTED, '50')
 assert json.loads(record_path.read_text()) == record  # Disconnection never renews evidence.
 record_path.write_text('{malformed')
 run(DISCONNECTED, '-INFINITY')
 assert not record_path.exists()
 run(CONNECTED, '100')
 run(dict(DISCONNECTED, identity=['999', '/new', '1789201000.123456']), '-INFINITY')
 assert not record_path.exists()
 print('PASS root CLI: fixed /run location, ownership/modes, persistence, disconnect, malformed and replaced data')
@@ -6,11 +6,12 @@ import re
 import subprocess
 import tempfile
 import xml.etree.ElementTree as ET
 from plugin_test_support import LOAD, restore_boundaries
 ROOT = Path(__file__).resolve().parents[1]
 REFERENCE = ROOT / 'reference/managed-container'
 AGENT = ROOT / 'refactored/managed-container'
-MODULES = sorted((AGENT.parent / 'managed-container.d').glob('*.sh'))
+MODULES = sorted((AGENT.parent / 'managed-container.d').rglob('*.sh'))
 SOURCE = REFERENCE.read_text()
 PATTERN = r'^([a-z_]+)\(\)\n\{\n.*?^\}'
@@ -41,34 +42,45 @@ for path in (REFERENCE, ROOT / 'reference/managed-container.pre-refactor-5eae1f3
    assert hashlib.sha256(path.read_bytes()).hexdigest() == expected_hash
 print('PASS both reference hashes')
-assert functions(SOURCE) == functions('\n'.join(p.read_text() for p in MODULES))
+current_functions = functions('\n'.join(p.read_text() for p in MODULES))
 for name, definition in functions(SOURCE).items():
    assert restore_boundaries(current_functions[name]) == definition, name
 assert len(functions(SOURCE)) == 29
-common = (AGENT.parent / 'managed-container.d/common.sh').read_text()
+common = (AGENT.parent / 'managed-container.d/applications/postgresql/plugin.sh').read_text()
 defaults = SOURCE[SOURCE.index('OCF_RESKEY_service_user_default='):SOURCE.index('\n###############################################################################\n# Metadata')].strip()
 assert defaults in common
-assert SOURCE[SOURCE.index('case "$1" in'):] == AGENT.read_text()[AGENT.read_text().index('case "$1" in'):]
+print('PASS all 29 original function bodies identical after exact adapter-call reversal; defaults unchanged')
 print('PASS all 29 function bodies, defaults/globals and dispatcher identical')
 for path in [AGENT, *MODULES]:
    subprocess.run(['bash', '-n', str(path)], check=True)
-print('PASS syntax for entrypoint and all nine modules')
+print('PASS syntax for entrypoint and all modules')
 metadata = []
 for path in (REFERENCE, AGENT):
    result = subprocess.run([str(path), 'meta-data'], cwd='/tmp', capture_output=True, text=True)
    assert result.returncode == 0 and not result.stderr, result
    metadata.append(result.stdout)
-assert metadata[0] == metadata[1]
+extended = ET.fromstring(metadata[1])
 parameters = extended.find('parameters')
 added = parameters.find("parameter[@name='disconnected_max_age']")
 assert added is not None and added.find('content').get('default') == '300'
 parameters.remove(added)
 for name, default in [('application', 'postgresql'), ('container_runtime', 'podman'), ('lifecycle_manager', 'systemd')]:
    added = parameters.find("parameter[@name='" + name + "']")
    assert added is not None and added.find('content').get('default') == default
    parameters.remove(added)
 def normalize(element):
    return (element.tag, element.attrib, (element.text or '').strip(),
            [normalize(child) for child in element])
 assert normalize(extended) == normalize(ET.fromstring(metadata[0]))
 xml = ET.fromstring(metadata[0])
 assert len(xml.findall('./parameters/parameter')) == 12
 assert len(xml.findall('./actions/action')) == 8
-print('PASS metadata byte equality and XML parsing from unrelated cwd')
+print('PASS original metadata preserved plus freshness parameter and XML parsing from unrelated cwd')
 # Sourcing production modules in a fresh shell tests their real loading order.
 # No dispatcher executes. All operational boundaries are overridden below.
-modular_definitions = SOURCE[:SOURCE.index('\n###############################################################################')]
+modular_definitions = LOAD
 for name in ('common', 'container', 'postgres', 'auth', 'replication', 'notify', 'promote', 'demote', 'rejoin'):
    modular_definitions += '\n. ' + "'" + str(AGENT.parent / 'managed-container.d' / (name + '.sh')).replace("'", "'\\''") + "'"
 base = '''
 ocf_log() { printf 'log %s\\n' "$*"; }
 ocf_exit_reason() { printf 'reason %s\\n' "$*" >&2; }