Freeze lab-tested PostgreSQL plugin engine
Extract PostgreSQL, Podman runtime and systemd lifecycle adapters while preserving existing lifecycle and scoring behavior. Update RPM layout and add guarded operator-run lab harnesses. Record successful manual stop/start, promotion, demotion and rewind-only rejoin with pinned installed hashes. The unfenced lab does not validate automatic failover, Pacemaker-managed transitions or divergent-data rewind. Validation: 103 offline tests, original-function differential checks, checksum manifest and whitespace checks passed.
This commit is contained in:
@@ -0,0 +1,11 @@
|
||||
a4b775a0ddcabdb0558da9b34c7fe2a883c07d0ef308c1c7224526abf53f63a0 managed-container
|
||||
93191771d3cb8aefadb2a16d2f89d64a8b6c584071a473439be818237bdf7f17 managed-container.d/auth.sh
|
||||
97e93d9164e854e5f3688cf9d3cc656b001eb30ef3c2073c907450ca43588f3d managed-container.d/common.sh
|
||||
ac6bc363a2301f1491d162ef609103dbfbceb3b099a3556614c6165952309374 managed-container.d/container.sh
|
||||
0af3bb362a303017372c631354ba94ef9e723bf14e64d2eff91b6662e54a5be0 managed-container.d/demote.sh
|
||||
6a98fcdda82cea9365db95b4ee779be5c567b3d846f48b90fbc8c254af697670 managed-container.d/notify.sh
|
||||
d2779ce6e7d1f426ffca2e41bbb8de77503005dc38d658d69f184020365796eb managed-container.d/postgres.sh
|
||||
3085ec9b4410905ab9d4370786ed776eac3470a0415aa1b4f608b083945aeb58 managed-container.d/promote.sh
|
||||
21beea72bcb7f24a51c17ea48666ac87df9436702ad77b166b45f4647e28e8ad managed-container.d/rejoin.sh
|
||||
6b2021887e3b4cd6a12490e7a8dff98320b2133b472ab25a22a0eead46137547 managed-container.d/replication.sh
|
||||
5166fc3370f49ab14cd975ded4b03a74b9bd7dbff5762b41a15552c42f934ea7 managed-container.d/promotion_state.py
|
||||
@@ -0,0 +1,15 @@
|
||||
01eb9b161c8e3db289f09bf63953be779921e15bdf8b1a4d8ce7e60032b826e4 /usr/lib/ocf/resource.d/loxthelion/managed-container
|
||||
10932919c4339b553c7c9dbf30b45a917c51c0a08f362e12678003d58b12fa82 /usr/lib/ocf/resource.d/loxthelion/managed-container.d/applications/postgresql/auth.sh
|
||||
da4e58cadf1df3953cb97a0512e3931ee1b9d3ee61c39a1ad1a6047ef537addc /usr/lib/ocf/resource.d/loxthelion/managed-container.d/applications/postgresql/demote.sh
|
||||
6a98fcdda82cea9365db95b4ee779be5c567b3d846f48b90fbc8c254af697670 /usr/lib/ocf/resource.d/loxthelion/managed-container.d/applications/postgresql/notify.sh
|
||||
d3a989de97ceb61a56828f006d5ccb87b7c34fc155a944585bedf928785021fc /usr/lib/ocf/resource.d/loxthelion/managed-container.d/applications/postgresql/plugin.sh
|
||||
fa222e6767249999767eb82fac2d85e7e35404da240517a8b630571e77282053 /usr/lib/ocf/resource.d/loxthelion/managed-container.d/applications/postgresql/postgres.sh
|
||||
3085ec9b4410905ab9d4370786ed776eac3470a0415aa1b4f608b083945aeb58 /usr/lib/ocf/resource.d/loxthelion/managed-container.d/applications/postgresql/promote.sh
|
||||
5166fc3370f49ab14cd975ded4b03a74b9bd7dbff5762b41a15552c42f934ea7 /usr/lib/ocf/resource.d/loxthelion/managed-container.d/applications/postgresql/promotion_state.py
|
||||
28b0e44554a88ccb87cd16a68315c045dd37d354aceb2fa25da7c7399a659b23 /usr/lib/ocf/resource.d/loxthelion/managed-container.d/applications/postgresql/rejoin.sh
|
||||
6b2021887e3b4cd6a12490e7a8dff98320b2133b472ab25a22a0eead46137547 /usr/lib/ocf/resource.d/loxthelion/managed-container.d/applications/postgresql/replication.sh
|
||||
a482ef3cc1048e6a4f427e76dd2dce173ffd00760dbed1afa601a3fd05b0ad99 /usr/lib/ocf/resource.d/loxthelion/managed-container.d/common.sh
|
||||
3a27ac9c63c61310da8b8711fa2c4400b37ea8eae2ee2a79e28bc63180b06c7c /usr/lib/ocf/resource.d/loxthelion/managed-container.d/container.sh
|
||||
53978e2367394695d61c6fb38f2ee46814474c5a4e0448d2692f1a3ce583af93 /usr/lib/ocf/resource.d/loxthelion/managed-container.d/engine.sh
|
||||
6c92c095ce10ac95d843fdc1c49e31c19491ca22fdac960affd8649606d76d82 /usr/lib/ocf/resource.d/loxthelion/managed-container.d/lifecycles/systemd.sh
|
||||
dc5820067a78c4a1c531426902793aae97a03fd64912b017b3b102031dcc1e04 /usr/lib/ocf/resource.d/loxthelion/managed-container.d/runtimes/podman.sh
|
||||
@@ -0,0 +1,119 @@
|
||||
# ELITEDESK plugin installation and basic checks
|
||||
|
||||
Status: installed on DAVID-ELITEDESK; operator-reported basic checks passed.
|
||||
All 15 installed code files were subsequently verified read-only against the
|
||||
working candidate and matched byte-for-byte.
|
||||
No production or peer-node actions are part of this procedure.
|
||||
|
||||
Read-only preflight on DAVID-ELITEDESK found its PostgreSQL container running as
|
||||
an asynchronous streaming standby of 192.168.1.2 using ha_david_elitedesk.
|
||||
All 11 installed RA/module/helper files match source freeze 488a167; their exact
|
||||
hashes are captured in lab-installed-before-plugin.sha256. This is a lab snapshot,
|
||||
not the outstanding production snapshot. Current cluster maintenance state could
|
||||
not be read without sudo; the installer verifies it live before any replacement.
|
||||
|
||||
Run in the operator's terminal:
|
||||
|
||||
```sh
|
||||
sudo python3 /home/david/codex-postgres-ha-refactor/scripts/test-elitedesk-plugin.py
|
||||
```
|
||||
|
||||
The command requires the existing postgres-ra-test cluster, exactly the two lab
|
||||
nodes, pg-ha under pg-ha-clone, expected local container/database settings, and
|
||||
maintenance-mode already true. It does not enable maintenance mode or modify CIB
|
||||
configuration. It copies parameters from the live resource and rejects ambiguous
|
||||
parameter sets. Native sudo prompts stay in the terminal; never share a password.
|
||||
|
||||
It verifies source checksums, stages root-owned code, backs up the installed agent
|
||||
and modules under /var/backups/managed-container-plugin-lab-*, validates staged
|
||||
metadata and configuration, then replaces the complete module tree and entrypoint.
|
||||
The installation remains in place after success. A failed post-install check
|
||||
restores the old agent and module tree and retains diagnostic files.
|
||||
|
||||
Basic checks are metadata (16 parameters), validate-all, monitor, status, an
|
||||
idempotent start on the already-running standby, and another monitor. Before and
|
||||
after checks require the same PostgreSQL system identifier, postmaster start time,
|
||||
recovery role, streaming sender and slot. No stop, promote, demote, rejoin or
|
||||
fencing is invoked. Normal monitors/start may refresh local promotion evidence
|
||||
and the pg-ha promotion score; those expected side effects are part of testing.
|
||||
A score of -INFINITY is recorded rather than mistaken for a failed role monitor;
|
||||
strict catch-up can fluctuate. Investigate that result before broader HA testing.
|
||||
|
||||
The script prints its backup/result directory and action results. It records
|
||||
results.json and installed-SHA256SUMS there with root-only access. Return the
|
||||
terminal output before progressing to disruptive stop/start or switchover tests.
|
||||
|
||||
Preparation validation: current structural/differential tests passed; two
|
||||
isolated installer tests passed, covering basic-action-only success and restoration
|
||||
of both agent and modules on an injected installed-monitor failure. These tests
|
||||
mock privilege/cluster boundaries and touch only temporary directories; they do
|
||||
not claim live installation success.
|
||||
|
||||
## Completed basic lab run
|
||||
|
||||
The operator supplied successful installer output for backup/result directory
|
||||
`/var/backups/managed-container-plugin-lab-mow_t6zi`:
|
||||
|
||||
- Staged metadata and validate-all: rc=0.
|
||||
- Installed validate-all, monitor, status, idempotent start, monitor: rc=0.
|
||||
- Final promotion score: 100.
|
||||
- The installer reported PostgreSQL was not restarted and maintenance remained true.
|
||||
- No stop, promote, demote, rejoin, fencing, or peer-node actions ran.
|
||||
|
||||
Codex independently compared every installed entrypoint/module/helper file with
|
||||
the working candidate using read-only filesystem access; all 15 matched. Exact
|
||||
installed SHA-256 values are recorded in [lab-installed-plugin.sha256](lab-installed-plugin.sha256).
|
||||
The root-only results.json was not independently read; the action results above
|
||||
are attributed to the operator-supplied output. This validates the installation
|
||||
and non-disruptive standby path, not actual stop/start, switchover or failover.
|
||||
|
||||
Installed-file comparison captured at 2026-09-13T03:01:11.114949+00:00 UTC.
|
||||
|
||||
## Controlled standby stop/start — operator-reported pass
|
||||
|
||||
The operator ran:
|
||||
|
||||
```sh
|
||||
sudo python3 /home/david/codex-postgres-ha-refactor/scripts/test-elitedesk-cycle.py
|
||||
```
|
||||
|
||||
This briefly interrupts only the ELITEDESK standby. The script requires the same
|
||||
lab identity, maintenance=true, streaming sender/slot and exact installed hashes.
|
||||
It validates/monitors, runs RA stop, checks systemd inactive and the container
|
||||
stopped/absent, expects monitor=7 and score=-INFINITY, then runs RA start. It
|
||||
requires the same database identifier, a new postmaster start time, recovery and
|
||||
streaming from DV6, monitor/status=0 and score=100 within the verification window.
|
||||
Each RA action has a 60-second timeout with a 5-second kill grace. Post-start
|
||||
streaming/eligibility checks have a 60-second retry window (an in-flight bounded
|
||||
check can finish after the deadline). No source files or cluster configuration
|
||||
are replaced, and no promotion, demotion, rejoin, fencing or peer action runs.
|
||||
|
||||
Failures halt without automatic recovery and may leave the standby stopped.
|
||||
Inspect the printed /var/backups/managed-container-cycle-lab-* result directory
|
||||
before proceeding. It contains root-only results.json and per-action output.
|
||||
Three mocked harness tests passed: successful cycle, stop failure without start,
|
||||
and start failure without retry. Those are preparation checks, not live results.
|
||||
|
||||
|
||||
### Completed cycle result
|
||||
|
||||
The operator supplied successful output with evidence directory
|
||||
`/var/backups/managed-container-cycle-lab-b1048r6q`:
|
||||
|
||||
| Check | Result |
|
||||
| --- | --- |
|
||||
| Initial validate-all and monitor | rc=0 |
|
||||
| RA stop | rc=0 |
|
||||
| Monitor while stopped | rc=7 |
|
||||
| Stopped lifecycle state | systemd inactive; container stopped/absent |
|
||||
| Stopped promotion score | -INFINITY |
|
||||
| RA start, subsequent monitor and status | rc=0 |
|
||||
| Database identity | Same system identifier; new postmaster start time |
|
||||
| Replication after restart | Streaming from DV6; score 100 |
|
||||
| Cluster maintenance | Remained true |
|
||||
|
||||
These results are attributed to the operator-supplied script output; Codex has
|
||||
not independently read the root-only results.json. No promotion, demotion,
|
||||
rejoin, fencing or peer-node actions ran. The basic standby lifecycle now has
|
||||
live lab coverage; this does not establish switchover/failover behavior for the
|
||||
plugin extraction or constitute production deployment validation.
|
||||
@@ -0,0 +1,113 @@
|
||||
# Staged lab role tests
|
||||
|
||||
**All five live stages passed according to operator-supplied terminal output.**
|
||||
The sequence followed the successful ELITEDESK stop/start cycle. Only the two-node postgres-ra-test lab is supported; maintenance
|
||||
must remain true throughout. DV6 retains the frozen RA and ELITEDESK has the
|
||||
plugin candidate. Neither implementation is installed or changed by this tool.
|
||||
|
||||
The operator confirmed that this lab has no fencing; fencing is available only
|
||||
in production. This checkpoint covers controlled, manual role transitions with
|
||||
a reachable primary. Abrupt-primary-loss automatic failover is not validated
|
||||
for the plugin candidate. The earlier production fencing report applies to the
|
||||
earlier implementation, not this extraction.
|
||||
|
||||
During maintenance, Pacemaker's recurring role monitors are paused. The harness
|
||||
invokes the RA directly, so its action results do not update Pacemaker's recorded
|
||||
operation history. A Cockpit cluster role display can therefore remain stale
|
||||
even while the RA explicitly updates promotion scores. Database roles were
|
||||
verified using direct SQL; systemd/container running state remains distinct
|
||||
from PostgreSQL role. Maintenance was not disabled to refresh the display.
|
||||
|
||||
`scripts/test-lab-roles.py` is a portable, operator-run harness with embedded
|
||||
hashes from both lab snapshots. It needs root locally and noninteractive SSH
|
||||
as david to the peer for read-only observations. SSH host keys must already be
|
||||
verified. Passwords are entered in the operator's terminal, never in chat.
|
||||
|
||||
Run `sudo python3 /path/to/test-lab-roles.py inspect` on each node first.
|
||||
Inspection checks cluster name, membership, maintenance, resource parameters,
|
||||
installed hashes, SQL readiness, roles, system identity and maintenance service.
|
||||
It changes no database or cluster state. Do not proceed if either check fails.
|
||||
|
||||
Each stage requires a separate invocation and review of its successful output:
|
||||
|
||||
| Order | Host | Stage argument | Expected result |
|
||||
| --- | --- | --- | --- |
|
||||
| 1 | DV6 | demote | DV6 streams from ELITEDESK; both are standbys |
|
||||
| 2 | ELITEDESK | promote | ELITEDESK primary, DV6 streaming standby |
|
||||
| 3 | ELITEDESK | demote | ELITEDESK streams from DV6; both are standbys |
|
||||
| 4 | DV6 | promote | DV6 primary, ELITEDESK streaming standby |
|
||||
| 5 | ELITEDESK | rejoin | Stop PostgreSQL only, run rewind-only rejoin to DV6, verify streaming |
|
||||
|
||||
The demote stage submits a **synthetic lab pre-demote notification** through the
|
||||
RA notify action, then invokes demote. This tests the notification consumer and
|
||||
controlled demotion; it does not prove Pacemaker generates/orders notifications.
|
||||
The RA retains its own catch-up and successor checks. The harness refuses promote
|
||||
unless the peer is already in recovery and streaming from the local standby.
|
||||
Between demote and promote there is deliberately no writable primary. Arrange
|
||||
both terminals before starting; do not perform other manual database operations
|
||||
during the sequence. Scores on both standbys can fluctuate during this interval.
|
||||
|
||||
The rejoin stage is restricted to the restored ELITEDESK standby. It explicitly
|
||||
sets basebackup fallback=false. It tests the stopped-PostgreSQL precondition,
|
||||
maintenance container survival, source authentication, pg_rewind invocation,
|
||||
recovery configuration, PostgreSQL restart, streaming and score restoration.
|
||||
Because it does not create divergent data, it does **not** validate divergent
|
||||
former-primary rewind repair. No fencing, data replacement fallback, VIP moves,
|
||||
Pacemaker-managed switchover or automatic failover are tested here.
|
||||
|
||||
Each modifying invocation writes root-only results/action logs beneath
|
||||
`/var/backups/managed-container-roles-lab-*`. Failures halt without automatic
|
||||
retry/recovery. A failure after demotion can leave no primary; a rejoin failure
|
||||
can leave ELITEDESK stopped. Stop the sequence and inspect the evidence rather
|
||||
than running the next stage. Do not clear failed actions or change maintenance.
|
||||
|
||||
The checks sample state; they cannot prevent a separate administrator from
|
||||
changing roles concurrently. Controlled demotion of a reachable primary is the
|
||||
basis for these manual lab promotions. This is not a procedure for promoting
|
||||
after an unreachable primary failure; that requires fencing and Pacemaker's
|
||||
ordering under the operating policy.
|
||||
|
||||
## Completed live sequence
|
||||
|
||||
The operator supplied successful output for each stage below. Evidence directories
|
||||
are on the host named in the table, under `/var/backups/`. Codex has not read the
|
||||
root-only results.json or action logs; these are operator-reported passes, not
|
||||
independently witnessed live actions.
|
||||
|
||||
| Host | Stage | Result | Evidence directory |
|
||||
| --- | --- | --- | --- |
|
||||
| DV6 | demote | rc=0; standby monitor=0; score=100 | managed-container-roles-lab-o1xjobi2 |
|
||||
| ELITEDESK | promote | rc=0; primary monitor=8; score=1000 | managed-container-roles-lab-qdz57g70 |
|
||||
| ELITEDESK | demote | rc=0; standby monitor=0; score=100 | managed-container-roles-lab-nwbzdxnl |
|
||||
| DV6 | promote | rc=0; primary monitor=8; score=1000 | managed-container-roles-lab-r69amyxg |
|
||||
| ELITEDESK | rejoin | stopped monitor=7; rejoin=0; standby monitor=0; score=100 | managed-container-roles-lab-7c4vw7_p |
|
||||
|
||||
Each stage reported preserved database identity and maintenance container,
|
||||
verified roles and streaming, and maintenance=true. The final reported state is
|
||||
DV6 primary and ELITEDESK streaming standby. The rejoin harness also checks the
|
||||
stopped score is -INFINITY before invoking rejoin, with basebackup disabled.
|
||||
This establishes successful pg_rewind invocation, not repair of deliberately
|
||||
divergent timelines or a measured amount of rewound data.
|
||||
|
||||
The supplied inspection after ELITEDESK demotion showed both nodes in recovery,
|
||||
streaming from each other, with replay LSN 0/A02B560 and system identifier
|
||||
7684451577990111248. An earlier inspection in the same paste showed DV6 primary;
|
||||
the later observation was used to resolve the role confusion before restoring DV6.
|
||||
|
||||
Both-way SSH access was established and independently checked before the role
|
||||
sequence. The installed code was pinned to the existing lab snapshots. No RA
|
||||
installation, maintenance change, fencing, VIP move or failure cleanup was part
|
||||
of these tests. The limitations above continue to apply.
|
||||
|
||||
Offline original-function comparisons and 97 tests passed before the live
|
||||
sequence (the 11 scheduler tests required execution outside the sandbox).
|
||||
Six additional role-harness safety-gate tests passed before staging the harness.
|
||||
|
||||
## Commit checkpoint verification
|
||||
|
||||
On 2026-09-13, all 103 tests in `python3 -m unittest discover -s tests -p
|
||||
'test_*.py'` passed, including offline scheduler simulations.
|
||||
`python3 tests/verify_refactor.py` also passed: frozen reference hashes, all 29
|
||||
original functions after exact adapter-call reversal, syntax, metadata and
|
||||
22 differential lifecycle cases. No additional live actions were run for this
|
||||
commit checkpoint.
|
||||
+3
-2
@@ -46,9 +46,10 @@ and set PGPASSWORD only for the child process.
|
||||
|
||||
Lifecycle success is 0; monitor's 8 is a role report, not a generic lifecycle
|
||||
success. Promoted monitoring must expect the correct role. Metadata advertises
|
||||
13 parameters and eight actions; status is an alias and rejoin stays private.
|
||||
16 parameters and eight actions; status is an alias and rejoin stays private.
|
||||
The scoring wrappers withdraw scores before lifecycle actions and reassess after
|
||||
successful actions. They do not change the original 29 function bodies.
|
||||
successful actions. The engine extraction preserves the original 29 functions with explicit
|
||||
adapter-call substitutions; see plugin-engine.md.
|
||||
|
||||
## Scoring and fencing
|
||||
|
||||
|
||||
@@ -0,0 +1,210 @@
|
||||
# Podman plugin engine: first extraction
|
||||
|
||||
Development baseline: freeze commit `488a167`. This branch extracts the existing PostgreSQL/Podman/systemd implementation.
|
||||
It is installed on DAVID-ELITEDESK in the lab only; see lab-plugin-basic-test.md. Production
|
||||
hash provenance is still pending as described in testing.md. Quadlet, bootstrap and cluster configuration remain unchanged. The lab RA files
|
||||
were replaced, and basic monitor/start checks refreshed standby scoring without
|
||||
restarting PostgreSQL. No production installation has occurred.
|
||||
|
||||
## Scope and supported configuration
|
||||
|
||||
The default and only supported combination is:
|
||||
|
||||
```text
|
||||
application=postgresql
|
||||
container_runtime=podman
|
||||
lifecycle_manager=systemd
|
||||
```
|
||||
|
||||
These are advertised OCF instance parameters, not Pacemaker meta-attributes.
|
||||
Existing resource configurations need no new parameters. Missing or empty
|
||||
selectors use the defaults. Explicit unsupported values, including Docker or
|
||||
direct-runtime lifecycle, fail with OCF_ERR_CONFIGURED (6) before operations.
|
||||
The entrypoint/provider and every existing parameter default remain unchanged.
|
||||
|
||||
The engine loads a fixed allowlist of packaged files beside the entrypoint.
|
||||
There is no arbitrary plugin pathname, dynamic shell evaluation, network plugin
|
||||
installation, or fallback to another runtime. Every implementation declares API
|
||||
version 1; missing files, incompatible versions and missing required hooks fail
|
||||
configuration. Packaged code is trusted/root-owned, as was the original RA.
|
||||
|
||||
## Boundaries
|
||||
|
||||
```text
|
||||
Pacemaker OCF action
|
||||
-> shared scoring/action wrappers
|
||||
-> applications/postgresql/plugin.sh
|
||||
-> existing PostgreSQL lifecycle and recovery modules
|
||||
-> runtimes/podman.sh (inspect / exec / info)
|
||||
-> lifecycles/systemd.sh (user service lifecycle)
|
||||
-> unchanged as_service_user privilege boundary
|
||||
```
|
||||
|
||||
`engine.sh` selects and checks plugins and adds selector metadata. `common.sh`
|
||||
contains the existing score-publication and action wrappers, now calling the
|
||||
application interface. `container.sh` retains service-user access plus
|
||||
compatibility health helpers that delegate to the selected adapters.
|
||||
|
||||
`applications/postgresql/plugin.sh` owns the existing PostgreSQL defaults, metadata,
|
||||
validation and seven PostgreSQL/HA modules. Application modules and the state
|
||||
helper live together under
|
||||
`applications/postgresql/`; runtime and lifecycle adapters occupy their own
|
||||
`runtimes/` and `lifecycles/` directories. The loader uses fixed relative paths.
|
||||
Future applications will have their own
|
||||
metadata and dependency validation rather than inherit PostgreSQL assumptions.
|
||||
|
||||
The packaged layout is:
|
||||
|
||||
```text
|
||||
managed-container.d/
|
||||
engine.sh
|
||||
common.sh
|
||||
container.sh
|
||||
applications/postgresql/
|
||||
plugin.sh
|
||||
postgres.sh, auth.sh, replication.sh, notify.sh
|
||||
promote.sh, demote.sh, rejoin.sh
|
||||
promotion_state.py
|
||||
lifecycles/systemd.sh
|
||||
runtimes/podman.sh
|
||||
```
|
||||
|
||||
The PostgreSQL plugin sets its own `MC_APPLICATION_DIR` for sibling modules and
|
||||
the Python state helper. Host evidence remains at its existing `/run` location;
|
||||
only packaged code paths changed. The public RA entrypoint is unchanged.
|
||||
|
||||
## Interface contract, API 1
|
||||
|
||||
All interfaces are sourced Bash functions in a single RA invocation. Arguments
|
||||
are passed as arrays using `"$@"`; adapters do not concatenate or evaluate command
|
||||
strings. Existing PostgreSQL shell snippets remain explicit arguments to `sh -c`;
|
||||
this extraction does not redesign or broaden their trusted configuration inputs.
|
||||
|
||||
| Interface | Contract |
|
||||
| --- | --- |
|
||||
| mc_runtime_exec ARGS... | Execute in the container through the service owner; preserve stdout, stderr, and exact exit status. |
|
||||
| mc_runtime_exec_bounded DURATION ARGS... | Same execution with timeout placed inside the service-user boundary; retain kill-after=1s. Current scoring callers pass 3s. |
|
||||
| mc_runtime_inspect ARGS... | Read container properties through the same runtime/user boundary. |
|
||||
| mc_runtime_info ARGS... | Validate runtime access through the service owner. |
|
||||
| mc_lifecycle_start SERVICE | Start the configured user-systemd service. |
|
||||
| mc_lifecycle_stop SERVICE | Stop that service, preserving failures. |
|
||||
| mc_lifecycle_running SERVICE | systemctl --user is-active --quiet. |
|
||||
| mc_lifecycle_cat SERVICE | Inspect service definition for existing validation. |
|
||||
| mc_application_monitor | Return the application's normalized OCF health/role code. |
|
||||
| mc_application_standby_score | Emit the eligible standby score, or fail; PostgreSQL owns evidence and freshness. |
|
||||
| mc_application_action ACTION | Dispatch only start/stop/promote/demote/rejoin to the existing application functions. |
|
||||
| mc_application_validate | Run existing application/configuration validation. |
|
||||
| mc_application_metadata | Return application metadata without runtime or CIB access. |
|
||||
| mc_application_notify | Process the existing transition notification. |
|
||||
|
||||
The runtime exec argument grammar currently follows the existing callers:
|
||||
`--user USER`, optional `--env KEY=VALUE` entries, container name, command and
|
||||
arguments. Inspect retains the existing Go-template argument. This is a narrow
|
||||
Podman adapter contract, not a claim that every future runtime accepts it.
|
||||
A second runtime must explicitly implement and test those semantics.
|
||||
|
||||
## Normalization and state
|
||||
|
||||
For this extraction the existing PostgreSQL monitor is the normalizer: it turns
|
||||
PostgreSQL's role/health observations into OCF codes. No extra JSON protocol or
|
||||
new independently maintained role state is introduced.
|
||||
|
||||
| Observation | Monitor code |
|
||||
| --- | ---: |
|
||||
| Healthy standby | 0 (Unpromoted) |
|
||||
| Healthy primary | 8 (Promoted) |
|
||||
| Container absent/stopped, or PostgreSQL stopped | 7 (not running) |
|
||||
| PostgreSQL unready or failed/invalid role query | 1 (generic error) |
|
||||
|
||||
Container availability, application health/role and promotion eligibility remain
|
||||
separate concepts. Container-running/PostgreSQL-stopped is a legitimate
|
||||
maintenance condition, but is not OCF start success. A healthy standby can have
|
||||
score -INFINITY without losing its Unpromoted role report. Core score validation
|
||||
still replaces malformed/failed score output with -INFINITY.
|
||||
|
||||
All existing scores, freshness checks, persisted identity/LSNs, timeout locations,
|
||||
notification handling, fencing assumptions, standby-only start and manual fenced
|
||||
rejoin rules are preserved. Direct application-plugin calls are internal;
|
||||
Pacemaker dispatch continues through the score/lifecycle wrappers.
|
||||
|
||||
## Maintenance and lifecycle ownership
|
||||
|
||||
Quadlet remains responsible for the bootstrap entrypoint, `Exec=sleep infinity`,
|
||||
`RunInit=true`, stop signal/timeout, mounts, image, environment and systemd service
|
||||
policies. `SuccessExitStatus=143` is systemd's classification of its managed
|
||||
service; the runtime exec adapter must still return 143 as an application-command
|
||||
failure. `Restart=on-abnormal` remains solely in the existing unit. No new
|
||||
restart loop, container creation, container replacement or supervision is added.
|
||||
|
||||
During demote/rejoin, PostgreSQL may be stopped while its container remains
|
||||
available. Ordinary stop still stops PostgreSQL before asking systemd to stop
|
||||
the container service. These ordering rules remain in the PostgreSQL lifecycle
|
||||
functions; the adapters do not reorder them.
|
||||
|
||||
## Preservation and validation
|
||||
|
||||
The original 29 functions are retained by name. Some now call adapter functions,
|
||||
so their literal hashes differ. `tests/verify_refactor.py` reverses only the
|
||||
explicit command-boundary substitutions and compares every remaining function
|
||||
byte with the immutable reference. It also retains all 22 differential lifecycle
|
||||
cases and verifies original metadata plus the four accumulated added parameters
|
||||
(freshness plus three selectors). Existing scoring tests use the actual engine
|
||||
loader and still exercise the original handlers against fake external boundaries.
|
||||
|
||||
`tests/test_plugin_engine.py` checks argument fidelity (including quotes, spaces,
|
||||
newlines and literal shell syntax), lifecycle ownership, exit-code propagation,
|
||||
timeout placement, application dispatch, metadata discovery, missing/incompatible
|
||||
plugins, and unsupported/path-like selections. It also checks that PostgreSQL
|
||||
modules no longer invoke Podman/systemd or service-user execution directly.
|
||||
|
||||
Run:
|
||||
|
||||
```sh
|
||||
python3 tests/verify_refactor.py
|
||||
python3 tests/test_plugin_engine.py
|
||||
python3 tests/test_promotion_scores.py
|
||||
python3 tests/test_disconnected_state.py
|
||||
python3 tests/test_promotion_scheduler.py
|
||||
python3 tests/test_promotion_cleanup.py
|
||||
python3 tests/verify_promotion_sql.py
|
||||
```
|
||||
|
||||
The last command uses only its own disposable PostgreSQL container. RPM checks
|
||||
include the plugin tests and verify discovery from the staged installation and
|
||||
an unrelated working directory. No live HA transitions are part of this branch's
|
||||
validation. Live lab authorization and actual configuration comparison remain
|
||||
necessary before deployment.
|
||||
|
||||
## Follow-on work
|
||||
|
||||
After this extraction passes lab validation, a separate change can add direct
|
||||
Podman lifecycle management for pre-created maintenance containers. That work
|
||||
must define exit/restart behavior and sole lifecycle ownership. Container creation,
|
||||
RA-coordinated standby provisioning, a second runtime or another application are
|
||||
separate extensions. The current PostgreSQL plugin is intentionally still the
|
||||
only supported application; this is not yet a general multi-application framework.
|
||||
|
||||
## Results for this extraction
|
||||
|
||||
Passed: 22 original differential lifecycle cases, 25 scoring tests, 16 state/monitor
|
||||
tests, 11 scheduler tests, 4 preference-planner tests, 11 engine contract tests,
|
||||
and all 9 PostgreSQL 18 SQL scenarios. Fedora 43 RPM build, disposable installation,
|
||||
15-file ownership/mode/payload checks, and the installed state-helper CLI passed.
|
||||
Folder-layout build log: `/tmp/mc-plugin-folders-rpm.log`; development artifacts:
|
||||
`dist/plugin-folders/`. Those build and offline checks did not run live HA transitions; subsequent
|
||||
operator-run lab transitions are recorded below.
|
||||
|
||||
The core still retains the existing score vocabulary and freshness-parameter
|
||||
validation for this sole supported PostgreSQL profile. API 1 is an internal
|
||||
extraction interface, not a stable third-party plugin SDK. Generalizing these
|
||||
remaining policy assumptions should be driven by a second real application,
|
||||
not assumed complete by this milestone.
|
||||
|
||||
Basic ELITEDESK installation checks subsequently passed with score 100; all 15
|
||||
installed files match the candidate. A controlled ELITEDESK standby stop/start subsequently passed according to
|
||||
operator-supplied output: stopped score -INFINITY, resumed streaming score 100,
|
||||
same database and new postmaster. The operator subsequently reported successful controlled switchovers in both
|
||||
directions and a rewind-only rejoin on ELITEDESK; see [role test results](lab-role-tests.md).
|
||||
Fencing, Pacemaker-managed transitions and divergent-data rewind remain untested
|
||||
with this extraction.
|
||||
See [the lab record](lab-plugin-basic-test.md) for evidence attribution.
|
||||
+5
-2
@@ -1,3 +1,6 @@
|
||||
> These production reports and source hashes describe the 488a167 freeze.
|
||||
> The current undeployed extraction is described in [plugin-engine.md](plugin-engine.md).
|
||||
|
||||
# Validation and freeze-point evidence
|
||||
|
||||
## Reported production fencing and recovery test
|
||||
@@ -69,7 +72,7 @@ whether any code changed since the test. Hash the running container's bootstrap;
|
||||
a host snapshot or image tag alone is not equivalent. Do not dump environment,
|
||||
secret files, or pgpass. Compare every node individually, preserving mismatches.
|
||||
|
||||
## Repository freeze snapshot
|
||||
## Repository freeze snapshot (488a167)
|
||||
|
||||
The exact repository code hashes appear below and in the reviewed checksum
|
||||
manifest. These identify the source freeze only, pending production comparison.
|
||||
@@ -94,7 +97,7 @@ not yet confirmed identical to the production bootstrap.
|
||||
|
||||
## Non-production checks
|
||||
|
||||
The source retains all 29 original lifecycle function bodies and original
|
||||
The frozen source retained all 29 original lifecycle function bodies and original
|
||||
parameter defaults. Added scoring changes public dispatch and metadata; it is
|
||||
not a claim of identical total behavior to the monolithic reference.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user