Freeze lab-tested PostgreSQL plugin engine
Extract PostgreSQL, Podman runtime and systemd lifecycle adapters while preserving existing lifecycle and scoring behavior. Update RPM layout and add guarded operator-run lab harnesses. Record successful manual stop/start, promotion, demotion and rewind-only rejoin with pinned installed hashes. The unfenced lab does not validate automatic failover, Pacemaker-managed transitions or divergent-data rewind. Validation: 103 offline tests, original-function differential checks, checksum manifest and whitespace checks passed.
This commit is contained in:
Lyncoln
@@ -0,0 +1,210 @@
|
||||
# Podman plugin engine: first extraction
|
||||
|
||||
Development baseline: freeze commit `488a167`. This branch extracts the existing PostgreSQL/Podman/systemd implementation.
|
||||
It is installed on DAVID-ELITEDESK in the lab only; see lab-plugin-basic-test.md. Production
|
||||
hash provenance is still pending as described in testing.md. Quadlet, bootstrap and cluster configuration remain unchanged. The lab RA files
|
||||
were replaced, and basic monitor/start checks refreshed standby scoring without
|
||||
restarting PostgreSQL. No production installation has occurred.
|
||||
|
||||
## Scope and supported configuration
|
||||
|
||||
The default and only supported combination is:
|
||||
|
||||
```text
|
||||
application=postgresql
|
||||
container_runtime=podman
|
||||
lifecycle_manager=systemd
|
||||
```
|
||||
|
||||
These are advertised OCF instance parameters, not Pacemaker meta-attributes.
|
||||
Existing resource configurations need no new parameters. Missing or empty
|
||||
selectors use the defaults. Explicit unsupported values, including Docker or
|
||||
direct-runtime lifecycle, fail with OCF_ERR_CONFIGURED (6) before operations.
|
||||
The entrypoint/provider and every existing parameter default remain unchanged.
|
||||
|
||||
The engine loads a fixed allowlist of packaged files beside the entrypoint.
|
||||
There is no arbitrary plugin pathname, dynamic shell evaluation, network plugin
|
||||
installation, or fallback to another runtime. Every implementation declares API
|
||||
version 1; missing files, incompatible versions and missing required hooks fail
|
||||
configuration. Packaged code is trusted/root-owned, as was the original RA.
|
||||
|
||||
## Boundaries
|
||||
|
||||
```text
|
||||
Pacemaker OCF action
|
||||
-> shared scoring/action wrappers
|
||||
-> applications/postgresql/plugin.sh
|
||||
-> existing PostgreSQL lifecycle and recovery modules
|
||||
-> runtimes/podman.sh (inspect / exec / info)
|
||||
-> lifecycles/systemd.sh (user service lifecycle)
|
||||
-> unchanged as_service_user privilege boundary
|
||||
```
|
||||
|
||||
`engine.sh` selects and checks plugins and adds selector metadata. `common.sh`
|
||||
contains the existing score-publication and action wrappers, now calling the
|
||||
application interface. `container.sh` retains service-user access plus
|
||||
compatibility health helpers that delegate to the selected adapters.
|
||||
|
||||
`applications/postgresql/plugin.sh` owns the existing PostgreSQL defaults, metadata,
|
||||
validation and seven PostgreSQL/HA modules. Application modules and the state
|
||||
helper live together under
|
||||
`applications/postgresql/`; runtime and lifecycle adapters occupy their own
|
||||
`runtimes/` and `lifecycles/` directories. The loader uses fixed relative paths.
|
||||
Future applications will have their own
|
||||
metadata and dependency validation rather than inherit PostgreSQL assumptions.
|
||||
|
||||
The packaged layout is:
|
||||
|
||||
```text
|
||||
managed-container.d/
|
||||
engine.sh
|
||||
common.sh
|
||||
container.sh
|
||||
applications/postgresql/
|
||||
plugin.sh
|
||||
postgres.sh, auth.sh, replication.sh, notify.sh
|
||||
promote.sh, demote.sh, rejoin.sh
|
||||
promotion_state.py
|
||||
lifecycles/systemd.sh
|
||||
runtimes/podman.sh
|
||||
```
|
||||
|
||||
The PostgreSQL plugin sets its own `MC_APPLICATION_DIR` for sibling modules and
|
||||
the Python state helper. Host evidence remains at its existing `/run` location;
|
||||
only packaged code paths changed. The public RA entrypoint is unchanged.
|
||||
|
||||
## Interface contract, API 1
|
||||
|
||||
All interfaces are sourced Bash functions in a single RA invocation. Arguments
|
||||
are passed as arrays using `"$@"`; adapters do not concatenate or evaluate command
|
||||
strings. Existing PostgreSQL shell snippets remain explicit arguments to `sh -c`;
|
||||
this extraction does not redesign or broaden their trusted configuration inputs.
|
||||
|
||||
| Interface | Contract |
|
||||
| --- | --- |
|
||||
| mc_runtime_exec ARGS... | Execute in the container through the service owner; preserve stdout, stderr, and exact exit status. |
|
||||
| mc_runtime_exec_bounded DURATION ARGS... | Same execution with timeout placed inside the service-user boundary; retain kill-after=1s. Current scoring callers pass 3s. |
|
||||
| mc_runtime_inspect ARGS... | Read container properties through the same runtime/user boundary. |
|
||||
| mc_runtime_info ARGS... | Validate runtime access through the service owner. |
|
||||
| mc_lifecycle_start SERVICE | Start the configured user-systemd service. |
|
||||
| mc_lifecycle_stop SERVICE | Stop that service, preserving failures. |
|
||||
| mc_lifecycle_running SERVICE | systemctl --user is-active --quiet. |
|
||||
| mc_lifecycle_cat SERVICE | Inspect service definition for existing validation. |
|
||||
| mc_application_monitor | Return the application's normalized OCF health/role code. |
|
||||
| mc_application_standby_score | Emit the eligible standby score, or fail; PostgreSQL owns evidence and freshness. |
|
||||
| mc_application_action ACTION | Dispatch only start/stop/promote/demote/rejoin to the existing application functions. |
|
||||
| mc_application_validate | Run existing application/configuration validation. |
|
||||
| mc_application_metadata | Return application metadata without runtime or CIB access. |
|
||||
| mc_application_notify | Process the existing transition notification. |
|
||||
|
||||
The runtime exec argument grammar currently follows the existing callers:
|
||||
`--user USER`, optional `--env KEY=VALUE` entries, container name, command and
|
||||
arguments. Inspect retains the existing Go-template argument. This is a narrow
|
||||
Podman adapter contract, not a claim that every future runtime accepts it.
|
||||
A second runtime must explicitly implement and test those semantics.
|
||||
|
||||
## Normalization and state
|
||||
|
||||
For this extraction the existing PostgreSQL monitor is the normalizer: it turns
|
||||
PostgreSQL's role/health observations into OCF codes. No extra JSON protocol or
|
||||
new independently maintained role state is introduced.
|
||||
|
||||
| Observation | Monitor code |
|
||||
| --- | ---: |
|
||||
| Healthy standby | 0 (Unpromoted) |
|
||||
| Healthy primary | 8 (Promoted) |
|
||||
| Container absent/stopped, or PostgreSQL stopped | 7 (not running) |
|
||||
| PostgreSQL unready or failed/invalid role query | 1 (generic error) |
|
||||
|
||||
Container availability, application health/role and promotion eligibility remain
|
||||
separate concepts. Container-running/PostgreSQL-stopped is a legitimate
|
||||
maintenance condition, but is not OCF start success. A healthy standby can have
|
||||
score -INFINITY without losing its Unpromoted role report. Core score validation
|
||||
still replaces malformed/failed score output with -INFINITY.
|
||||
|
||||
All existing scores, freshness checks, persisted identity/LSNs, timeout locations,
|
||||
notification handling, fencing assumptions, standby-only start and manual fenced
|
||||
rejoin rules are preserved. Direct application-plugin calls are internal;
|
||||
Pacemaker dispatch continues through the score/lifecycle wrappers.
|
||||
|
||||
## Maintenance and lifecycle ownership
|
||||
|
||||
Quadlet remains responsible for the bootstrap entrypoint, `Exec=sleep infinity`,
|
||||
`RunInit=true`, stop signal/timeout, mounts, image, environment and systemd service
|
||||
policies. `SuccessExitStatus=143` is systemd's classification of its managed
|
||||
service; the runtime exec adapter must still return 143 as an application-command
|
||||
failure. `Restart=on-abnormal` remains solely in the existing unit. No new
|
||||
restart loop, container creation, container replacement or supervision is added.
|
||||
|
||||
During demote/rejoin, PostgreSQL may be stopped while its container remains
|
||||
available. Ordinary stop still stops PostgreSQL before asking systemd to stop
|
||||
the container service. These ordering rules remain in the PostgreSQL lifecycle
|
||||
functions; the adapters do not reorder them.
|
||||
|
||||
## Preservation and validation
|
||||
|
||||
The original 29 functions are retained by name. Some now call adapter functions,
|
||||
so their literal hashes differ. `tests/verify_refactor.py` reverses only the
|
||||
explicit command-boundary substitutions and compares every remaining function
|
||||
byte with the immutable reference. It also retains all 22 differential lifecycle
|
||||
cases and verifies original metadata plus the four accumulated added parameters
|
||||
(freshness plus three selectors). Existing scoring tests use the actual engine
|
||||
loader and still exercise the original handlers against fake external boundaries.
|
||||
|
||||
`tests/test_plugin_engine.py` checks argument fidelity (including quotes, spaces,
|
||||
newlines and literal shell syntax), lifecycle ownership, exit-code propagation,
|
||||
timeout placement, application dispatch, metadata discovery, missing/incompatible
|
||||
plugins, and unsupported/path-like selections. It also checks that PostgreSQL
|
||||
modules no longer invoke Podman/systemd or service-user execution directly.
|
||||
|
||||
Run:
|
||||
|
||||
```sh
|
||||
python3 tests/verify_refactor.py
|
||||
python3 tests/test_plugin_engine.py
|
||||
python3 tests/test_promotion_scores.py
|
||||
python3 tests/test_disconnected_state.py
|
||||
python3 tests/test_promotion_scheduler.py
|
||||
python3 tests/test_promotion_cleanup.py
|
||||
python3 tests/verify_promotion_sql.py
|
||||
```
|
||||
|
||||
The last command uses only its own disposable PostgreSQL container. RPM checks
|
||||
include the plugin tests and verify discovery from the staged installation and
|
||||
an unrelated working directory. No live HA transitions are part of this branch's
|
||||
validation. Live lab authorization and actual configuration comparison remain
|
||||
necessary before deployment.
|
||||
|
||||
## Follow-on work
|
||||
|
||||
After this extraction passes lab validation, a separate change can add direct
|
||||
Podman lifecycle management for pre-created maintenance containers. That work
|
||||
must define exit/restart behavior and sole lifecycle ownership. Container creation,
|
||||
RA-coordinated standby provisioning, a second runtime or another application are
|
||||
separate extensions. The current PostgreSQL plugin is intentionally still the
|
||||
only supported application; this is not yet a general multi-application framework.
|
||||
|
||||
## Results for this extraction
|
||||
|
||||
Passed: 22 original differential lifecycle cases, 25 scoring tests, 16 state/monitor
|
||||
tests, 11 scheduler tests, 4 preference-planner tests, 11 engine contract tests,
|
||||
and all 9 PostgreSQL 18 SQL scenarios. Fedora 43 RPM build, disposable installation,
|
||||
15-file ownership/mode/payload checks, and the installed state-helper CLI passed.
|
||||
Folder-layout build log: `/tmp/mc-plugin-folders-rpm.log`; development artifacts:
|
||||
`dist/plugin-folders/`. Those build and offline checks did not run live HA transitions; subsequent
|
||||
operator-run lab transitions are recorded below.
|
||||
|
||||
The core still retains the existing score vocabulary and freshness-parameter
|
||||
validation for this sole supported PostgreSQL profile. API 1 is an internal
|
||||
extraction interface, not a stable third-party plugin SDK. Generalizing these
|
||||
remaining policy assumptions should be driven by a second real application,
|
||||
not assumed complete by this milestone.
|
||||
|
||||
Basic ELITEDESK installation checks subsequently passed with score 100; all 15
|
||||
installed files match the candidate. A controlled ELITEDESK standby stop/start subsequently passed according to
|
||||
operator-supplied output: stopped score -INFINITY, resumed streaming score 100,
|
||||
same database and new postmaster. The operator subsequently reported successful controlled switchovers in both
|
||||
directions and a rewind-only rejoin on ELITEDESK; see [role test results](lab-role-tests.md).
|
||||
Fencing, Pacemaker-managed transitions and divergent-data rewind remain untested
|
||||
with this extraction.
|
||||
See [the lab record](lab-plugin-basic-test.md) for evidence attribution.
|
||||
Reference in New Issue
Block a user