Lyncoln
ec2077f979
Extract PostgreSQL, Podman runtime and systemd lifecycle adapters while preserving existing lifecycle and scoring behavior. Update RPM layout and add guarded operator-run lab harnesses. Record successful manual stop/start, promotion, demotion and rewind-only rejoin with pinned installed hashes. The unfenced lab does not validate automatic failover, Pacemaker-managed transitions or divergent-data rewind. Validation: 103 offline tests, original-function differential checks, checksum manifest and whitespace checks passed.
211 lines
11 KiB
Markdown
211 lines
11 KiB
Markdown
# Podman plugin engine: first extraction
|
|
|
|
Development baseline: freeze commit `488a167`. This branch extracts the existing PostgreSQL/Podman/systemd implementation.
|
|
It is installed on DAVID-ELITEDESK in the lab only; see lab-plugin-basic-test.md. Production
|
|
hash provenance is still pending as described in testing.md. Quadlet, bootstrap and cluster configuration remain unchanged. The lab RA files
|
|
were replaced, and basic monitor/start checks refreshed standby scoring without
|
|
restarting PostgreSQL. No production installation has occurred.
|
|
|
|
## Scope and supported configuration
|
|
|
|
The default and only supported combination is:
|
|
|
|
```text
|
|
application=postgresql
|
|
container_runtime=podman
|
|
lifecycle_manager=systemd
|
|
```
|
|
|
|
These are advertised OCF instance parameters, not Pacemaker meta-attributes.
|
|
Existing resource configurations need no new parameters. Missing or empty
|
|
selectors use the defaults. Explicit unsupported values, including Docker or
|
|
direct-runtime lifecycle, fail with OCF_ERR_CONFIGURED (6) before operations.
|
|
The entrypoint/provider and every existing parameter default remain unchanged.
|
|
|
|
The engine loads a fixed allowlist of packaged files beside the entrypoint.
|
|
There is no arbitrary plugin pathname, dynamic shell evaluation, network plugin
|
|
installation, or fallback to another runtime. Every implementation declares API
|
|
version 1; missing files, incompatible versions and missing required hooks fail
|
|
configuration. Packaged code is trusted/root-owned, as was the original RA.
|
|
|
|
## Boundaries
|
|
|
|
```text
|
|
Pacemaker OCF action
|
|
-> shared scoring/action wrappers
|
|
-> applications/postgresql/plugin.sh
|
|
-> existing PostgreSQL lifecycle and recovery modules
|
|
-> runtimes/podman.sh (inspect / exec / info)
|
|
-> lifecycles/systemd.sh (user service lifecycle)
|
|
-> unchanged as_service_user privilege boundary
|
|
```
|
|
|
|
`engine.sh` selects and checks plugins and adds selector metadata. `common.sh`
|
|
contains the existing score-publication and action wrappers, now calling the
|
|
application interface. `container.sh` retains service-user access plus
|
|
compatibility health helpers that delegate to the selected adapters.
|
|
|
|
`applications/postgresql/plugin.sh` owns the existing PostgreSQL defaults, metadata,
|
|
validation and seven PostgreSQL/HA modules. Application modules and the state
|
|
helper live together under
|
|
`applications/postgresql/`; runtime and lifecycle adapters occupy their own
|
|
`runtimes/` and `lifecycles/` directories. The loader uses fixed relative paths.
|
|
Future applications will have their own
|
|
metadata and dependency validation rather than inherit PostgreSQL assumptions.
|
|
|
|
The packaged layout is:
|
|
|
|
```text
|
|
managed-container.d/
|
|
engine.sh
|
|
common.sh
|
|
container.sh
|
|
applications/postgresql/
|
|
plugin.sh
|
|
postgres.sh, auth.sh, replication.sh, notify.sh
|
|
promote.sh, demote.sh, rejoin.sh
|
|
promotion_state.py
|
|
lifecycles/systemd.sh
|
|
runtimes/podman.sh
|
|
```
|
|
|
|
The PostgreSQL plugin sets its own `MC_APPLICATION_DIR` for sibling modules and
|
|
the Python state helper. Host evidence remains at its existing `/run` location;
|
|
only packaged code paths changed. The public RA entrypoint is unchanged.
|
|
|
|
## Interface contract, API 1
|
|
|
|
All interfaces are sourced Bash functions in a single RA invocation. Arguments
|
|
are passed as arrays using `"$@"`; adapters do not concatenate or evaluate command
|
|
strings. Existing PostgreSQL shell snippets remain explicit arguments to `sh -c`;
|
|
this extraction does not redesign or broaden their trusted configuration inputs.
|
|
|
|
| Interface | Contract |
|
|
| --- | --- |
|
|
| mc_runtime_exec ARGS... | Execute in the container through the service owner; preserve stdout, stderr, and exact exit status. |
|
|
| mc_runtime_exec_bounded DURATION ARGS... | Same execution with timeout placed inside the service-user boundary; retain kill-after=1s. Current scoring callers pass 3s. |
|
|
| mc_runtime_inspect ARGS... | Read container properties through the same runtime/user boundary. |
|
|
| mc_runtime_info ARGS... | Validate runtime access through the service owner. |
|
|
| mc_lifecycle_start SERVICE | Start the configured user-systemd service. |
|
|
| mc_lifecycle_stop SERVICE | Stop that service, preserving failures. |
|
|
| mc_lifecycle_running SERVICE | systemctl --user is-active --quiet. |
|
|
| mc_lifecycle_cat SERVICE | Inspect service definition for existing validation. |
|
|
| mc_application_monitor | Return the application's normalized OCF health/role code. |
|
|
| mc_application_standby_score | Emit the eligible standby score, or fail; PostgreSQL owns evidence and freshness. |
|
|
| mc_application_action ACTION | Dispatch only start/stop/promote/demote/rejoin to the existing application functions. |
|
|
| mc_application_validate | Run existing application/configuration validation. |
|
|
| mc_application_metadata | Return application metadata without runtime or CIB access. |
|
|
| mc_application_notify | Process the existing transition notification. |
|
|
|
|
The runtime exec argument grammar currently follows the existing callers:
|
|
`--user USER`, optional `--env KEY=VALUE` entries, container name, command and
|
|
arguments. Inspect retains the existing Go-template argument. This is a narrow
|
|
Podman adapter contract, not a claim that every future runtime accepts it.
|
|
A second runtime must explicitly implement and test those semantics.
|
|
|
|
## Normalization and state
|
|
|
|
For this extraction the existing PostgreSQL monitor is the normalizer: it turns
|
|
PostgreSQL's role/health observations into OCF codes. No extra JSON protocol or
|
|
new independently maintained role state is introduced.
|
|
|
|
| Observation | Monitor code |
|
|
| --- | ---: |
|
|
| Healthy standby | 0 (Unpromoted) |
|
|
| Healthy primary | 8 (Promoted) |
|
|
| Container absent/stopped, or PostgreSQL stopped | 7 (not running) |
|
|
| PostgreSQL unready or failed/invalid role query | 1 (generic error) |
|
|
|
|
Container availability, application health/role and promotion eligibility remain
|
|
separate concepts. Container-running/PostgreSQL-stopped is a legitimate
|
|
maintenance condition, but is not OCF start success. A healthy standby can have
|
|
score -INFINITY without losing its Unpromoted role report. Core score validation
|
|
still replaces malformed/failed score output with -INFINITY.
|
|
|
|
All existing scores, freshness checks, persisted identity/LSNs, timeout locations,
|
|
notification handling, fencing assumptions, standby-only start and manual fenced
|
|
rejoin rules are preserved. Direct application-plugin calls are internal;
|
|
Pacemaker dispatch continues through the score/lifecycle wrappers.
|
|
|
|
## Maintenance and lifecycle ownership
|
|
|
|
Quadlet remains responsible for the bootstrap entrypoint, `Exec=sleep infinity`,
|
|
`RunInit=true`, stop signal/timeout, mounts, image, environment and systemd service
|
|
policies. `SuccessExitStatus=143` is systemd's classification of its managed
|
|
service; the runtime exec adapter must still return 143 as an application-command
|
|
failure. `Restart=on-abnormal` remains solely in the existing unit. No new
|
|
restart loop, container creation, container replacement or supervision is added.
|
|
|
|
During demote/rejoin, PostgreSQL may be stopped while its container remains
|
|
available. Ordinary stop still stops PostgreSQL before asking systemd to stop
|
|
the container service. These ordering rules remain in the PostgreSQL lifecycle
|
|
functions; the adapters do not reorder them.
|
|
|
|
## Preservation and validation
|
|
|
|
The original 29 functions are retained by name. Some now call adapter functions,
|
|
so their literal hashes differ. `tests/verify_refactor.py` reverses only the
|
|
explicit command-boundary substitutions and compares every remaining function
|
|
byte with the immutable reference. It also retains all 22 differential lifecycle
|
|
cases and verifies original metadata plus the four accumulated added parameters
|
|
(freshness plus three selectors). Existing scoring tests use the actual engine
|
|
loader and still exercise the original handlers against fake external boundaries.
|
|
|
|
`tests/test_plugin_engine.py` checks argument fidelity (including quotes, spaces,
|
|
newlines and literal shell syntax), lifecycle ownership, exit-code propagation,
|
|
timeout placement, application dispatch, metadata discovery, missing/incompatible
|
|
plugins, and unsupported/path-like selections. It also checks that PostgreSQL
|
|
modules no longer invoke Podman/systemd or service-user execution directly.
|
|
|
|
Run:
|
|
|
|
```sh
|
|
python3 tests/verify_refactor.py
|
|
python3 tests/test_plugin_engine.py
|
|
python3 tests/test_promotion_scores.py
|
|
python3 tests/test_disconnected_state.py
|
|
python3 tests/test_promotion_scheduler.py
|
|
python3 tests/test_promotion_cleanup.py
|
|
python3 tests/verify_promotion_sql.py
|
|
```
|
|
|
|
The last command uses only its own disposable PostgreSQL container. RPM checks
|
|
include the plugin tests and verify discovery from the staged installation and
|
|
an unrelated working directory. No live HA transitions are part of this branch's
|
|
validation. Live lab authorization and actual configuration comparison remain
|
|
necessary before deployment.
|
|
|
|
## Follow-on work
|
|
|
|
After this extraction passes lab validation, a separate change can add direct
|
|
Podman lifecycle management for pre-created maintenance containers. That work
|
|
must define exit/restart behavior and sole lifecycle ownership. Container creation,
|
|
RA-coordinated standby provisioning, a second runtime or another application are
|
|
separate extensions. The current PostgreSQL plugin is intentionally still the
|
|
only supported application; this is not yet a general multi-application framework.
|
|
|
|
## Results for this extraction
|
|
|
|
Passed: 22 original differential lifecycle cases, 25 scoring tests, 16 state/monitor
|
|
tests, 11 scheduler tests, 4 preference-planner tests, 11 engine contract tests,
|
|
and all 9 PostgreSQL 18 SQL scenarios. Fedora 43 RPM build, disposable installation,
|
|
15-file ownership/mode/payload checks, and the installed state-helper CLI passed.
|
|
Folder-layout build log: `/tmp/mc-plugin-folders-rpm.log`; development artifacts:
|
|
`dist/plugin-folders/`. Those build and offline checks did not run live HA transitions; subsequent
|
|
operator-run lab transitions are recorded below.
|
|
|
|
The core still retains the existing score vocabulary and freshness-parameter
|
|
validation for this sole supported PostgreSQL profile. API 1 is an internal
|
|
extraction interface, not a stable third-party plugin SDK. Generalizing these
|
|
remaining policy assumptions should be driven by a second real application,
|
|
not assumed complete by this milestone.
|
|
|
|
Basic ELITEDESK installation checks subsequently passed with score 100; all 15
|
|
installed files match the candidate. A controlled ELITEDESK standby stop/start subsequently passed according to
|
|
operator-supplied output: stopped score -INFINITY, resumed streaming score 100,
|
|
same database and new postmaster. The operator subsequently reported successful controlled switchovers in both
|
|
directions and a rewind-only rejoin on ELITEDESK; see [role test results](lab-role-tests.md).
|
|
Fencing, Pacemaker-managed transitions and divergent-data rewind remain untested
|
|
with this extraction.
|
|
See [the lab record](lab-plugin-basic-test.md) for evidence attribution.
|