ocf-managed-container/docs/plugin-engine.md

Podman plugin engine: first extraction

Development baseline: freeze commit 488a167. This branch extracts the existing PostgreSQL/Podman/systemd implementation. It is installed on DAVID-ELITEDESK in the lab only; see lab-plugin-basic-test.md. Production hash provenance is still pending as described in testing.md. Quadlet, bootstrap and cluster configuration remain unchanged. The lab RA files were replaced, and basic monitor/start checks refreshed standby scoring without restarting PostgreSQL. No production installation has occurred.

Scope and supported configuration

The default and only supported combination is:

application=postgresql
container_runtime=podman
lifecycle_manager=systemd

These are advertised OCF instance parameters, not Pacemaker meta-attributes. Existing resource configurations need no new parameters. Missing or empty selectors use the defaults. Explicit unsupported values, including Docker or direct-runtime lifecycle, fail with OCF_ERR_CONFIGURED (6) before operations. The entrypoint/provider and every existing parameter default remain unchanged.

The engine loads a fixed allowlist of packaged files beside the entrypoint. There is no arbitrary plugin pathname, dynamic shell evaluation, network plugin installation, or fallback to another runtime. Every implementation declares API version 1; missing files, incompatible versions and missing required hooks fail configuration. Packaged code is trusted/root-owned, as was the original RA.

Boundaries

Pacemaker OCF action
  -> shared scoring/action wrappers
     -> applications/postgresql/plugin.sh
        -> existing PostgreSQL lifecycle and recovery modules
           -> runtimes/podman.sh (inspect / exec / info)
           -> lifecycles/systemd.sh (user service lifecycle)
              -> unchanged as_service_user privilege boundary

engine.sh selects and checks plugins and adds selector metadata. common.sh contains the existing score-publication and action wrappers, now calling the application interface. container.sh retains service-user access plus compatibility health helpers that delegate to the selected adapters.

applications/postgresql/plugin.sh owns the existing PostgreSQL defaults, metadata, validation and seven PostgreSQL/HA modules. Application modules and the state helper live together under applications/postgresql/; runtime and lifecycle adapters occupy their own runtimes/ and lifecycles/ directories. The loader uses fixed relative paths. Future applications will have their own metadata and dependency validation rather than inherit PostgreSQL assumptions.

The packaged layout is:

managed-container.d/
  engine.sh
  common.sh
  container.sh
  applications/postgresql/
    plugin.sh
    postgres.sh, auth.sh, replication.sh, notify.sh
    promote.sh, demote.sh, rejoin.sh
    promotion_state.py
  lifecycles/systemd.sh
  runtimes/podman.sh

The PostgreSQL plugin sets its own MC_APPLICATION_DIR for sibling modules and the Python state helper. Host evidence remains at its existing /run location; only packaged code paths changed. The public RA entrypoint is unchanged.

Interface contract, API 1

All interfaces are sourced Bash functions in a single RA invocation. Arguments are passed as arrays using "$@"; adapters do not concatenate or evaluate command strings. Existing PostgreSQL shell snippets remain explicit arguments to sh -c; this extraction does not redesign or broaden their trusted configuration inputs.

Interface	Contract
mc_runtime_exec ARGS...	Execute in the container through the service owner; preserve stdout, stderr, and exact exit status.
mc_runtime_exec_bounded DURATION ARGS...	Same execution with timeout placed inside the service-user boundary; retain kill-after=1s. Current scoring callers pass 3s.
mc_runtime_inspect ARGS...	Read container properties through the same runtime/user boundary.
mc_runtime_info ARGS...	Validate runtime access through the service owner.
mc_lifecycle_start SERVICE	Start the configured user-systemd service.
mc_lifecycle_stop SERVICE	Stop that service, preserving failures.
mc_lifecycle_running SERVICE	systemctl --user is-active --quiet.
mc_lifecycle_cat SERVICE	Inspect service definition for existing validation.
mc_application_monitor	Return the application's normalized OCF health/role code.
mc_application_standby_score	Emit the eligible standby score, or fail; PostgreSQL owns evidence and freshness.
mc_application_action ACTION	Dispatch only start/stop/promote/demote/rejoin to the existing application functions.
mc_application_validate	Run existing application/configuration validation.
mc_application_metadata	Return application metadata without runtime or CIB access.
mc_application_notify	Process the existing transition notification.

The runtime exec argument grammar currently follows the existing callers: --user USER, optional --env KEY=VALUE entries, container name, command and arguments. Inspect retains the existing Go-template argument. This is a narrow Podman adapter contract, not a claim that every future runtime accepts it. A second runtime must explicitly implement and test those semantics.

Normalization and state

For this extraction the existing PostgreSQL monitor is the normalizer: it turns PostgreSQL's role/health observations into OCF codes. No extra JSON protocol or new independently maintained role state is introduced.

Observation	Monitor code
Healthy standby	0 (Unpromoted)
Healthy primary	8 (Promoted)
Container absent/stopped, or PostgreSQL stopped	7 (not running)
PostgreSQL unready or failed/invalid role query	1 (generic error)

Container availability, application health/role and promotion eligibility remain separate concepts. Container-running/PostgreSQL-stopped is a legitimate maintenance condition, but is not OCF start success. A healthy standby can have score -INFINITY without losing its Unpromoted role report. Core score validation still replaces malformed/failed score output with -INFINITY.

All existing scores, freshness checks, persisted identity/LSNs, timeout locations, notification handling, fencing assumptions, standby-only start and manual fenced rejoin rules are preserved. Direct application-plugin calls are internal; Pacemaker dispatch continues through the score/lifecycle wrappers.

Maintenance and lifecycle ownership

Quadlet remains responsible for the bootstrap entrypoint, Exec=sleep infinity, RunInit=true, stop signal/timeout, mounts, image, environment and systemd service policies. SuccessExitStatus=143 is systemd's classification of its managed service; the runtime exec adapter must still return 143 as an application-command failure. Restart=on-abnormal remains solely in the existing unit. No new restart loop, container creation, container replacement or supervision is added.

During demote/rejoin, PostgreSQL may be stopped while its container remains available. Ordinary stop still stops PostgreSQL before asking systemd to stop the container service. These ordering rules remain in the PostgreSQL lifecycle functions; the adapters do not reorder them.

Preservation and validation

The original 29 functions are retained by name. Some now call adapter functions, so their literal hashes differ. tests/verify_refactor.py reverses only the explicit command-boundary substitutions and compares every remaining function byte with the immutable reference. It also retains all 22 differential lifecycle cases and verifies original metadata plus the four accumulated added parameters (freshness plus three selectors). Existing scoring tests use the actual engine loader and still exercise the original handlers against fake external boundaries.

tests/test_plugin_engine.py checks argument fidelity (including quotes, spaces, newlines and literal shell syntax), lifecycle ownership, exit-code propagation, timeout placement, application dispatch, metadata discovery, missing/incompatible plugins, and unsupported/path-like selections. It also checks that PostgreSQL modules no longer invoke Podman/systemd or service-user execution directly.

Run:

python3 tests/verify_refactor.py
python3 tests/test_plugin_engine.py
python3 tests/test_promotion_scores.py
python3 tests/test_disconnected_state.py
python3 tests/test_promotion_scheduler.py
python3 tests/test_promotion_cleanup.py
python3 tests/verify_promotion_sql.py

The last command uses only its own disposable PostgreSQL container. RPM checks include the plugin tests and verify discovery from the staged installation and an unrelated working directory. No live HA transitions are part of this branch's validation. Live lab authorization and actual configuration comparison remain necessary before deployment.

Follow-on work

After this extraction passes lab validation, a separate change can add direct Podman lifecycle management for pre-created maintenance containers. That work must define exit/restart behavior and sole lifecycle ownership. Container creation, RA-coordinated standby provisioning, a second runtime or another application are separate extensions. The current PostgreSQL plugin is intentionally still the only supported application; this is not yet a general multi-application framework.

Results for this extraction

Passed: 22 original differential lifecycle cases, 25 scoring tests, 16 state/monitor tests, 11 scheduler tests, 4 preference-planner tests, 11 engine contract tests, and all 9 PostgreSQL 18 SQL scenarios. Fedora 43 RPM build, disposable installation, 15-file ownership/mode/payload checks, and the installed state-helper CLI passed. Folder-layout build log: /tmp/mc-plugin-folders-rpm.log; development artifacts: dist/plugin-folders/. Those build and offline checks did not run live HA transitions; subsequent operator-run lab transitions are recorded below.

The core still retains the existing score vocabulary and freshness-parameter validation for this sole supported PostgreSQL profile. API 1 is an internal extraction interface, not a stable third-party plugin SDK. Generalizing these remaining policy assumptions should be driven by a second real application, not assumed complete by this milestone.

Basic ELITEDESK installation checks subsequently passed with score 100; all 15 installed files match the candidate. A controlled ELITEDESK standby stop/start subsequently passed according to operator-supplied output: stopped score -INFINITY, resumed streaming score 100, same database and new postmaster. The operator subsequently reported successful controlled switchovers in both directions and a rewind-only rejoin on ELITEDESK; see role test results. Fencing, Pacemaker-managed transitions and divergent-data rewind remain untested with this extraction. See the lab record for evidence attribution.