david/ocf-managed-container
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add PostgreSQL HA OCF agent modular baseline

Browse Source
This commit is contained in:
Lyncoln
2026-09-12 01:28:33 -04:00
commit 4a358ae0f6
30 changed files with 7119 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
notes/REFACTOR-PLAN.md
+101
View File
@@ -0,0 +1,101 @@
# Refactor inventory and module plan
Prepared before implementation. The frozen reference is authoritative.
## Parameters and globals
| OCF parameter | Default | Runtime global |
| --- | --- | --- |
| `service_user` | `david` | `SERVICE_USER` |
| `service_name` | `postgres.service` | `SERVICE_NAME` |
| `container_name` | `postgres` | `CONTAINER_NAME` |
| `pg_os_user` | `postgres` | `PG_OS_USER` |
| `db_user` | `david` | `DB_USER` |
| `db_name` | `david` | `DB_NAME` |
| `rejoin_source_host` | `` | `REJOIN_SOURCE_HOST` |
| `rejoin_source_port` | `5432` | `REJOIN_SOURCE_PORT` |
| `replication_user` | `replicator` | `REPLICATION_USER` |
| `replication_slot` | `` | `REPLICATION_SLOT` |
| `application_name` | `` | `APPLICATION_NAME` |
| `rejoin_allow_basebackup` | `false` | `REJOIN_ALLOW_BASEBACKUP` |
`application_name` falls back to `hostname -s`. Each parameter has an OCF_RESKEY input and an OCF_RESKEY_*_default global. Other framework globals: OCF_ROOT, OCF_FUNCTIONS_DIR, OCF_RUNNING_PROMOTED (normalized from OCF_RUNNING_MASTER), OCF_SUCCESS, OCF_NOT_RUNNING, OCF_ERR_GENERIC, OCF_ERR_CONFIGURED, OCF_ERR_INSTALLED, OCF_ERR_UNIMPLEMENTED, OCF_RESOURCE_INSTANCE. The validate_all loop also assigns the existing nonlocal variable cmd.
Notification inputs: OCF_RESKEY_CRM_meta_notify_{type,operation,promote_uname,demote_uname,promoted_uname,unpromoted_uname}. State persists under /run/loxthelion; demote accepts matching pre-demote state aged 0–120 seconds.
Container inputs: PGDATA, POSTGRES_REPLICATION_PASSWORD, POSTGRES_REPLICATION_USER, POSTGRES_HA_SLOTS. Container command locals include BINDIR, blocks and the explicitly injected RA_PRIMARY_HOST, RA_PRIMARY_PORT, RA_REPLICATION_USER, RA_REPLICATION_SLOT, RA_APPLICATION_NAME. PGPASSWORD is exported only in the container authentication subprocess. The rootless wrapper sets HOME and XDG_RUNTIME_DIR for that subprocess.
## Actions
Dispatcher: start, stop, monitor (status alias), promote, demote, notify, rejoin, meta-data, validate-all. Start/promote/demote/rejoin call validate_all first. Metadata advertises eight actions and omits rejoin/status; preserve this exactly.
## Module ownership and dependencies
All modules are sourced once by the entrypoint before dispatch. common initializes original defaults. Source order: common, container, postgres, auth, replication, notify, promote, demote, rejoin. Functions resolve dependencies when invoked; common validation can therefore call container functions without a source cycle.
Initial extraction retains original function names and bodies to enable exact comparison, including existing unused service_home. Namespacing and recovery deduplication are deferred from this first mechanical extraction so it can serve as a verified modular baseline. No shell option, SQL, quoting, log, or error-handling cleanup is included.
### common.sh
- `meta_data` → external commands / OCF library only
- `validate_all` → `service_uid`, `as_service_user`
### container.sh
- `service_uid` → external commands / OCF library only
- `service_home` → external commands / OCF library only
- `as_service_user` → external commands / OCF library only
- `service_running` → `as_service_user`
- `container_running` → `as_service_user`
### postgres.sh
- `pg_command` → `as_service_user`
- `pgdata_path` → `pg_command`
- `pg_running` → `pg_command`
- `pg_ready` → `as_service_user`
- `pg_role` → `as_service_user`
- `pg_wal_receiver_status` → `as_service_user`
- `managed_monitor` → `service_running`, `container_running`, `pg_running`, `pg_ready`, `pg_role`
- `managed_start` → `as_service_user`, `service_running`, `container_running`, `pg_command`, `pg_running`, `managed_monitor`
- `managed_stop` → `as_service_user`, `service_running`, `container_running`, `pg_command`, `pg_running`
### auth.sh
- `pg_replication_exec` → `as_service_user`
- `ensure_rewind_privileges` → `pg_command`
### replication.sh
- `pg_remote_query` → `pg_replication_exec`
- `ensure_local_ha_slots` → `pg_command`
- `ensure_remote_slot` → `pg_replication_exec`
- `candidate_caught_up` → `pg_command`
- `candidate_is_standby` → `pg_replication_exec`
### notify.sh
- `demote_target` → external commands / OCF library only
- `managed_notify` → external commands / OCF library only
### promote.sh
- `managed_promote` → `pg_command`, `pg_role`, `managed_monitor`, `ensure_rewind_privileges`, `ensure_local_ha_slots`
### demote.sh
- `node_to_addr` → external commands / OCF library only
- `managed_demote` → `as_service_user`, `service_running`, `container_running`, `pg_command`, `pg_running`, `pg_role`, `pg_wal_receiver_status`, `managed_monitor`, `node_to_addr`, `candidate_caught_up`, `demote_target`, `ensure_remote_slot`, `candidate_is_standby`
### rejoin.sh
- `managed_rejoin` → `as_service_user`, `service_running`, `container_running`, `pg_command`, `pg_replication_exec`, `pgdata_path`, `pg_running`, `pg_role`, `pg_remote_query`, `pg_wal_receiver_status`, `ensure_remote_slot`
## Incremental implementation and validation
1. Extract verbatim functions to the modules above in a new refactored/ tree; preserve both reference files.
2. Keep original OCF initialization and dispatcher; add script-relative module loading with a configured-error failure if any module cannot load.
3. Compare every function body, original defaults and dispatcher against the reference; run bash -n on every shell file.
4. Compare real meta-data output and parsed XML; exercise validate-all and record environmental limitations.
5. Exercise isolated action scenarios using mocked external boundaries; do not operate the live cluster.
6. Report hashes, install commands, and the operator-run lab comparison sequence.
notes/REFACTOR-REPORT.md
+137
View File
@@ -0,0 +1,137 @@
# PostgreSQL HA resource-agent modular refactor
The refactored agent is in `refactored/managed-container`, with nine sourced modules beside it. Both reference copies remain unchanged at SHA256 `5eae1f3dba0586dc06105e4dd7a9b8809a5c9961452abe7654dcfce77c3c5ce2`.
## Final file tree
```text
refactored/
managed-container
managed-container.d/
common.sh
container.sh
postgres.sh
auth.sh
replication.sh
notify.sh
promote.sh
demote.sh
rejoin.sh
notes/
REFACTOR-PLAN.md
REFACTOR-REPORT.md
REFACTOR-SHA256SUMS
tests/
verify_refactor.py
```
The SHA256 manifest lists every delivered code, test, and documentation file (excluding the manifest itself). Existing context files are unchanged.
## Function assignments
| Module | Functions |
| --- | --- |
| common.sh | meta_data, validate_all; original parameter defaults and runtime globals |
| container.sh | service_uid, service_home, as_service_user, service_running, container_running |
| postgres.sh | pg_command, pgdata_path, pg_running, pg_ready, pg_role, pg_wal_receiver_status, managed_monitor, managed_start, managed_stop |
| auth.sh | pg_replication_exec, ensure_rewind_privileges |
| replication.sh | pg_remote_query, ensure_local_ha_slots, ensure_remote_slot, candidate_caught_up, candidate_is_standby |
| notify.sh | demote_target, managed_notify |
| promote.sh | managed_promote |
| demote.sh | node_to_addr, managed_demote |
| rejoin.sh | managed_rejoin |
The entrypoint retains OCF initialization and the original dispatcher. See REFACTOR-PLAN.md for parameters, globals, function dependencies, and source order.
## Validation results
Run from the workspace root:
```bash
python3 tests/verify_refactor.py
sha256sum -c notes/REFACTOR-SHA256SUMS
```
- `bash -n`: entrypoint and all nine modules pass.
- All 29 original function definitions are byte-identical. Original defaults/global initialization and dispatcher are unchanged. This also preserves function/variable references inside original code.
- Real OCF-library metadata execution succeeds from `/tmp`; output is byte-identical to the reference and parses as XML with the same 12 parameters and eight advertised actions.
- All modules load in the installed OCF-library environment. The loader uses `command dirname` because this OCF library defines a dirname shell function that does not support `--`.
- Missing module directory produces OCF_ERR_CONFIGURED (6).
- 22 isolated differential scenarios pass, comparing return codes, stdout, and stderr: eight monitor states; three start cases; two demote cases; rejoin while PostgreSQL is running; existing-primary promotion reconciliation; already-stopped stop; and six validate-all paths.
- Direct `validate-all` on both original and refactored agents returns 6 with `User service 'postgres.service' cannot be found for david`. The execution environment cannot validate the target service. Stubbed validation success and missing user/command/runtime/service/Podman failure paths all match.
- No `.pgpass`, PGPASSFILE, or host environment-file sourcing was introduced into the RA.
- Both frozen and working reference hashes still match the handoff.
No live cluster actions or destructive HA tests were executed. Offline comparisons establish extraction equivalence; they do not replace controlled live validation of a multi-file installation.
## Behavior differences and limits
No intended action behavior differences. Function bodies, SQL, quoting, log messages, OCF return paths, authentication, and recovery flow are preserved exactly. The new packaging requires all nine modules beside the executable. Missing/unloadable modules now fail explicitly with configured error. The only added persistent shell global is MC_MODULE_DIR; the loader unsets its loop variable MC_MODULE.
Original function names and duplicated demote/rejoin recovery configuration remain in this first modular extraction. Namespacing and deduplication were preferences in the handoff; retaining them makes exact body equivalence checkable. The existing unused service_home helper remains too. No global strict-mode options were added. The executable should be installed as a regular file at the public OCF path, with modules in its sibling managed-container.d directory; a symlink elsewhere does not automatically locate modules at its target.
The bootstrap script's existing pgpass handling and the Quadlet host environment-file injection are outside this RA refactor and were not changed. The metadata continues to omit the dispatched rejoin action and status alias, matching the reference.
## Installation commands for the second lab node
Target the handoff's standby, DAVID-ELITEDESK (192.168.1.6). These commands are provided for operator execution and have not been run. Run the first block from the workspace root on the machine holding these files. The staging path must be unused; it is created with mkdir so an existing staging directory stops the sequence.
```bash
set -e
ssh david@192.168.1.6 'mkdir /tmp/managed-container-refactor'
scp -r refactored david@192.168.1.6:/tmp/managed-container-refactor/
```
Then log into DAVID-ELITEDESK and run the following block. Confirm the saved lab roles still hold before applying. Leave maintenance mode enabled throughout.
```bash
ssh david@192.168.1.6
```
On DAVID-ELITEDESK:
```bash
set -e
sudo pcs property config maintenance-mode
# Confirm maintenance-mode=true in the output before continuing.
sudo bash -n /tmp/managed-container-refactor/refactored/managed-container
for module in /tmp/managed-container-refactor/refactored/managed-container.d/*.sh; do
sudo bash -n "$module"
done
mc_backup_dir="/var/backups/managed-container-$(date -u +%Y%m%d-%H%M%S)"
sudo mkdir -m 0700 "$mc_backup_dir"
sudo cp -a /usr/lib/ocf/resource.d/loxthelion/managed-container "$mc_backup_dir/"
if sudo test -d /usr/lib/ocf/resource.d/loxthelion/managed-container.d; then
sudo cp -a /usr/lib/ocf/resource.d/loxthelion/managed-container.d "$mc_backup_dir/"
fi
sudo install -d -o root -g root -m 0755 /usr/lib/ocf/resource.d/loxthelion/managed-container.d
sudo install -o root -g root -m 0644 \
/tmp/managed-container-refactor/refactored/managed-container.d/*.sh \
/usr/lib/ocf/resource.d/loxthelion/managed-container.d/
sudo install -o root -g root -m 0755 \
/tmp/managed-container-refactor/refactored/managed-container \
/usr/lib/ocf/resource.d/loxthelion/managed-container
sudo restorecon -RF /usr/lib/ocf/resource.d/loxthelion/managed-container \
/usr/lib/ocf/resource.d/loxthelion/managed-container.d
sudo /usr/lib/ocf/resource.d/loxthelion/managed-container meta-data
sudo /usr/lib/ocf/resource.d/loxthelion/managed-container validate-all
printf 'Original installation saved at %s\n' "$mc_backup_dir"
```
Module files must be root-owned and not writable by the service user because the RA sources them as root. Install modules before the entrypoint, with resource management still disabled. No service restart is required just to replace the agent files. No commands here disable maintenance mode.
## Recommended behavioral comparison sequence
Perform in the isolated lab with maintenance mode retained and a recoverable data baseline. Compare original and refactored agents from equivalent initial states, recording OCF return codes, logs, service/container state, PostgreSQL role, receiver status, slot state and data-directory names. Do not invoke both implementations sequentially against already-mutated state and call that an equivalence test.
1. Validate metadata/parameters and validate-all on the target node. Run monitor on healthy primary and standby (8 and 0); verify stopped cases return 7.
2. Verify standby start succeeds, primary start is refused, and missing standby.signal prevents startup of stopped PostgreSQL.
3. Verify full stop shuts down PostgreSQL and the maintenance service/container, then restore the standby setup.
4. Promote a healthy standby and repeat promotion on the primary; check configured physical slots and all four rewind grants without deleting inactive slots.
5. Reproduce controlled switchover using the proven lab orchestration: persist a fresh matching pre-demote notification, resolve the successor through Corosync ring0, verify standby/catch-up checks, and coordinate successor promotion so streaming can complete. Repeat in the opposite direction. Keep the maintenance container alive throughout demote.
6. Rejoin a stopped divergent node using successful pg_rewind; verify normalized recovery settings, permissions, recovery=true and streaming. Verify rejoin refuses running PostgreSQL.
7. With a recoverable lab baseline, force rewind failure with fallback disabled; verify failure and no basebackup replacement.
8. Enable fallback for the explicit lab test; force rewind failure and successful basebackup. Verify old PGDATA preservation and streaming standby recovery.
9. Force basebackup failure; verify partial-copy quarantine, restoration of canonical old PGDATA, PostgreSQL stopped, and failure returned. Also exercise source-role/slot/container/free-space preflight failures.
10. Restore the intended DAVID-DV6 primary / DAVID-ELITEDESK standby topology and verify maintenance mode remains true. Decide separately when to deploy beyond this lab.
notes/REFACTOR-SHA256SUMS
+13
View File
@@ -0,0 +1,13 @@
6bdc385dff0e66de372f802a577f67d852f7a2a0e45610865f558ee3b93adf20 notes/REFACTOR-PLAN.md
08135e93b355d2f8239d9abbe0297eafdd6109d025a27d13ab35ccd74811e03a notes/REFACTOR-REPORT.md
c91f09adfdeb45a3be1a72df3f5ca5b02fd4555dec4397ad944eab439ace0b89 refactored/managed-container
93191771d3cb8aefadb2a16d2f89d64a8b6c584071a473439be818237bdf7f17 refactored/managed-container.d/auth.sh
e0bb13e9fda84ba7b657b2e774c08fa64988d463f45c699681e0e6ce49109440 refactored/managed-container.d/common.sh
ac6bc363a2301f1491d162ef609103dbfbceb3b099a3556614c6165952309374 refactored/managed-container.d/container.sh
0af3bb362a303017372c631354ba94ef9e723bf14e64d2eff91b6662e54a5be0 refactored/managed-container.d/demote.sh
6a98fcdda82cea9365db95b4ee779be5c567b3d846f48b90fbc8c254af697670 refactored/managed-container.d/notify.sh
310132d8032891727076295f39548b1cc0e51a10301a0ee4d934161f4f903e3e refactored/managed-container.d/postgres.sh
3085ec9b4410905ab9d4370786ed776eac3470a0415aa1b4f608b083945aeb58 refactored/managed-container.d/promote.sh
21beea72bcb7f24a51c17ea48666ac87df9436702ad77b166b45f4647e28e8ad refactored/managed-container.d/rejoin.sh
6b2021887e3b4cd6a12490e7a8dff98320b2133b472ab25a22a0eead46137547 refactored/managed-container.d/replication.sh
e3bc103b61b317e91d28260cc7bb86a3cc8390854762d6c5b9f325a1a8b1c362 tests/verify_refactor.py
notes/SHA256SUMS
+5
View File
@@ -0,0 +1,5 @@
5eae1f3dba0586dc06105e4dd7a9b8809a5c9961452abe7654dcfce77c3c5ce2 /home/david/codex-postgres-ha-refactor/reference/managed-container
5eae1f3dba0586dc06105e4dd7a9b8809a5c9961452abe7654dcfce77c3c5ce2 /home/david/codex-postgres-ha-refactor/reference/managed-container.pre-refactor-5eae1f3
3f1a15f7054fb69b9aa92204de30306fd6053c63560310f4231f314b148e2dcf /home/david/codex-postgres-ha-refactor/runtime-context/postgres-ha-bootstrap
21d9c28704d0dc4a0cb49b8aff553b337455ccb1bd708518042419b85f6ecf12 /home/david/codex-postgres-ha-refactor/runtime-context/postgres.container