ocf-managed-container/HANDOFF.md

# PostgreSQL HA OCF Resource Agent — Codex Refactor Handoff

## Objective

Perform a behavior-preserving modular refactor of the proven PostgreSQL HA
OCF resource agent contained in:

    reference/managed-container

Known-good SHA256:

    5eae1f3dba0586dc06105e4dd7a9b8809a5c9961452abe7654dcfce77c3c5ce2

A frozen identical reference copy is:

    reference/managed-container.pre-refactor-5eae1f3

DO NOT MODIFY THE FROZEN REFERENCE COPY.

The objective is maintainability and modularity, not redesign.

## Architecture

Control path:

    Pacemaker
      -> OCF resource agent
      -> runuser david
      -> rootless systemd / Podman
      -> persistent PostgreSQL maintenance container
      -> PostgreSQL

The maintenance container PID 1 is intentionally `sleep infinity`.

PostgreSQL itself is independently started/stopped with `pg_ctl`.

The maintenance container must remain alive during demote/rejoin operations so
the RA can perform:

- SQL checks
- pg_rewind
- pg_basebackup
- replication slot management
- recovery configuration

PostgreSQL storage is NOT DRBD-backed.

Quadlet owns container construction and environment injection.

The RA may consume environment variables from the already-running container,
but must not gain a dependency on parsing a host-side env file.

## Public OCF entrypoint

The public executable must remain:

    /usr/lib/ocf/resource.d/loxthelion/managed-container

Preferred modular layout:

    managed-container
    managed-container.d/
        common.sh
        container.sh
        postgres.sh
        auth.sh
        replication.sh
        promote.sh
        demote.sh
        rejoin.sh
        notify.sh

Helper files should normally be sourced shell modules rather than separate
executables.

Namespaced helper functions such as these are preferred:

    mc_container_*
    mc_pg_*
    mc_auth_*
    mc_replication_*
    mc_promote_*
    mc_demote_*
    mc_rejoin_*

## Preserve these actions

- start
- stop
- monitor
- promote
- demote
- notify
- rejoin
- meta-data
- validate-all

Do not change OCF return codes or action semantics.

## start()

start must NOT promote PostgreSQL.

It should:

- succeed for an already healthy standby
- refuse an already-running primary
- start/ensure the maintenance container where appropriate
- require standby.signal for stopped PostgreSQL
- start PostgreSQL with pg_ctl
- succeed only as Unpromoted

Promotion belongs exclusively to promote().

## stop()

stop performs a full resource shutdown:

1. stop PostgreSQL cleanly
2. stop the rootless postgres.service
3. verify service/container disappear

Demote and rejoin intentionally behave differently and preserve the maintenance
container.

## promote()

Preserve current behavior.

Promotion must:

- promote a standby
- verify primary role
- ensure configured local physical HA slots
- ensure pg_rewind privileges

POSTGRES_HA_SLOTS is obtained from the running container environment.

Do not automatically delete inactive replication slots.

## pg_rewind privileges

Preserve grants for:

- pg_catalog.pg_ls_dir(text, boolean, boolean)
- pg_catalog.pg_stat_file(text, boolean)
- pg_catalog.pg_read_binary_file(text)
- pg_catalog.pg_read_binary_file(text, bigint,bigint,boolean)

## demote()

The proven controlled demotion logic must remain behaviorally identical.

It:

1. determines the current role
2. retrieves the persisted pre-demote target from Pacemaker notify state
3. resolves that Corosync node to its ring0 address
4. determines the local physical replication slot
5. verifies the target is a standby
6. verifies the target is caught up
7. ensures this node's slot on the destination
8. stops the old primary
9. keeps the maintenance container alive
10. rewrites recovery settings
11. creates standby.signal
12. starts PostgreSQL
13. verifies recovery mode
14. verifies WAL receiver is streaming

Do not replace Corosync ring0 addressing with normal hostname resolution.

## notify()

Pacemaker notification state is persisted because notification environment
variables may no longer exist by demote time.

Preserve this mechanism.

## Replication authentication

Runtime replication authentication comes from the running container environment.

POSTGRES_REPLICATION_PASSWORD is exposed inside the running container.

The RA sets PGPASSWORD only inside the container process executing libpq tools.

Preserve this design.

Do NOT:

- introduce .pgpass
- place passwords in connection strings
- parse a host environment file for runtime credentials

## rejoin()

Preconditions include:

- maintenance service alive
- maintenance container alive
- PostgreSQL stopped
- source reachable
- source confirmed primary
- physical replication slot ensured on source

Repair order:

1. pg_rewind
2. optional pg_basebackup fallback

## pg_basebackup fallback

Parameter:

    rejoin_allow_basebackup

Default:

    false

If rewind fails and fallback is disabled, fail safely.

If enabled:

1. revalidate source primary
2. revalidate PostgreSQL stopped
3. revalidate maintenance container alive
4. reensure replication slot
5. calculate approximate source size
6. check conservative free-space requirement
7. rename old canonical PGDATA to:

       $PGDATA.pre-basebackup-TIMESTAMP

8. create new canonical PGDATA
9. perform pg_basebackup

If basebackup fails:

- quarantine partial copy as:

      $PGDATA.failed-basebackup-TIMESTAMP

- restore the old PGDATA to its canonical pathname
- leave PostgreSQL stopped
- return failure

Never blindly:

    rm -rf "$PGDATA"

## Recovery normalization

After successful rewind or basebackup:

- normalize primary_conninfo
- normalize primary_slot_name
- create standby.signal
- apply expected permissions
- start PostgreSQL
- verify pg_is_in_recovery() = true
- verify WAL receiver state = streaming

Prefer extracting common recovery-configuration logic rather than duplicating it.

## Behavior already proven

The reference build has successfully passed:

- standby start
- promotion
- controlled demotion
- bidirectional switchover
- slot reconciliation after promotion
- rewind privilege reconciliation
- successful pg_rewind rejoin
- pg_rewind failure with fallback=false
- pg_rewind failure followed by successful pg_basebackup
- pg_rewind failure followed by failed pg_basebackup and successful rollback
- refusal to run rejoin while PostgreSQL is running

Do not change these semantics.

## Current lab

DAVID-DV6:
- primary

DAVID-ELITEDESK:
- standby
- streaming from 192.168.1.2
- slot ha_david_elitedesk

Pacemaker is intentionally in:

    maintenance-mode=true

Do NOT disable maintenance mode.

Do NOT touch production systems.

## Refactor procedure

Before editing:

1. read the complete reference implementation
2. inventory global variables
3. inventory OCF parameters
4. inventory action entrypoints
5. map helper/function dependencies
6. produce a proposed module dependency map

Then refactor incrementally.

Avoid changing shell semantics unless required to modularize the code.

Preserve quoting, SQL, command invocation, OCF error handling and logging.

## Validation

At minimum:

- bash -n main entrypoint
- bash -n every module
- meta-data action works
- validate-all works
- all modules resolve correctly in OCF execution environment
- no missing functions
- no missing variables
- same metadata parameters
- same OCF actions
- no .pgpass usage
- no host env-file parsing dependency
- frozen reference remains unchanged

Do NOT automatically perform destructive HA tests.

When finished, report:

1. final file tree
2. SHA256 of every file
3. functions assigned to each module
4. syntax validation results
5. metadata validation results
6. validate-all results
7. any behavior differences, ideally none
8. exact commands to install the refactored version on the second lab node
9. recommended behavioral comparison test sequence

The frozen source implementation is authoritative whenever this document and
the implementation differ.