Lyncoln
ec2077f979
Extract PostgreSQL, Podman runtime and systemd lifecycle adapters while preserving existing lifecycle and scoring behavior. Update RPM layout and add guarded operator-run lab harnesses. Record successful manual stop/start, promotion, demotion and rewind-only rejoin with pinned installed hashes. The unfenced lab does not validate automatic failover, Pacemaker-managed transitions or divergent-data rewind. Validation: 103 offline tests, original-function differential checks, checksum manifest and whitespace checks passed.
126 lines
6.8 KiB
Markdown
126 lines
6.8 KiB
Markdown
> These production reports and source hashes describe the 488a167 freeze.
|
|
> The current undeployed extraction is described in [plugin-engine.md](plugin-engine.md).
|
|
|
|
# Validation and freeze-point evidence
|
|
|
|
## Reported production fencing and recovery test
|
|
|
|
Evidence source: operator-supplied production handoff received during the
|
|
2026-09-12 documentation review. The operator reports a successful real
|
|
three-node Pacemaker/Corosync test with SBD fencing. Codex did not execute or
|
|
independently witness this production test. Its exact start/end timestamps and
|
|
raw cluster logs were not supplied; no timestamps or measurements are invented.
|
|
|
|
1. david-eve was primary and was fenced successfully through SBD.
|
|
2. david-prodesk promoted automatically; the PostgreSQL VIP moved with it.
|
|
3. david-macbookpro remained/reconnected as a streaming standby.
|
|
4. david-eve rebooted with former-primary PGDATA. Ordinary resource start
|
|
correctly refused it because standby.signal was absent.
|
|
5. The operator manually started its maintenance container with PostgreSQL stopped.
|
|
6. The private rejoin action used the PostgreSQL VIP as rewind source.
|
|
pg_rewind detected divergence, found the common checkpoint and completed.
|
|
7. david-eve started as a streaming standby. Only after recovery, Pacemaker
|
|
cleanup allowed the cluster to adopt it as Unpromoted.
|
|
|
|
Reported final state:
|
|
|
|
| Node | Role | Score |
|
|
| --- | --- | ---: |
|
|
| david-prodesk | Promoted; PostgreSQL VIP local | 1000 |
|
|
| david-macbookpro | Unpromoted; asynchronous streaming | 100 |
|
|
| david-eve | Unpromoted; asynchronous streaming | 100 |
|
|
|
|
No PostgreSQL failed actions remained. The successful run demonstrates the
|
|
reported fencing/failover/VIP/rejoin sequence, not every possible partition,
|
|
latency, concurrent-write, or recovery failure. The intermediate disconnected
|
|
score and lost-transaction count were not supplied. The quoting/authentication
|
|
incident and required fix are documented in [operations](operations.md).
|
|
|
|
## Validated production snapshot — exact deployed hashes pending
|
|
|
|
The production outcome is operator-reported validated; **the byte-level production
|
|
snapshot is not yet established**. Exact hashes from david-prodesk,
|
|
david-macbookpro and david-eve have not been supplied or collected. Repository
|
|
hashes below must not be represented as deployed hashes. A later capture also
|
|
requires operator confirmation that those bytes are the ones used during the test.
|
|
|
|
| Artifact on each production node | Snapshot status |
|
|
| --- | --- |
|
|
| /usr/lib/ocf/resource.d/loxthelion/managed-container | Pending production SHA-256 |
|
|
| managed-container.d/*.sh (all nine modules) | Pending production SHA-256 |
|
|
| managed-container.d/promotion_state.py | Pending production SHA-256 |
|
|
| Running container /usr/local/sbin/postgres-ha-bootstrap | Pending production SHA-256 |
|
|
|
|
Run the following read-only commands on **each production node**, using the
|
|
rootless container owner for podman. They hash code, not environment or secrets:
|
|
|
|
```sh
|
|
hostname
|
|
date -u '+%Y-%m-%dT%H:%M:%SZ'
|
|
sha256sum /usr/lib/ocf/resource.d/loxthelion/managed-container \
|
|
/usr/lib/ocf/resource.d/loxthelion/managed-container.d/*.sh \
|
|
/usr/lib/ocf/resource.d/loxthelion/managed-container.d/promotion_state.py
|
|
podman exec postgres sha256sum /usr/local/sbin/postgres-ha-bootstrap
|
|
podman inspect --format '{{.Image}}' postgres
|
|
```
|
|
|
|
If permissions require it, run only the host sha256sum command with sudo. Do not
|
|
run rootful podman instead of the actual container owner. If the installed
|
|
bootstrap path/container name differs, record the actual path/name. Capture
|
|
node identity, UTC capture time, all filenames/digests, immutable image ID, and
|
|
whether any code changed since the test. Hash the running container's bootstrap;
|
|
a host snapshot or image tag alone is not equivalent. Do not dump environment,
|
|
secret files, or pgpass. Compare every node individually, preserving mismatches.
|
|
|
|
## Repository freeze snapshot (488a167)
|
|
|
|
The exact repository code hashes appear below and in the reviewed checksum
|
|
manifest. These identify the source freeze only, pending production comparison.
|
|
The bootstrap is a saved runtime-context artifact, not part of the RA RPM and
|
|
not yet confirmed identical to the production bootstrap.
|
|
|
|
| Repository file | SHA-256 |
|
|
| --- | --- |
|
|
| `refactored/managed-container` | `a4b775a0ddcabdb0558da9b34c7fe2a883c07d0ef308c1c7224526abf53f63a0` |
|
|
| `refactored/managed-container.d/auth.sh` | `93191771d3cb8aefadb2a16d2f89d64a8b6c584071a473439be818237bdf7f17` |
|
|
| `refactored/managed-container.d/common.sh` | `97e93d9164e854e5f3688cf9d3cc656b001eb30ef3c2073c907450ca43588f3d` |
|
|
| `refactored/managed-container.d/container.sh` | `ac6bc363a2301f1491d162ef609103dbfbceb3b099a3556614c6165952309374` |
|
|
| `refactored/managed-container.d/demote.sh` | `0af3bb362a303017372c631354ba94ef9e723bf14e64d2eff91b6662e54a5be0` |
|
|
| `refactored/managed-container.d/notify.sh` | `6a98fcdda82cea9365db95b4ee779be5c567b3d846f48b90fbc8c254af697670` |
|
|
| `refactored/managed-container.d/postgres.sh` | `d2779ce6e7d1f426ffca2e41bbb8de77503005dc38d658d69f184020365796eb` |
|
|
| `refactored/managed-container.d/promote.sh` | `3085ec9b4410905ab9d4370786ed776eac3470a0415aa1b4f608b083945aeb58` |
|
|
| `refactored/managed-container.d/rejoin.sh` | `21beea72bcb7f24a51c17ea48666ac87df9436702ad77b166b45f4647e28e8ad` |
|
|
| `refactored/managed-container.d/replication.sh` | `6b2021887e3b4cd6a12490e7a8dff98320b2133b472ab25a22a0eead46137547` |
|
|
| `refactored/managed-container.d/promotion_state.py` | `5166fc3370f49ab14cd975ded4b03a74b9bd7dbff5762b41a15552c42f934ea7` |
|
|
| `runtime-context/postgres-ha-bootstrap` | `3f1a15f7054fb69b9aa92204de30306fd6053c63560310f4231f314b148e2dcf` |
|
|
|
|
|
|
## Non-production checks
|
|
|
|
The frozen source retained all 29 original lifecycle function bodies and original
|
|
parameter defaults. Added scoring changes public dispatch and metadata; it is
|
|
not a claim of identical total behavior to the monolithic reference.
|
|
|
|
Run from the repository root:
|
|
|
|
```sh
|
|
python3 tests/verify_refactor.py
|
|
python3 tests/test_promotion_scores.py
|
|
python3 tests/test_disconnected_state.py
|
|
python3 tests/test_promotion_scheduler.py
|
|
python3 tests/test_promotion_cleanup.py
|
|
sha256sum -c notes/REFACTOR-SHA256SUMS
|
|
git diff --check
|
|
```
|
|
|
|
The baseline suite covers 22 differential lifecycle cases, 25 scoring tests,
|
|
16 state/monitor tests, 11 real scheduler simulations and 4 planner tests.
|
|
Scheduler failure simulations use synthetic CIBs, enable fencing, and assert
|
|
fencing prerequisites for promotion. They do not operate a live cluster.
|
|
|
|
`tests/verify_promotion_sql.py` explicitly creates/removes a disposable PostgreSQL
|
|
18 container and checks nine strict/observation SQL scenarios. Previous candidate
|
|
RPM build/install and root-helper CLI verification passed in disposable Fedora
|
|
containers. They are separate from the operator's production validation.
|
|
Current freeze review results are recorded in [review.md](review.md).
|