Freeze HA scoring and document reported production failover
This commit is contained in:
Lyncoln
+122
@@ -0,0 +1,122 @@
|
||||
# Validation and freeze-point evidence
|
||||
|
||||
## Reported production fencing and recovery test
|
||||
|
||||
Evidence source: operator-supplied production handoff received during the
|
||||
2026-09-12 documentation review. The operator reports a successful real
|
||||
three-node Pacemaker/Corosync test with SBD fencing. Codex did not execute or
|
||||
independently witness this production test. Its exact start/end timestamps and
|
||||
raw cluster logs were not supplied; no timestamps or measurements are invented.
|
||||
|
||||
1. david-eve was primary and was fenced successfully through SBD.
|
||||
2. david-prodesk promoted automatically; the PostgreSQL VIP moved with it.
|
||||
3. david-macbookpro remained/reconnected as a streaming standby.
|
||||
4. david-eve rebooted with former-primary PGDATA. Ordinary resource start
|
||||
correctly refused it because standby.signal was absent.
|
||||
5. The operator manually started its maintenance container with PostgreSQL stopped.
|
||||
6. The private rejoin action used the PostgreSQL VIP as rewind source.
|
||||
pg_rewind detected divergence, found the common checkpoint and completed.
|
||||
7. david-eve started as a streaming standby. Only after recovery, Pacemaker
|
||||
cleanup allowed the cluster to adopt it as Unpromoted.
|
||||
|
||||
Reported final state:
|
||||
|
||||
| Node | Role | Score |
|
||||
| --- | --- | ---: |
|
||||
| david-prodesk | Promoted; PostgreSQL VIP local | 1000 |
|
||||
| david-macbookpro | Unpromoted; asynchronous streaming | 100 |
|
||||
| david-eve | Unpromoted; asynchronous streaming | 100 |
|
||||
|
||||
No PostgreSQL failed actions remained. The successful run demonstrates the
|
||||
reported fencing/failover/VIP/rejoin sequence, not every possible partition,
|
||||
latency, concurrent-write, or recovery failure. The intermediate disconnected
|
||||
score and lost-transaction count were not supplied. The quoting/authentication
|
||||
incident and required fix are documented in [operations](operations.md).
|
||||
|
||||
## Validated production snapshot — exact deployed hashes pending
|
||||
|
||||
The production outcome is operator-reported validated; **the byte-level production
|
||||
snapshot is not yet established**. Exact hashes from david-prodesk,
|
||||
david-macbookpro and david-eve have not been supplied or collected. Repository
|
||||
hashes below must not be represented as deployed hashes. A later capture also
|
||||
requires operator confirmation that those bytes are the ones used during the test.
|
||||
|
||||
| Artifact on each production node | Snapshot status |
|
||||
| --- | --- |
|
||||
| /usr/lib/ocf/resource.d/loxthelion/managed-container | Pending production SHA-256 |
|
||||
| managed-container.d/*.sh (all nine modules) | Pending production SHA-256 |
|
||||
| managed-container.d/promotion_state.py | Pending production SHA-256 |
|
||||
| Running container /usr/local/sbin/postgres-ha-bootstrap | Pending production SHA-256 |
|
||||
|
||||
Run the following read-only commands on **each production node**, using the
|
||||
rootless container owner for podman. They hash code, not environment or secrets:
|
||||
|
||||
```sh
|
||||
hostname
|
||||
date -u '+%Y-%m-%dT%H:%M:%SZ'
|
||||
sha256sum /usr/lib/ocf/resource.d/loxthelion/managed-container \
|
||||
/usr/lib/ocf/resource.d/loxthelion/managed-container.d/*.sh \
|
||||
/usr/lib/ocf/resource.d/loxthelion/managed-container.d/promotion_state.py
|
||||
podman exec postgres sha256sum /usr/local/sbin/postgres-ha-bootstrap
|
||||
podman inspect --format '{{.Image}}' postgres
|
||||
```
|
||||
|
||||
If permissions require it, run only the host sha256sum command with sudo. Do not
|
||||
run rootful podman instead of the actual container owner. If the installed
|
||||
bootstrap path/container name differs, record the actual path/name. Capture
|
||||
node identity, UTC capture time, all filenames/digests, immutable image ID, and
|
||||
whether any code changed since the test. Hash the running container's bootstrap;
|
||||
a host snapshot or image tag alone is not equivalent. Do not dump environment,
|
||||
secret files, or pgpass. Compare every node individually, preserving mismatches.
|
||||
|
||||
## Repository freeze snapshot
|
||||
|
||||
The exact repository code hashes appear below and in the reviewed checksum
|
||||
manifest. These identify the source freeze only, pending production comparison.
|
||||
The bootstrap is a saved runtime-context artifact, not part of the RA RPM and
|
||||
not yet confirmed identical to the production bootstrap.
|
||||
|
||||
| Repository file | SHA-256 |
|
||||
| --- | --- |
|
||||
| `refactored/managed-container` | `a4b775a0ddcabdb0558da9b34c7fe2a883c07d0ef308c1c7224526abf53f63a0` |
|
||||
| `refactored/managed-container.d/auth.sh` | `93191771d3cb8aefadb2a16d2f89d64a8b6c584071a473439be818237bdf7f17` |
|
||||
| `refactored/managed-container.d/common.sh` | `97e93d9164e854e5f3688cf9d3cc656b001eb30ef3c2073c907450ca43588f3d` |
|
||||
| `refactored/managed-container.d/container.sh` | `ac6bc363a2301f1491d162ef609103dbfbceb3b099a3556614c6165952309374` |
|
||||
| `refactored/managed-container.d/demote.sh` | `0af3bb362a303017372c631354ba94ef9e723bf14e64d2eff91b6662e54a5be0` |
|
||||
| `refactored/managed-container.d/notify.sh` | `6a98fcdda82cea9365db95b4ee779be5c567b3d846f48b90fbc8c254af697670` |
|
||||
| `refactored/managed-container.d/postgres.sh` | `d2779ce6e7d1f426ffca2e41bbb8de77503005dc38d658d69f184020365796eb` |
|
||||
| `refactored/managed-container.d/promote.sh` | `3085ec9b4410905ab9d4370786ed776eac3470a0415aa1b4f608b083945aeb58` |
|
||||
| `refactored/managed-container.d/rejoin.sh` | `21beea72bcb7f24a51c17ea48666ac87df9436702ad77b166b45f4647e28e8ad` |
|
||||
| `refactored/managed-container.d/replication.sh` | `6b2021887e3b4cd6a12490e7a8dff98320b2133b472ab25a22a0eead46137547` |
|
||||
| `refactored/managed-container.d/promotion_state.py` | `5166fc3370f49ab14cd975ded4b03a74b9bd7dbff5762b41a15552c42f934ea7` |
|
||||
| `runtime-context/postgres-ha-bootstrap` | `3f1a15f7054fb69b9aa92204de30306fd6053c63560310f4231f314b148e2dcf` |
|
||||
|
||||
|
||||
## Non-production checks
|
||||
|
||||
The source retains all 29 original lifecycle function bodies and original
|
||||
parameter defaults. Added scoring changes public dispatch and metadata; it is
|
||||
not a claim of identical total behavior to the monolithic reference.
|
||||
|
||||
Run from the repository root:
|
||||
|
||||
```sh
|
||||
python3 tests/verify_refactor.py
|
||||
python3 tests/test_promotion_scores.py
|
||||
python3 tests/test_disconnected_state.py
|
||||
python3 tests/test_promotion_scheduler.py
|
||||
python3 tests/test_promotion_cleanup.py
|
||||
sha256sum -c notes/REFACTOR-SHA256SUMS
|
||||
git diff --check
|
||||
```
|
||||
|
||||
The baseline suite covers 22 differential lifecycle cases, 25 scoring tests,
|
||||
16 state/monitor tests, 11 real scheduler simulations and 4 planner tests.
|
||||
Scheduler failure simulations use synthetic CIBs, enable fencing, and assert
|
||||
fencing prerequisites for promotion. They do not operate a live cluster.
|
||||
|
||||
`tests/verify_promotion_sql.py` explicitly creates/removes a disposable PostgreSQL
|
||||
18 container and checks nine strict/observation SQL scenarios. Previous candidate
|
||||
RPM build/install and root-helper CLI verification passed in disposable Fedora
|
||||
containers. They are separate from the operator's production validation.
|
||||
Current freeze review results are recorded in [review.md](review.md).
|
||||
Reference in New Issue
Block a user