Freeze HA scoring and document reported production failover
This commit is contained in:
@@ -0,0 +1,198 @@
|
||||
> Production fencing/rejoin was subsequently reported successful by the operator.
|
||||
> See [../docs/testing.md](../docs/testing.md) for evidence and snapshot status,
|
||||
> and [../docs/operations.md](../docs/operations.md) for current operating policy.
|
||||
> Deployment statements below describe the original development session only.
|
||||
|
||||
# Disconnected standby implementation and original candidate validation
|
||||
|
||||
This revision uses the accepted promotion-scoring candidate as its baseline.
|
||||
The user explicitly accepts asynchronous replication data-loss risk after an
|
||||
unexpected primary failure, provided Pacemaker fences the failed primary before
|
||||
promotion. No installed RA, live cluster attributes, constraints, fencing
|
||||
configuration, or PostgreSQL instance was changed during this revision.
|
||||
|
||||
## State model and scores
|
||||
|
||||
Every monitor first runs the original role/health monitor. Primary remains
|
||||
1000 with OCF return 8; stopped, unready, and failed role detection publish
|
||||
-INFINITY with their original return codes. Healthy standbys report OCF return
|
||||
0 regardless of eligibility. Each standby monitor collects a new local SQL
|
||||
observation and evaluates one of these states:
|
||||
|
||||
| State | Evidence | Score |
|
||||
| --- | --- | --- |
|
||||
| Connected and verified | In recovery; replay unpaused; receiver streaming; flush non-NULL; replay >= flush | 100 |
|
||||
| Disconnected and verified recently | Same instance; recovery and replay healthy; no streaming receiver; valid fresh stored observation; current replay >= saved replay | 50 |
|
||||
| Ineligible | Missing, expired, invalid or incompatible evidence; replay paused/regressed; connected strict check failed; query/helper failure | -INFINITY |
|
||||
|
||||
The strict connected predicate has not been relaxed. A streaming receiver with
|
||||
lag or NULL flush cannot fall back to disconnected eligibility. A missing
|
||||
receiver, or a PostgreSQL receiver in starting/waiting/restarting/stopping state,
|
||||
can use the disconnected path. A valid connected observation replaces the
|
||||
record atomically; disconnected observations never renew its timestamp.
|
||||
Reconnection must pass the strict checks to create fresh evidence.
|
||||
|
||||
Score publication still uses bounded `crm_attribute --promotion` query/update
|
||||
calls. A missing transient attribute is recreated from current eligibility,
|
||||
including disconnected eligibility. This is not retention of an old +100.
|
||||
Lifecycle wrappers continue to withdraw scores before actions and re-evaluate
|
||||
after successful actions. The original 29 function bodies, defaults, and all
|
||||
original safety checks remain byte-identical. An added metadata wrapper
|
||||
advertises the new parameter without modifying the original metadata function.
|
||||
|
||||
## Exact persistence and ownership
|
||||
|
||||
Host-side directory: `/run/managed-container/promotion/`.
|
||||
Both `/run/managed-container` and its `promotion` child are root-owned mode 0700.
|
||||
Each record is `<sha256>.json`, root-owned mode 0600, with a corresponding
|
||||
root-owned mode 0600 `<sha256>.lock`. The key is SHA-256 of the UTF-8 compact JSON
|
||||
array `[OCF_RESOURCE_INSTANCE, service_user, service_name, container_name]`.
|
||||
An instance suffix change conservatively selects a different record.
|
||||
|
||||
The RA's root process runs `managed-container.d/promotion_state.py` with Python
|
||||
3; the rootless service user cannot read/write the host state directory. There
|
||||
are no credentials, connection strings, passwords, or upstream secrets in the
|
||||
record. Contents are schema version, resource context, host boot ID, PostgreSQL
|
||||
system identifier, actual SQL `data_directory`, postmaster start timestamp,
|
||||
verified replay and flush LSNs, wall-clock observation timestamp for audit,
|
||||
and boot-relative observation time for expiry.
|
||||
|
||||
A nonblocking per-resource lock serializes evaluation and same-directory atomic
|
||||
replacement. Temporary files use mode 0600; JSON is bounded and parsed as data,
|
||||
with duplicate keys, invalid types, invalid LSNs, and extra/missing fields
|
||||
rejected. Unsafe directory/file modes, ownership or symlinks fail closed.
|
||||
Failed persistence withdraws eligibility. Runtime state survives separate RA
|
||||
invocations and Pacemaker transient score loss, but not host reboot; boot ID
|
||||
also guards against restored state from a previous boot.
|
||||
|
||||
## Freshness and identity policy
|
||||
|
||||
OCF parameter `disconnected_max_age`: integer seconds 0–99999, default **300**.
|
||||
Zero disables disconnected eligibility. Ages from zero through the threshold
|
||||
are accepted; negative, future, or greater ages are rejected. Invalid parameter
|
||||
values fail validation and cannot yield a positive standby score.
|
||||
|
||||
Expiry uses Linux CLOCK_BOOTTIME, which includes suspended time. Changing the
|
||||
wall clock does not extend or prematurely expire an observation. Choose the
|
||||
threshold to cover the monitoring, failure detection, fencing, and scheduling
|
||||
time needed by the cluster. Slow fencing may outlast this window and leave no
|
||||
eligible candidate; freshness is deliberately not extended in response.
|
||||
|
||||
The data identity is `(system_identifier, data_directory, postmaster_start_time)`
|
||||
plus host boot/resource context. A new PostgreSQL system ID or data path
|
||||
invalidates the observation. **Any PostgreSQL restart also requires a new strict
|
||||
streaming verification**, even when restarting the same data. This is deliberate:
|
||||
a basebackup or same-cluster replacement can preserve the system identifier,
|
||||
but cannot preserve the running postmaster incarnation. Normal reinitialization,
|
||||
rewind/rejoin restart, or container restart therefore cannot inherit eligibility.
|
||||
Current replay must also be at least the stored replay LSN.
|
||||
|
||||
Malformed, stale or mismatching records are deleted when observed by the
|
||||
state evaluator. A subsequent strict streaming verification creates a new
|
||||
record. A stopped PostgreSQL process is immediately scored -INFINITY; its file
|
||||
may remain until the next standby evaluation, but its replacement/restarted
|
||||
process cannot use it. A healthy primary keeps 1000 and does not consume standby
|
||||
evidence. Stale unused resource-key files disappear at host reboot.
|
||||
|
||||
## Candidate selection and fencing
|
||||
|
||||
100 ranks currently verified connected evidence above recent disconnected
|
||||
proof at 50, while preserving primary 1000. Two disconnected candidates receive
|
||||
equal 50. Pacemaker resolves ties using its configured scheduling rules; no
|
||||
preferred hostname is hardcoded. Tests show repeatable selection for an
|
||||
identical synthetic CIB, not a guarantee of a particular host in every cluster.
|
||||
|
||||
We intentionally do not map LSNs or local observation timestamps into unequal
|
||||
scores. A 64-bit LSN cannot be represented faithfully in Pacemaker's bounded
|
||||
score range, and comparison needs compatible timeline/history context. Local
|
||||
observation age is not a measure of WAL completeness: a more recent observation
|
||||
can contain less WAL than an older observation on another node. Cross-node
|
||||
clock timestamps introduce further uncertainty. Replay and flush LSNs are
|
||||
stored for a future explicit comparison policy; this revision has no cross-node
|
||||
shell access or invented distributed ranking mechanism.
|
||||
|
||||
Fencing remains mandatory and is entirely Pacemaker's responsibility. The RA
|
||||
does not inspect fencing completion or create/modify constraints. A score of 50
|
||||
means locally eligible under the accepted asynchronous policy; it is not
|
||||
permission to run the promote action manually against an unfenced primary.
|
||||
Configured fencing, quorum, and action ordering must be validated in the lab.
|
||||
The synthetic failure tests use fencing enabled and check that every promotion
|
||||
has a fencing action as a transitive prerequisite in the scheduler graph.
|
||||
|
||||
## Verification and review artifacts
|
||||
|
||||
- Original frozen hashes and all 29 original function bodies/defaults: unchanged.
|
||||
- All 22 original differential lifecycle cases pass.
|
||||
- All 25 accepted scoring/action tests pass; mock boundary and metadata handler
|
||||
were adjusted for the added helper without removing existing assertions.
|
||||
- 16 state/monitor tests cover streaming-disconnect, never verified, exact
|
||||
expiry/disabled policy, malformed and unsafe state, system/path/postmaster
|
||||
changes, reinitialization replacement, replay pause/regression, connected
|
||||
failures, reconnect, clock changes, stopped PostgreSQL, score loss/reprobe, and query/helper failures.
|
||||
- 10 real `crm_simulate` synthetic-CIB tests pass, including the original six,
|
||||
unexpected primary failure with one/two disconnected candidates, mandatory
|
||||
fencing dependencies, all-stale refusal, and connected preference.
|
||||
- All 4 cleanup-planner tests pass. No generated cleanup commands were executed.
|
||||
- PostgreSQL 18 integration: all nine original strict SQL cases pass, and the
|
||||
new JSON observation query agrees with each. Only a disposable container with
|
||||
tmpfs data and no network/host ports was used.
|
||||
- The root helper CLI also passed in a disposable Fedora container, using the
|
||||
actual fixed /run path: never-verified refusal, 100 -> 50, ownership/modes,
|
||||
unchanged disconnected timestamp, malformed-state deletion and identity change.
|
||||
- Fedora 43 RPM build and disposable installation validation passed, recorded in
|
||||
`/tmp/mc-disconnected-rpm.log`; artifacts are in `dist/disconnected-review/`.
|
||||
- Incremental diff from the accepted candidate: `DISCONNECTED-STANDBY.patch`.
|
||||
Full diff from committed baseline: `PROMOTION-SCORING.patch`.
|
||||
|
||||
Run the offline suite with:
|
||||
|
||||
```sh
|
||||
python3 tests/verify_refactor.py
|
||||
python3 tests/test_promotion_scores.py
|
||||
python3 tests/test_disconnected_state.py
|
||||
python3 tests/test_promotion_scheduler.py
|
||||
python3 tests/test_promotion_cleanup.py
|
||||
```
|
||||
|
||||
The optional disposable SQL test is `python3 tests/verify_promotion_sql.py`.
|
||||
During the original candidate-development session, no live HA transitions or
|
||||
deployment were performed. Subsequent operator-reported production results are
|
||||
recorded in ../docs/testing.md. The previous scoring
|
||||
report is retained as historical context; its strict-disconnection blocker is
|
||||
superseded by this policy. Existing lab/production cleanup prerequisites remain.
|
||||
|
||||
## Remaining limitations
|
||||
|
||||
Asynchronous replication does not guarantee zero data loss. The 300-second
|
||||
window bounds observation age, **not lost bytes or transactions**, and does not
|
||||
prove that a standby has the failed primary's last acknowledged transaction.
|
||||
Nor does this design guarantee selection of the most advanced of two eligible
|
||||
standbys. It does not establish common timelines across peers.
|
||||
|
||||
The SQL user must be allowed to execute `pg_control_system()` and read the
|
||||
required monitoring fields. Insufficient privileges fail closed; no grants are
|
||||
changed automatically. Restarting a disconnected standby forfeits eligibility
|
||||
until it streams again. Loss of /run state likewise requires fresh verification.
|
||||
Replacing files underneath a running PostgreSQL process or deliberately restoring
|
||||
fabricated identity/state is outside supported PostgreSQL operation; root and
|
||||
the database monitoring boundary are trusted.
|
||||
|
||||
The scoring SQL retains its 3-second external bound (1-second kill grace) and
|
||||
2-second statement timeout. State evaluation adds a 1-second external bound
|
||||
(1-second kill grace); crm_attribute retains each 2-second bound and 1-second
|
||||
grace. Original lifecycle/role queries and operation timeouts are unchanged;
|
||||
validate the aggregate monitor budget on actual hardware.
|
||||
|
||||
Eligibility is sampled, not an atomic transaction with future promotion.
|
||||
Publication/IPC failure can leave an old Pacemaker score until the next
|
||||
successful update, as in the accepted baseline. Scheduling and fencing can
|
||||
also outlast an observation's expiry before the next monitor. This revision
|
||||
preserves the existing promote implementation rather than adding a new final
|
||||
promotion guard. Lab testing must cover detection/fencing latency, failover in
|
||||
both directions, reconnect/rejoin, monitoring load, and score updates during
|
||||
failure before a production rollout is approved.
|
||||
|
||||
Sources: PostgreSQL documents the system identifier and server start timestamp
|
||||
in [system information functions](https://www.postgresql.org/docs/18/functions-info.html).
|
||||
Pacemaker's fencing, resource requirements and promotion scheduling are described
|
||||
in [Pacemaker Explained](https://clusterlabs.org/projects/pacemaker/doc/3.0/Pacemaker_Explained/pdf/Pacemaker_Explained.pdf).
|
||||
@@ -0,0 +1,280 @@
|
||||
> Current production report and required operating policy: [../docs/testing.md](../docs/testing.md)
|
||||
> and [../docs/operations.md](../docs/operations.md). The historical lab plan below
|
||||
> is not a production runbook.
|
||||
|
||||
> Historical accepted scoring baseline. The current disconnected policy, tests,
|
||||
> and limitations are in [DISCONNECTED-STANDBY.md](DISCONNECTED-STANDBY.md).
|
||||
> The pending decision and strict-disconnection blocker below are superseded.
|
||||
|
||||
# Promotion scoring — undeployed review candidate
|
||||
|
||||
This enhancement is based on the canonical modular source at c8636ed. This task changed no
|
||||
installed agent, production configuration, fencing setting or live cluster
|
||||
attribute. The earlier baseline installer is now pinned to that
|
||||
baseline entrypoint and refuses this candidate. Do not deploy it using the old
|
||||
installation instructions.
|
||||
|
||||
## Decision still required before rollout
|
||||
|
||||
**Strict live-streaming eligibility can prevent failover when the primary is
|
||||
lost.** A previously healthy standby can lose its WAL receiver before Pacemaker
|
||||
chooses a successor. This candidate then replaces its positive score with
|
||||
-INFINITY. With all candidates ineligible, no instance is promoted. The SQL and
|
||||
scheduler tests confirm this outcome; it is not a claim of preserved automatic
|
||||
failover. Controlled switchovers and abrupt primary-loss failovers must both be
|
||||
reviewed. The user's choice between this conservative first candidate and a
|
||||
separate disconnected-standby eligibility policy is pending.
|
||||
|
||||
There is no inferred grace period, persisted last-known-good eligibility, or
|
||||
cross-node WAL ranking. Retaining an old positive score merely because a
|
||||
standby once streamed would contradict the requested stale-score cleanup.
|
||||
Resolving disconnected-standby promotion safely requires an explicit data-loss
|
||||
policy and validated fencing/recovery assumptions. Do not remove the production
|
||||
preference or reset production metadata until this decision and lab validation
|
||||
are complete.
|
||||
|
||||
## Scoring policy
|
||||
|
||||
| Observed state | Published score | Monitor result |
|
||||
| --- | --- | --- |
|
||||
| Original monitor reports healthy primary | 1000 | 8 (Promoted) |
|
||||
| Healthy standby; in recovery, replay unpaused, streaming receiver, replay LSN at least receiver's flushed LSN | 100 | 0 (Unpromoted) |
|
||||
| Healthy standby but eligibility SQL false, empty, malformed, failed or timed out | -INFINITY | 0 (Unpromoted) |
|
||||
| Service/container/PostgreSQL stopped | -INFINITY | Original result, normally 7 |
|
||||
| Unready, failed role query or unexpected role | -INFINITY | Original failure code |
|
||||
|
||||
The eligibility query is local to the running container. It uses the durable
|
||||
`flushed_lsn`, not `written_lsn`. SQL NULL values cannot yield eligibility.
|
||||
A CASE expression avoids invoking recovery-only functions on a primary.
|
||||
Standbys receive equal scores when eligible; no preferred hostname is encoded.
|
||||
Requiring replay to catch up to locally flushed WAL is conservative and can
|
||||
make eligibility fluctuate under sustained write load. This is a real state
|
||||
change, not artificial periodic CIB rewriting.
|
||||
|
||||
A positive score is an observation of local eligibility, not proof that a node
|
||||
has every acknowledged transaction from the primary, is freshest among peers,
|
||||
or is safe to promote without fencing. Asynchronous replication can lose
|
||||
transactions. These scores do not implement consensus or replace PostgreSQL
|
||||
recovery checks, Pacemaker constraints, quorum, or fencing.
|
||||
|
||||
## Placement and scope
|
||||
|
||||
All 29 original function definitions, defaults and metadata remain byte-for-byte
|
||||
identical. The existing nine-module layout remains. Six helpers were added:
|
||||
|
||||
- common.sh: mc_promotion_score_set, mc_promotion_score_from_monitor,
|
||||
mc_promotion_monitor, mc_promotion_action, mc_promotion_validate.
|
||||
- postgres.sh: mc_pg_promotion_eligible.
|
||||
|
||||
The dispatcher now routes monitor/status through mc_promotion_monitor, which
|
||||
calls the original managed_monitor once, publishes a score and returns the
|
||||
original role code. Internal lifecycle monitor calls remain unchanged and do
|
||||
not publish scores mid-transition.
|
||||
|
||||
Start/promote/demote/stop/rejoin publish -INFINITY before validation or mutation.
|
||||
After a successful non-stop action, the original monitor is called again and
|
||||
eligibility is reassessed. Failure leaves the negative score until a subsequent
|
||||
fresh monitor observation. Successful stop leaves -INFINITY. Stop still skips
|
||||
validate_all and attempts shutdown even if publication fails. Notify retains
|
||||
its original notification persistence behavior and does not affect scores.
|
||||
Metadata and validate-all publish no scores. validate-all additionally checks
|
||||
that crm_attribute and timeout exist; deployments require Pacemaker >=2.1.0
|
||||
for the modern --promotion option, enforced by the RPM dependency.
|
||||
|
||||
The entrypoint captures the explicitly supplied OCF_RESOURCE_INSTANCE before
|
||||
ocf-shellfuncs can supply a dummy standalone default. If none was supplied,
|
||||
publication logs a warning and performs no write. Pacemaker resolves the node
|
||||
from its execution context (including OCF_RESKEY_CRM_meta_on_node); there are no
|
||||
node-name literals or manual node-resolution rules in the RA scoring code.
|
||||
|
||||
The helper uses:
|
||||
|
||||
```text
|
||||
crm_attribute --promotion=<explicit resource instance> --lifetime=reboot --query --quiet
|
||||
crm_attribute --promotion=<explicit resource instance> --lifetime=reboot --update=<score>
|
||||
```
|
||||
|
||||
Only a missing or different value causes an update. No local cache or durable
|
||||
eligibility file is used, so reprobes restore missing transient scores. Values
|
||||
are replaced, never incremented. A negative value is used instead of deletion:
|
||||
a missing score alone is not an explicit promotion prohibition.
|
||||
|
||||
Pacemaker currently implements this API with an internal master-<resource>
|
||||
attribute name. Seeing that prefix for the NEW resource is expected. The RA
|
||||
never constructs, reads, deletes or migrates master-postgresql,
|
||||
postgresql-data-status, or postgresql-status. Stale attributes for the old
|
||||
primitive are not inputs to this code. Independently configured location rules
|
||||
that reference old attributes remain cluster configuration and require their
|
||||
own operator review.
|
||||
|
||||
Each score query/write is bounded by a two-second timeout plus a one-second
|
||||
kill grace. The new eligibility command has a three-second outer timeout plus
|
||||
a one-second kill grace and a two-second SQL statement timeout. The existing
|
||||
monitor's own command timeouts are unchanged. Account for these additions in
|
||||
lab operation timing before production rollout.
|
||||
|
||||
If attribute publication fails, the helper logs an error explicitly warning
|
||||
that the prior score may remain. It retries on the next observation. Role
|
||||
return codes and existing lifecycle results are preserved; shutdown is never
|
||||
blocked by a score-write failure. **No RA can guarantee stale-score removal
|
||||
while the cluster attribute service is unreachable.** This failure mode needs
|
||||
lab validation and is not hidden by reporting a false PostgreSQL role.
|
||||
Pacemaker normally serializes actions for a resource; parallel direct/manual
|
||||
calls that race score updates are not supported by this implementation.
|
||||
|
||||
## Tests and evidence
|
||||
|
||||
Commands run from the source tree:
|
||||
|
||||
```bash
|
||||
bash -n refactored/managed-container
|
||||
for module in refactored/managed-container.d/*.sh; do bash -n "$module"; done
|
||||
python3 tests/verify_refactor.py
|
||||
python3 tests/test_promotion_scores.py
|
||||
python3 tests/test_promotion_scheduler.py
|
||||
python3 tests/test_promotion_cleanup.py
|
||||
python3 tests/verify_promotion_sql.py
|
||||
```
|
||||
|
||||
- Original 22 differential scenarios pass. All 29 original definitions and the
|
||||
same 12 metadata parameters / eight advertised actions are preserved.
|
||||
- 25 scoring tests pass, including failures, stop/start, promote/demote/rejoin,
|
||||
stale positive replacement, repeated monitors, loss/recreation of transient
|
||||
attributes, missing explicit resource identity and score-publication errors.
|
||||
- A real crm_attribute test uses CIB_file with a synthetic CIB. Repeated equal
|
||||
observations do not change the file; negative replacement and recreation
|
||||
after loss of transient attributes work.
|
||||
- Six real crm_simulate tests on synthetic CIBs pass: absent and explicit
|
||||
Started target-role each permit one promotion; Unpromoted prevents it;
|
||||
larger scores win, ineligible nodes are excluded, and all-negative scores
|
||||
permit no promotion. Old resource attributes are seeded and do not alter
|
||||
this new resource's election. These are scheduler simulations, not live HA
|
||||
transitions. The synthetic CIB disables fencing only inside its test file;
|
||||
the live lab/production fencing configuration was not touched.
|
||||
- Nine SQL cases pass in a separate ephemeral PostgreSQL 18 container with no
|
||||
host ports or data mounts, evaluating the actual production SQL against
|
||||
controlled state fixtures. Includes replay pause/lag, missing positions,
|
||||
stopped/absent receiver, and recovery-only function handling on a primary.
|
||||
- Four cleanup-command planner tests pass, including ambiguous-rule rejection
|
||||
and preservation of unrelated rules.
|
||||
|
||||
The RPM %check now runs the regression, scoring and scheduler suites. PostgreSQL
|
||||
integration is an explicit opt-in local test needing Podman and a local
|
||||
postgres:18 image; it does not touch the lab postgres container.
|
||||
|
||||
## Specific disposable two-node lab sequence (after review)
|
||||
|
||||
Keep maintenance-mode=true as required by the lab handoff. Use direct RA
|
||||
operations only after installing the reviewed version under a separate approved
|
||||
installation procedure. Automatic scheduling claims must first be exercised
|
||||
with a copied CIB and crm_simulate; do not disable maintenance mode as part of
|
||||
these instructions. Do not use this sequence on production.
|
||||
|
||||
1. Save the lab CIB, installed executable/modules, and current state on both
|
||||
nodes. Verify the cluster is postgres-ra-test, the roles and streaming source,
|
||||
current PGDATA backup/recovery baseline, and that production is not targeted.
|
||||
Establish timing and monitor outputs from the existing implementation.
|
||||
2. Run metadata and validate-all for the candidate on each lab node. For direct
|
||||
actions always supply the actual primitive ID (pg-ha in saved lab context),
|
||||
not the clone ID. Example observation commands on each lab node:
|
||||
|
||||
```bash
|
||||
sudo env OCF_RESOURCE_INSTANCE=pg-ha \
|
||||
/usr/lib/ocf/resource.d/loxthelion/managed-container monitor
|
||||
# Interpret rc=8 as healthy primary, rc=0 as healthy standby.
|
||||
sudo crm_attribute --promotion=pg-ha --lifetime=reboot --query --quiet
|
||||
```
|
||||
|
||||
Expect primary=1000 and a caught-up streaming standby=100. Repeat monitors;
|
||||
verify no writes for identical scores and no unrelated attribute changes.
|
||||
3. On the standby, test full stop and start: stopped service/container and
|
||||
PostgreSQL must yield monitor=7 and score=-INFINITY; standby start restores
|
||||
monitor=0 and score=100 after streaming/catch-up. Verify primary start refusal
|
||||
remains intact. Do not use a successful role-return code as a score assertion.
|
||||
4. Separately stop only standby PostgreSQL while keeping the maintenance
|
||||
container alive; observe -INFINITY, then start it through the proven standby
|
||||
path. Verify no rejoin is attempted while PostgreSQL is running.
|
||||
5. Pause replay on the standby (lab only), observe monitor=0 and -INFINITY,
|
||||
resume replay and await catch-up/100. Test receiver interruption, SQL errors,
|
||||
stopped container, and a seeded prior positive score becoming ineligible.
|
||||
6. Exercise the existing controlled switchover sequence in both directions:
|
||||
persist the fresh matching pre-demote notification on the old primary,
|
||||
demote through the Corosync-resolved successor, and promote the successor.
|
||||
Verify -INFINITY during transitions, 100 for a successful eligible demotion,
|
||||
1000 after promotion/invariant reconciliation, and persistent container
|
||||
identity throughout demotion. Return to DV6 primary / ELITEDESK standby.
|
||||
7. Test two eligible standbys using the existing cascading-standby intermediate
|
||||
state. Both should have the same positive score; a copied-CIB simulation
|
||||
should select one with promoted-max=1. Do not promote both manually.
|
||||
8. Reproduce the abrupt-primary-loss case in the isolated lab only after a
|
||||
recoverable baseline exists. Allow the surviving receiver to disappear,
|
||||
run its monitor, and inspect the score. This candidate is EXPECTED to deny
|
||||
promotion then. That is a rollout blocker unless the strict policy is
|
||||
explicitly chosen; it cannot be counted as a passed failover test.
|
||||
9. Remove only the new resource's transient score in the lab, then reprobe:
|
||||
fresh eligibility must recreate it. Exercise score-service/query/update
|
||||
failure and recovery without confusing role codes. Repeated successful
|
||||
monitoring must replace any stale positive score after ineligibility.
|
||||
10. With a saved CIB, simulate removal of only the temporary node preference.
|
||||
Retain target-role=Promoted, promoted-max=1, and the VIP constraints.
|
||||
Earlier absent/Started scheduler cases are compatibility tests, not
|
||||
production configuration recommendations.
|
||||
11. Restore the baseline topology, streaming, test-induced settings and intended
|
||||
agent version. Confirm maintenance mode still true. Record both nodes'
|
||||
installed hashes, OCF codes, scores, receiver status and action durations.
|
||||
|
||||
## Production cleanup and rollout: gated on lab validation
|
||||
|
||||
The conceptual description does not include the actual preference constraint
|
||||
ID, so inventing a deletion command would be unsafe. Read-only discovery is:
|
||||
|
||||
```bash
|
||||
sudo pcs constraint config --full
|
||||
sudo pcs resource config postgresql-ha-clone
|
||||
sudo cibadmin --query > production-before-scoring.xml
|
||||
```
|
||||
|
||||
Do not commit that CIB snapshot: it may contain sensitive deployment settings.
|
||||
After lab validation and policy approval, scripts/plan-promotion-cleanup.py can
|
||||
read the supplied snapshot and PRINT the exact commands using the real IDs:
|
||||
|
||||
```bash
|
||||
python3 scripts/plan-promotion-cleanup.py production-before-scoring.xml \
|
||||
--clone postgresql-ha-clone --preferred-node david-eve --lab-validated
|
||||
```
|
||||
|
||||
The planner never executes commands. It matches only the simple rule described
|
||||
in the task (Promoted, score=200, #uname eq david-eve), refuses zero/multiple
|
||||
matches, preserves other rules, and emits only the preference constraint/rule
|
||||
removal. It requires and preserves direct clone-level target-role=Promoted.
|
||||
Rule-based/inherited settings require manual inspection. Do not confuse
|
||||
preference removal with clearing failed actions: former-primary failure state
|
||||
must remain until rejoin succeeds and streaming standby status is verified.
|
||||
|
||||
There is no RA code that deletes constraints or changes clone metadata. No
|
||||
production installation, cleanup, or restart was executed during that original
|
||||
source-review session. Subsequent production validation is in ../docs/testing.md. Exact production
|
||||
rollout commands are intentionally deferred until the policy, real constraint
|
||||
ID, candidate hashes and successful lab results are known, as requested.
|
||||
|
||||
## Sources used for API semantics
|
||||
|
||||
- https://clusterlabs.org/projects/pacemaker/man/crm_attribute.8.html
|
||||
- https://www.postgresql.org/docs/18/monitoring-stats.html#MONITORING-PG-STAT-WAL-RECEIVER-VIEW
|
||||
- https://www.postgresql.org/docs/18/functions-admin.html
|
||||
- Installed pcs help: `pcs constraint remove --help`, `pcs resource meta --help`.
|
||||
|
||||
The source diff is provided in PROMOTION-SCORING.patch. Source and test file
|
||||
hashes are recorded in REFACTOR-SHA256SUMS, which is refreshed for this candidate.
|
||||
The frozen reference hash remains unchanged.
|
||||
|
||||
RPM validation of this candidate also passed in the disposable Fedora 43 build
|
||||
container: binary/source packages built, %check ran the regression, scoring,
|
||||
scheduler and cleanup tests, DNF resolved the new Pacemaker dependency, and the
|
||||
installed-RPM integrity/permissions/metadata checks passed. Local artifacts are
|
||||
under dist/scoring-review/ and are not releases or deployment approval.
|
||||
|
||||
A final read-only inspection found that ELITEDESK's installed executable and
|
||||
all nine modules match the earlier modular baseline at c8636ed exactly. Its
|
||||
entrypoint SHA256 is c91f09adfdeb45a3be1a72df3f5ca5b02fd4555dec4397ad944eab439ace0b89.
|
||||
The promotion-scoring candidate is not installed there.
|
||||
@@ -1,5 +1,10 @@
|
||||
# PostgreSQL HA resource-agent modular refactor
|
||||
|
||||
Historical baseline report (commit c8636ed and earlier). The later promotion
|
||||
scoring candidate changes dispatcher behavior and adds helpers; see
|
||||
[DISCONNECTED-STANDBY.md](DISCONNECTED-STANDBY.md). Do not use the installation
|
||||
commands below to deploy that unreviewed candidate.
|
||||
|
||||
The refactored agent is in `refactored/managed-container`, with nine sourced modules beside it. Both reference copies remain unchanged at SHA256 `5eae1f3dba0586dc06105e4dd7a9b8809a5c9961452abe7654dcfce77c3c5ce2`.
|
||||
|
||||
## Final file tree
|
||||
|
||||
@@ -1,13 +1,39 @@
|
||||
e2df02e66e120481eb62894b2356f40030c63d5bcf734de2cb7ee64bda8b3376 .gitea/workflows/build-rpm.yml
|
||||
bb9026b0c528721ab80038344929a8bcd0ab8c5c8e438a22978bc362572d6112 .gitignore
|
||||
4351008e41de847f0dd8323843a28ea3b0fbe100d0268faa77183cdb4f7c7da0 HANDOFF.md
|
||||
6e3cf9e32b85808ff503ac4d07f1dff69e002f0dbcb1630ad3c36ca32e530afb README.md
|
||||
13b689315ab43e830473aa3c97e37390a5417b4681fab2774c6072b52b3266b4 docs/operations.md
|
||||
3e5f075baa7863bee3d12dd50319466291e30e7be895dfe16e9724da9ae4f021 docs/review.md
|
||||
6e0ce0054b478dcae608b6f35c6614bbfae8ef8bb978144cfc6d560de6d57d0f docs/testing.md
|
||||
bfdeecfb1791c4f1d8ec08f28774f94769fffeb51b3268342fd6c63e2322eb23 notes/DISCONNECTED-STANDBY.md
|
||||
0a6af2a5fbf1ea1d9551893578df094442ee10e032e1b2de7c92f9662b133ad3 notes/PROMOTION-SCORING.md
|
||||
6bdc385dff0e66de372f802a577f67d852f7a2a0e45610865f558ee3b93adf20 notes/REFACTOR-PLAN.md
|
||||
08135e93b355d2f8239d9abbe0297eafdd6109d025a27d13ab35ccd74811e03a notes/REFACTOR-REPORT.md
|
||||
c91f09adfdeb45a3be1a72df3f5ca5b02fd4555dec4397ad944eab439ace0b89 refactored/managed-container
|
||||
05b860269ba536361f7b806f415ffb6c2f6632ecf8374bfcd02e7bb7ad1069bb notes/REFACTOR-REPORT.md
|
||||
2d728176c301e24f673b8d36a9ae0fd3bd3816d37b1f945100b37a74779887ef packaging/Containerfile
|
||||
676455c64d130eb1ad36e007f023f0f30c98beedf87f18be74ae33202b646ffa packaging/PROVENANCE.md
|
||||
ba3fe4e1ee60ec9445c4c2cc6591e90b674c7113474ec8b33edb4f1016dc0ae2 packaging/README.md
|
||||
e9dd8507f4bf0c6f42458e41aea833ad0bd3f6127272335eee9bf4d58541ed67 packaging/VERSION
|
||||
4b9e13339a1eacbced38c69edf2a8c518997160104aab1bbfe6d2f019a93d6fa packaging/ocf-managed-container.spec
|
||||
a4b775a0ddcabdb0558da9b34c7fe2a883c07d0ef308c1c7224526abf53f63a0 refactored/managed-container
|
||||
93191771d3cb8aefadb2a16d2f89d64a8b6c584071a473439be818237bdf7f17 refactored/managed-container.d/auth.sh
|
||||
e0bb13e9fda84ba7b657b2e774c08fa64988d463f45c699681e0e6ce49109440 refactored/managed-container.d/common.sh
|
||||
97e93d9164e854e5f3688cf9d3cc656b001eb30ef3c2073c907450ca43588f3d refactored/managed-container.d/common.sh
|
||||
ac6bc363a2301f1491d162ef609103dbfbceb3b099a3556614c6165952309374 refactored/managed-container.d/container.sh
|
||||
0af3bb362a303017372c631354ba94ef9e723bf14e64d2eff91b6662e54a5be0 refactored/managed-container.d/demote.sh
|
||||
6a98fcdda82cea9365db95b4ee779be5c567b3d846f48b90fbc8c254af697670 refactored/managed-container.d/notify.sh
|
||||
310132d8032891727076295f39548b1cc0e51a10301a0ee4d934161f4f903e3e refactored/managed-container.d/postgres.sh
|
||||
d2779ce6e7d1f426ffca2e41bbb8de77503005dc38d658d69f184020365796eb refactored/managed-container.d/postgres.sh
|
||||
3085ec9b4410905ab9d4370786ed776eac3470a0415aa1b4f608b083945aeb58 refactored/managed-container.d/promote.sh
|
||||
5166fc3370f49ab14cd975ded4b03a74b9bd7dbff5762b41a15552c42f934ea7 refactored/managed-container.d/promotion_state.py
|
||||
21beea72bcb7f24a51c17ea48666ac87df9436702ad77b166b45f4647e28e8ad refactored/managed-container.d/rejoin.sh
|
||||
6b2021887e3b4cd6a12490e7a8dff98320b2133b472ab25a22a0eead46137547 refactored/managed-container.d/replication.sh
|
||||
e3bc103b61b317e91d28260cc7bb86a3cc8390854762d6c5b9f325a1a8b1c362 tests/verify_refactor.py
|
||||
3f1a15f7054fb69b9aa92204de30306fd6053c63560310f4231f314b148e2dcf runtime-context/postgres-ha-bootstrap
|
||||
17ce35741b0ef55a6e4785a093a5097f13139edd428fd4c65e8f50d222161b65 scripts/build-rpm.sh
|
||||
42744b1b407ab4a79725bd90d7d067f46dd8b36728f104c52544d464b2e323c3 scripts/install-elitedesk-ra.sh
|
||||
470fd174e42efef876e540dfbfa42b4f36bf70b29a26ab3e5f0c10c813f3b6ae scripts/plan-promotion-cleanup.py
|
||||
52162183beb2eb258b2336c4fd6392319e2ac7415372b623226bd1dd4ff0720d tests/test_disconnected_state.py
|
||||
e44ac0e6fc322d7e314814c69d9c3b1ab582b43afbf8b66bd30a07713682ae1c tests/test_promotion_cleanup.py
|
||||
eb88cdaffeccb00c92e1c43725112ae16aa15faf923b84db62ae5950c5c26d2e tests/test_promotion_scheduler.py
|
||||
29bac1cc5939aaf772259bb08f3d2cfae2186e081fcb203c4979a32d0a60e0c0 tests/test_promotion_scores.py
|
||||
40211e06e4b23f0e1842759cfeb4508a7dbd2ec985da9bdef090efa017f4e02d tests/verify_installed_rpm.py
|
||||
d2280256e451afca90d035e85998ed9ebab8ec21aa7c0aa4605fed7619ba24b9 tests/verify_promotion_sql.py
|
||||
c7744c533055dac5744fcb54a56ff1dd3f95940185ddc860a69dc2dd88669819 tests/verify_promotion_state_cli.py
|
||||
dc7d1c8b47c4f0d4edbfacd38714405d6da8d3ae6b183f363afd264e4a0229e5 tests/verify_refactor.py
|
||||
|
||||
Reference in New Issue
Block a user