Freeze HA scoring and document reported production failover
This commit is contained in:
Lyncoln
@@ -0,0 +1,280 @@
|
||||
> Current production report and required operating policy: [../docs/testing.md](../docs/testing.md)
|
||||
> and [../docs/operations.md](../docs/operations.md). The historical lab plan below
|
||||
> is not a production runbook.
|
||||
|
||||
> Historical accepted scoring baseline. The current disconnected policy, tests,
|
||||
> and limitations are in [DISCONNECTED-STANDBY.md](DISCONNECTED-STANDBY.md).
|
||||
> The pending decision and strict-disconnection blocker below are superseded.
|
||||
|
||||
# Promotion scoring — undeployed review candidate
|
||||
|
||||
This enhancement is based on the canonical modular source at c8636ed. This task changed no
|
||||
installed agent, production configuration, fencing setting or live cluster
|
||||
attribute. The earlier baseline installer is now pinned to that
|
||||
baseline entrypoint and refuses this candidate. Do not deploy it using the old
|
||||
installation instructions.
|
||||
|
||||
## Decision still required before rollout
|
||||
|
||||
**Strict live-streaming eligibility can prevent failover when the primary is
|
||||
lost.** A previously healthy standby can lose its WAL receiver before Pacemaker
|
||||
chooses a successor. This candidate then replaces its positive score with
|
||||
-INFINITY. With all candidates ineligible, no instance is promoted. The SQL and
|
||||
scheduler tests confirm this outcome; it is not a claim of preserved automatic
|
||||
failover. Controlled switchovers and abrupt primary-loss failovers must both be
|
||||
reviewed. The user's choice between this conservative first candidate and a
|
||||
separate disconnected-standby eligibility policy is pending.
|
||||
|
||||
There is no inferred grace period, persisted last-known-good eligibility, or
|
||||
cross-node WAL ranking. Retaining an old positive score merely because a
|
||||
standby once streamed would contradict the requested stale-score cleanup.
|
||||
Resolving disconnected-standby promotion safely requires an explicit data-loss
|
||||
policy and validated fencing/recovery assumptions. Do not remove the production
|
||||
preference or reset production metadata until this decision and lab validation
|
||||
are complete.
|
||||
|
||||
## Scoring policy
|
||||
|
||||
| Observed state | Published score | Monitor result |
|
||||
| --- | --- | --- |
|
||||
| Original monitor reports healthy primary | 1000 | 8 (Promoted) |
|
||||
| Healthy standby; in recovery, replay unpaused, streaming receiver, replay LSN at least receiver's flushed LSN | 100 | 0 (Unpromoted) |
|
||||
| Healthy standby but eligibility SQL false, empty, malformed, failed or timed out | -INFINITY | 0 (Unpromoted) |
|
||||
| Service/container/PostgreSQL stopped | -INFINITY | Original result, normally 7 |
|
||||
| Unready, failed role query or unexpected role | -INFINITY | Original failure code |
|
||||
|
||||
The eligibility query is local to the running container. It uses the durable
|
||||
`flushed_lsn`, not `written_lsn`. SQL NULL values cannot yield eligibility.
|
||||
A CASE expression avoids invoking recovery-only functions on a primary.
|
||||
Standbys receive equal scores when eligible; no preferred hostname is encoded.
|
||||
Requiring replay to catch up to locally flushed WAL is conservative and can
|
||||
make eligibility fluctuate under sustained write load. This is a real state
|
||||
change, not artificial periodic CIB rewriting.
|
||||
|
||||
A positive score is an observation of local eligibility, not proof that a node
|
||||
has every acknowledged transaction from the primary, is freshest among peers,
|
||||
or is safe to promote without fencing. Asynchronous replication can lose
|
||||
transactions. These scores do not implement consensus or replace PostgreSQL
|
||||
recovery checks, Pacemaker constraints, quorum, or fencing.
|
||||
|
||||
## Placement and scope
|
||||
|
||||
All 29 original function definitions, defaults and metadata remain byte-for-byte
|
||||
identical. The existing nine-module layout remains. Six helpers were added:
|
||||
|
||||
- common.sh: mc_promotion_score_set, mc_promotion_score_from_monitor,
|
||||
mc_promotion_monitor, mc_promotion_action, mc_promotion_validate.
|
||||
- postgres.sh: mc_pg_promotion_eligible.
|
||||
|
||||
The dispatcher now routes monitor/status through mc_promotion_monitor, which
|
||||
calls the original managed_monitor once, publishes a score and returns the
|
||||
original role code. Internal lifecycle monitor calls remain unchanged and do
|
||||
not publish scores mid-transition.
|
||||
|
||||
Start/promote/demote/stop/rejoin publish -INFINITY before validation or mutation.
|
||||
After a successful non-stop action, the original monitor is called again and
|
||||
eligibility is reassessed. Failure leaves the negative score until a subsequent
|
||||
fresh monitor observation. Successful stop leaves -INFINITY. Stop still skips
|
||||
validate_all and attempts shutdown even if publication fails. Notify retains
|
||||
its original notification persistence behavior and does not affect scores.
|
||||
Metadata and validate-all publish no scores. validate-all additionally checks
|
||||
that crm_attribute and timeout exist; deployments require Pacemaker >=2.1.0
|
||||
for the modern --promotion option, enforced by the RPM dependency.
|
||||
|
||||
The entrypoint captures the explicitly supplied OCF_RESOURCE_INSTANCE before
|
||||
ocf-shellfuncs can supply a dummy standalone default. If none was supplied,
|
||||
publication logs a warning and performs no write. Pacemaker resolves the node
|
||||
from its execution context (including OCF_RESKEY_CRM_meta_on_node); there are no
|
||||
node-name literals or manual node-resolution rules in the RA scoring code.
|
||||
|
||||
The helper uses:
|
||||
|
||||
```text
|
||||
crm_attribute --promotion=<explicit resource instance> --lifetime=reboot --query --quiet
|
||||
crm_attribute --promotion=<explicit resource instance> --lifetime=reboot --update=<score>
|
||||
```
|
||||
|
||||
Only a missing or different value causes an update. No local cache or durable
|
||||
eligibility file is used, so reprobes restore missing transient scores. Values
|
||||
are replaced, never incremented. A negative value is used instead of deletion:
|
||||
a missing score alone is not an explicit promotion prohibition.
|
||||
|
||||
Pacemaker currently implements this API with an internal master-<resource>
|
||||
attribute name. Seeing that prefix for the NEW resource is expected. The RA
|
||||
never constructs, reads, deletes or migrates master-postgresql,
|
||||
postgresql-data-status, or postgresql-status. Stale attributes for the old
|
||||
primitive are not inputs to this code. Independently configured location rules
|
||||
that reference old attributes remain cluster configuration and require their
|
||||
own operator review.
|
||||
|
||||
Each score query/write is bounded by a two-second timeout plus a one-second
|
||||
kill grace. The new eligibility command has a three-second outer timeout plus
|
||||
a one-second kill grace and a two-second SQL statement timeout. The existing
|
||||
monitor's own command timeouts are unchanged. Account for these additions in
|
||||
lab operation timing before production rollout.
|
||||
|
||||
If attribute publication fails, the helper logs an error explicitly warning
|
||||
that the prior score may remain. It retries on the next observation. Role
|
||||
return codes and existing lifecycle results are preserved; shutdown is never
|
||||
blocked by a score-write failure. **No RA can guarantee stale-score removal
|
||||
while the cluster attribute service is unreachable.** This failure mode needs
|
||||
lab validation and is not hidden by reporting a false PostgreSQL role.
|
||||
Pacemaker normally serializes actions for a resource; parallel direct/manual
|
||||
calls that race score updates are not supported by this implementation.
|
||||
|
||||
## Tests and evidence
|
||||
|
||||
Commands run from the source tree:
|
||||
|
||||
```bash
|
||||
bash -n refactored/managed-container
|
||||
for module in refactored/managed-container.d/*.sh; do bash -n "$module"; done
|
||||
python3 tests/verify_refactor.py
|
||||
python3 tests/test_promotion_scores.py
|
||||
python3 tests/test_promotion_scheduler.py
|
||||
python3 tests/test_promotion_cleanup.py
|
||||
python3 tests/verify_promotion_sql.py
|
||||
```
|
||||
|
||||
- Original 22 differential scenarios pass. All 29 original definitions and the
|
||||
same 12 metadata parameters / eight advertised actions are preserved.
|
||||
- 25 scoring tests pass, including failures, stop/start, promote/demote/rejoin,
|
||||
stale positive replacement, repeated monitors, loss/recreation of transient
|
||||
attributes, missing explicit resource identity and score-publication errors.
|
||||
- A real crm_attribute test uses CIB_file with a synthetic CIB. Repeated equal
|
||||
observations do not change the file; negative replacement and recreation
|
||||
after loss of transient attributes work.
|
||||
- Six real crm_simulate tests on synthetic CIBs pass: absent and explicit
|
||||
Started target-role each permit one promotion; Unpromoted prevents it;
|
||||
larger scores win, ineligible nodes are excluded, and all-negative scores
|
||||
permit no promotion. Old resource attributes are seeded and do not alter
|
||||
this new resource's election. These are scheduler simulations, not live HA
|
||||
transitions. The synthetic CIB disables fencing only inside its test file;
|
||||
the live lab/production fencing configuration was not touched.
|
||||
- Nine SQL cases pass in a separate ephemeral PostgreSQL 18 container with no
|
||||
host ports or data mounts, evaluating the actual production SQL against
|
||||
controlled state fixtures. Includes replay pause/lag, missing positions,
|
||||
stopped/absent receiver, and recovery-only function handling on a primary.
|
||||
- Four cleanup-command planner tests pass, including ambiguous-rule rejection
|
||||
and preservation of unrelated rules.
|
||||
|
||||
The RPM %check now runs the regression, scoring and scheduler suites. PostgreSQL
|
||||
integration is an explicit opt-in local test needing Podman and a local
|
||||
postgres:18 image; it does not touch the lab postgres container.
|
||||
|
||||
## Specific disposable two-node lab sequence (after review)
|
||||
|
||||
Keep maintenance-mode=true as required by the lab handoff. Use direct RA
|
||||
operations only after installing the reviewed version under a separate approved
|
||||
installation procedure. Automatic scheduling claims must first be exercised
|
||||
with a copied CIB and crm_simulate; do not disable maintenance mode as part of
|
||||
these instructions. Do not use this sequence on production.
|
||||
|
||||
1. Save the lab CIB, installed executable/modules, and current state on both
|
||||
nodes. Verify the cluster is postgres-ra-test, the roles and streaming source,
|
||||
current PGDATA backup/recovery baseline, and that production is not targeted.
|
||||
Establish timing and monitor outputs from the existing implementation.
|
||||
2. Run metadata and validate-all for the candidate on each lab node. For direct
|
||||
actions always supply the actual primitive ID (pg-ha in saved lab context),
|
||||
not the clone ID. Example observation commands on each lab node:
|
||||
|
||||
```bash
|
||||
sudo env OCF_RESOURCE_INSTANCE=pg-ha \
|
||||
/usr/lib/ocf/resource.d/loxthelion/managed-container monitor
|
||||
# Interpret rc=8 as healthy primary, rc=0 as healthy standby.
|
||||
sudo crm_attribute --promotion=pg-ha --lifetime=reboot --query --quiet
|
||||
```
|
||||
|
||||
Expect primary=1000 and a caught-up streaming standby=100. Repeat monitors;
|
||||
verify no writes for identical scores and no unrelated attribute changes.
|
||||
3. On the standby, test full stop and start: stopped service/container and
|
||||
PostgreSQL must yield monitor=7 and score=-INFINITY; standby start restores
|
||||
monitor=0 and score=100 after streaming/catch-up. Verify primary start refusal
|
||||
remains intact. Do not use a successful role-return code as a score assertion.
|
||||
4. Separately stop only standby PostgreSQL while keeping the maintenance
|
||||
container alive; observe -INFINITY, then start it through the proven standby
|
||||
path. Verify no rejoin is attempted while PostgreSQL is running.
|
||||
5. Pause replay on the standby (lab only), observe monitor=0 and -INFINITY,
|
||||
resume replay and await catch-up/100. Test receiver interruption, SQL errors,
|
||||
stopped container, and a seeded prior positive score becoming ineligible.
|
||||
6. Exercise the existing controlled switchover sequence in both directions:
|
||||
persist the fresh matching pre-demote notification on the old primary,
|
||||
demote through the Corosync-resolved successor, and promote the successor.
|
||||
Verify -INFINITY during transitions, 100 for a successful eligible demotion,
|
||||
1000 after promotion/invariant reconciliation, and persistent container
|
||||
identity throughout demotion. Return to DV6 primary / ELITEDESK standby.
|
||||
7. Test two eligible standbys using the existing cascading-standby intermediate
|
||||
state. Both should have the same positive score; a copied-CIB simulation
|
||||
should select one with promoted-max=1. Do not promote both manually.
|
||||
8. Reproduce the abrupt-primary-loss case in the isolated lab only after a
|
||||
recoverable baseline exists. Allow the surviving receiver to disappear,
|
||||
run its monitor, and inspect the score. This candidate is EXPECTED to deny
|
||||
promotion then. That is a rollout blocker unless the strict policy is
|
||||
explicitly chosen; it cannot be counted as a passed failover test.
|
||||
9. Remove only the new resource's transient score in the lab, then reprobe:
|
||||
fresh eligibility must recreate it. Exercise score-service/query/update
|
||||
failure and recovery without confusing role codes. Repeated successful
|
||||
monitoring must replace any stale positive score after ineligibility.
|
||||
10. With a saved CIB, simulate removal of only the temporary node preference.
|
||||
Retain target-role=Promoted, promoted-max=1, and the VIP constraints.
|
||||
Earlier absent/Started scheduler cases are compatibility tests, not
|
||||
production configuration recommendations.
|
||||
11. Restore the baseline topology, streaming, test-induced settings and intended
|
||||
agent version. Confirm maintenance mode still true. Record both nodes'
|
||||
installed hashes, OCF codes, scores, receiver status and action durations.
|
||||
|
||||
## Production cleanup and rollout: gated on lab validation
|
||||
|
||||
The conceptual description does not include the actual preference constraint
|
||||
ID, so inventing a deletion command would be unsafe. Read-only discovery is:
|
||||
|
||||
```bash
|
||||
sudo pcs constraint config --full
|
||||
sudo pcs resource config postgresql-ha-clone
|
||||
sudo cibadmin --query > production-before-scoring.xml
|
||||
```
|
||||
|
||||
Do not commit that CIB snapshot: it may contain sensitive deployment settings.
|
||||
After lab validation and policy approval, scripts/plan-promotion-cleanup.py can
|
||||
read the supplied snapshot and PRINT the exact commands using the real IDs:
|
||||
|
||||
```bash
|
||||
python3 scripts/plan-promotion-cleanup.py production-before-scoring.xml \
|
||||
--clone postgresql-ha-clone --preferred-node david-eve --lab-validated
|
||||
```
|
||||
|
||||
The planner never executes commands. It matches only the simple rule described
|
||||
in the task (Promoted, score=200, #uname eq david-eve), refuses zero/multiple
|
||||
matches, preserves other rules, and emits only the preference constraint/rule
|
||||
removal. It requires and preserves direct clone-level target-role=Promoted.
|
||||
Rule-based/inherited settings require manual inspection. Do not confuse
|
||||
preference removal with clearing failed actions: former-primary failure state
|
||||
must remain until rejoin succeeds and streaming standby status is verified.
|
||||
|
||||
There is no RA code that deletes constraints or changes clone metadata. No
|
||||
production installation, cleanup, or restart was executed during that original
|
||||
source-review session. Subsequent production validation is in ../docs/testing.md. Exact production
|
||||
rollout commands are intentionally deferred until the policy, real constraint
|
||||
ID, candidate hashes and successful lab results are known, as requested.
|
||||
|
||||
## Sources used for API semantics
|
||||
|
||||
- https://clusterlabs.org/projects/pacemaker/man/crm_attribute.8.html
|
||||
- https://www.postgresql.org/docs/18/monitoring-stats.html#MONITORING-PG-STAT-WAL-RECEIVER-VIEW
|
||||
- https://www.postgresql.org/docs/18/functions-admin.html
|
||||
- Installed pcs help: `pcs constraint remove --help`, `pcs resource meta --help`.
|
||||
|
||||
The source diff is provided in PROMOTION-SCORING.patch. Source and test file
|
||||
hashes are recorded in REFACTOR-SHA256SUMS, which is refreshed for this candidate.
|
||||
The frozen reference hash remains unchanged.
|
||||
|
||||
RPM validation of this candidate also passed in the disposable Fedora 43 build
|
||||
container: binary/source packages built, %check ran the regression, scoring,
|
||||
scheduler and cleanup tests, DNF resolved the new Pacemaker dependency, and the
|
||||
installed-RPM integrity/permissions/metadata checks passed. Local artifacts are
|
||||
under dist/scoring-review/ and are not releases or deployment approval.
|
||||
|
||||
A final read-only inspection found that ELITEDESK's installed executable and
|
||||
all nine modules match the earlier modular baseline at c8636ed exactly. Its
|
||||
entrypoint SHA256 is c91f09adfdeb45a3be1a72df3f5ca5b02fd4555dec4397ad944eab439ace0b89.
|
||||
The promotion-scoring candidate is not installed there.
|
||||
Reference in New Issue
Block a user