CRE-2025-0020

Self-hosted PostgreSQL HA: WAL Streaming & HA Controller Crisis (Replication Slot Loss, Disk Full, Etcd Quorum Failure)High

Impact: 10/10

Mitigation: 6/10

CRE-2025-0020View on GitHub

High AvailabilityPatroniZalandoEtcdReplicationWALStorageQuorumCrashData LossTimeout

Description

Detects high-severity failures in self-hosted PostgreSQL high-availability clusters managed by Patroni, Zalando, or similar HA controllers.\nThis rule targets catastrophic conditions that break replication or cluster consensus:\n - WAL streaming failures due to missing replication slots (usually after disk full or crash events)\n - Persistent errors resolving HA controller endpoints (etcd/consul) and loss of HA controller quorum\n - Disk saturation leading to WAL write errors and replication breakage\n

Mitigation

PREVENTION:\n - Monitor disk usage on all PostgreSQL nodes, especially WAL and archive directories\n - Set up alerting for replication lag and missing replication slots\n - Ensure HA controllers (etcd/consul) are running on redundant, reliable nodes\nRESPONSE:\n - Restore or recreate missing replication slots\n - Free up disk space and restart affected PostgreSQL instances\n - Restore etcd/consul cluster quorum; check container/network status\n - Perform manual failover if automatic recovery fails\n

References

• https://patroni.readthedocs.io/en/latest/
• https://www.postgresql.org/docs/current/warm-standby.html
• https://etcd.io/docs/latest/op-guide/clustering/