Tag: High Availability

Critical Kafka cluster degradation detected: Multiple partitions have lost replicas due to broker failure,resulting in an under-replicated state. This pattern indicates a broker has become unavailable, causingpartition leadership changes and In-Sync Replica (ISR) shrinkage across multiple topics.

CoreDNS deployment is unavailable or has no ready endpoints, indicating an imminent cluster-wide DNS outage.

Detects critical Nginx upstream failure cascades that lead to complete service unavailability.This advanced rule identifies comprehensive upstream failure patterns including DNS resolutionfailures, connection timeouts, SSL/TLS handshake errors, protocol violations, and serverunavailability, followed by HTTP 5xx error responses within a 60-second window.The rule uses optimized regex patterns for maximum detection coverage while maintaininghigh performance and low false-positive rates. It captures both the root cause (upstreamfailures) and the user-facing impact (HTTP errors) to provide complete incident context.

Detects when Slurm's accounting daemon (slurmdbd) or controller (slurmctld) loses connection to its MySQL database, causing job scheduling and recording to halt.

During rolling updates, when a deployment's maxUnavailable setting conflicts with a Pod Disruption Budget's minAvailable requirement, it can cause service outages by terminating too many pods simultaneously, violating the availability guarantees.This can also occur during node drains, cluster autoscaling, or maintenance operations.

Critical NGINX Ingress Controller SSL certificate validation failure detected. This pattern indicatescascading SSL failures where certificate verification errors lead to upstream connection failuresand service unavailability. The failure sequence shows SSL handshake failures, certificate verificationerrors, and resulting HTTP error responses that affect client connectivity.

Critical AWS VPC CNI IP address exhaustion detected. This pattern indicates cascading failureswhere subnet IP exhaustion leads to ENI allocation failures, pod scheduling failures, andcomplete service unavailability. The failure sequence shows IP allocation errors, ENI attachmentfailures, and resulting pod startup failures that affect cluster scalability and workload deployment.