Skip to main content

CRE-2025-0119

Kubernetes Pod Disruption Budget (PDB) Violation During Rolling UpdatesHigh
Impact: 8/10
Mitigation: 7/10

CRE-2025-0119View on GitHub

Description

During rolling updates, when a deployment's maxUnavailable setting conflicts with

a Pod Disruption Budget's minAvailable requirement, it can cause service outages

by terminating too many pods simultaneously, violating the availability guarantees.

This can also occur during node drains, cluster autoscaling, or maintenance operations.


Cause

  • Deployment's rolling update strategy sets maxUnavailable higher than PDB allows
  • PDB requires minAvailable pods but rolling update violates this constraint
  • Concurrent pod terminations exceed the allowed disruption threshold
  • Deployment configuration conflicts with PDB policy
  • Node drains or cluster autoscaling events trigger multiple simultaneous pod evictions
  • Resource pressure or node failures force pod relocations
  • Maintenance operations affecting multiple nodes simultaneously

Mitigation

Immediate Actions:

  1. Pause the rolling update:
   kubectl rollout pause deployment/<deployment-name>
  1. Verify PDB and deployment settings:
   kubectl get pdb   kubectl get deployment <deployment-name> -o yaml
  1. Adjust maxUnavailable to respect PDB:
   kubectl patch deployment/<deployment-name> -p '{"spec":{"strategy":{"rollingUpdate":{"maxUnavailable":"1"}}}}'
  1. Check node conditions and drain status:
   kubectl get nodes   kubectl get pods -o wide

Long-term fixes:

  • Ensure deployment's maxUnavailable setting respects PDB requirements
  • Implement pre-deployment validation checks
  • Use progressive delivery (canary/blue-green) for critical services
  • Monitor PDB violations through metrics/alerts
  • Configure cluster autoscaler to respect PDBs
  • Implement node maintenance windows
  • Use pod anti-affinity to spread critical workloads
  • Set up automated rollback triggers on PDB violations

References