CRE-2025-0119
Kubernetes Pod Disruption Budget (PDB) Violation During Rolling UpdatesHighImpact: 8/10Mitigation: 7/10
Description
During rolling updates, when a deployment's maxUnavailable setting conflicts with \na Pod Disruption Budget's minAvailable requirement, it can cause service outages \nby terminating too many pods simultaneously, violating the availability guarantees.\nThis can also occur during node drains, cluster autoscaling, or maintenance operations.\n
Mitigation
**Immediate Actions:**\n1. Pause the rolling update:\n ```\n kubectl rollout pause deployment/<deployment-name>\n ```\n2. Verify PDB and deployment settings:\n ```\n kubectl get pdb\n kubectl get deployment <deployment-name> -o yaml\n ```\n3. Adjust maxUnavailable to respect PDB:\n ```\n kubectl patch deployment/<deployment-name> -p '{\"spec\":{\"strategy\":{\"rollingUpdate\":{\"maxUnavailable\":\"1\"}}}}'\n ```\n4. Check node conditions and drain status:\n ```\n kubectl get nodes\n kubectl get pods -o wide\n ```\n\n**Long-term fixes:**\n- Ensure deployment's maxUnavailable setting respects PDB requirements\n- Implement pre-deployment validation checks\n- Use progressive delivery (canary/blue-green) for critical services\n- Monitor PDB violations through metrics/alerts\n- Configure cluster autoscaler to respect PDBs\n- Implement node maintenance windows\n- Use pod anti-affinity to spread critical workloads\n- Set up automated rollback triggers on PDB violations\n