Skip to main content

CRE-2025-0058

Celery Worker Stops Consuming Tasks After Redis RestartMedium
Impact: 7/10
Mitigation: 4/10

CRE-2025-0058View on GitHub

Description

  • When Redis is restarted, Celery workers using Redis as a broker may stop consuming tasks without exiting or logging a fatal error.
  • Although Celery Beat continues to publish tasks successfully, the worker remains in a broken state until manually restarted.
  • This results in a silent backlog of scheduled but unprocessed tasks.

Cause

  • After Redis restarts, open connections from the Celery worker are closed by the server.
  • The Celery worker does not detect or recover from the dropped connection, entering a non‑responsive state.
  • This issue is caused by a bug in Kombu (Celery's messaging library) affecting connection recovery.

Mitigation

  • Monitor Redis availability and trigger Celery worker restarts automatically when Redis restarts.
  • Consider using process managers like `supervisord`, `systemd`, or Kubernetes liveness probes to enforce recovery.
  • Upgrade to Celery 5.4.0 or later, which includes Kombu 5.4.0 where the issue is fixed.
  • If upgrading is not possible, implement monitoring for task queue depth and alert on worker stalls.

References