Skip to main content

CRE-2024-0008

RabbitMQ memory alarmHigh
Impact: 9/10
Mitigation: 6/10

CRE-2024-0008View on GitHub

Description

A RabbitMQ node has entered the “memory alarm” state because the total memory used by the Erlang VM (plus allocated binaries, ETS tables,

and processes) has exceeded the configured `vm_memory_high_watermark`. While the alarm is active the broker

applies flow-control, blocking publishers and pausing most ingress activity to protect itself from running out of RAM.


Cause

  • Large or rapidly growing queues keeping many messages in RAM.
  • Messages with very large payloads or headers.
  • Insufficient memory limits / cgroup constraints on the container or VM.
  • Consumers that are slow, offline, or ACKing messages late, preventing RAM from being reclaimed.
  • A too-low `vm_memory_high_watermark` setting relative to available memory on the host.

Mitigation

  • Inspect memory usage to identify queues or processes consuming RAM:

`rabbitmq-diagnostics memory_breakdown -n <node>`

  • Purge or truncate unused / dead-letter queues holding many messages.
  • Scale or speed up consumers so messages are ACKed and cleared from memory faster.
  • Enable “lazy queues” or set TTLs / max-length limits for queues that can tolerate disk-based storage.
  • Increase the node’s memory allocation or raise `vm_memory_high_watermark` (e.g., to 0.8) only after confirming the host has sufficient physical RAM.
  • In Kubernetes, also raise the Pod memory limit so the broker can use the additional headroom without OOM-killing.
  • Long-term: review message sizes, batch patterns, and queue topology to prevent unbounded growth.

References