Skip to main content

CRE-2025-0038

Loki fails to cache entries due to Memcached out-of-memory errorLow
Impact: 5/10
Mitigation: 3/10

CRE-2025-0038View on GitHub

Description

Grafana Loki may emit errors when attempting to write to a Memcached backend that has run out of available memory. This results in dropped index or query cache entries, which can degrade query performance but does not interrupt ingestion.


Cause

Loki uses Memcached to cache query results and index lookups. When Memcached exhausts its allocated memory, it may respond with `SERVER_ERROR out of memory storing object\\r\

`, preventing new items from being cached. Loki logs this as an exporter or client error when the storage backend rejects the cache operation.


Mitigation

  • Increase the memory limit allocated to your Memcached deployment.
  • Use Memcached eviction policies (e.g., LRU) or tune object size limits.
  • Monitor Memcached memory metrics (e.g., `memcached_mem_bytes`) and set alerts for high usage.
  • Consider using a more robust backend like Redis for larger or persistent caches.

References