Skip to main content

PREQUEL-2025-0101

Loki Ingester Memcache Out of MemoryMedium
Impact: 6/10
Mitigation: 5/10

PREQUEL-2025-0101View on GitHub

Description

Loki ingester reports memcached errors indicating out-of-memory conditions while caching

objects, logging messages such as "SERVER_ERROR out of memory storing object". When this

occurs, cache writes fail and can lead to degraded ingestion performance, retries, and

increased memory pressure on the ingester.


Cause

  • Memcached instance memory exhausted due to workload/cache size
  • Insufficient memcached memory limits or large working set
  • Cache configuration leading to high churn/eviction pressure

Mitigation

  • Verify memcached memory allocation; consider increasing memory requests/limits
  • Reduce volume of cached data or adjust cache policies to fit capacity
  • Monitor cache hit/miss and eviction metrics to tune sizing appropriately
  • Review object sizes; consider compression or truncation strategies where applicable

References