Skip to main content

PREQUEL-2025-0096

Loki Ingester Memcache Object Size Limit ExceededMedium
Impact: 7/10
Mitigation: 6/10

PREQUEL-2025-0096View on GitHub

Description

Loki ingester encounters "object too large for cache" errors when attempting to store log entries

exceeding memcache's configured size limit (typically 1MB). Large log lines remain in the ingester

buffer causing continuous failed ingest attempts, pod health degradation, and eventual recycling.

The accumulation of oversized entries can lead to buffer exhaustion and ingester instability.


Cause

  • Log lines exceeding memcache's maximum object size limit (default 1MB)
  • Applications generating extremely large single log entries or stack traces
  • Serialization bloat increasing entry size beyond application expectations

Mitigation

  • Increase memcache max_item_size configuration to handle larger objects
  • Implement log truncation or filtering at source to prevent oversized entries
  • Configure log rotation and size limits in application logging frameworks
  • Monitor log entry sizes and set alerts for approaching size limits
  • Consider alternative storage backends for very large log entries
  • Implement chunking for large log entries at the application level

References