PREQUEL-2025-0096
Loki Ingester Memcache Object Size Limit ExceededMediumImpact: 7/10Mitigation: 6/10
Description
Loki ingester encounters "object too large for cache" errors when attempting to store log entries \nexceeding memcache's configured size limit (typically 1MB). Large log lines remain in the ingester \nbuffer causing continuous failed ingest attempts, pod health degradation, and eventual recycling.\nThe accumulation of oversized entries can lead to buffer exhaustion and ingester instability.\n
Mitigation
- Increase memcache max_item_size configuration to handle larger objects\n- Implement log truncation or filtering at source to prevent oversized entries\n- Configure log rotation and size limits in application logging frameworks\n- Monitor log entry sizes and set alerts for approaching size limits\n- Consider alternative storage backends for very large log entries\n- Implement chunking for large log entries at the application level\n