Skip to main content

Tag: Storage

Failures in block, object, or ephemeral storage backends.

IDTitleDescriptionCategoryTechnologyTags
prequel-2025-0096
Medium
Impact: 7/10
Mitigation: 6/10
Loki Ingester Memcache Object Size Limit ExceededLoki ingester encounters "object too large for cache" errors when attempting to store log entries exceeding memcache's configured size limit (typically 1MB). Large log lines remain in the ingester buffer causing continuous failed ingest attempts, pod health degradation, and eventual recycling. The accumulation of oversized entries can lead to buffer exhaustion and ingester instability.Observability ProblemsingesterLokiIngesterMemcachedObject Size LimitCacheStorageObservabilityTelemetryThreshold ExceededData LossConfiguration
prequel-2025-0097
Medium
Impact: 6/10
Mitigation: 5/10
Loki Compactor Schema Table MismatchLoki compactor encounters schema configuration mismatches when it finds index tables in object storage that don't correspond to any configured schema period in the Loki configuration. This causes the compactor to skip compaction for those tables, leading to storage inefficiency and potential query performance degradation. The issue typically occurs after schema migrations, configuration changes, or when legacy data exists with different table naming conventions.Observability ProblemscompactorLokiCompactorSchemaConfigurationStorageObservabilityIndex
prequel-2025-0101
Medium
Impact: 6/10
Mitigation: 5/10
Loki Ingester Memcache Out of MemoryLoki ingester reports memcached errors indicating out-of-memory conditions while caching objects, logging messages such as "SERVER_ERROR out of memory storing object". When this occurs, cache writes fail and can lead to degraded ingestion performance, retries, and increased memory pressure on the ingester.Observability ProblemsingesterLokiIngesterMemcachedStorageCacheMemoryData LossThreshold ExceededObservabilityConfiguration