Tag: Loki
Problems with Grafana Loki
| ID | Title | Description | Category | Technology | Tags |
|---|---|---|---|---|---|
| prequel-2025-0003 Low Impact: 4/10 Mitigation: 5/10 | Loki WAL Out of Disk Space | Loki is experiencing an out of disk space error due to the WAL (Write-Ahead Logging) filling up the disk. This can happen when the WAL is not properly configured or when the disk is full. | Storage Problems | loki | LokiWALDisk SpaceOut of Disk SpaceDisk Full |
| prequel-2025-0096 Medium Impact: 7/10 Mitigation: 6/10 | Loki Ingester Memcache Object Size Limit Exceeded | Loki ingester encounters "object too large for cache" errors when attempting to store log entries exceeding memcache's configured size limit (typically 1MB). Large log lines remain in the ingester buffer causing continuous failed ingest attempts, pod health degradation, and eventual recycling. The accumulation of oversized entries can lead to buffer exhaustion and ingester instability. | Observability Problems | ingester | LokiIngesterMemcachedObject Size LimitCacheStorageObservabilityTelemetryThreshold ExceededData LossConfiguration |
| prequel-2025-0097 Medium Impact: 6/10 Mitigation: 5/10 | Loki Compactor Schema Table Mismatch | Loki compactor encounters schema configuration mismatches when it finds index tables in object storage that don't correspond to any configured schema period in the Loki configuration. This causes the compactor to skip compaction for those tables, leading to storage inefficiency and potential query performance degradation. The issue typically occurs after schema migrations, configuration changes, or when legacy data exists with different table naming conventions. | Observability Problems | compactor | LokiCompactorSchemaConfigurationStorageObservabilityIndex |
| prequel-2025-0098 Medium Impact: 6/10 Mitigation: 4/10 | Loki Pattern Ingester Empty Ring | Loki distributor encounters "empty ring" errors when attempting to send streams to pattern ingesters. This occurs when pattern ingestion is enabled in the configuration but no pattern-ingester pods are running or properly registered in the ring. The distributor's pattern-tee component cannot find any available pattern ingesters to process pattern extraction, leading to high error spam in logs while normal log ingestion continues to function. | Observability Problems | distributor | LokiConfigurationObservabilityDeploymentReplication |
| prequel-2025-0101 Medium Impact: 6/10 Mitigation: 5/10 | Loki Ingester Memcache Out of Memory | Loki ingester reports memcached errors indicating out-of-memory conditions while caching objects, logging messages such as "SERVER_ERROR out of memory storing object". When this occurs, cache writes fail and can lead to degraded ingestion performance, retries, and increased memory pressure on the ingester. | Observability Problems | ingester | LokiIngesterMemcachedStorageCacheMemoryData LossThreshold ExceededObservabilityConfiguration |