Tag: Threshold Exceeded
An external API limit has been exceeded and API requests are currently failing or being throttled
| ID | Title | Description | Category | Technology | Tags |
|---|---|---|---|---|---|
| prequel-2025-0096 Medium Impact: 7/10 Mitigation: 6/10 | Loki Ingester Memcache Object Size Limit Exceeded | Loki ingester encounters "object too large for cache" errors when attempting to store log entries exceeding memcache's configured size limit (typically 1MB). Large log lines remain in the ingester buffer causing continuous failed ingest attempts, pod health degradation, and eventual recycling. The accumulation of oversized entries can lead to buffer exhaustion and ingester instability. | Observability Problems | ingester | LokiIngesterMemcachedObject Size LimitCacheStorageObservabilityTelemetryThreshold ExceededData LossConfiguration |
| prequel-2025-0100 Medium Impact: 6/10 Mitigation: 4/10 | Prometheus ingestion failure due to too many labels | Grafana Mimir's distributor rejects incoming Prometheus series when the number of label names on a single series exceeds the configured per-tenant limit. When this occurs, logs contain the message "received a series whose number of labels exceeds the limit" and the affected samples are dropped. This typically arises from excessive or dynamic labeling in scrape targets or relabeling rules that generate many unique label names per series. To adjust the per-tenant limit, configure the distributor with `-validation.max-label-names-per-series`. When deploying via the `mimir-distributed` Helm chart, set `mimir.structuredConfig.limits.max_label_names_per_series` to a higher value (default is 30). Increase limits cautiously to avoid cardinality explosions and memory pressure. Prefer reducing label names at the source where possible. | Observability Problems | distributor | PrometheusGrafanaMetricsConfigurationConfiguration IssueThreshold ExceededObservability |
| prequel-2025-0101 Medium Impact: 6/10 Mitigation: 5/10 | Loki Ingester Memcache Out of Memory | Loki ingester reports memcached errors indicating out-of-memory conditions while caching objects, logging messages such as "SERVER_ERROR out of memory storing object". When this occurs, cache writes fail and can lead to degraded ingestion performance, retries, and increased memory pressure on the ingester. | Observability Problems | ingester | LokiIngesterMemcachedStorageCacheMemoryData LossThreshold ExceededObservabilityConfiguration |