Skip to main content

Tag: Configuration

Problems caused by incorrect or missing configuration settings

IDTitleDescriptionCategoryTechnologyTags
prequel-2025-0093
Medium
Impact: 8/10
Mitigation: 5/10
aws-load-balancer-controller rejects Ingress resource with wildcard path and Prefix pathTypeThe aws-load-balancer-controller is unable to translate an Ingress resource into an AWS ALB Listener Rule when the path contains a wildcard (*) and the pathType is set to Prefix.Kubernetes Networking Problemsaws-load-balancer-controllerKubernetesAWS Loadbalancer ControllerIngress ResourceAWSNetworkingConfigurationPath ValidationALBRouting
prequel-2025-0096
Medium
Impact: 7/10
Mitigation: 6/10
Loki Ingester Memcache Object Size Limit ExceededLoki ingester encounters "object too large for cache" errors when attempting to store log entries exceeding memcache's configured size limit (typically 1MB). Large log lines remain in the ingester buffer causing continuous failed ingest attempts, pod health degradation, and eventual recycling. The accumulation of oversized entries can lead to buffer exhaustion and ingester instability.Observability ProblemsingesterLokiIngesterMemcachedObject Size LimitCacheStorageObservabilityTelemetryThreshold ExceededData LossConfiguration
prequel-2025-0097
Medium
Impact: 6/10
Mitigation: 5/10
Loki Compactor Schema Table MismatchLoki compactor encounters schema configuration mismatches when it finds index tables in object storage that don't correspond to any configured schema period in the Loki configuration. This causes the compactor to skip compaction for those tables, leading to storage inefficiency and potential query performance degradation. The issue typically occurs after schema migrations, configuration changes, or when legacy data exists with different table naming conventions.Observability ProblemscompactorLokiCompactorSchemaConfigurationStorageObservabilityIndex
prequel-2025-0098
Medium
Impact: 6/10
Mitigation: 4/10
Loki Pattern Ingester Empty RingLoki distributor encounters "empty ring" errors when attempting to send streams to pattern ingesters. This occurs when pattern ingestion is enabled in the configuration but no pattern-ingester pods are running or properly registered in the ring. The distributor's pattern-tee component cannot find any available pattern ingesters to process pattern extraction, leading to high error spam in logs while normal log ingestion continues to function.Observability ProblemsdistributorLokiConfigurationObservabilityDeploymentReplication
prequel-2025-0099
Medium
Impact: 6/10
Mitigation: 3/10
DataDog Agent Remote Configuration ErrorDataDog Agent encounters "empty targets meta in director local store" errors when attempting to retrieve remote configuration. This issue affects APM (Application Performance Monitoring) remote configuration functionality in DataDog Agent versions between 7.61.0 and 7.68.0. The error prevents proper retrieval and parsing of remote configuration from DataDog's backend, causing APM tracer libraries to fail when attempting to fetch dynamic configuration updates.Observability ProblemsdatadogDatadogObservabilityConfiguration
prequel-2025-0100
Medium
Impact: 6/10
Mitigation: 4/10
Prometheus ingestion failure due to too many labelsGrafana Mimir's distributor rejects incoming Prometheus series when the number of label names on a single series exceeds the configured per-tenant limit. When this occurs, logs contain the message "received a series whose number of labels exceeds the limit" and the affected samples are dropped. This typically arises from excessive or dynamic labeling in scrape targets or relabeling rules that generate many unique label names per series. To adjust the per-tenant limit, configure the distributor with `-validation.max-label-names-per-series`. When deploying via the `mimir-distributed` Helm chart, set `mimir.structuredConfig.limits.max_label_names_per_series` to a higher value (default is 30). Increase limits cautiously to avoid cardinality explosions and memory pressure. Prefer reducing label names at the source where possible.Observability ProblemsdistributorPrometheusGrafanaMetricsConfigurationConfiguration IssueThreshold ExceededObservability
prequel-2025-0101
Medium
Impact: 6/10
Mitigation: 5/10
Loki Ingester Memcache Out of MemoryLoki ingester reports memcached errors indicating out-of-memory conditions while caching objects, logging messages such as "SERVER_ERROR out of memory storing object". When this occurs, cache writes fail and can lead to degraded ingestion performance, retries, and increased memory pressure on the ingester.Observability ProblemsingesterLokiIngesterMemcachedStorageCacheMemoryData LossThreshold ExceededObservabilityConfiguration