Tag: Metrics

Problems related to metrics collection or reporting, such as missing, delayed, or incorrect data

ID	Title	Description	Category	Technology	Tags
prequel-2025-0002 Medium Impact: 7/10 Mitigation: 3/10	Envoy metrics scraping failure with unexpected EOF	Prometheus is failing to scrape and write Envoy metrics from Istio sidecars due to an unexpected EOF error. This occurs when trying to collect metrics from services that don't have proper protocol selection configured in their Kubernetes Service definition	Service Mesh Monitoring	prometheus	Prometheus Istio Envoy Metrics Service Mesh Kubernetes
prequel-2025-0100 Medium Impact: 6/10 Mitigation: 4/10	Prometheus ingestion failure due to too many labels	Grafana Mimir's distributor rejects incoming Prometheus series when the number of label names on a single series exceeds the configured per-tenant limit. When this occurs, logs contain the message "received a series whose number of labels exceeds the limit" and the affected samples are dropped. This typically arises from excessive or dynamic labeling in scrape targets or relabeling rules that generate many unique label names per series. To adjust the per-tenant limit, configure the distributor with `-validation.max-label-names-per-series`. When deploying via the `mimir-distributed` Helm chart, set `mimir.structuredConfig.limits.max_label_names_per_series` to a higher value (default is 30). Increase limits cautiously to avoid cardinality explosions and memory pressure. Prefer reducing label names at the source where possible.	Observability Problems	distributor	Prometheus Grafana Metrics Configuration Configuration Issue Threshold Exceeded Observability