Tag: Prometheus
Problems with scraping, rule evaluation, or querying Prometheus data.
| ID | Title | Description | Category | Technology | Tags |
|---|---|---|---|---|---|
| prequel-2025-0002 Medium Impact: 7/10 Mitigation: 3/10 | Envoy metrics scraping failure with unexpected EOF | Prometheus is failing to scrape and write Envoy metrics from Istio sidecars due to an unexpected EOF error. This occurs when trying to collect metrics from services that don't have proper protocol selection configured in their Kubernetes Service definition | Service Mesh Monitoring | prometheus | PrometheusIstioEnvoyMetricsService MeshKubernetes |
| prequel-2025-0075 Low Impact: 2/10 Mitigation: 5/10 | Prometheus Config Reload Failed | The **prometheus-config-reloader** sidecar (used by the Prometheus Operator / kube-prometheus-stack) detected a change in the ConfigMap/Secret but cannot POST to the Prometheus `/-/reload` endpoint. It logs repeatedly: ``` Failed to trigger reload. Retrying. ``` While the main Prometheus container keeps serving traffic, **new scrape configs, alerting rules, and recording rules are NOT applied**, leaving the instance frozen on an outdated configuration set. | Prometheus Problems | prometheus | Prometheus |
| prequel-2025-0100 Medium Impact: 6/10 Mitigation: 4/10 | Prometheus ingestion failure due to too many labels | Grafana Mimir's distributor rejects incoming Prometheus series when the number of label names on a single series exceeds the configured per-tenant limit. When this occurs, logs contain the message "received a series whose number of labels exceeds the limit" and the affected samples are dropped. This typically arises from excessive or dynamic labeling in scrape targets or relabeling rules that generate many unique label names per series. To adjust the per-tenant limit, configure the distributor with `-validation.max-label-names-per-series`. When deploying via the `mimir-distributed` Helm chart, set `mimir.structuredConfig.limits.max_label_names_per_series` to a higher value (default is 30). Increase limits cautiously to avoid cardinality explosions and memory pressure. Prefer reducing label names at the source where possible. | Observability Problems | distributor | PrometheusGrafanaMetricsConfigurationConfiguration IssueThreshold ExceededObservability |