PREQUEL-2025-0100
Prometheus ingestion failure due to too many labelsMediumImpact: 6/10Mitigation: 4/10
Description
Grafana Mimir's distributor rejects incoming Prometheus series when the number of label\nnames on a single series exceeds the configured per-tenant limit. When this occurs, logs\ncontain the message "received a series whose number of labels exceeds the limit" and the\naffected samples are dropped. This typically arises from excessive or dynamic labeling in\nscrape targets or relabeling rules that generate many unique label names per series.\n\nTo adjust the per-tenant limit, configure the distributor with\n`-validation.max-label-names-per-series`. When deploying via the `mimir-distributed` Helm\nchart, set `mimir.structuredConfig.limits.max_label_names_per_series` to a higher value\n(default is 30). Increase limits cautiously to avoid cardinality explosions and memory\npressure. Prefer reducing label names at the source where possible.\n
Mitigation
- Reduce the number of label names produced by scrape targets and relabeling rules\n- If necessary, raise the per-tenant limit:\n - CLI: set `-validation.max-label-names-per-series`\n - Helm (mimir-distributed): set `mimir.structuredConfig.limits.max_label_names_per_series`\n to a value higher than 30\n- Apply changes per-tenant where appropriate to avoid broad impact\n- Audit exporters and relabeling configs to remove dynamic or unnecessary label keys\n