Skip to main content

PREQUEL-2025-0100

Prometheus ingestion failure due to too many labelsMedium
Impact: 6/10
Mitigation: 4/10

PREQUEL-2025-0100View on GitHub

Description

Grafana Mimir's distributor rejects incoming Prometheus series when the number of label

names on a single series exceeds the configured per-tenant limit. When this occurs, logs

contain the message "received a series whose number of labels exceeds the limit" and the

affected samples are dropped. This typically arises from excessive or dynamic labeling in

scrape targets or relabeling rules that generate many unique label names per series.


To adjust the per-tenant limit, configure the distributor with

`-validation.max-label-names-per-series`. When deploying via the `mimir-distributed` Helm

chart, set `mimir.structuredConfig.limits.max_label_names_per_series` to a higher value

(default is 30). Increase limits cautiously to avoid cardinality explosions and memory

pressure. Prefer reducing label names at the source where possible.


Cause

  • Excessive label names on scraped targets or exporters
  • Misconfigured relabeling adding many distinct label keys per series
  • Multi-tenant limit too low for specific workloads
  • Client libraries or sidecars attaching dynamic label keys

Mitigation

  • Reduce the number of label names produced by scrape targets and relabeling rules
  • If necessary, raise the per-tenant limit:
  • CLI: set `-validation.max-label-names-per-series`
  • Helm (mimir-distributed): set `mimir.structuredConfig.limits.max_label_names_per_series`

to a value higher than 30

  • Apply changes per-tenant where appropriate to avoid broad impact
  • Audit exporters and relabeling configs to remove dynamic or unnecessary label keys

References