Skip to main content

PREQUEL-2025-0075

Prometheus Config Reload FailedLow
Impact: 2/10
Mitigation: 5/10

PREQUEL-2025-0075View on GitHub

Description

The **prometheus-config-reloader** sidecar (used by the Prometheus\nOperator / kube-prometheus-stack) detected a change in the\nConfigMap/Secret but cannot POST to the Prometheus `/-/reload`\nendpoint. It logs repeatedly:\n\n```\nFailed to trigger reload. Retrying.\n```\n\nWhile the main Prometheus container keeps serving traffic, **new\nscrape configs, alerting rules, and recording rules are NOT applied**,\nleaving the instance frozen on an outdated configuration set.\n

Mitigation

1. **Check Prometheus logs** for parse errors:\n ```bash\n kubectl logs -l app=prometheus -c prometheus | grep 'error loading config'\n ```\n Then run `promtool check config prometheus.yml`.\n2. **Verify the reload endpoint** is reachable inside the pod:\n ```bash\n curl -sv -X POST http://localhost:9090/-/reload\n ```\n3. **Align web settings** – if you enabled TLS or auth on the main\n container, mirror the same options in the `--web.enable-lifecycle`\n and sidecar args.\n4. **Loosen or update NetworkPolicies / mesh policies** so the\n sidecar can talk to `127.0.0.1:9090`.\n5. After corrections, **restart the Prometheus pod** to force a clean\n load and watch for a single\n `Reloaded configuration file` entry in logs.\n

References