Skip to main content

PREQUEL-2025-0098

Loki Pattern Ingester Empty RingMedium
Impact: 6/10
Mitigation: 4/10

PREQUEL-2025-0098View on GitHub

Description

Loki distributor encounters "empty ring" errors when attempting to send streams to pattern ingesters.

This occurs when pattern ingestion is enabled in the configuration but no pattern-ingester pods are

running or properly registered in the ring. The distributor's pattern-tee component cannot find any

available pattern ingesters to process pattern extraction, leading to high error spam in logs while

normal log ingestion continues to function.


Cause

  • Pattern ingester feature enabled but no pattern-ingester pods deployed (replicas set to 0)
  • Pattern-ingester pods failed to start or join the ring properly
  • Ring configuration mismatch between distributor and pattern-ingester components
  • Network connectivity issues preventing pattern-ingester registration in the ring
  • KV store or gossip configuration inconsistencies between components
  • Incorrect replication factor settings for pattern ingester (must be 1)
  • Pattern-ingester pods not listening on expected gRPC/HTTP ports

Mitigation

  • Verify pattern-ingester pods are deployed with replicas > 0 in Helm values
  • Check pattern-ingester pod status: kubectl get pods -l app.kubernetes.io/component=pattern-ingester -A
  • Ensure pattern_ingester.enabled: true AND patternIngester.replicas: 1+ in Helm configuration
  • Verify ring membership: curl http://localhost:3100/ring from distributor pod
  • Check pattern-ingester logs for startup or ring join errors
  • Validate KV store and gossip configuration consistency across components
  • Confirm network policies allow distributor to pattern-ingester communication
  • Verify pattern-ingester replication factor is set to 1
  • Review pattern-ingester port configuration and health check endpoints

References