Prometheus Rate Limit

[opencow]

Component(s)

prometheus.scrape

Background

Today, relabeling can statically keep or drop samples based on labels, but it cannot rate-limit samples by label value.
Our use case is an Alloy deployment that scrapes metrics from multiple products. A single noisy product can affect metric ingestion for other products. We’d like to apply independent rate limits using a label such as product or namespace.

Proposal

A new rate limit component prometheus.limit, taking inspiration from the loki.process's stage.limit configurations.

prometheus.limit "metrics" {
  forward_to         = [prometheus.remote_write.mimir.receiver]
  rate               = 1000
  burst              = 2000
  drop               = true
  by_label_name      = "namespace"
  max_distinct_labels = 10000
}

Will put out a PR if the proposal is considered!

AI disclosure

• This issue was substantially generated with AI assistance.

Tip

_{React with 👍 if this issue is important to you.}

[kgeckhart]

Thanks for the proposal, looking at the premise

A single noisy product can affect metric ingestion for other products.

Are these products individually scraped? The scrape component has a number of limit arguments which can be applied to prevent label explosions as described, https://grafana.com/docs/alloy/latest/reference/components/prometheus/prometheus.scrape/#arguments. Since it's failing at scrape time you can also much more easily monitor the up metric for failures.

I think the proposal itself needs a bit more details, how are you proposing rate / burst / max_distict_labels operate? How does someone know their metrics are being rate limited?