prometheus.remote_write silently drops any data older than the age of the alloy process itself

[ellisonch]

Component(s)

prometheus.remote_write

What's wrong?

I have an instance of alloy running that receives data from various scrapers via a prometheus.receive_http. I then configured alloy to send data it receives to a backend via prometheus.remote_write. This is working fine, for any sample younger than the alloy instance.

However, I noticed that if I send alloy samples with timestamps older than when alloy started up, those samples do not get put in the backend. I've verified via the live debugging tool for prometheus.remote_write that it is seeing these samples; they simply do not make it to the backend. Any sample with a timestamp younger than when alloy started up works fine.

E.g., live_debugger shows

sample: ts=1769032229115, labels={__name__="test_backfill", ...}, value=1769035229.115777

but because the ts is ~50 minutes ago, and alloy only started up 5 minutes ago, this sample never makes it to the backend.

I imagine what's going on is that there's a datastructure somewhere that gets initted on startup, and given some kind of "minimum valid timestamp" that is now(). If this is related to the wal, it seems like this minimum valid timestamp should actually be something like now() - max_keepalive_time).

I'd like to have samples pass through alloy without being discarded. Is there any way to work around this limitation?

Steps to reproduce

prometheus.remote_write "primary" {
  endpoint {
    url = ...
   ...
  }

  wal {
    truncate_frequency = "2h"
    min_keepalive_time = "7h" // adjusting this does not help
    max_keepalive_time = "8h"
  }
}

prometheus.receive_http "metrics_ingress" {
  http {
    listen_address = "0.0.0.0"
    listen_port    = 8080
  }

  forward_to = [prometheus.remote_write.primary.receiver]
}

start it up, and send it metrics with timestamps older than alloy's startup time.

System information

linux arm

Software version

alloy v1.11.3 via helm chart 1.4.0

Tip

_{React with 👍 if this issue is important to you.}

[bboreham]

Since Alloy re-uses this code from Prometheus, most of the discussion is in the Prometheus repo.

• I think this is the best hope currently: proposal: Remote Write: Restart from segment-based savepoint prometheus/proposals#72