Alloy doesn't replay wal for recoverable error

[amanpruthi]

Component(s)

Alloy

What's wrong?

Ideally if the remote write endpoint goes down alloy should replay the metrics and then start ingesting new metrics.
but that is not happening I tried 2 use cases

1. Bought down prometheus to with alloy was writing by scaling it down and then bought it back up again. the metrics stopped coming in altogether .
2. Bought down prometheus shifted endpoint to Amazon managed prometheus after 10 mins but alloy started writing the latest data to the new endpoint .

We have seen in the past while using grafana agent that that actually used to replay the data from the WAL .
WAL settings
wal {
truncate_frequency = "15m"
min_keepalive_time = "1h"
max_keepalive_time = "2h"
}
queue_config {
max_shards = 20 // default 50
capacity = 20000 // default 10000
max_samples_per_send = 3500 // default 2000
min_shards = 3
min_backoff = "30s"
max_backoff = "15m"
retry_on_http_429 = true
batch_send_deadline = "55s"
}

Steps to reproduce

1. Install alloy with helm chart and enable persistent storage and store wal on a pvc
2. Add prometheus as a remote write endpoint
3. Validate metrics are coming into prometheus usingsum(scrape_samples_scraped)
4. Scale down prometheus .You will notice alloy is still scraping .After 10 mins scale prometheus and you will see that no data comes in

System information

No response

Software version

docker.io/grafana/alloy:v1.10.0

Configuration

logging {
      level = "debug"
    }
    prometheus.exporter.self "alloy" {}
    prometheus.scrape "demo" {
      targets    = prometheus.exporter.self.alloy.targets
      clustering { enabled = true }
      forward_to = [prometheus.remote_write.amp.receiver]
    }
    prometheus.remote_write "amp" {
    wal {
      truncate_frequency = "15m"
      min_keepalive_time = "1h"
      max_keepalive_time = "2h"
    }
    external_labels = {
      collector = "alloy",
    }
    endpoint {
      url = "http://prometheus-server.monitoring.svc.cluster.local:9090/api/v1/write"
      
      queue_config {
        max_shards           = 20   // default 50
        capacity             = 20000 // default 10000
        max_samples_per_send = 3500 // default 2000
        min_shards           = 3
        min_backoff          = "30s"
        max_backoff          = "15m"
        retry_on_http_429    = true
        batch_send_deadline  = "55s"
      }
    }

    // Write-ahead-log defaults are fine; keep them if you see back-pressure
    }
    //////////////////////////////////////////////////////////////////////////
    //  Discover *only* ServiceMonitors labelled team=loadgen
    //  Target auto-distribution is ON via `clustering { enabled = true }`
    prometheus.operator.servicemonitors "avalanche" {
      selector {
        match_labels = {
          "team"= "loadgen",
        }
      }
      clustering { enabled = true }
      scrape {
        default_scrape_interval = "1m"
        default_scrape_timeout  = "30s"
      }
      forward_to = [prometheus.remote_write.amp.receiver]
    }
    //////////////////////////////////////////////////////////////////////////
    //  Discover *only* ServiceMonitors labelled o11y=agent
    //  Target auto-distribution is ON via `clustering { enabled = true }`
    prometheus.operator.servicemonitors "o11y_agent" {
      selector {
        match_labels = {
          "service-monitor-selector"= "o11y-agent",
        }
      }

      clustering { enabled = true }

      //  (`sample_limit` is honoured by the scrape jobs derived from each SM.)

      scrape {
        default_scrape_interval = "1m"
        default_scrape_timeout  = "30s"
      }
      forward_to = [prometheus.remote_write.amp.receiver]
    }
    prometheus.operator.servicemonitors "o11y_agent_hugepayload" {
      selector {
        match_labels = {
          "service-monitor-selector"= "o11y-agent-hugepayload",
        }
      }

      clustering { enabled = true }

      //  (`sample_limit` is honoured by the scrape jobs derived from each SM.)

      scrape {
        default_scrape_interval = "1m"
        default_scrape_timeout  = "30s"
      }
      forward_to = [prometheus.remote_write.amp.receiver]
    }
    //////////////////////////////////////////////////////////////////////////
    //  Discover *only* PodMonitors labelled pod-monitor-selector=alloy
    //  Target auto-distribution is ON via `clustering { enabled = true }`
    prometheus.operator.podmonitors "alloy_pods_t10" {
      selector {
        match_labels = {
          "pod-monitor-selector" = "alloy",
        }
      }

      clustering { enabled = true }
      scrape {
        default_scrape_interval = "1m"
        default_scrape_timeout  = "30s"
      }
      forward_to = [prometheus.remote_write.amp.receiver]
    }
    //////////////////////////////////////////////////////////////////////////
    // Reusable Kubernetes node discovery
    discovery.kubernetes "nodes" {
      role = "node"
    }

    //////////////////////////////////////////////////////////////////////////
    // Target relabeling for kubelet /metrics via apiserver proxy
    discovery.relabel "nodes_kubelet" {
      targets = discovery.kubernetes.nodes.targets

      // expose node labels as normal labels
      rule {
        action = "labelmap"
        regex  = "__meta_kubernetes_node_label_(.+)"
      }

      // scrape through apiserver
      rule {
        action       = "replace"
        target_label = "__address__"
        replacement  = "kubernetes.default.svc:443"
      }
      rule {
        action        = "replace"
        source_labels = ["__meta_kubernetes_node_name"]
        target_label  = "__metrics_path__"
        regex         = "(.+)"
        replacement   = "/api/v1/nodes/$1/proxy/metrics"
      }
    }

    prometheus.scrape "kubernetes_nodes" {
      job_name          = "kubernetes-nodes"
      targets           = discovery.relabel.nodes_kubelet.output
      scheme            = "https"
      bearer_token_file = "/var/run/secrets/kubernetes.io/serviceaccount/token"
      sample_limit      = 20000

      tls_config {
        ca_file              = "/var/run/secrets/kubernetes.io/serviceaccount/ca.crt"
        insecure_skip_verify = true
      }

      clustering { enabled = true }

      forward_to = [prometheus.relabel.kubernetes_nodes_filter.receiver]
    }

    prometheus.relabel "kubernetes_nodes_filter" {
      

      // drop noisy kubelet buckets/counters
      rule {
        action        = "drop"
        source_labels = ["__name__"]
        regex         = "(storage_operation_duration_seconds_bucket|kubelet_runtime_operations_duration_seconds_bucket|rest_client_request_duration_seconds_bucket|rest_client_rate_limiter_duration_seconds_bucket|rest_client_response_size_bytes_bucket|rest_client_request_size_bytes_bucket|kubelet_http_requests_duration_seconds_bucket|csi_operations_seconds_bucket|kubelet_pod_worker_duration_seconds_bucket|kubelet_cgroup_manager_duration_seconds_bucket|authentication_token_cache_request_duration_seconds_bucket|storage_operation_duration_seconds_count|storage_operation_duration_seconds_sum)"
      }

      forward_to = [prometheus.remote_write.amp.receiver]
    }

    //////////////////////////////////////////////////////////////////////////
    // Target relabeling for node-exporter (:9100 on node InternalIP)
    discovery.relabel "nodes_node_exporter" {
      targets = discovery.kubernetes.nodes.targets


      rule {
        action = "labelmap"
        regex  = "__meta_kubernetes_node_label_(.+)"
      }

      rule {
        action        = "replace"
        source_labels = ["__meta_kubernetes_node_address_InternalIP"]
        target_label  = "__address__"
        regex         = "(.+)"
        replacement   = "$1:9100"
      }
    }

    prometheus.scrape "kubernetes_node_exporter" {
      job_name     = "kubernetes-node-exporter"
      targets      = discovery.relabel.nodes_node_exporter.output
      scheme       = "http"
      sample_limit = 20000

      clustering { enabled = true }

      forward_to = [prometheus.relabel.kubernetes_node_exporter_filter.receiver]
    }

    prometheus.relabel "kubernetes_node_exporter_filter" {
      // normalize labels
      rule {
        regex         = "(.+)"
        source_labels = ["kops_k8s_io_instancegroup"]
        target_label  = "instancegroup"
        replacement   = "$1"
      }
      rule {
        regex         = "(.+)"
        source_labels = ["instance"]
        target_label  = "kubernetes_node"
        replacement   = "$1"
      }

      forward_to = [prometheus.remote_write.amp.receiver]
    }

    //////////////////////////////////////////////////////////////////////////
    // Target relabeling for cAdvisor via apiserver proxy
    discovery.relabel "nodes_cadvisor" {
      targets = discovery.kubernetes.nodes.targets
      rule {
        action       = "replace"
        target_label = "__address__"
        replacement  = "kubernetes.default.svc:443"
      }
      rule {
        action        = "replace"
        source_labels = ["__meta_kubernetes_node_name"]
        target_label  = "__metrics_path__"
        regex         = "(.+)"
        replacement   = "/api/v1/nodes/$1/proxy/metrics/cadvisor"
      }
    }

    prometheus.scrape "kubernetes_nodes_cadvisor" {
      job_name          = "kubernetes-nodes-cadvisor"
      targets           = discovery.relabel.nodes_cadvisor.output
      scheme            = "https"
      bearer_token_file = "/var/run/secrets/kubernetes.io/serviceaccount/token"
      sample_limit      = 50000

      tls_config {
        ca_file              = "/var/run/secrets/kubernetes.io/serviceaccount/ca.crt"
        insecure_skip_verify = true
      }

      clustering { enabled = true }

      forward_to = [prometheus.relabel.kubernetes_nodes_cadvisor_filter.receiver]
    }

    prometheus.relabel "kubernetes_nodes_cadvisor_filter" {
      // rename labels
      rule {
        regex         = "(.+)"
        source_labels = ["container_name"]
        target_label  = "container"
        replacement   = "$1"
      }
      rule {
        regex         = "(.+)"
        source_labels = ["pod_name"]
        target_label  = "pod"
        replacement   = "$1"
      }
      rule {
        regex         = "(.+)"
        source_labels = ["namespace"]
        target_label  = "kubernetes_namespace"
        replacement   = "$1"
      }



      forward_to = [prometheus.remote_write.amp.receiver]
    }

Logs

ts=2025-09-19T08:15:59.528472949Z level=debug msg=QueueManager.calculateDesiredShards component_path=/ component_id=prometheus.remote_write.amp subcomponent=rw remote_name=705773 url=http://prometheus-server.monitoring.svc.cluster.local:9090/api/v1/write dataInRate=7820.7951877227815 dataOutRate=7721.815736708949 dataKeptRatio=1 dataPendingRate=98.97945101383266 dataPending=7820.7951877227815 dataOutDuration=0.10527633413408446 timePerSample=1.3633624230841e-05 desiredShards=0.11195707191457109 highestSent=1.758269758e+09 highestRecv=1.758269759e+09
ts=2025-09-19T08:15:59.52855142Z level=debug msg=QueueManager.updateShardsLoop component_path=/ component_id=prometheus.remote_write.amp subcomponent=rw remote_name=705773 url=http://prometheus-server.monitoring.svc.cluster.local:9090/api/v1/write lowerBound=2.0999999999999996 desiredShards=0.11195707191457109 upperBound=3.9000000000000004
ts=2025-09-19T08:16:03.664811121Z level=debug msg="merging remote state" service=cluster remote_time=182
ts=2025-09-19T08:16:09.528724154Z level=debug msg=QueueManager.calculateDesiredShards component_path=/ component_id=prometheus.remote_write.amp subcomponent=rw remote_name=705773 url=http://prometheus-server.monitoring.svc.cluster.local:9090/api/v1/write dataInRate=7246.036150178225 dataOutRate=7507.452589367159 dataKeptRatio=1 dataPendingRate=-261.4164391889335 dataPending=0 dataOutDuration=0.10244502150726756 timePerSample=1.3645776684938502e-05 desiredShards=0.09887779115632357 highestSent=1.758269769e+09 highestRecv=1.758269769e+09
ts=2025-09-19T08:16:09.528801745Z level=debug msg=QueueManager.updateShardsLoop component_path=/ component_id=prometheus.remote_write.amp subcomponent=rw remote_name=705773 url=http://prometheus-server.monitoring.svc.cluster.local:9090/api/v1/write lowerBound=2.0999999999999996 desiredShards=0.09887779115632357 upperBound=3.9000000000000004
ts=2025-09-19T08:16:10.888438367Z level=debug msg="merging remote state" service=cluster remote_time=183
ts=2025-09-19T08:16:12.735926248Z level=debug msg="merging remote state" service=cluster remote_time=184
ts=2025-09-19T08:16:19.527970329Z level=debug msg=QueueManager.calculateDesiredShards component_path=/ component_id=prometheus.remote_write.amp subcomponent=rw remote_name=705773 url=http://prometheus-server.monitoring.svc.cluster.local:9090/api/v1/write dataInRate=7384.668920142581 dataOutRate=7265.962071493727 dataKeptRatio=1 dataPendingRate=118.70684864885334 dataPending=0 dataOutDuration=0.09824196834581404 timePerSample=1.3520847945414279e-05 desiredShards=0.0998469855964745 highestSent=1.758269779e+09 highestRecv=1.758269779e+09
ts=2025-09-19T08:16:19.52804064Z level=debug msg=QueueManager.updateShardsLoop component_path=/ component_id=prometheus.remote_write.amp subcomponent=rw remote_name=705773 url=http://prometheus-server.monitoring.svc.cluster.local:9090/api/v1/write lowerBound=2.0999999999999996 desiredShards=0.0998469855964745 upperBound=3.9000000000000004
ts=2025-09-19T08:16:26.825575815Z level=debug msg="merging remote state" service=cluster remote_time=185
ts=2025-09-19T08:16:26.825651876Z level=debug msg="merging remote state" service=cluster remote_time=185
ts=2025-09-19T08:16:26.828206469Z level=debug msg="merging remote state" service=cluster remote_time=185
ts=2025-09-19T08:16:29.528245016Z level=debug msg=QueueManager.calculateDesiredShards component_path=/ component_id=prometheus.remote_write.amp subcomponent=rw remote_name=705773 url=http://prometheus-server.monitoring.svc.cluster.local:9090/api/v1/write dataInRate=7601.315136114064 dataOutRate=7562.769657194982 dataKeptRatio=1 dataPendingRate=38.5454789190826 dataPending=7601.315136114064 dataOutDuration=0.09623002763665123 timePerSample=1.2724178045684758e-05 desiredShards=0.10155651153193752 highestSent=1.758269788e+09 highestRecv=1.758269789e+09
ts=2025-09-19T08:16:29.528314617Z level=debug msg=QueueManager.updateShardsLoop component_path=/ component_id=prometheus.remote_write.amp subcomponent=rw remote_name=705773 url=http://prometheus-server.monitoring.svc.cluster.local:9090/api/v1/write lowerBound=2.0999999999999996 desiredShards=0.10155651153193752 upperBound=3.9000000000000004
ts=2025-09-19T08:16:33.667144271Z level=debug msg="merging remote state" service=cluster remote_time=187
ts=2025-09-19T08:16:39.528146572Z level=debug msg=QueueManager.calculateDesiredShards component_path=/ component_id=prometheus.remote_write.amp subcomponent=rw remote_name=705773 url=http://prometheus-server.monitoring.svc.cluster.local:9090/api/v1/write dataInRate=7741.592108891252 dataOutRate=7380.215725755986 dataKeptRatio=1 dataPendingRate=361.3763831352662 dataPending=7741.592108891252 dataOutDuration=0.09070875740932098 timePerSample=1.229080026655038e-05 desiredShards=0.09990788047325916 highestSent=1.758269798e+09 highestRecv=1.758269799e+09
ts=2025-09-19T08:16:39.528206543Z level=debug msg=QueueManager.updateShardsLoop component_path=/ component_id=prometheus.remote_write.amp subcomponent=rw remote_name=705773 url=http://prometheus-server.monitoring.svc.cluster.local:9090/api/v1/write lowerBound=2.0999999999999996 desiredShards=0.09990788047325916 upperBound=3.9000000000000004
ts=2025-09-19T08:16:42.739831793Z level=debug msg="merging remote state" service=cluster remote_time=187
ts=2025-09-19T08:16:49.528892739Z level=debug msg=QueueManager.calculateDesiredShards component_path=/ component_id=prometheus.remote_write.amp subcomponent=rw remote_name=705773 url=http://prometheus-server.monitoring.svc.cluster.local:9090/api/v1/write dataInRate=7530.753687113001 dataOutRate=7864.172580604789 dataKeptRatio=1 dataPendingRate=-333.41889349178746 dataPending=0 dataOutDuration=0.09833378006745679 timePerSample=1.250402112359219e-05 desiredShards=0.09416470318023074 highestSent=1.758269809e+09 highestRecv=1.758269809e+09
ts=2025-09-19T08:16:49.528961809Z level=debug msg=QueueManager.updateShardsLoop component_path=/ component_id=prometheus.remote_write.amp subcomponent=rw remote_name=705773 url=http://prometheus-server.monitoring.svc.cluster.local:9090/api/v1/write lowerBound=2.0999999999999996 desiredShards=0.09416470318023074 upperBound=3.9000000000000004
ts=2025-09-19T08:16:59.527921447Z level=debug msg=QueueManager.calculateDesiredShards component_path=/ component_id=prometheus.remote_write.amp subcomponent=rw remote_name=705773 url=http://prometheus-server.monitoring.svc.cluster.local:9090/api/v1/write dataInRate=7821.182949690401 dataOutRate=7691.338064483831 dataKeptRatio=1 dataPendingRate=129.84488520657032 dataPending=7821.182949690401 dataOutDuration=0.09736584121396544 timePerSample=1.2659155064782568e-05 desiredShards=0.10396004613767253 highestSent=1.758269818e+09 highestRecv=1.758269819e+09
ts=2025-09-19T08:16:59.528030058Z level=debug msg=QueueManager.updateShardsLoop component_path=/ component_id=prometheus.remote_write.amp subcomponent=rw remote_name=705773 url=http://prometheus-server.monitoring.svc.cluster.local:9090/api/v1/write lowerBound=2.0999999999999996 desiredShards=0.10396004613767253 upperBound=3.9000000000000004
ts=2025-09-19T08:17:03.670729806Z level=debug msg="merging remote state" service=cluster remote_time=189

Tip

_{React with 👍 if this issue is important to you.}

[github-actions]

This issue has not had any activity in the past 90 days, so the needs-attention label has been added to it.
If the opened issue is a bug, check to see if a newer release fixed your issue. If it is no longer relevant, please feel free to close this issue.
The needs-attention label signals to maintainers that something has fallen through the cracks. No action is needed by you; your issue will be kept open and you do not have to respond to this comment. The label will be removed the next time this job runs if there is new activity.
Thank you for your contributions!

[kgeckhart]

Bought down prometheus to with alloy was writing by scaling it down and then bought it back up again. the metrics stopped coming in altogether .

Do you have logs from this? Were there sending errors? Did you make any config changes during this time?

Bought down prometheus shifted endpoint to Amazon managed prometheus after 10 mins but alloy started writing the latest data to the new endpoint .

This is the current expected behavior based on upstream prometheus remote write implementation, see prometheus/proposals#72