Helm: gateway nginx re-resolves the distributor per request, so one DNS timeout stalls a push for resolver_timeout

[peterD4N]

Describe the bug

The gateway nginx config proxies the distributor through an nginx variable:

location = /api/v1/push {
  set $distributor {{ template "mimir.fullname" . }}-distributor-headless.{{ .Release.Namespace }}.svc.{{ .Values.global.clusterDomain }};
  proxy_pass      http://$distributor:{{ include "mimir.serverHttpListenPort" . }}$request_uri;
}

(gateway.nginx.config.file, chart 6.1.0 — same pattern for /distributor, = /api/v1/push and /otlp/v1/metrics.)

Because proxy_pass contains variables, nginx does not resolve the upstream at startup. It performs a DNS lookup through the resolver directive on every request whose cache entry has expired. nginx's resolver is a bare UDP client: if a response is lost or slow, the request blocks until resolver_timeout and never reaches an upstream.

With the chart defaults (gateway.nginx.config.resolver: null → no valid=, and no resolver_timeout → nginx's default of 30s) a single dropped DNS packet stalls a push for 30 seconds.

That is particularly bad for POST /api/v1/push, because Prometheus remote write has a fixed number of shards. Every stalled request occupies one shard for the full 30s, so a small DNS error rate collapses total write throughput.

Evidence

We hit roughly 5% of pushes stalling. With $request_time, $upstream_connect_time, $upstream_response_time and $upstream_addr added to logFormat, the stalled requests look like this:

499 "POST /api/v1/push HTTP/1.1" ... rt=29.984 uct=- urt=- ua=-

ua=- / uct=- / urt=- means nginx never selected an upstream — 596 of 597 ua=- requests were the 499s, 1:1. With errorLogLevel: warn the cause is visible:

mimir-distributed-distributor.system-mimir.svc.cluster.local could not be resolved (110: Operation timed out)

Note the error is logged far less often than it occurs: the client aborts at its own 30s remote-write timeout at the same moment resolver_timeout expires, so nginx logs a 499 and usually never gets as far as logging the resolver failure. That made this very hard to attribute.

Downstream effect on a Prometheus with maxShards: 50 pushing ~120k samples/s: throughput fell to ~30k samples/s, the queue fell behind sampleAgeLimit, and batches that aged out were re-sent as empty bodies which the distributor rejected with 400 ... snappy: corrupt input.

Lowering resolver_timeout does not fix it, it only changes the failure shape — the same DNS failures then return fast 502s instead of 30s stalls.

Expected behaviour

A transient DNS timeout should not stall or fail a push. The gateway should not depend on a successful DNS round-trip per request.

Proposed fix

Point the distributor locations at the ClusterIP service instead of the headless one and use a static upstream block, so nginx resolves once at startup and the resolver leaves the request path entirely:

upstream distributor_backend {
  server {{ template "mimir.fullname" . }}-distributor.{{ .Release.Namespace }}.svc.{{ .Values.global.clusterDomain }}:{{ include "mimir.serverHttpListenPort" . }};
}
...
location = /api/v1/push {
  proxy_pass http://distributor_backend;   # no URI part => original request URI is forwarded
}

The ClusterIP is a stable VIP, so start-up resolution is safe and kube-proxy handles load balancing across healthy endpoints — which also removes the stale-pod-IP window that the headless service has whenever distributors scale down.

We verified this on a test cluster under real load (~120k samples/s):

	per-request DNS (current)	upstream block + ClusterIP
499s on /api/v1/push	~5% (596 in 6 min)	0 in 28k pushes
$request_time p99 / max	30.00s / 30.05s	0.45s / 0.91s
requests > 5s	596	0
resolver errors	present	0

It also survived forcing the distributor deployment 3 → 6 → 3 twice, with no stalls.

If keeping the headless service is preferred for client-side balancing, then an upstream block cannot be used directly (nginx would pin a single pod IP at startup), and the per-request resolution would need to stay — in which case it would help a lot to at least expose the upstream host as a value so operators can opt into the ClusterIP.

Happy to open a PR if the approach looks reasonable.

Environment

• Chart: mimir-distributed 6.1.0 (latest at time of writing)
• Mimir: 3.1.2
• Kubernetes: AKS, CoreDNS, IPv4 only
• Gateway image: nginx-unprivileged:1.29-alpine