wrangler dev: steady ~5s request cadence phase-locks with kj's 5s keep-alive timeouts — POSTs intermittently fail with 503 "worker restarted mid-request" (worker never restarted)

[powfan]

Summary

wrangler dev intermittently fails POST requests with 503 "Your worker restarted mid-request" even though the worker never restarts. The trigger is surprisingly specific: a client sending requests on a steady ~5.000s cadence phase-locks with KJ's two default 5-second connection timeouts inside the local dev proxy chain, so every request rides exactly on the millisecond where an idle keep-alive connection is being closed by the other side.

We hit this in production-like local dev (a timer service firing every 5s) with 30–47% of requests failing. Changing the cadence to 4s or 6s drops the failure rate to 0%.

This is the request-drop mechanism underlying the misleading 503. It is related to but distinct from PR #14593, which fixes the error classification (origin vs href comparison) — with that PR merged, the error becomes honest, but the request still fails.

Minimal reproduction

Any worker project (the failure happens in the ProxyWorker forwarding layer, worker code is irrelevant):

npx wrangler dev --port 8787   # any worker
node repro.mjs 8787 5000       # steady 5s cadence -> ~40% failures
node repro.mjs 8787 4000       # 4s -> 0%
node repro.mjs 8787 6000       # 6s -> 0%

// repro.mjs — POST wrangler dev on a steady cadence (send-time aligned)
import http from "node:http";
const port = Number(process.argv[2] ?? 8787);
const interval = Number(process.argv[3] ?? 5000);
let sent = 0, failed = 0;
function shoot() {
  const body = Buffer.from(JSON.stringify({ hello: "world" }));
  const req = http.request(
    { hostname: "127.0.0.1", port, path: "/", method: "POST",
      headers: { "content-type": "application/json", "content-length": body.length } },
    (res) => {
      res.resume();
      res.on("end", () => {
        sent++;
        if (res.statusCode >= 500) failed++;
        console.log(new Date().toISOString(), res.statusCode, `sent=${sent} failed=${failed} reused=${req.reusedSocket}`);
      });
    });
  req.on("error", (e) => { sent++; failed++; console.log(new Date().toISOString(), "SOCKET-ERR", e.code, `sent=${sent} failed=${failed}`); });
  req.end(body);
}
shoot();
setInterval(shoot, interval);

Measured on our project (wrangler 4.99.0; an earlier investigation also reproduced on 4.108.0):

Client	Cadence	Result
node, keep-alive globalAgent	5000ms (send-time aligned)	43.3% failed (30 shots: 9× HTTP 503 + 4× ECONNRESET)
node, new connection per request	5000ms	40.0% failed (20 shots: 8× 503, 0× ECONNRESET) — inbound reuse is not required
node, keep-alive	4000ms	0% (20 shots)
node, keep-alive	6000ms	0% (20 shots)
bun fetch	5000ms	33.3% failed
any client	5s measured after the response (i.e. interval ≈ 5015ms)	0% — the ~15ms systematic offset de-phases the race

Mechanism (directly observed at TCP level)

We placed a transparent TCP observer proxy between ProxyWorker and the UserWorker socket (by temporarily patching the inner-fetch port in ProxyWorker.js). Failing request, millisecond timestamps:

48706  C3  OPEN   (ProxyWorker opens inner connection)
48706  C3  DATA   A->B 960B   <- previous request
48712  C3  DATA   B->A 72B    <- previous response, fine
53706  C3  DATA   A->B 960B   <- THIS request (connection idle for exactly 5000ms, pool reuses it)
53706  C3  CLOSE  by-B        <- same millisecond: UserWorker closes the idle connection. No response.

Why exactly 5 seconds, twice:

• kj/compat/http.h: HttpServerSettings.pipelineTimeout = 5 * kj::SECONDS (server closes an idle keep-alive connection 5s after the last request) and HttpClientSettings.idleTimeout = 5 * kj::SECONDS (client pool keeps an idle connection up to 5s). workerd does not override either default.
• The UserWorker closes the inner connection at last request + 5.000s; a client on a 5.000s cadence makes ProxyWorker's next reuse attempt land on that exact millisecond. The ±1ms jitter between the two clocks decides each shot — hence the ~"coin-flip" failure rates and the bursty all-green/all-red streaks.
• The failed inner fetch() rejects with kj DISCONNECTED, which decodeTunneledException (workerd src/workerd/jsg/util.c++) renders as "Network connection lost.". ProxyWorker's catch then misclassifies it (see [wrangler] compare origins, not hrefs, when classifying dev proxy fetch errors #14593) and synthesizes the 503 with the misleading "worker restarted" text. GET/HEAD are silently retried, so only non-GET traffic surfaces it.
• The same race on the inbound connection (client → ProxyWorker, same 5s pipelineTimeout) produces the ECONNRESET flavor: the client reuses a socket the proxy just closed. Any keep-alive reverse proxy in front of wrangler dev turns that into a 502.

A steady ~5s cadence is not exotic — it is a default polling/heartbeat interval in lots of tooling, and anyone hitting this sees only the misleading "restarted mid-request" text (we lost two days to it before instrumenting).

Suggested fix directions

1. Retry once on stale-connection reuse in ProxyWorker's inner fetch (or in kj's pooled HttpClient): connection was reused + zero response bytes received → replay on a fresh connection. This is what curl, Go's net/http, undici and browsers do for this classic race; it would eliminate both the 503 and the drop.
2. Alternatively (cheaper): disable keep-alive / set Connection: close on the ProxyWorker→UserWorker hop in local dev — the loopback reconnect cost is negligible compared to a 40% failure rate at unlucky cadences.
3. Merge [wrangler] compare origins, not hrefs, when classifying dev proxy fetch errors #14593 regardless, so that whatever still fails is at least reported truthfully instead of as a phantom worker restart.

Environment

• wrangler 4.99.0 (earlier investigation also reproduced on 4.108.0), local dev (wrangler dev, miniflare/workerd)
• macOS (darwin 25.4.0), node v24.13.1, bun 1.3.9
• Reproduces with and without a reverse proxy in front; with keep-alive and non-keep-alive inbound clients; worker code irrelevant