Local dev: containers never start on hosts with net.ipv4.conf.all.src_valid_mark=1 (e.g. Tailscale) — sidecar handshake packets dropped as martians

[Flo-90]

Summary

On Linux hosts where net.ipv4.conf.all.src_valid_mark=1 is set in the init
netns — which Tailscale does by default — wrangler dev with a containers
config can never start the application container. workerd creates the
cloudflare/proxy-everything egress-interceptor sidecar, but the TCP handshake
between workerd and the sidecar never completes: the kernel inside the sidecar
netns drops every packet that the sidecar's own TPROXY rules have fwmark-marked,
because the inherited src_valid_mark=1 makes source validation use the fwmark
and classify the peer as a martian source. workerd times out after ~20 s
(kj/timer.c++:30: overloaded: operation timed out), destroys the sidecar,
retries, and the Durable Object eventually fails with
Container failed to start (outcome: 'unrecognized_error') / HTTP 500
"Container is starting. Please retry in a moment." after ~151 s.

The application container is never even created (no container create in
docker events for the app image over the whole retry budget — only the
…-proxy sidecar churns create/destroy every ~20–45 s).

A one-line fix is possible on workerd's side: pass
HostConfig.Sysctls: {"net.ipv4.conf.all.src_valid_mark": "0"} when creating
the sidecar (namespaced net.* sysctls are settable via the Docker API without
extra privileges). Optionally also net.ipv4.conf.all.rp_filter: "0", which is
inherited the same way and can interfere with TPROXY setups too.

Environment

• workerd 1.20260701.1 (via wrangler 4.107.0, miniflare 4.20260701.0)
• @cloudflare/sandbox 0.12.3 (but any containers config reproduces it)
• Sidecar image: cloudflare/proxy-everything:3cb1195@sha256:0ef6716c…
• Linux 7.1.3 (CachyOS/Arch), Docker 29.6.1 (iptables backend), iptables 1.8.11 (nf_tables)
• Tailscale installed and running on the host (this is what sets
  net.ipv4.conf.all.src_valid_mark=1 in the init netns)

Reproduction

Tailscale is not required — the sysctl is the trigger:

# on any Linux host with working `wrangler dev` containers:
sudo sysctl -w net.ipv4.conf.all.src_valid_mark=1

# any Worker project with a `containers` config:
wrangler dev
curl -X POST http://localhost:8787/...   # anything that starts the container DO

Result: sidecar create/destroy loop, ~20 s per attempt, app container never
created, request fails after the retry budget. Setting the sysctl back to 0
(and recreating the containers) fixes it immediately.

Note on inheritance: with the kernel default
net.core.devconf_inherit_init_net=0, new network namespaces inherit all
IPv4 conf/{all,default} values from init_net, so a host-level
src_valid_mark=1 silently propagates into every container netns that Docker
creates.

Root cause analysis

The sidecar sets up transparent interception inside its netns:

# mangle PREROUTING (inside the sidecar netns)
-A PREROUTING -p tcp -m socket -j DIVERT          # existing-socket match → mark
-A DIVERT -j MARK --set-xmark 0x1/0xffffffff
-A DIVERT -j ACCEPT
...
# policy routing
32765:  from all fwmark 0x1 lookup 100
# table 100
local default dev lo scope host

When workerd connects from the docker bridge gateway (e.g. 172.17.0.1) to the
sidecar's ingress (<container-ip>:39001) to push the egress configuration:

1. The SYN does not match -m socket (xt_socket ignores non-transparent
   wildcard listeners), stays unmarked, and is delivered normally — the
   listener replies SYN-ACK and goes to SYN_RECV.
2. The final ACK and the request payload match -m socket (the request
   socket for the 4-tuple now exists), get fwmark 0x1, and then hit source
   validation. With src_valid_mark=1, fib_validate_source() includes the
   fwmark in the reverse-path lookup, which therefore resolves via table 100
   (local default dev lo) — the source 172.17.0.1 is classified as a
   local/martian source and the packet is dropped.
3. The sidecar's TCP stack never sees the ACK: it stays in SYN_RECV
   retransmitting SYN-ACK; workerd's side sits in FIN-WAIT-1 with ~164 bytes
   unacked. After ~20 s the kj timer fires, workerd tears the sidecar down and
   retries.

Evidence gathered while debugging:

• docker events: only …-proxy create/start/kill/destroy cycles; zero
  create events for the app image.
• Docker API capture (socat proxy on the docker socket): per attempt exactly
  GET /networks/bridge → POST /containers/create?name=…-proxy →
  POST /containers/…-proxy/start → GET /containers/…-proxy/json, then ~20 s
  of silence, then force-DELETEs — i.e. workerd blocks between "inspect
  sidecar" and "create app container".
• tcpdump on both ends: ACK+data visible on the sidecar's eth0, never
  processed by TCP; /proc/net/tcp6 inside the sidecar shows the connection
  stuck in SYN_RECV; host side stuck in FIN-WAIT-1 with Send-Q > 0.
• Routing verdict inside the sidecar netns:
  ip route get <container-ip> from 172.17.0.1 iif eth0 mark 0x1 →
  RTNETLINK answers: Invalid argument, while the same query without
  mark 0x1 returns a normal local route.
• Live fix: nsenter -t <sidecar-pid> -n sysctl -w net.ipv4.conf.all.src_valid_mark=0
  → the very next attempt succeeds; app container is created and the DO
  request returns HTTP 200 in ~1.6 s.

Suggested fix

Set the sysctl explicitly when creating the sidecar container (and, if it uses
mark-based routing too, the app container), so local dev does not depend on
host-inherited values:

"HostConfig": {
  "Sysctls": {
    "net.ipv4.conf.all.src_valid_mark": "0"
  }
}

rp_filter may deserve the same treatment ("net.ipv4.conf.all.rp_filter": "0"),
since strict rp_filter in the container netns (also inherited from hosts that
set it, e.g. via ufw's /etc/ufw/sysctl.conf) is the classic TPROXY breaker.

Workarounds for affected users (until fixed)

• net.core.devconf_inherit_init_net=2 (new netns get kernel defaults instead
  of inheriting init_net's IPv4 conf values) — verified working:

  echo 'net.core.devconf_inherit_init_net = 2' | sudo tee /etc/sysctl.d/99-docker-netns-defaults.conf
  sudo sysctl -w net.core.devconf_inherit_init_net=2

• Not recommended: clearing net.ipv4.conf.all.src_valid_mark on the host —
  Tailscale needs it and re-sets it.

Related, but distinct

While debugging this we also hit an independent local-dev pitfall with
identical symptoms: host firewalls with a default-deny INPUT policy (e.g. plain
ufw) block the sidecar's dial-back to workerd's egress listener on the bridge
gateway (172.17.0.1:<random port>), because that's container→host INPUT
traffic. ufw allow in on docker0 fixes it. Possibly worth a note in the local
development docs — both issues produce the same "sidecar churns, app container
never created, 20 s timeout" signature that is otherwise hard to attribute.

Related issues (similar symptoms, different causes as far as I can tell):
cloudflare/workers-sdk#14242, #5996.