Skip to content

fix(start-os): route v6 replies out the interface that owns the source address - #3632

Open
dr-bonez wants to merge 1 commit into
masterfrom
fix/v6-reply-source-routing
Open

fix(start-os): route v6 replies out the interface that owns the source address#3632
dr-bonez wants to merge 1 commit into
masterfrom
fix/v6-reply-source-routing

Conversation

@dr-bonez

@dr-bonez dr-bonez commented Aug 5, 2026

Copy link
Copy Markdown
Member

An inbound IPv6 connection to a tunnel-delegated GUA hangs until it times out. Reported against tunnel.thebarsonists.run, whose AAAA points at a tunnel client.

Root cause

The reply path is at fault, not the inbound one. The SYN arrives fine and conntrack shows the interface mark set correctly, but the entry stays [UNREPLIED]:

tcp 6 104 SYN_SENT src=2001:470:... dst=...:f002 dport=443 [UNREPLIED] ... mark=1005

StartOS sends a reply back out the interface its connection arrived on by restoring a CONNMARK in a type route output chain. That works for forwarded traffic. It cannot catch a locally generated reply: the kernel routes a SYN-ACK before the output hook runs (ir_mark = sk_mark = 0, since tcp_fwmark_accept is 0). So it falls to the priority-75 catch-all, reaches the default outbound's table, and hits the blackhole default leak guard installed when that gateway carries no v6 of its own.

A blackhole lookup returns EINVAL, so tcp_v6_send_synack drops the packet before it is ever built — leaving the reroute chain no packet to act on.

IPv4 escapes this by accident: its catch-all table holds a real default, so the packet is built, hits the output hook, gets its mark restored, and is rerouted correctly. That is the exact v4/v6 asymmetry observed — over v4 the box replies to a source outside the tunnel subnet; the identical v6 test fails.

Tell-tale on an affected box:

$ ip -6 route get <external>              # blackhole
RTNETLINK answers: Invalid argument
$ ip -6 route get <server tunnel addr>    # fine
... dev wg1 table 1002 ...

Fix

Install a priority-60 from <global v6 addr> lookup <iface table> rule per global v6 address on the interface, sitting between the priority-50 CONNMARK rule and the priority-75 catch-all. The source address is known at SYN-ACK route-lookup time, so this routes the reply out the interface that owns it.

Rules reconcile as addresses change, and are GC'd for removed interfaces alongside the existing priority-50 rules.

Verification

Reproduced and fixed against a live tunnel client. The rule alone takes HTTPS over IPv6 from a hard timeout to a normal response:

result
before code=000, 10.0s timeout
after code=200, 0.25s

Confirmed the server side was never at fault — proxy NDP, routing, the nft ruleset and forwarding into the tunnel all behaved correctly; the SYN was observed arriving on eth0 and leaving on the wg interface.

Also verified the printed rule form (from 2604:… lookup 1005 — bare compressed address) matches what the reconciler compares against, so it does not churn on every poll.

cargo check -p start-core and pinned rustfmt are clean.

Scope

This does not change unbound outbound v6, which still blackholes by design — a from rule cannot match a socket whose source address is unselected at route-lookup time, and the blackhole is the deliberate leak guard against v6 escaping via an arbitrary interface. A socket bound to its tunnel GUA does now get working outbound v6 (verified: works for an address with the rule, fails for one without).

No docs change: gateways.md and clearnet.md already describe delegated-GUA IPv6 as working, so this restores documented behavior.

Left out deliberately

The changelog entry is filed under a new ## [0.4.0.2] heading, but the version mechanics are not included — no manifest bump, no version/v0_4_0_2.rs migration node. Cutting the number is a release decision. Outstanding when it is cut: root package.json + package-lock.json, the projects/start-os/Cargo.toml label + Cargo.lock, the migration node with its 5 mod.rs updates, and the docs release links.

…e address

An inbound IPv6 connection to a tunnel-delegated GUA hung until it timed out.
The reply path was at fault, not the inbound one: the SYN arrived and conntrack
showed the interface mark set correctly, but the entry stayed UNREPLIED.

StartOS sends a reply back out the interface its connection arrived on by
restoring a CONNMARK in a `type route` output chain. That cannot catch a
locally generated reply — the kernel routes a SYN-ACK before the output hook
runs (ir_mark = sk_mark = 0, tcp_fwmark_accept being 0), so it fell to the
priority-75 catch-all, reached the default outbound's table, and hit the
`blackhole default` leak guard installed when that gateway carries no v6. A
blackhole lookup returns EINVAL, so tcp_v6_send_synack dropped the packet
before it was ever built, leaving the reroute chain no packet to act on. IPv4
escapes this because its catch-all table holds a real default: the packet is
built and only then rerouted.

Install a priority-60 `from <global v6 addr> lookup <iface table>` rule per
global v6 address on the interface, between the CONNMARK rule and the
catch-all. The source address is known at SYN-ACK route-lookup time, so it
routes the reply out the interface that owns it. Rules reconcile as addresses
change and are GC'd for removed interfaces alongside the priority-50 rules.

This does not change unbound outbound v6, which still blackholes by design —
a `from` rule cannot match a socket whose source is unselected at lookup time.
A socket bound to its tunnel GUA does now get working outbound v6.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant