fix(start-os): route v6 replies out the interface that owns the source address - #3632
Open
dr-bonez wants to merge 1 commit into
Open
fix(start-os): route v6 replies out the interface that owns the source address#3632dr-bonez wants to merge 1 commit into
dr-bonez wants to merge 1 commit into
Conversation
…e address An inbound IPv6 connection to a tunnel-delegated GUA hung until it timed out. The reply path was at fault, not the inbound one: the SYN arrived and conntrack showed the interface mark set correctly, but the entry stayed UNREPLIED. StartOS sends a reply back out the interface its connection arrived on by restoring a CONNMARK in a `type route` output chain. That cannot catch a locally generated reply — the kernel routes a SYN-ACK before the output hook runs (ir_mark = sk_mark = 0, tcp_fwmark_accept being 0), so it fell to the priority-75 catch-all, reached the default outbound's table, and hit the `blackhole default` leak guard installed when that gateway carries no v6. A blackhole lookup returns EINVAL, so tcp_v6_send_synack dropped the packet before it was ever built, leaving the reroute chain no packet to act on. IPv4 escapes this because its catch-all table holds a real default: the packet is built and only then rerouted. Install a priority-60 `from <global v6 addr> lookup <iface table>` rule per global v6 address on the interface, between the CONNMARK rule and the catch-all. The source address is known at SYN-ACK route-lookup time, so it routes the reply out the interface that owns it. Rules reconcile as addresses change and are GC'd for removed interfaces alongside the priority-50 rules. This does not change unbound outbound v6, which still blackholes by design — a `from` rule cannot match a socket whose source is unselected at lookup time. A socket bound to its tunnel GUA does now get working outbound v6.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
An inbound IPv6 connection to a tunnel-delegated GUA hangs until it times out. Reported against
tunnel.thebarsonists.run, whose AAAA points at a tunnel client.Root cause
The reply path is at fault, not the inbound one. The SYN arrives fine and conntrack shows the interface mark set correctly, but the entry stays
[UNREPLIED]:StartOS sends a reply back out the interface its connection arrived on by restoring a CONNMARK in a
type routeoutput chain. That works for forwarded traffic. It cannot catch a locally generated reply: the kernel routes a SYN-ACK before the output hook runs (ir_mark=sk_mark= 0, sincetcp_fwmark_acceptis 0). So it falls to the priority-75 catch-all, reaches the default outbound's table, and hits theblackhole defaultleak guard installed when that gateway carries no v6 of its own.A blackhole lookup returns
EINVAL, sotcp_v6_send_synackdrops the packet before it is ever built — leaving the reroute chain no packet to act on.IPv4 escapes this by accident: its catch-all table holds a real default, so the packet is built, hits the output hook, gets its mark restored, and is rerouted correctly. That is the exact v4/v6 asymmetry observed — over v4 the box replies to a source outside the tunnel subnet; the identical v6 test fails.
Tell-tale on an affected box:
Fix
Install a priority-60
from <global v6 addr> lookup <iface table>rule per global v6 address on the interface, sitting between the priority-50 CONNMARK rule and the priority-75 catch-all. The source address is known at SYN-ACK route-lookup time, so this routes the reply out the interface that owns it.Rules reconcile as addresses change, and are GC'd for removed interfaces alongside the existing priority-50 rules.
Verification
Reproduced and fixed against a live tunnel client. The rule alone takes HTTPS over IPv6 from a hard timeout to a normal response:
code=000, 10.0s timeoutcode=200, 0.25sConfirmed the server side was never at fault — proxy NDP, routing, the nft ruleset and forwarding into the tunnel all behaved correctly; the SYN was observed arriving on
eth0and leaving on the wg interface.Also verified the printed rule form (
from 2604:… lookup 1005— bare compressed address) matches what the reconciler compares against, so it does not churn on every poll.cargo check -p start-coreand pinned rustfmt are clean.Scope
This does not change unbound outbound v6, which still blackholes by design — a
fromrule cannot match a socket whose source address is unselected at route-lookup time, and the blackhole is the deliberate leak guard against v6 escaping via an arbitrary interface. A socket bound to its tunnel GUA does now get working outbound v6 (verified: works for an address with the rule, fails for one without).No docs change:
gateways.mdandclearnet.mdalready describe delegated-GUA IPv6 as working, so this restores documented behavior.Left out deliberately
The changelog entry is filed under a new
## [0.4.0.2]heading, but the version mechanics are not included — no manifest bump, noversion/v0_4_0_2.rsmigration node. Cutting the number is a release decision. Outstanding when it is cut: rootpackage.json+package-lock.json, theprojects/start-os/Cargo.tomllabel +Cargo.lock, the migration node with its 5mod.rsupdates, and the docs release links.