fix(tunnel): publish ports on a VPS whose public address is NAT'd at the edge - #3573
fix(tunnel): publish ports on a VPS whose public address is NAT'd at the edge#3573helix-nine wants to merge 1 commit into
Conversation
…the edge
AWS, Google Cloud, Azure and Oracle Cloud assign your dedicated public IPv4
at their edge and hand the server a private address. StartTunnel looked for
a public address on the server itself, found none, and refused to publish:
`no WAN IP available for device 10.59.203.3`. Ports published before the
upgrade kept working, because their external IP is embedded in the forward
key and `resync_forward_keys` falls back to it — so only new adds failed,
along with every automatic PCP/UPnP publish from a StartOS server on the
tunnel, and both `set-wan` escape hatches were themselves unreachable.
`default_wan` filtered the host's addresses through `is_wan_candidate`,
which rejects RFC1918. It now falls back to any address the host holds when
no public one exists — the private address is what inbound packets carry by
the time they arrive, so it is what the DNAT rule (`ip daddr $sip`) has to
match. A host that does hold a public address resolves exactly as before.
Deliberately not the WAN IP override: pinning `wan_ip` also sets the
`Endpoint` in every WireGuard config generated afterwards (api.rs:1167),
so directing these users to pin their private address would hand their
devices an endpoint they cannot dial. Resolving it in `default_wan` leaves
that chain untouched.
`subnet set-wan` and `subnet set-ipv6` were also uninvokable: both declared
their own `<SUBNET>` positional instead of inheriting the parent's, so the
CLI asked for the subnet twice and `combine` rejected it as a duplicate key
(`ParentHandler::cli_parse` -> `util::combine`). They now take `SubnetParams`
as inherited params like their `add`/`remove`/`set-dns` siblings. These two
were the only such commands in the repo; the regenerated man pages lose the
second `<SUBNET>`, and the RPC payload is unchanged, so the web UI keeps
sending `{subnet, wanIp}` and only its type annotations move.
The UI's WAN IP selectors mirrored the same public-only filter, leaving the
menu empty on these hosts, so they now list what the backend accepts.
Regression: 49ea23e (#3306), shipped in start-tunnel 1.1.0, which replaced
`port-forward add`'s caller-supplied external IP with this inference.
Closes #3565
helix-nine
left a comment
There was a problem hiding this comment.
Two things from the analysis on #3565, one of which I think is a real defect in the second pass.
The private fallback can select a bridge address. default_wan_of filters out only wg0 and Loopback, and net_iface holds every interface ip -o addr show reports (tunnel/context.rs:242-252), so on a host running Docker or libvirt the second pass sees docker0 = 172.17.0.1 and virbr0 = 192.168.122.1 alongside the uplink. OrdMap<GatewayId, _> iterates by interface name, and docker0 sorts before ens3/eth0 (as does virbr0 before wlan0), so the fallback picks the container bridge rather than the address inbound packets actually carry — and the resulting DNAT rule never matches.
That only bites when the first pass finds nothing, which is exactly the NAT'd-VPS case this PR targets, and Docker on such a VPS is common.
net/gateway.rs:2314-2318 already draws this line for WAN probing:
let forwardable = !subnets.is_empty()
&& !matches!(
device_type,
Some(NetworkInterfaceType::Bridge | NetworkInterfaceType::Loopback)
);Adding Bridge to egress()'s filter would match it. Worth considering whether the first pass should stay bridge-inclusive — a public address on a bridge is plausible on a virtualization host, and it was reachable before this PR — in which case the exclusion belongs on the fallback closure only.
external_ipv4 also answers downstream StartOS boxes, and one of those paths won't filter the private address out. It backs GatewayBackend::external_ipv4, so it feeds UPnP GetExternalIPAddress (net/port_map/server/igd.rs:199) and the PCP MAP external address (net/port_map/server/mod.rs:338). A downstream UPnP client discards a private answer (net/port_map/upnp.rs:104-105), but the PCP client returns it unfiltered (net/port_map/client.rs:328-337, :361), and net/gateway.rs:652-675 short-circuits the reachability probe once auto-forwarding has succeeded — so the attached server would report open_externally: true for an RFC1918 address. Today those paths fail closed on this topology (CANNOT_PROVIDE_EXTERNAL), which is less useful but honest. If the intent is dataplane-only, the relaxation could be confined to the callers that key rules (add_forward, resync_forward_keys) and left out of the GatewayBackend impl.
Unrelated to the above: this needs a rebase — GitHub now reports it conflicting, because #3570 landed the same CLI fix (api.rs, the four man pages, both bindings, the three api-service files, CHANGELOG/Cargo.toml at 1.2.2) on master about ten minutes before this went up. Apologies for the collision; two sessions picked up #3565 in parallel and I didn't see this PR until after mine was merged. The forward/igd.rs, wan.ts, help-content and docs work here is all additional to what landed.
Fixes #3565.
What breaks, and why
The reporter runs on a 1:1-NAT VPS: their dedicated public IPv4 lives at the provider's edge and
ens3holds only172.16.0.12/20.port-forward addfails withno WAN IP available for device 10.59.203.3, from the CLI and the UI alike.add_forwardderives the external IP fromexternal_ipv4→default_wan, which filtered the host's addresses throughis_wan_candidate— a predicate that rejects RFC1918. No public address on the NIC, soNone, so a hard error. Ports published before the upgrade keep working because their external IP is embedded in the forward key andresync_forward_keysfalls back to it (.unwrap_or(*src.ip())), which is why only new adds fail.Three things the issue reports are worth correcting:
wanIp: nullis not a migration defect. StartTunnel never populatesgateways.<iface>.ipInfo.wanIpon any install:TunnelContext::initfillsgatewaysonce fromload_ip_info()(ip -o addr show), which sets name/scope_id/subnets/device_type and nothing else. The UPnP + echoip WAN probe lives innet/gateway.rs's watcher, a StartOS-server componentstart-tunneldnever starts. Sodefault_wan'sip_info.wan_ipbranch is dead code in this product, and a backfill migration would be clobbered on the next boot —gatewaysis derived from the kernel at every start.external_ipv4caller is affected, so automatic publishing (PCPMAP→CANNOT_PROVIDE_EXTERNAL, UPnPGetExternalIPAddress→ SOAP 501) from a StartOS server on such a tunnel is dead too.port-forward add's caller-suppliedsource: SocketAddrV4with a server-derived IP, on the rationale that a forward's inbound IP must equal the device's egress WAN.The fix
default_wankeeps today's behaviour exactly wherever the host holds a public address, and otherwise falls back to any address it does hold. That private address is what inbound packets carry once the provider has translated them, so it is what the DNAT rule has to match —build/lib/scripts/forward-portemitsip daddr $sip, a destination match that needs no address ownership beyond the packet actually carrying it. Extracted asdefault_wan_of(&OrdMap<GatewayId, NetworkInterfaceInfo>)so it is unit-testable; there was no coverage of this path at all before.Why not just document the WAN IP override. Pinning
wan_ipreaches the dataplane correctly (assigned_wan_forapplies no filter), but it is also read at precedence #2 byshow_configto build the WireGuardEndpoint. Telling these users to pin their private address would stamp an endpoint their devices cannot dial into every config generated afterwards. Resolving it indefault_wanleaves that chain untouched. Two related footguns are deliberately left alone here, noted below.Second defect: the escape hatch was uninvokable
subnet set-wanandsubnet set-ipv6each declared their own<SUBNET>positional instead of inheriting the parent's, soParentHandler::cli_commandappended a second one andcli_parse→util::combinerejected it asduplicate key: subnet. Nothing catches this at compile time —impl<A, B> OrEmpty<Flat<A, B>> for Emptylets a 2-arg handler register under a params-carrying parent by silently discarding the inherited params. Both now takeSubnetParamsas inherited params like theiradd/remove/set-dnssiblings. A repo-wide audit found these two were the only instances;host_api, thes9pkparents,logsand theaddress/bindingfamilies all inherit correctly.set-ipv6has been equally unusable from the CLI since 1.1.0 — the CLI reference documents the inherited form the code never implemented.The RPC path never calls
combine, so the web UI was unaffected and its payload is unchanged:{subnet, wanIp}. Only the generated binding and six type annotations move (mirroring the already-correctsetSubnetDns); the two real call sites already passsubnet.UI
wanOptions()mirrorsis_wan_candidatein TypeScript, so on these hosts the WAN IP menu was empty — the setting was unreachable from the UI even once the CLI worked. It now lists what the backend accepts, withdefaultWanIpmirroring the same two passes so theSystem default (…)parenthetical stays truthful.Verification
cargo test -p start-core --features test tunnel::— 45 passed, including 4 newdefault_wan_ofcases (public wins over private; private-only host; wg interface excluded; loopback/link-local yieldNone)npm run check:tunnel,npm run check:i18n:tunnel— cleanmake manpages— all four affected pages lose the duplicate<SUBNET>make start-core-ts-bindings—SetSubnetWanParamsbecomes{ wanIp }make start-core-format-check,make start-tunnel-format-check, prettier — cleantunnelboxbefore the change (Usage: start-tunnel subnet <SUBNET> set-wan [OPTIONS] <SUBNET>, then both reported errors verbatim) and after (help loses the positional, invocation reaches transport)Not verified on real hardware: the live PCP/UPnP path on a provider-NAT'd VPS. #3306's own body flagged that path as needing a router + VPS pair to exercise end to end.
Deliberately out of scope
set-wanstill accepts an address the host does not hold. Now that the CLI works, a user who pins their real public IP loses every working forward in one command:resync_forward_keysre-keys them onto an address no packet carries, and the SNI listener still comes up becausebuild_listen_socketsetsIP_FREEBIND, so the bind succeeds and silently blackholes. A validation belongs here, but rejecting outright forecloses pinning an address that is about to be assigned — worth a decision rather than a guess.Endpointis already wrong on this topology, independent of any pin:show_config's chain falls through to the webserver listen address because itslocal_addrbranch is dead (theConnectInfomiddleware is registered nowhere). Registering it would makedevices.md's existing promise — "StartTunnel uses the address you are accessing it over" — true, and is the cheaper of the two ways to separate the advertised endpoint from the dataplane address.m_04deletes any port-80 forward with no publicness check, on the assumption the HTTP→HTTPS redirect replaces it — but the redirect binds only public addresses, so on this host nothing takes over. Worth asking the reporter whether they had a172.16.0.12:80forward before upgrading; if so it is a separate 1.2.1 data-loss bug.