Skip to content

fix(tunnel): publish ports on a VPS whose public address is NAT'd at the edge - #3573

Open
helix-nine wants to merge 1 commit into
masterfrom
fix/tunnel-wan-on-nat-vps
Open

fix(tunnel): publish ports on a VPS whose public address is NAT'd at the edge#3573
helix-nine wants to merge 1 commit into
masterfrom
fix/tunnel-wan-on-nat-vps

Conversation

@helix-nine

Copy link
Copy Markdown
Contributor

Fixes #3565.

What breaks, and why

The reporter runs on a 1:1-NAT VPS: their dedicated public IPv4 lives at the provider's edge and ens3 holds only 172.16.0.12/20. port-forward add fails with no WAN IP available for device 10.59.203.3, from the CLI and the UI alike.

add_forward derives the external IP from external_ipv4default_wan, which filtered the host's addresses through is_wan_candidate — a predicate that rejects RFC1918. No public address on the NIC, so None, so a hard error. Ports published before the upgrade keep working because their external IP is embedded in the forward key and resync_forward_keys falls back to it (.unwrap_or(*src.ip())), which is why only new adds fail.

Three things the issue reports are worth correcting:

  • wanIp: null is not a migration defect. StartTunnel never populates gateways.<iface>.ipInfo.wanIp on any install: TunnelContext::init fills gateways once from load_ip_info() (ip -o addr show), which sets name/scope_id/subnets/device_type and nothing else. The UPnP + echoip WAN probe lives in net/gateway.rs's watcher, a StartOS-server component start-tunneld never starts. So default_wan's ip_info.wan_ip branch is dead code in this product, and a backfill migration would be clobbered on the next boot — gateways is derived from the kernel at every start.
  • The blast radius is larger than manual publishing. Every external_ipv4 caller is affected, so automatic publishing (PCP MAPCANNOT_PROVIDE_EXTERNAL, UPnP GetExternalIPAddress → SOAP 501) from a StartOS server on such a tunnel is dead too.
  • This is a 1.1.0 regression, not 1.2.x. 49ea23e (feat(net): automatic gateway configuration — port forwarding (PCP/NAT-PMP/UPnP) + private-domain DNS (RFC 2136) #3306) replaced port-forward add's caller-supplied source: SocketAddrV4 with a server-derived IP, on the rationale that a forward's inbound IP must equal the device's egress WAN.

The fix

default_wan keeps today's behaviour exactly wherever the host holds a public address, and otherwise falls back to any address it does hold. That private address is what inbound packets carry once the provider has translated them, so it is what the DNAT rule has to match — build/lib/scripts/forward-port emits ip daddr $sip, a destination match that needs no address ownership beyond the packet actually carrying it. Extracted as default_wan_of(&OrdMap<GatewayId, NetworkInterfaceInfo>) so it is unit-testable; there was no coverage of this path at all before.

Why not just document the WAN IP override. Pinning wan_ip reaches the dataplane correctly (assigned_wan_for applies no filter), but it is also read at precedence #2 by show_config to build the WireGuard Endpoint. Telling these users to pin their private address would stamp an endpoint their devices cannot dial into every config generated afterwards. Resolving it in default_wan leaves that chain untouched. Two related footguns are deliberately left alone here, noted below.

Second defect: the escape hatch was uninvokable

subnet set-wan and subnet set-ipv6 each declared their own <SUBNET> positional instead of inheriting the parent's, so ParentHandler::cli_command appended a second one and cli_parseutil::combine rejected it as duplicate key: subnet. Nothing catches this at compile time — impl<A, B> OrEmpty<Flat<A, B>> for Empty lets a 2-arg handler register under a params-carrying parent by silently discarding the inherited params. Both now take SubnetParams as inherited params like their add/remove/set-dns siblings. A repo-wide audit found these two were the only instances; host_api, the s9pk parents, logs and the address/binding families all inherit correctly.

set-ipv6 has been equally unusable from the CLI since 1.1.0 — the CLI reference documents the inherited form the code never implemented.

The RPC path never calls combine, so the web UI was unaffected and its payload is unchanged: {subnet, wanIp}. Only the generated binding and six type annotations move (mirroring the already-correct setSubnetDns); the two real call sites already pass subnet.

UI

wanOptions() mirrors is_wan_candidate in TypeScript, so on these hosts the WAN IP menu was empty — the setting was unreachable from the UI even once the CLI worked. It now lists what the backend accepts, with defaultWanIp mirroring the same two passes so the System default (…) parenthetical stays truthful.

Verification

  • cargo test -p start-core --features test tunnel:: — 45 passed, including 4 new default_wan_of cases (public wins over private; private-only host; wg interface excluded; loopback/link-local yield None)
  • npm run check:tunnel, npm run check:i18n:tunnel — clean
  • make manpages — all four affected pages lose the duplicate <SUBNET>
  • make start-core-ts-bindingsSetSubnetWanParams becomes { wanIp }
  • make start-core-format-check, make start-tunnel-format-check, prettier — clean
  • Behaviour was reproduced against a built tunnelbox before the change (Usage: start-tunnel subnet <SUBNET> set-wan [OPTIONS] <SUBNET>, then both reported errors verbatim) and after (help loses the positional, invocation reaches transport)

Not verified on real hardware: the live PCP/UPnP path on a provider-NAT'd VPS. #3306's own body flagged that path as needing a router + VPS pair to exercise end to end.

Deliberately out of scope

  • set-wan still accepts an address the host does not hold. Now that the CLI works, a user who pins their real public IP loses every working forward in one command: resync_forward_keys re-keys them onto an address no packet carries, and the SNI listener still comes up because build_listen_socket sets IP_FREEBIND, so the bind succeeds and silently blackholes. A validation belongs here, but rejecting outright forecloses pinning an address that is about to be assigned — worth a decision rather than a guess.
  • The WireGuard Endpoint is already wrong on this topology, independent of any pin: show_config's chain falls through to the webserver listen address because its local_addr branch is dead (the ConnectInfo middleware is registered nowhere). Registering it would make devices.md's existing promise — "StartTunnel uses the address you are accessing it over" — true, and is the cheaper of the two ways to separate the advertised endpoint from the dataplane address.
  • Migration m_04 deletes any port-80 forward with no publicness check, on the assumption the HTTP→HTTPS redirect replaces it — but the redirect binds only public addresses, so on this host nothing takes over. Worth asking the reporter whether they had a 172.16.0.12:80 forward before upgrading; if so it is a separate 1.2.1 data-loss bug.

…the edge

AWS, Google Cloud, Azure and Oracle Cloud assign your dedicated public IPv4
at their edge and hand the server a private address. StartTunnel looked for
a public address on the server itself, found none, and refused to publish:
`no WAN IP available for device 10.59.203.3`. Ports published before the
upgrade kept working, because their external IP is embedded in the forward
key and `resync_forward_keys` falls back to it — so only new adds failed,
along with every automatic PCP/UPnP publish from a StartOS server on the
tunnel, and both `set-wan` escape hatches were themselves unreachable.

`default_wan` filtered the host's addresses through `is_wan_candidate`,
which rejects RFC1918. It now falls back to any address the host holds when
no public one exists — the private address is what inbound packets carry by
the time they arrive, so it is what the DNAT rule (`ip daddr $sip`) has to
match. A host that does hold a public address resolves exactly as before.

Deliberately not the WAN IP override: pinning `wan_ip` also sets the
`Endpoint` in every WireGuard config generated afterwards (api.rs:1167),
so directing these users to pin their private address would hand their
devices an endpoint they cannot dial. Resolving it in `default_wan` leaves
that chain untouched.

`subnet set-wan` and `subnet set-ipv6` were also uninvokable: both declared
their own `<SUBNET>` positional instead of inheriting the parent's, so the
CLI asked for the subnet twice and `combine` rejected it as a duplicate key
(`ParentHandler::cli_parse` -> `util::combine`). They now take `SubnetParams`
as inherited params like their `add`/`remove`/`set-dns` siblings. These two
were the only such commands in the repo; the regenerated man pages lose the
second `<SUBNET>`, and the RPC payload is unchanged, so the web UI keeps
sending `{subnet, wanIp}` and only its type annotations move.

The UI's WAN IP selectors mirrored the same public-only filter, leaving the
menu empty on these hosts, so they now list what the backend accepts.

Regression: 49ea23e (#3306), shipped in start-tunnel 1.1.0, which replaced
`port-forward add`'s caller-supplied external IP with this inference.

Closes #3565

@helix-nine helix-nine left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two things from the analysis on #3565, one of which I think is a real defect in the second pass.

The private fallback can select a bridge address. default_wan_of filters out only wg0 and Loopback, and net_iface holds every interface ip -o addr show reports (tunnel/context.rs:242-252), so on a host running Docker or libvirt the second pass sees docker0 = 172.17.0.1 and virbr0 = 192.168.122.1 alongside the uplink. OrdMap<GatewayId, _> iterates by interface name, and docker0 sorts before ens3/eth0 (as does virbr0 before wlan0), so the fallback picks the container bridge rather than the address inbound packets actually carry — and the resulting DNAT rule never matches.

That only bites when the first pass finds nothing, which is exactly the NAT'd-VPS case this PR targets, and Docker on such a VPS is common.

net/gateway.rs:2314-2318 already draws this line for WAN probing:

let forwardable = !subnets.is_empty()
    && !matches!(
        device_type,
        Some(NetworkInterfaceType::Bridge | NetworkInterfaceType::Loopback)
    );

Adding Bridge to egress()'s filter would match it. Worth considering whether the first pass should stay bridge-inclusive — a public address on a bridge is plausible on a virtualization host, and it was reachable before this PR — in which case the exclusion belongs on the fallback closure only.

external_ipv4 also answers downstream StartOS boxes, and one of those paths won't filter the private address out. It backs GatewayBackend::external_ipv4, so it feeds UPnP GetExternalIPAddress (net/port_map/server/igd.rs:199) and the PCP MAP external address (net/port_map/server/mod.rs:338). A downstream UPnP client discards a private answer (net/port_map/upnp.rs:104-105), but the PCP client returns it unfiltered (net/port_map/client.rs:328-337, :361), and net/gateway.rs:652-675 short-circuits the reachability probe once auto-forwarding has succeeded — so the attached server would report open_externally: true for an RFC1918 address. Today those paths fail closed on this topology (CANNOT_PROVIDE_EXTERNAL), which is less useful but honest. If the intent is dataplane-only, the relaxation could be confined to the callers that key rules (add_forward, resync_forward_keys) and left out of the GatewayBackend impl.

Unrelated to the above: this needs a rebase — GitHub now reports it conflicting, because #3570 landed the same CLI fix (api.rs, the four man pages, both bindings, the three api-service files, CHANGELOG/Cargo.toml at 1.2.2) on master about ten minutes before this went up. Apologies for the collision; two sessions picked up #3565 in parallel and I didn't see this PR until after mine was merged. The forward/igd.rs, wan.ts, help-content and docs work here is all additional to what landed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[bug]: StartTunnel 1.2.1 — port forwarding is impossible on a NAT'd VPS: subnet set-wan is uninvokable and upgraded installs are left with wanIp: null

1 participant