fix: hotspot/AP network resilience — Windows drain, watcher v2, rotation handoff#111
Merged
Conversation
Rotation on Windows was a hard TerminateProcess: graceful_stop_child had
no non-unix drain path, and the worker only listened for Ctrl+C. Every
network-change rotation reset all in-flight connections — the top cause
of "hotspot/AP works badly on Windows".
Worker (soth-proxy): new drain_signal module. On startup the worker
creates a named kernel event Local\soth-worker-drain-{pid} and waits on
it from a detached thread (GenerateConsoleCtrlEvent cannot cross the
CREATE_NO_WINDOW console boundary; a named event can). Event fires ->
same graceful path SIGUSR1 takes on Unix (stop accepting, drain 30s).
Both duplicated shutdown-wait blocks in runtime.rs now share
drain_signal::wait_for_shutdown_signal().
Supervisor (soth-cli): Windows graceful_stop_child opens the child's
drain event, sets it, and waits up to 30s for exit — mirroring the Unix
SIGUSR1 flow. Falls back to the old hard kill when the event can't be
opened (older worker, dead child) or the drain window elapses.
Part of docs/common/2026-07-02/network-resilience-impl-plan.md (Phase 2).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… edge, debounce
The watcher compared only the IPv4 address set (Unix) or the
default-route IPv4 (Windows UDP-connect trick). Hotspot/AP transitions
it missed entirely:
- toggle hotspot off/on or roam APs: same IP comes back, no event —
the worker kept its startup-pinned DNS nameservers and dead sockets
("tunnel connection failed" until `soth up`)
- IPv6-only / IPv6-primary tethering: invisible
- DNS-server change without an address change: invisible
- offline ticks were skipped outright, so drop -> recover-same-address
never fired
Replace with a NetworkSignature {v4, v6 (global+ULA), dns} poll:
- Unix: getifaddrs extended to AF_INET6; DNS parsed from
/etc/resolv.conf (configd-managed on macOS; inert-but-harmless on
systemd-resolved stubs)
- Windows: adapter enumeration via the ipconfig crate (what
hickory-resolver uses), replacing the UDP trick — covers addresses,
IPv6, and per-adapter DNS in one call
- offline marker: recovery fires even with an identical signature
- 10s debounce with trailing delivery: a flapping hotspot causes one
rotation per window instead of a rotation storm
Watcher core now takes an injectable signature source; new tests cover
offline-recovery, DNS-only change, and debounce/trailing behavior.
117 soth-cli unit tests pass.
Part of docs/common/2026-07-02/network-resilience-impl-plan.md (Phase 3).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…rotation
Readiness (both platforms): wait_for_listener_start TCP-probed
127.0.0.1:<port>, which cannot distinguish the new child's listener
from the old child's during rotation — and on Unix proved nothing about
the worker at all, since the supervisor holds the inherited listener
and the kernel completes connects from its backlog. The supervisor now
passes SOTH_WORKER_READY_FILE to each spawned worker; the worker writes
{"pid": ...} there (atomic temp+rename) once its listener is live, and
the supervisor waits on that file. The TCP probe survives only as a
late fallback (after half the startup budget) for version-skewed
workers across hot-updates. Probe files are cleaned up on drop.
Rotation order (Windows): with SO_EXCLUSIVEADDRUSE (soth-mitm 0.3.4)
the new child can no longer bind over the old child's live listener,
so the overlapped Unix flow cannot work there. Windows rotation is now
sequenced: signal the drain event first (the old worker closes its
listener within milliseconds and keeps draining in-flight flows), then
spawn the new child into the freed port — its bind-retry loop absorbs
the close latency — then wait up to 30s for the old child to finish
draining. If the new child fails to start, the supervisor loop observes
the draining old child's exit and respawns via the normal restart path.
Part of docs/common/2026-07-02/network-resilience-impl-plan.md (Phase 4).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The per-PR lanes are ubuntu-only and the existing test-windows job skipped exactly the two crates that carry Windows-divergent runtime code: soth-proxy (named drain event) and soth-cli (ipconfig network watcher, sequenced rotation, drain-event signaling). Add them so cfg(windows) API mistakes surface at the release gate — and via workflow_dispatch for branch validation. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
`cargo test --lib -p soth-cli-update-sidecar` fails with "no library targets found" (the crate is binary-only), so the job has been broken since it was added — masked because it only runs on tag push / workflow_dispatch. Test the bin target instead. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
dirs::home_dir() reads USERPROFILE on Windows, not HOME, so the with_temp_home sandboxes leaked: `~` expansion (e.g. the default bundle_dir "~/.soth/bundle") resolved into the real runner profile and start_forwards_allow_daemon_child_fallback_flag failed on the new native Windows lane with "Bundle directory ... missing". Override and restore both vars in all three helpers (command_graph, proxy/start, proxy/system); harmless on Unix where USERPROFILE is unused. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
secxena
force-pushed
the
fix/windows-network-resilience
branch
from
July 3, 2026 09:02
28c9139 to
77d8d7f
Compare
secxena
added a commit
that referenced
this pull request
Jul 4, 2026
Re-applies a fix that was orphaned off PR #111 before its merge (the env-aware home_dir landed on the branch after the merge commit, so staging never got it). dirs::home_dir() resolves via the known-folder OS API on Windows and ignores the environment, so `~` expansion (default bundle_dir, config/device-id paths, bootstrap init dir) escaped both HOME and SOTH_HOME_DIR — the recurring start_forwards_allow_daemon_child_fallback_flag failure on the native Windows lane ("Bundle directory C:\Users\runneradmin\... missing"). Add an env-aware home_dir() that prefers a non-empty $HOME and falls back to dirs::home_dir(); route cli_config's path helpers and the command-graph bootstrap dir through it. Also fixes home-relocation via $HOME on Windows in the product, not just tests. Unchanged where HOME is unset (normal Windows) or equals the profile dir (Unix). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
secxena
added a commit
that referenced
this pull request
Jul 4, 2026
) * fix(cli): fail-open safety net so a dead proxy never strands the user SOTH installs itself as the machine-wide system proxy, so any way the worker dies without cleanup leaves the OS proxy pointed at a dead 127.0.0.1:<port> — total internet outage until the user manually runs `soth off`. This converts that fail-CLOSED posture to fail-OPEN (Tier 0 of the 2026-07-04 robustness audit, docs/common/2026-07-04/network-robustness-hardening-plan.md). Three layers, all signature-gated so they only ever clear soth's own loopback proxy, never a foreign one: 1. Health watchdog (0.1): a reconciler task spawned in supervise_proxy that every 5s checks the local listener. If it's dead for 3 consecutive ticks (~15s, above a normal rotation window) AND the OS proxy still carries soth's signature, it disables the system proxy so connectivity falls back to direct. Re-enables (quietly) once the listener recovers. Aborted via Drop guard when the supervisor exits, so it can never re-enable a proxy being torn down. This covers the cases the restart loop can't see — a wedged-but-not-exited worker, a listener that stops accepting. 2. Give-up fail-open (0.2): when the supervisor exhausts its restart budget and bails, it now disables the system proxy first. 3. Startup crash-repair (0.3): if worker startup fails while a dangling soth proxy from a prior crashed/SIGKILLed session is still active, disable it before returning the error (fail_startup_open). And `soth doctor` now reads the *live* OS proxy setting (new os_signature_active), emitting an error-level `dangling_system_proxy` finding — pointing at `soth doctor --reset-network` — when the proxy is set but no listener answers. Previously doctor only read soth's own state file and couldn't see this at all. New system.rs surface: soth_proxy_signature_active(port) (cross-platform signature check) and enable_quiet(). 121 soth-cli unit tests pass incl. new doctor findings coverage; the Windows cfg paths are exercised by the release-ci test-windows lane. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(cli): honor $HOME in tilde expansion and default soth paths Re-applies a fix that was orphaned off PR #111 before its merge (the env-aware home_dir landed on the branch after the merge commit, so staging never got it). dirs::home_dir() resolves via the known-folder OS API on Windows and ignores the environment, so `~` expansion (default bundle_dir, config/device-id paths, bootstrap init dir) escaped both HOME and SOTH_HOME_DIR — the recurring start_forwards_allow_daemon_child_fallback_flag failure on the native Windows lane ("Bundle directory C:\Users\runneradmin\... missing"). Add an env-aware home_dir() that prefers a non-empty $HOME and falls back to dirs::home_dir(); route cli_config's path helpers and the command-graph bootstrap dir through it. Also fixes home-relocation via $HOME on Windows in the product, not just tests. Unchanged where HOME is unset (normal Windows) or equals the profile dir (Unix). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * review: gate startup crash-repair on a dead listener Addresses adversarial-review finding 1 on this PR: fail_startup_open disabled the OS proxy whenever it carried soth's signature, without checking the listener was actually dead. On a Windows bind-conflict — a second 'soth start' whose worker fails to bind because a HEALTHY prior instance already holds the port — this tore down the running instance's system proxy, silently turning off interception with nothing to re-enable it. Add !is_local_listener_ready(port) to the guard so we only clear a soth-signature proxy when nothing is answering (the genuine dangling-from-a-crash case), mirroring the watchdog's dead-AND-signature rule. A bind conflict against a healthy instance now leaves that instance's proxy untouched. 121 soth-cli tests pass. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * review: fix Windows signature false-positive, soth-off defeat, blocking probes Addresses the deeper adversarial-review findings on this PR: 1. (loses-internet) The Windows signature used starts_with — a PREFIX match — so a foreign loopback proxy whose port has soth's as a numeric prefix (soth :1080 vs foreign :10800; soth :8 vs :8080) falsely matched soth's signature, letting the watchdog / disable path clear someone else's proxy. Replaced with exact host:port token matching that also handles the scheme=host:port;... list form (windows_proxy_value_is_soth), and fixed the same pre-existing bug in the sentinel-missing disable path. macOS/Linux already matched exactly. 2. (misconfigured) The watchdog re-enabled the proxy after a user ran → Removing system proxy configuration... during a watchdog-induced outage — silently undoing the user's intent, since the in-process disabled_by_watchdog flag can't observe an external → Removing system proxy configuration.... Added a user_proxy_off sentinel set by the user-facing disable (not the watchdog's own quiet disable) and cleared by any enable; the watchdog checks it before re-enabling and stands down if the user turned it off. 3. (runtime) The watchdog's synchronous probes (500ms connect; subprocess reads for the signature) ran directly on the async workers. Wrapped both in spawn_blocking so a per-tick block can't starve the supervisor on a small runtime; a probe panic resolves to the safe direction (not-ready / not-ours). New tests: exact-match truth table (Windows-gated, run by the CI Windows lane) and the sentinel round-trip. 121 soth-cli tests pass. Known narrow limitations left as-is (documented for reviewers): a co-located instance B can still disable instance A's proxy if B starts precisely during A's multi-second restart gap (cross-process, would need IPC); and a shutdown at the exact instant of a watchdog re-enable can race (very narrow interleave). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
secxena
added a commit
that referenced
this pull request
Jul 4, 2026
…y/PAC (#113) * fix(cli): honor $HOME in tilde expansion and default soth paths Re-applies a fix that was orphaned off PR #111 before its merge (the env-aware home_dir landed on the branch after the merge commit, so staging never got it). dirs::home_dir() resolves via the known-folder OS API on Windows and ignores the environment, so `~` expansion (default bundle_dir, config/device-id paths, bootstrap init dir) escaped both HOME and SOTH_HOME_DIR — the recurring start_forwards_allow_daemon_child_fallback_flag failure on the native Windows lane ("Bundle directory C:\Users\runneradmin\... missing"). Add an env-aware home_dir() that prefers a non-empty $HOME and falls back to dirs::home_dir(); route cli_config's path helpers and the command-graph bootstrap dir through it. Also fixes home-relocation via $HOME on Windows in the product, not just tests. Unchanged where HOME is unset (normal Windows) or equals the profile dir (Unix). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(cli): complete proxy bypass lists + warn before clobbering a proxy/PAC Tier 1.4/1.5/1.1/1.2 of the network-robustness audit (docs/common/2026-07-04/network-robustness-hardening-plan.md). Fixes OS-integration cases where soth silently broke connectivity. Bypass completeness (1.4/1.5): - Add link-local (169.254.*), CGNAT 100.64.0.0/10 — Tailscale's whole range incl. MagicDNS 100.100.100.100 and *.ts.net — and IPv6 private (fc00::/7, fe80::/10) to PROXY_BYPASS_DOMAINS. Previously only ::1 was bypassed for v6, and tailnet/overlay traffic routed through soth. - Derive the Linux no_proxy / KDE NoProxyFor and the Windows ProxyOverride from that single source instead of hardcoded subsets. This fixes two silent breakages: Linux env-var no_proxy omitted all RFC1918 (curl/pip/npm/apt proxied LAN + internal-registry traffic), and the Windows ProxyOverride omitted the entire captive-portal allow-list (NCSI probes + hotel/airport login pages went through soth, whose CA isn't trusted pre-login, and failed). Pre-existing proxy / PAC detection (1.1/1.2): - Before overwriting the OS proxy, detect_conflicting_proxy checks for a manual proxy pointing somewhere other than our loopback, or a PAC / auto-config URL (macOS -getautoproxyurl, GNOME mode=auto, Windows AutoConfigURL). On corp/managed networks where the only egress is that proxy, silently replacing it with soth (which dials direct) takes the user fully offline while status reports healthy. We warn loudly and log rather than clobber silently. Full upstream-proxy chaining is a follow-up; this is the audit's stated minimum. Best-effort throughout: any probe error resolves to "no conflict" so a failed read never blocks `soth on`. 119 soth-cli unit tests pass (new coverage: bypass list contains the private/link-local/CGNAT/v6/captive entries; joined form keeps RFC1918 + captive in sync). Windows/Linux cfg paths compile+run via the release-ci test-windows lane. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Phases 2–4 of the network-resilience plan (
docs/common/2026-07-02/network-resilience-impl-plan.mdin the workspace docs). Fixes the "hotspot and access-point connections work badly" complaints, which hit Windows hardest. Companion crate PR: soth-ai/soth-mitm#70 (0.3.4 — TLS handshake timeouts, Windows keepalive, exclusive bind, connect retry).Windows graceful drain (was: hard TerminateProcess on every network change)
soth-proxy): newdrain_signalmodule. Creates a named kernel eventLocal\soth-worker-drain-{pid}and waits on it from a detached thread; event → same stop-accepting/drain-30s path SIGUSR1 takes on Unix. (GenerateConsoleCtrlEventcan't cross theCREATE_NO_WINDOWconsole boundary; a named event can.)soth-cli): Windowsgraceful_stop_childopens + sets the event and waits ≤30s, falling back to the old hard kill on failure.Network watcher v2 (was: IPv4-set on Unix / default-route-IP UDP trick on Windows)
getifaddrsextended to AF_INET6 +/etc/resolv.confparse. Windows: adapter enumeration via theipconfigcrate (what hickory-resolver uses).soth up).Rotation handoff correctness
SOTH_WORKER_READY_FILE; the worker writes{"pid": …}once its listener is live. Replaces the TCP probe, which couldn't distinguish old/new child during rotation (and on Unix only ever proved the supervisor's own inherited listener existed). TCP probe kept as a late fallback for version-skewed workers across hot-updates.SO_EXCLUSIVEADDRUSE(soth-mitm 0.3.4) makes listener overlap impossible.Follow-ups (not in this PR)
soth-mitm0.3.4 publish + dep bump incrates/soth-proxy(currently 0.3.3; this PR compiles and works against either — the sequenced-rotation benefit fully lands with 0.3.4)soth off)Testing
🤖 Generated with Claude Code