fix(cli): fail-open safety net — dead proxy never strands the user#112
Merged
Conversation
SOTH installs itself as the machine-wide system proxy, so any way the worker dies without cleanup leaves the OS proxy pointed at a dead 127.0.0.1:<port> — total internet outage until the user manually runs `soth off`. This converts that fail-CLOSED posture to fail-OPEN (Tier 0 of the 2026-07-04 robustness audit, docs/common/2026-07-04/network-robustness-hardening-plan.md). Three layers, all signature-gated so they only ever clear soth's own loopback proxy, never a foreign one: 1. Health watchdog (0.1): a reconciler task spawned in supervise_proxy that every 5s checks the local listener. If it's dead for 3 consecutive ticks (~15s, above a normal rotation window) AND the OS proxy still carries soth's signature, it disables the system proxy so connectivity falls back to direct. Re-enables (quietly) once the listener recovers. Aborted via Drop guard when the supervisor exits, so it can never re-enable a proxy being torn down. This covers the cases the restart loop can't see — a wedged-but-not-exited worker, a listener that stops accepting. 2. Give-up fail-open (0.2): when the supervisor exhausts its restart budget and bails, it now disables the system proxy first. 3. Startup crash-repair (0.3): if worker startup fails while a dangling soth proxy from a prior crashed/SIGKILLed session is still active, disable it before returning the error (fail_startup_open). And `soth doctor` now reads the *live* OS proxy setting (new os_signature_active), emitting an error-level `dangling_system_proxy` finding — pointing at `soth doctor --reset-network` — when the proxy is set but no listener answers. Previously doctor only read soth's own state file and couldn't see this at all. New system.rs surface: soth_proxy_signature_active(port) (cross-platform signature check) and enable_quiet(). 121 soth-cli unit tests pass incl. new doctor findings coverage; the Windows cfg paths are exercised by the release-ci test-windows lane. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Re-applies a fix that was orphaned off PR #111 before its merge (the env-aware home_dir landed on the branch after the merge commit, so staging never got it). dirs::home_dir() resolves via the known-folder OS API on Windows and ignores the environment, so `~` expansion (default bundle_dir, config/device-id paths, bootstrap init dir) escaped both HOME and SOTH_HOME_DIR — the recurring start_forwards_allow_daemon_child_fallback_flag failure on the native Windows lane ("Bundle directory C:\Users\runneradmin\... missing"). Add an env-aware home_dir() that prefers a non-empty $HOME and falls back to dirs::home_dir(); route cli_config's path helpers and the command-graph bootstrap dir through it. Also fixes home-relocation via $HOME on Windows in the product, not just tests. Unchanged where HOME is unset (normal Windows) or equals the profile dir (Unix). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Addresses adversarial-review finding 1 on this PR: fail_startup_open disabled the OS proxy whenever it carried soth's signature, without checking the listener was actually dead. On a Windows bind-conflict — a second 'soth start' whose worker fails to bind because a HEALTHY prior instance already holds the port — this tore down the running instance's system proxy, silently turning off interception with nothing to re-enable it. Add !is_local_listener_ready(port) to the guard so we only clear a soth-signature proxy when nothing is answering (the genuine dangling-from-a-crash case), mirroring the watchdog's dead-AND-signature rule. A bind conflict against a healthy instance now leaves that instance's proxy untouched. 121 soth-cli tests pass. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ng probes Addresses the deeper adversarial-review findings on this PR: 1. (loses-internet) The Windows signature used starts_with — a PREFIX match — so a foreign loopback proxy whose port has soth's as a numeric prefix (soth :1080 vs foreign :10800; soth :8 vs :8080) falsely matched soth's signature, letting the watchdog / disable path clear someone else's proxy. Replaced with exact host:port token matching that also handles the scheme=host:port;... list form (windows_proxy_value_is_soth), and fixed the same pre-existing bug in the sentinel-missing disable path. macOS/Linux already matched exactly. 2. (misconfigured) The watchdog re-enabled the proxy after a user ran → Removing system proxy configuration... during a watchdog-induced outage — silently undoing the user's intent, since the in-process disabled_by_watchdog flag can't observe an external → Removing system proxy configuration.... Added a user_proxy_off sentinel set by the user-facing disable (not the watchdog's own quiet disable) and cleared by any enable; the watchdog checks it before re-enabling and stands down if the user turned it off. 3. (runtime) The watchdog's synchronous probes (500ms connect; subprocess reads for the signature) ran directly on the async workers. Wrapped both in spawn_blocking so a per-tick block can't starve the supervisor on a small runtime; a probe panic resolves to the safe direction (not-ready / not-ours). New tests: exact-match truth table (Windows-gated, run by the CI Windows lane) and the sentinel round-trip. 121 soth-cli tests pass. Known narrow limitations left as-is (documented for reviewers): a co-located instance B can still disable instance A's proxy if B starts precisely during A's multi-second restart gap (cross-process, would need IPC); and a shutdown at the exact instant of a watchdog re-enable can race (very narrow interleave). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
SOTH installs itself as the machine-wide system proxy, so any way the worker dies without cleanup leaves the OS proxy pointed at a dead
127.0.0.1:<port>— a total internet outage until the user manually runssoth off. This is Tier 0 of the six-agent robustness audit (docs/common/2026-07-04/network-robustness-hardening-plan.md), converting that fail-closed posture to fail-open.Three layers, all signature-gated so they only ever clear soth's own loopback proxy, never a foreign one (Charles, a corporate proxy, etc.):
1. Health watchdog (audit 0.1) — the structural fix
A reconciler task spawned in
supervise_proxythat every 5s checks the local listener. If it's dead for 3 consecutive ticks (~15s, deliberately above a normal rotation window) and the OS proxy still carries soth's signature, it disables the system proxy → connectivity falls back to direct. Re-enables (quietly) once the listener recovers. Aborted via aDropguard when the supervisor exits, so it can never re-enable a proxy that's being torn down. This catches what the restart loop can't see: a wedged-but-not-exited worker, or a listener that silently stops accepting.2. Give-up fail-open (audit 0.2)
When the supervisor exhausts its restart budget and bails, it now disables the system proxy first (one-liner, biggest bang-for-buck).
3. Startup crash-repair + doctor detection (audit 0.3)
fail_startup_opendisables it before returning the error.soth doctornow reads the live OS proxy setting (os_signature_active), emitting an error-leveldangling_system_proxyfinding — pointing atsoth doctor --reset-network— when the proxy is set but no listener answers. Previously doctor only read soth's own state file and was blind to this.New surface (system.rs)
soth_proxy_signature_active(port) -> bool— cross-platform check of whether the live OS proxy is soth's loopback+port signature (macOSnetworksetup, Linux gsettings, Windows registry). Best-effort; resolves tofalseon any read error (safe direction, sincetrueauthorizes a disable).enable_quiet(port)— non-printing enable for the watchdog.Testing
compute_findingscoverage: dead-listener-without-proxy → warn, dead-listener-with-active-proxy →dangling_system_proxyerror, healthy → no finding.release-citest-windowslane (soth-cli was added to it in the prior PR).Scope / follow-ups
This is Tier 0 only (stop total outages). Tiers 1–3 from the plan — corp-proxy/PAC chaining, DNS fail-open to getaddrinfo, bypass-list completeness, captive-portal probing, the data-plane tunnel-idle / connection-cap / h2-permit-leak fixes, and the TLS-backoff rework — are separate follow-up PRs.
🤖 Generated with Claude Code