fix(proxy): probe upstream liveness with TCP keepalive - #114
Open
cleverhu wants to merge 2 commits into
Open
Conversation
envd streams a command's lifecycle as a burst of tiny Connect-RPC frames. With Nagle enabled, every frame after the first waits for the peer's delayed ACK, which puts a ~40ms floor under each short-lived command regardless of how long the command itself runs. Every hop on that path exchanges small frames only, so the coalescing Nagle trades latency for never materializes. Set TCP_NODELAY on the API server's accepted connections, on the sandbox proxy's connector and on both envd client connectors.
Tearing down a VM never delivers FIN/RST to the host side of a proxied connection, so a request to a sandbox that is already gone hangs until some higher-level timeout fires. Short keepalive probes turn that into a prompt connection error instead.
Contributor
|
✅ OpenCodeReview: Review complete: 0 finding(s) across 3 selected item(s). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Enable TCP keepalive probes on the sandbox proxy's connector: 3 s idle before the first probe, 1 s between probes, 2 retries.
Why
Tearing down a VM never delivers FIN or RST to the host side of a proxied connection. The socket stays open from the host's point of view, so a request addressed to a sandbox that is already gone hangs until some higher-level timeout fires instead of failing immediately.
Keepalive probes are what detect that silence. With the values above the failure surfaces in roughly 5 s, which is a prompt error for a caller and still far above any plausible pause in normal proxied traffic.
Related issue
None. Small, self-contained reliability fix on an existing code path.
Scope and non-goals
Included: keepalive parameters on the proxy connector.
Excluded:
pool_max_idle_per_host(0)already handles the separate problem of interaction IPs being reused across runtime generations.Design and behavior changes
Three constants next to the existing
PROXY_CONNECT_TIMEOUT, applied viaHttpConnector::set_keepalive{,_interval,_retries}.Failure mode change: a request to a torn-down sandbox now returns a connection error after ~5 s instead of hanging. Callers that previously relied on a longer timeout to mask this will see the error sooner, which is the intent.
Compatibility and operations
[orchestrator]or a proxy section.TCP_KEEPIDLE/TCP_KEEPINTVL/TCP_KEEPCNTare Linux socket options and AgentENV is Linux-only.Validation
make fmtmake clippymake test-unitmake -C services test(required whenservices/changes)maketargetCommands and results:
Skipped checks and reasons:
make test-unitnot run. Observing this behavior in a test requires a peer that vanishes without sending FIN/RST, which is not reproducible with an in-process test server; it needs a real VM teardown or an injected netfilter drop.Risks and reviewer notes
The risk worth reviewing is the aggressiveness of the values. A 3 s idle threshold with 1 s probes means an idle-but-alive sandbox connection sends a probe every second. That is negligible traffic on a host-local link, but if the proxy is ever used across a slower network the numbers deserve revisiting.
The other question is whether ~5 s to detect is the right target. It is well under typical client timeouts while leaving room for a transient stall.
Most important file:
src/api/proxy.rs.Checklist