Make the Headscale e2e reliable under CI load - #83
Merged
Conversation
Two timing-sensitive test classes flaked intermittently on contended CI runners (30s "Future not completed" timeouts, a different test each run, never reproducible locally). Root causes and principled fixes: 1. Data-plane tests raced WireGuard/DERP path establishment. The two-node group never established the tunnel before its timed tests, so the first data-plane op -- or a peer->self inbound request -- raced establishment (endpoint discovery + DERP negotiation) inside its own 30s window. Added awaitDataPath(): the group's setUpAll now brings the path up once, up front, with a retrying 90s-budget probe against the peer's /hello. A successful outbound round-trip warms the tunnel both directions, so inbound tests are fast too. This is a structural readiness guarantee, not a bigger timeout. 2. Control-plane transition budgets were too tight. recordUntil defaulted to 30s; a credential-reconnect (up without authkey) against a containerized Headscale on a loaded runner can exceed that. The timeout IS the wait mechanism here, so it was simply under-budgeted: recordUntil 30s -> 60s, the broadcast reconnect wait 10s -> 60s, and a local onNodeChanges first- emit 2s -> 10s for headroom. Verified: e2e green locally (36/36), and green again under full CPU saturation (10 hogs on 10 cores, load avg ~39) to simulate a contended runner -- the condition that triggered the flakes. Test-only; no library or assertion changes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The e2e suite flaked intermittently on CI — 30s "Future not completed" timeouts, a different test each run, never reproducible locally, and it even failed on a changelog-only PR. Two root causes, both fixed at the source rather than by blanket timeout inflation:
1. Data-plane tests raced path establishment (the structural fix)
The two-node group never established the WireGuard/DERP tunnel before its timed tests, so the first data-plane op — or a peer→self inbound request — raced establishment (endpoint discovery + DERP negotiation) inside its 30s window.
awaitDataPath()now brings the path up once insetUpAllwith a retrying 90s-budget probe against the peer's/hello; a successful outbound round-trip warms the tunnel bidirectionally, so inbound tests are fast too. Tests now start from a guaranteed-warm precondition.2. Control-plane transition budgets were under-sized
recordUntildefaulted to 30s; a credential-reconnect against a containerized Headscale on a loaded runner can exceed it. Here the timeout is the wait mechanism, so it was right-sized:recordUntil30s→60s, the broadcast reconnect wait 10s→60s, a localonNodeChangesfirst-emit 2s→10s.Verification
yeshogs on 10 cores, load avg ~39) — deliberately simulating the contended-runner condition that triggered the flakes.Honest caveat: local can't perfectly reproduce a network-latency flake, so the ultimate proof is CI staying green across subsequent runs — but the structural warm-up removes the race entirely, and the stress run demonstrates the margins hold under heavy scheduling contention.
🤖 Generated with Claude Code