test: stub Slack name resolution instead of relying on a dead port - #252
Open
Dhevenddra wants to merge 1 commit into
Open
test: stub Slack name resolution instead of relying on a dead port#252Dhevenddra wants to merge 1 commit into
Dhevenddra wants to merge 1 commit into
Conversation
Both relay test modules pinned SLACK_API_URL at http://127.0.0.1:9/ on the stated assumption that lookups "fail instantly at a dead loopback port". That holds on POSIX, where a refused connection returns in microseconds. It does not hold on Windows, where refusing a loopback connect takes about 2.0s: socket.create_connection(("127.0.0.1", 9)) median 2.033s over 5 trials urllib.urlopen("http://127.0.0.1:9/...") median 2.032s over 3 trials RelayHub._run only increments _dispatched after awaiting the frame handler, and _dispatch_slack_event awaits both _display_name and _channel_name, so a single frame cost roughly 4s against the 2.0s wait_dispatched budget. The result was a deterministic "only 0 frames dispatched" across five tests, reproduced identically on three consecutive runs. Stub _slack_get, the single chokepoint both lookups go through, so the tests are hermetic by construction rather than by depending on how fast the host OS refuses a connection. SLACK_API_URL still points at the dead port as a backstop so an unstubbed path can never reach slack.com. The stub goes on the class, so the two tests that exercise real name mapping keep working: they set _slack_get on the instance, which shadows it. On Windows this takes the two modules from 5 failed / 29 passed in 21.2s to 34 passed in 10.2s. Linux is unaffected: 889 passed, 1 skipped.
This was referenced Jul 28, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Five tests fail deterministically on Windows because of an assumption written into the test
fixture itself:
tests/test_github_installs.py:218uses the same pattern inline."Fails instantly" is true on POSIX. It is not true on Windows. Measured on this host, 5 trials
each:
socket.create_connection(("127.0.0.1", 9))ConnectionRefusedErrorurllib.urlopen("http://127.0.0.1:9/...")URLErrorsocket.create_connection(("localhost", 9))::1, then IPv4The budget is
wait_dispatched(at_least, timeout: float = 2.0).It compounds, because
RelayHub._runonly counts a frame after awaiting the handler:and
_dispatch_slack_eventawaits two lookups per frame (relay_client.py:335and:339),each funnelling into
_slack_get. So one frame costs roughly 4s against a 2.0s budget.That is why the diagnostic is
only 0 frames dispatched (< 1). Zero rather than partial is whatdistinguishes this from ordinary timing jitter.
This is not flakiness
Three consecutive runs of both modules produced the identical five failures each time:
I mention this because #209, #210 and #216 are addressing genuinely flaky tests, and this is a
different problem that happens to share a symptom.
Fix
Stub
_slack_get, the single chokepoint both_display_nameand_channel_nameroute through,so the tests are hermetic by construction rather than by depending on how quickly the host OS
refuses a connection. The module docstring already claims hermeticity ("no live WebSocket ... no
network"); name resolution was the one path still escaping it.
SLACK_API_URLstays pinned at the dead port as a backstop, so any path I have not stubbed stillcannot reach slack.com.
One detail worth flagging for review: the stub is installed on the class, while
test_relay_resolves_names_and_mentionsandtest_relay_name_cache_is_per_workspaceset_slack_geton theinstance. Instance attributes shadow class attributes, so those two tests still exercise real
name mapping and still assert
user_name == "Rohit"/chat_name == "ocw-test". Their stubs take(team_id, method, params)because they are instance-bound; the class-level one takesself.Verification
The two-fold speedup on Windows is just the removed connect stalls.
The 3 remaining Windows failures are unrelated to this PR: two are the
user-only filepermission tests covered by #250, and one is
test_workspace_command_trust_controls_live_enginefailing withWinError 32on a directoryrename, which I have not investigated and am not claiming to fix.
Alternative considered
The repo already has a
fake_slackfixture intests/conftest.pythat boots an in-process fakeSlack and points
SLACK_API_URLat it. That would be more realistic than a stub, but it addsasync startup cost to tests that never assert anything about name resolution, and the dead-port
approach suggests speed was the original priority. Happy to switch to
fake_slackif you wouldrather these go through the real Web API surface.
Note
Found by running the suite on Windows. CI is
ubuntu-latestonly, so this cannot surfaceupstream today.