Symptom
Main CI run 31192788714 (post-merge of #1022, SHA 02e0ca1): the Python 3.11 Run tests step hung ~19.5 min with zero output (typical full-suite runtime ~10 min) while 3.12/3.13/lint/frontend/Docker/CodeQL all passed. Cancelled to expose the partial log; the runner's cleanup then reported:
Complete job: Cleaning up orphan processes
Terminate orphan process: pid (2156) (pytest)
Terminate orphan process: pid (2423) (claude)
Terminate orphan process: pid (2446) (python)
Terminate orphan process: pid (2447) (python)
Terminate orphan process: pid (2448) (python)
A live claude process was running as a child of the hung pytest. Rerun of the same SHA: 3.11 passed in normal time. The same code + test files also passed all three matrices on the PR run 20 minutes earlier. So: non-deterministic, likely test-ordering or timing dependent.
Why this matters beyond the flake
- Some test can spawn a real
claude CLI process instead of a mock under some interleaving. On a bare runner it hangs (no auth/TTY) and burns a 20+ min CI slot until timeout/cancel; the job produces no partial output while hung.
- Hermeticity: a test that can exec the real binary would, on any runner or dev box where
claude IS authenticated, start a real session with real side effects.
Diagnosis pointers
- Suspects: tests exercising spawn paths where the command comes from
_build_claude_cmd() / PINKY_CLAUDE_BIN / dream-runner or SDK-runner startup without the subprocess layer mocked — the specimen had 1×claude + 3×python children, which smells like a partially-mocked spawn chain rather than a single stray Popen.
- CI (
ci.yml Run tests) shows no live output while hung — pytest -q buffers; consider -v --timeout (pytest-timeout) as a structural guard so ANY future hang self-reports the test name instead of requiring a cancel-to-read.
- Grep starting points:
subprocess / Popen / create_subprocess in tests + fixtures that set PINKY_CLAUDE_BIN; any test relying on shutil.which("claude") absence as its "mock".
Fix shape
- Find and mock the leaking spawn path (the actual bug).
- Structural guards: pytest-timeout (per-test cap) so a future leak fails loudly with a name in minutes, not silently for 20; optionally a conftest-level guard that stubs
claude out of PATH / sets PINKY_CLAUDE_BIN=/bin/false for the whole suite so no test can ever exec a real claude.
Both CI runs referenced: hung 31192788714 (pre-rerun logs in the cancelled attempt), passing rerun same id attempt 2.
🤖 Opened by Barsik
Symptom
Main CI run 31192788714 (post-merge of #1022, SHA 02e0ca1): the Python 3.11
Run testsstep hung ~19.5 min with zero output (typical full-suite runtime ~10 min) while 3.12/3.13/lint/frontend/Docker/CodeQL all passed. Cancelled to expose the partial log; the runner's cleanup then reported:A live
claudeprocess was running as a child of the hung pytest. Rerun of the same SHA: 3.11 passed in normal time. The same code + test files also passed all three matrices on the PR run 20 minutes earlier. So: non-deterministic, likely test-ordering or timing dependent.Why this matters beyond the flake
claudeCLI process instead of a mock under some interleaving. On a bare runner it hangs (no auth/TTY) and burns a 20+ min CI slot until timeout/cancel; the job produces no partial output while hung.claudeIS authenticated, start a real session with real side effects.Diagnosis pointers
_build_claude_cmd()/PINKY_CLAUDE_BIN/ dream-runner or SDK-runner startup without the subprocess layer mocked — the specimen had 1×claude + 3×python children, which smells like a partially-mocked spawn chain rather than a single stray Popen.ci.ymlRun tests) shows no live output while hung — pytest-qbuffers; consider-v --timeout(pytest-timeout) as a structural guard so ANY future hang self-reports the test name instead of requiring a cancel-to-read.subprocess/Popen/create_subprocessin tests + fixtures that setPINKY_CLAUDE_BIN; any test relying onshutil.which("claude")absence as its "mock".Fix shape
claudeout of PATH / setsPINKY_CLAUDE_BIN=/bin/falsefor the whole suite so no test can ever exec a real claude.Both CI runs referenced: hung 31192788714 (pre-rerun logs in the cancelled attempt), passing rerun same id attempt 2.
🤖 Opened by Barsik