fix(coordinator): reject Stop/StopByName on a dataflow with a pending… - #3114
fix(coordinator): reject Stop/StopByName on a dataflow with a pending…#3114GuTS805 wants to merge 1 commit into
Conversation
… restart initiate_restart() sends StopDataflow and registers a PendingRestart under the dataflow's UUID, but leaves the entry in running_dataflows until DataflowFinishedOnDaemon fires. Stop/StopByName only checked running_dataflows, not pending_restarts, so a concurrent Stop would fall through to stop_dataflow(), succeed, and get queued — then silently lose when the pending restart resolved first and spawned a new incarnation under a fresh UUID. The caller saw a clean 'stop succeeded' while the dataflow kept running under a different UUID. Reproduced live 3/3 with dora restart + dora stop fired concurrently on a real coordinator+daemon+node setup. Now rejects the stop with a clear error instead of silently losing.
|
Merging to
After your PR is submitted to the merge queue, this comment will be automatically updated with its status. If the PR fails, failure details will also be posted here |
|
🤖 Automated review by Claude — this is a fully automated review with no human in the loop. Treat it as advisory. Reviewed the diff — no correctness issues found. The pending-restart guard is correctly placed in both the One observation: this change ships without an automated test. The race is subtle and was verified only by manual repro, and the repo's TDD policy asks for a regression test at the tier that reproduces the bug. An integration test in Generated by Claude Code |
initiate_restart() sends StopDataflow and registers a PendingRestart under
the dataflow's UUID, but leaves the entry in running_dataflows until
DataflowFinishedOnDaemon fires. Stop/StopByName only checked
running_dataflows, not pending_restarts, so a concurrent Stop would fall
through to stop_dataflow(), succeed, and get queued — then silently lose
when the pending restart resolved first and spawned a new incarnation
under a fresh UUID. The caller saw a clean 'stop succeeded' while the
dataflow kept running under a different UUID.
Reproduced live 3/3 with dora restart + dora stop fired concurrently on
a real coordinator+daemon+node setup. Now rejects the stop with a clear
error instead of silently losing.
Why not a duplicate
SpawnedNodeResult{restart:true}leaks the node inrunning_nodes→ dataflow never finishes (stop-vs-restart race and respawn-failure both hit it) #2936 (open) — daemon-level node restart-loop leak in a single daemon'srunning_nodes, not this coordinator-level Restart control-request path
dataflow.start()on an already-stopping dataflow (barrier-completed-by-node-death start isn't gated onstop_sent) #3053 (open) — dataflow.start() firing on an already-stopping dataflow viathe startup barrier, a different trigger mechanism
mid-restart; already fixed by draining pending_restarts on disconnect.
Doesn't cover a concurrent Stop racing a healthy in-progress Restart.
node-lifecycle-e2e.rs, ws_control_tests.rs) exercises Restart racing Stop
on the same UUID.
this branch is based cleanly on origin/main and touches neither file.
Verification
pre-existing unrelated large_enum_variant warning tracked by fix(coordinator): box CachedResult::Cached to fix large_enum_variant on Windows #3001)
3/3 runs — 2/3 hit the new guard directly (clear rejection error), 1/3
the restart had already resolved before stop arrived so it correctly
reported the now-genuinely-finished old UUID (no silent loss in any run)