`dora run` orphan containment: remaining gaps after #3018 — pre-`init` window, `path: shell` nodes, Windows Job Object

[heyong4725]

Summary

When the dora run process is killed abnormally (SIGKILL, supervisor timeout, kill to its process group), the dataflow's node processes survive it and busy-spin at 100–170% CPU indefinitely. They never exit on their own — we have observed an orphaned node accumulate 59 CPU-hours before being found.

Environment

• dora-cli 0.5.0, dora run <graph>.yaml --uv
• macOS 26 (arm64), Python 3.13 nodes (from dora import Node, pyarrow payloads)
• Nodes are plain Python processes spawned by the runtime (python3 -u <node>.py)

Reproduction

1. dora run dataflow.yaml --uv with any dataflow of a few Python nodes (ours have timer-driven inputs, e.g. dora/timer/millis/10).
2. Kill the dora run process hard while the dataflow is running — any of:
   • kill -9 <pid of dora run>
   • timeout 30 dora run ... (the pattern that first bit us: a test harness wrapping runs in timeout)
   • os.killpg(pid, SIGKILL) on the process group dora run was started in (start_new_session=True)
3. Observe surviving python3 -u <node>.py processes: ps aux | awk '$3 > 50'.

The killpg case is notable: the node processes are evidently not (all) in the dora run process group, so even a group kill misses them.

Expected

Node processes exit (or are reaped) when their runtime/daemon dies — ideally via a parent-death signal (PR_SET_PDEATHSIG equivalent; on macOS e.g. a watchdog on the daemon connection) or a bounded read-timeout → exit path.

Actual

Orphaned nodes spin in their event loop at 100–170% CPU each, apparently busy-polling the dead dataflow's channels, forever. On a laptop-class machine a handful of orphans dominate the cores and silently corrupt any timing-sensitive workload that runs afterwards (this is how we usually notice — unexplained perf regressions, then ps shows orphans from hours or days earlier).

Relationship to existing issues

• Nightly Examples (macOS): node fails to exit after dataflow stop — daemon spins on TCP read timeouts until step timeout #2152 describes nodes failing to exit after a clean dataflow stop with the daemon spinning on TCP read timeouts — plausibly the same underlying no-exit-on-dead-peer loop, but the trigger here is the runtime being killed, not a stop.
• Daemon does not exit when its coordinator is permanently gone (lingering daemon processes) #1996 (closed) covered lingering daemons when the coordinator is gone; this is the node-level analogue.

Workaround we use

We launch dora run with cwd set to a per-run directory and, after every run, kill any process whose working directory is that run dir. It works for supervised runs but misses anything killed before the reaper fires, which is how the 59-CPU-hour orphan happened.

Happy to provide a minimal reproduction dataflow or logs if useful.

[heyong4725]

Environment correction (important): the setup is a version-mismatched pair, which dora itself warns about:

• CLI/daemon: dora-cli 0.5.0 (PyPI wheel via uv tool install dora-rs-cli) — the side that spawns nodes and misses them on kill
• Python node API: dora-rs 1.0.0rc1, editable install built from a local source checkout (branch state near current main, commit d0cc9b3c7) — the side whose event loop busy-spins after the runtime dies
• dora --version prints: WARNING: Version mismatch detected! CLI version (0.5.0) differs from Python dora-rs version (1.0.0rc1)

So the orphan/busy-spin behavior reported above was observed under 0.5.0-CLI + 1.0.0rc1-node-API. We will re-test with matched versions (both 0.5.0, and both built from main) and report back whether the behavior reproduces in either matched configuration — treat the report as unconfirmed for matched-version setups until then.

[heyong4725]

Retested with matched CLI and Python node API built from latest main (7eb4a5f). I launched dora run in a new session and SIGKILLed its process group after the node was ready. Although the node had a separate PGID and was not directly signaled, both idiomatic Python iterator nodes and Rust nodes exited through the daemon-disconnect path and were gone before the 10-second check.

So the orphan/busy-spin behavior reported here does not reproduce with matched latest-main components. The original observation used a mismatched 0.5.0 CLI/daemon and 1.0.0rc1 Python API. Note that shutdown remains cooperative through the daemon connection rather than being enforced by a parent-death mechanism; guaranteed containment for nodes blocked outside the event loop could be tracked separately if needed.

[heyong4725]

Matched-version results as promised — two corrections and a retest:

Final environment correction: the mismatch I reported earlier turned out to be a red herring — the 1.0.0rc1 python API lived in an unused conda env that dora --version probed as the ambient python. Everything in the original report actually ran matched dora-cli 0.5.0 + dora-rs 0.5.0 (PyPI wheels). So the orphan + busy-spin behavior is confirmed for matched 0.5.0.

Matched current-main retest (both halves built from rev 7eb4a5f8 — dora-cli 1.0.0-rc.4 via cargo install --git, dora-rs 1.0.0rc4 via uv git source; macOS arm64, Python 3.13):

• Two-node timer dataflow, dora run dataflow.yaml --uv, kill -9 the dora run process after 30 s.
• Nodes still survive: both node processes alive 10 s and 30 s after the kill (never exit on their own; manually reaped).
• But the busy-spin is gone: survivors idle at 0.0 % CPU, vs 100–170 % sustained under 0.5.0.

Summary: on current main the orphan leak reproduces (nodes are not reaped when the runtime dies hard) but the pathological CPU spin appears fixed. Happy to run any variant that would help (longer observation, SIGTERM vs SIGKILL, daemon-alive cases).