fix(daemon): reclaim memory pools once no live node can reach them

[phil-opp]

Fixes #2881.

Pools registered by a node stayed in the daemon's table until the daemon itself
exited: neither RemoveNode nor a node crash released them, and finish_dataflow
did not either — so a long-running daemon (dora up) accumulated table entries and
/dev/shm segments across dataflows, walking toward the 512-entry pool cap.

Eagerly freeing a node's pools when it goes away would be wrong, as @SaitejaKommi
pointed out on the issue: a pool deliberately outlives its registrar. The normal
lifecycle is that a sender registers a pool and a receiver reads and frees it,
possibly well after the sender exited.

So reclaim by reachability instead. Each entry records who can still reach it:

• touched_by — nodes that have opened the pool, and
• potential_readers — the registrar's transitive downstream consumers as of
  registration time, which can still learn the pool id from a message already in
  flight (the daemon cannot see pool ids inside payloads, so the whole downstream
  closure counts).

A pool is released once none of those is still running locally. Every path that ends
an incarnation now checks: RemoveNode, ReplaceNode (whose outgoing exit event the
generation guard drops), and a crash, clean exit or restart. finish_dataflow
releases the remainder unconditionally — a finished dataflow has no node left to
serve, and its routing, channels and listeners are discarded in the same breath.

Nodes on other daemons never count as live: pools are host-local shared memory and
each daemon keeps its own table, so a remote consumer can never open this one's
segments.

Because a reclaimed pool has no live node in touched_by, there is nobody to send a
FreeMemoryPool notification to — unlike free_memory_pool, which must tell the
other holders to drop their per-process buffers.

finish_dataflow also sweeps the /dev/shm segments no table entry covers — a node
that died between creating a segment and registering it. That sweep is now scoped to
the nodes of this daemon, which it was not before: /dev/shm is host-wide, the
segment name embeds only the dataflow id, and finish_dataflow is a per-daemon
finish. With two daemons serving one dataflow on one machine (a machine: split),
whichever finished first would have unlinked the other's live segments, leaving a
consumer over there with ENOENT. RunningDataflow tracks its local nodes for this,
and the pre-existing spawn-time sweep — same hazard, narrower window — is scoped the
same way.

register_memory_pool gains a potential_readers parameter, MemoryPoolEntry a
field, and cleanup_orphans an is_local_node predicate, so this is a breaking
change to the dora-memory-pool crate surface. Its only consumer is dora-daemon.

Reachability cannot be a registration-time snapshot, because a running dataflow can
be rewired. dora graph connect S/out C/in after S registered a pool gives C a
path to it, and S is in its own reader set, so reclamation on S's exit would have
unlinked the segment while C still had the id in flight. AddMapping and AddNode
therefore widen the reader set of every pool the edge's source can reach, by the new
target's downstream closure. Recomputing the whole set at reclaim time is not an
option — RemoveNode tears the node's mappings down first — but widening on the two
paths that create edges covers the same ground.

The closure also follows edges whose receiver is on another daemon. mappings holds
only what this daemon delivers, so a local -> remote -> local chain used to stop at
the first hop and drop the local node at its end, even though a pool id can travel
the whole way. A remote node never counts as live, so following through one costs
nothing.

Tests: reachability rules and dataflow-scoped cleanup in dora-memory-pool; the
downstream closure (transitive, cycle-safe) on RunningDataflow; and the daemon
wiring — a pool survives an exited sender while its consumer runs, and is reclaimed
once the last consumer is gone, when the exiting node is its only reference, and
regardless of unrelated long-lived nodes. The finish path is covered too: it releases
a pool a live consumer could still reach, and its /dev/shm sweep unlinks this
daemon's orphan while leaving a co-located daemon's segment alone. Plus the two
reachability gaps: a consumer connected after registration keeps the pool alive until
it too is gone, and the closure reaches a local node whose only path runs through
another daemon.

[trunk-io]

😎 Merged successfully - details.

[phil-opp]

🤖 Automated review by Claude — fully automated review, no human in the loop; please verify before acting.

I reviewed the current diff. No issues found.

What I traced:

• Reachability is touched_by ∪ potential_readers (libraries/extensions/memory-pool/src/lib.rs:64-70), and potential_readers is the full transitive downstream closure (binaries/daemon/src/running_dataflow.rs:1021-1039), so a pool id forwarded through any chain of edges keeps the pool alive — the closure follows every queued node's outputs, not just the registrar's direct consumers, and the cycle guard via reachable.insert terminates.
• The exited node is correctly excluded from the live set in reclaim_memory_pools_after_exit (binaries/daemon/src/lib.rs:658-662), which matters because the SpawnedNodeResult path runs before the node leaves running_nodes; otherwise a consumer-less sender could never be reclaimed.
• release_matching drops the table lock before the remove_file unlinks, matching the existing free_memory_pool locking discipline, and skipping FreeMemoryPool notifications is justified — reclamation only fires when no live node remains in either set, so there is nobody to notify.
• Local-only liveness is correct: pools are host-local shared memory, so a remote consumer in the closure never pins one, and unlinking a segment a live node already mapped is safe (existing mappings survive shm_unlink).
• finish_dataflow's unconditional cleanup_dataflow + cleanup_orphans is safe for dynamic nodes that outlive the finish, since their routing/channels are already torn down.
• The tests exercise the actual change, including the leak scenario (unrelated node running), the in-flight-transfer scenario (consumer still running), dataflow scoping, and the late-reader-kept-via-touched_by case.

---

Generated by Claude Code

[github-actions]

@phil-opp the Trunk merge queue failed for this PR.

See the Trunk merge-status comment for details.

Posted as a new comment so GitHub sends an email — Trunk's sticky comment is edited in place and won't trigger a notification.

[github-actions]

@phil-opp the Trunk merge queue failed for this PR.

See the Trunk merge-status comment for details.

Posted as a new comment so GitHub sends an email — Trunk's sticky comment is edited in place and won't trigger a notification.

[phil-opp]

Rebased onto main and addressed all three points.

The finish-time sweep crossing daemon boundaries. Correct, and the spawn-time
sweep had the same defect with a narrower window. cleanup_orphans now takes an
is_local_node predicate and attributes each /dev/shm file to a node before
unlinking it — the segment name's last component is always the counter, so the node
id splits off unambiguously even when it contains _. A name that does not fit the
pattern is left alone: leaving an orphan is recoverable, unlinking a peer's live
segment is not. RunningDataflow tracks the nodes this daemon owns (spawn_nodes
plus anything AddNode adds later) to scope both call sites. The false justification
on cleanup_orphans is gone.

The "known limitation" not covering its own case. Also correct, and it was a
regression rather than a limitation, so it is fixed rather than documented.
AddMapping and AddNode — the only two paths that add an edge to a running
dataflow; ReplaceNode rejects any input change — now widen the reader set of every
pool the edge's source can reach by the new target's downstream closure. Recomputing
the whole set at reclaim time is indeed impossible, but updating where edges are
created covers the same ground.

While in there: downstream_closure walked mappings, which holds only edges this
daemon delivers, so a local -> remote -> local chain stopped at the first hop and
dropped the local node at its end — even though a pool id travels the whole chain.
Remote edges are recorded at spawn and followed now. A remote node never counts as
live, so following through one only widens what the closure reaches.

Test coverage of the finish path. The finish-time release is a free function now,
so it can be exercised without a whole Daemon: one test that it drops the table
entries a finished dataflow still holds, one that its sweep spares a co-located
daemon's segments. Plus the late-connected consumer (both halves: pinned while it
runs, reclaimed once it is gone), the reader-set widening not leaking into unrelated
pools, the segment-name parsing edge cases, and the closure's remote hop.

Two residual gaps, both erring toward retention rather than premature release:

• Runtime rewiring only reaches the target node's daemon, since that is where the
  coordinator routes AddMapping. A local -> remote -> local chain built at
  runtime through two dora graph connect calls therefore still misses the source
  daemon's reader sets.
• potential_readers only grows: RemoveMapping drops the edge but nothing removes
  the node, so repeated connect/disconnect cycles widen every pool's reader set and
  decay reclamation back toward teardown-only.

[phil-opp]

🤖 Automated review by Claude — fully automated, no human reviewed this; please verify before acting.

The two commits since my last review address both issues raised on 2026-08-11, and I traced them against the diff:

• Cross-daemon finish/spawn sweep — cleanup_orphans now takes an is_local_node predicate and attributes each /dev/shm/dora_pool_* segment to a node (rsplit_once('_') + counter.parse::<u64>() guard) before unlinking, skipping any name that doesn't fit the pattern. Both call sites pass a per-daemon owner set, so a co-located daemon's live segments are no longer destroyed by a peer's finish.
• Rewiring regression — AddMapping/AddNode now call extend_pool_readers_for_new_edge after recording the edge, widening every reachable pool's readers by the target's downstream_closure; and downstream_closure now follows remote_edges, so a local -> remote -> local chain keeps the trailing local reader. A consumer connected after registration now pins the pool it can reach.

No new issues. The two residual gaps you documented (runtime local -> remote -> local rewiring reaching only the target daemon; potential_readers growing monotonically across connect/disconnect cycles) both err toward retention rather than premature free, so they're safe as follow-ups.

---

Generated by Claude Code