Memory-pool ids collide across node restarts: a restarted node cannot re-register its pool

[phil-opp]

Summary

Memory-pool shared-memory names are derived from a process-local counter, so a
node that crashes and restarts regenerates the exact same name as its previous
incarnation. If the old pool has not been reclaimed by then, the restarted node
cannot register a pool at all — ShmemConf::create() fails on the leftover segment,
and the daemon would independently reject the registration as a duplicate.

Split out of #2881, which fixed when pools are reclaimed. This is the separate
question of pool ids not being unique across incarnations.

Evidence

The counter is a process-global static seeded at 0
(apis/python/node/src/lib.rs:137-138):

/// Counter to make pinned memory buffer IDs unique across registrations.
static PINNED_COUNTER: LazyLock<std::sync::Mutex<u64>> = LazyLock::new(|| std::sync::Mutex::new(0));

It is unique across registrations within one process, but a fresh process always
starts at 1. The segment name mixes it with two values that do not change across a
restart (apis/python/node/src/lib.rs:1965-1968):

let shmem_name = format!("dora_pool_{}_{}_{}", self.dataflow_id, self.node_id, pool_counter);

So incarnation 2 of a node re-derives dora_pool_{dataflow}_{node}_1 and calls
ShmemConf::new().os_id(&shmem_name)…create(). That fails while the old segment
exists — confirmed against shared_memory_extended 0.13.0:

first create: Ok
second create: ERR <-- collision: Shared memory OS specific ID already exists

which is exactly what the call site's own error message anticipates: "name collision
with another node or leftover segment". Independently, the daemon would reject the
registration too — MemoryPoolManager::register_memory_pool returns
"Memory pool with ID {} already registered" for the still-present entry.

Impact

A node using the memory-pool transport under restart_policy: Always fails on its
first registration after any crash, for the remaining life of the dataflow. Since
register_memory_pool raises, the restarted node typically dies immediately — a
crash-restart loop that never recovers.

Why #2881 does not cover it

After #2881, a pool is reclaimed on node exit once no live node can still reach it,
which frees both the table entry and the /dev/shm segment — so the restart succeeds
whenever the crashed node's consumers are gone too. The collision remains in the
common case: sender crashes while its receiver keeps running. The pool is then
deliberately retained (the receiver may still read it), and the restarted sender walks
straight into its own leftover name.

Reclaiming it eagerly is not an option, for the reason established in #2881 — it would
cut off the receiver's in-flight transfer.

Suggested direction

Make the name unique per incarnation rather than per registration. The cheapest form
that preserves the on-wire id format is to seed PINNED_COUNTER with a random u64
instead of 0: the component stays a u64, so both existing parsers keep working —
the writer fast path (buffer_id.rsplit_once('_') then parse::<u64>(),
apis/python/node/src/lib.rs:2459-2462) and try_doradma_read, which reconstructs
dora_pool_{dataflow_id}_{node_id}_{counter} from pool_{node_id}_{counter}
(apis/python/node/src/lib.rs:3481-3512). Any scheme that adds a new component to
the name would need both parsers updated.

An alternative — letting a re-registration by the same node id take over a stale entry
whose registrar has exited — fixes the daemon table but not the /dev/shm collision,
which happens first, node-side. It would also weaken the duplicate-registration check
for the Rust API, where the caller supplies shared_memory_id directly.

Happy to prepare a PR if the random-seed direction looks right.

[phil-opp]

Note for whoever picks up the follow-up (raised in review of #3056): the random-seed fix removes the name collision, but as a consequence nothing reclaims a dead incarnation's pool in the pathological case. Under restart_policy: Always, a node that crashes repeatedly while its receiver never calls free_memory_pool now registers a fresh id each restart and leaves the previous entry + its /dev/shm segment (data region up to 1 GiB) behind. MemoryPoolManager::cleanup_orphans runs only at dataflow spawn and the daemon registry is capped (512), so a long enough crash loop can exhaust it and then fail unrelated dataflows' pool registrations on that daemon. Pre-fix this loop failed fast on the first restart instead.

The complete fix is an owner-death pool reclaim (reclaim a pool once its registering node has exited and no live consumer can still reach it — the same reachability rule #2881 established, just triggered on registrar death rather than only at spawn). Tracking it here so #3056 can land as the narrow "unique id across restarts" fix.

---

Generated by Claude Code

[phil-opp]

Removed from the 1.0 milestone: the memory-pool transport is being parked out of the tree in #3152, so this no longer blocks 1.0.

The code is preserved under external/dora-pool, staged for lifting into its own repository. This bug is recorded in its README as a re-entry criterion — it must be fixed before the transport comes back. Not closing it: the defect is real and still unfixed.

[phil-opp]

Correction to my earlier note on this issue. The memory-pool transport is not being moved out of the repository after all.

It now lives at libraries/extensions/memory-pool as an opt-in feature (--features memory-pool, off by default) that is explicitly outside the 1.0 compatibility guarantees — see #3152. It reaches dora only through a new generic extension channel, so none of it is in dora's stable surface.

This stays off the 1.0 milestone for that reason, and stays open: the defect is real and unfixed. It is named in the extension's README as a known open defect.