refactor(tensor-pool): move behind a generic extension seam, opt-in and outside the 1.0 guarantees

[phil-opp]

Moves the pinned/CUDA tensor-pool transport (#2168, #2386, #2619) out of dora's core and behind a generic extension seam. It stays in the tree and stays usable — as an opt-in feature, explicitly outside the 1.0 compatibility guarantees.

Three commits, each independently reviewable:

1. Remove the transport from dora's core — 3,048 lines from the Python binding, 973 from the daemon, five wire-protocol variants.
2. Add a generic extension channel — the one thing an out-of-core transport genuinely needs from the daemon.
3. Reinstate the transport at libraries/extensions/tensor-pool, on that channel, behind --features tensor-pool.

Why

#1872 — the design issue for this feature — explicitly declined to commit to an architecture:

This issue documents the gap and invites proposals. It does NOT commit to an architecture. dora has historically moved AWAY from custom shared-memory infrastructure (see #1745…). Any new transport has to clear a high bar.

Priority: not currently scheduled. Filed for visibility.

It was filed 2026-05-19, the same day #1623 (+2,717 lines) was closed for being "7× larger than the feature needed", and listed five questions a proposal had to answer before writing code. #2168 landed the same architecture 24 days later, merged 5 days after opening, answering none of them. Question 3 — "who owns pinned memory, and when is it freed? what if the producer crashes mid-write?" — is the direct ancestor of #2881, #2935 and #3015.

The problem was never the feature. It was that the feature had grown into dora's core: ~3,000 lines inside apis/python/node/src/lib.rs with 64 unsafe sites (~40 hand-parsing a 256-byte binary header at hardcoded offsets, out of memory another process writes), 950 lines of daemon lifecycle logic, pool-specific variants frozen into the wire protocol — and no CI able to exercise any of its GPU paths.

So: keep the feature, remove it from the core, and make the boundary explicit.

1. The core is clean again

apis/python/node/src/lib.rs	−3,048 lines (4,064 → 1,016); unsafe count 64 → 0
binaries/daemon/src/lib.rs	−973 lines: handlers, reclamation, orphan sweep
Wire protocol	3 DaemonRequest + 2 DaemonReply pool variants removed
Release workflows	dora-memory-pool dropped from cargo-release.yml / release.yml, which would otherwise publish a crate that no longer exists

grep -ri 'cuda\|doradma\|pinned\|seqlock' over dora's core now returns nothing.

2. The seam

A transport outside the core still needs one thing only the daemon can provide: reclamation after a crash. A node that dies cannot withdraw the descriptor it published, so its readers keep mappings to memory nobody owns. That is #2881, and send_output cannot fix it.

So dora gained a dataflow-scoped table of opaque byte values whose lifetime the daemon brokers — and nothing else:

node.extension_store(namespace, key, value)          # bytes in
node.extension_load(namespace, key, remove=False)    # bytes out, or None
node.extension_drop(namespace, key)                  # withdraw + notify
node.drain_dropped_extension_keys(namespace)         # what went away

Same four on DoraNode for Rust nodes. Guarantees and limits: docs/extensions.md.

Deliberately generic — the variants are ExtensionStore / ExtensionLoad / ExtensionDrop, and adding a second extension needs no change to dora at all. A pool-shaped accessor would have frozen this transport's architecture into the framework, which is exactly what #1872 declined to do.

17 tests: ownership (a foreign node cannot hijack a key), namespace and dataflow scoping, the reclamation asymmetry (an owner's exit reclaims, a reader's must not), idempotent drop, per-dataflow cap.

3. The transport, opt-in

Lives at libraries/extensions/tensor-pool — beside ros2-bridge, which has the same python/ subcrate shape.

maturin develop -m apis/python/node/Cargo.toml --features tensor-pool
cargo build -p dora-daemon --features tensor-pool   # orphaned-segment reclamation

Both flags are off by default and independent. The default build neither compiles nor exposes any of it: process_pending_tensor_pool_frees becomes an empty no-op, and default cargo check on the daemon and Python crate is warning-clean.

register_tensor_pool and friends are still methods on Node, #[cfg(feature)]-gated, delegating to a Pool context struct (PyO3 cannot let another crate add #[pymethods]).

Renamed from memory-pool. dora already has an unrelated shared_memory_pool_size descriptor key (and DORA_NODE_SHM_POOL_SIZE) for the Zenoh SHM buffer pool, and "memory pool" for both was a persistent source of confusion. tensor-pool says what this one actually pools and stays accurate on the CPU path, which gpu- would not — the CPU path works without CUDA and is the only one with CI coverage.

This renames the four Python methods (register_memory_pool → register_tensor_pool, and likewise for write/read/free), the crates, the feature flags and the example env keys. That is a user-visible break for anyone on the current methods — permissible precisely because this feature sits outside the 1.0 guarantees, and better done now than after more users arrive.

Its six former daemon calls now go through the seam:

Before	Now
register_pinned_memory(id, meta)	extension_store("dora-tensor-pool", id, bytes)
read_pinned_memory(id, free) ×3	extension_load("dora-tensor-pool", id, remove=free)
free_pinned_memory(id)	extension_drop("dora-tensor-pool", id)
drain_freed_pools()	drain_dropped_extension_keys("dora-tensor-pool")

The descriptor is JSON that dora never parses. That boundary is what keeps the unsafe pointer arithmetic, the seqlock and the embedded libcudart bindings on the extension's side.

Guarantee status is stated where it gets read: the README leads with it, and it is repeated in the crate description, the module header, every #[pymethods] docstring and the .pyi stubs. Named open defects: #3015, #2935, #2890. (#2881 is fixed.)

Three latent bugs surfaced

Returning the crate to the workspace put it under -D warnings for the first time, which caught: a deprecated downcast_into, a dead initializer in the device-to-host copy path, and seqlock_begin_write with no callers at all — every write path uses begin_if_even. All fixed; the dead function is deleted.

Known gap

The smoke tests are not wired into any automated suite. They need a feature-built wheel plus torch, so as a cargo test target they broke the build. They ship as smoke-tests.rs.example with instructions instead. The extension therefore has 53 unit tests but no in-tree end-to-end coverage — a real regression against the old nightly memory-pool-smoke job, and worth tracking separately.

Validation

cargo fmt --all -- --check, cargo clippy --all -- -D warnings, cargo check --examples — clean, on both the default and --features tensor-pool builds.

cargo test --all: two failures, dora-ros2-bridge --test rmw_zenoh_pubsub::{transient_local_delivers_history_to_late_joiner, two_sessions_preserve_order_payload_and_metadata}, both Elapsed(()) on Zenoh peer discovery. They reproduce identically on an unmodified checkout — the dev container has no multicast — and this branch touches zero lines in that crate. The apis/c++ build-script failure in that environment is likewise pre-existing on main.

[trunk-io]

😎 Merged successfully - details.

[phil-opp]

🤖 Automated review by Claude — fully automated review; no human has verified these findings.

No issues found.

I checked the new generic extension channel and its plumbing (the rest of the diff is code movement into libraries/extensions/memory-pool and the removal of the in-tree pool API):

• binaries/daemon/src/extension_table.rs — store/load/drop/reclaim_owner/reclaim_dataflow, owner-only overwrite, and the per-dataflow cap all look correct and are well covered by unit tests (including the overwrite-at-cap edge case).
• apis/rust/node/src/event_stream/extensions.rs — the bounded, namespace-scoped drop-notification queue is sound.
• libraries/message/src/{node_to_daemon,daemon_to_node}.rs — every match arm (payload_len, wants_reply, expects_tcp_reply, encode_size_hint) is updated consistently for the three new variants.
• Daemon handlers in lib.rs — ExtensionLoad { remove: true } only broadcasts a drop on a hit, ExtensionDrop is idempotent, and inc_pending accounting matches the prior pool path.
• cuda.py — the remaining functions don't reference the removed _DTYPE_MAP/_TORCH_TO_NUMPY_DTYPE_MAP, so no dangling references.

One observation, not a bug: unlike the removed notify_memory_pool_freed (which excluded the initiator), notify_extension_dropped now also delivers ExtensionDropped to the node that triggered the drop / load-with-remove, since it's in touched_by. That's consumed out-of-band by the event-stream thread and left for the extension to drain, and the eager owner-exit reclamation deliberately replaces the old reachability logic (downstream_closure/remote_edges) — both are documented as intentional, with the pool extension marked opt-in and outside the 1.0 guarantees.

---

Generated by Claude Code