zenoh per-node listener bind race (#2716) reintroduces the #1858 silent-partition class on a path the #1858 fix does not cover

[phil-opp]

Automated review notice: This issue was created by a scheduled automated Claude review of recent commits/PRs. It is a code-path analysis (not observed at runtime), so please sanity-check the reasoning before acting. Probability is low; impact (silent data loss) is high.

Summary

Closed issue #1858 ("zenoh listener bind race: open Ok ≠ actually bound") documented a silent-partition race for the daemon's zenoh listener and was fixed by verifying the bind post-open via session.info().locators() and only advertising an endpoint that actually bound.

Commit fd8af08 (PR #2716, "restore reliable node links … under zenoh 1.9", merged 2026-07-16) added a new per-node listener mechanism (plan_zenoh_peering + DORA_ZENOH_LISTEN/DORA_ZENOH_CONNECT) that reintroduces the same failure class on a path the #1858 fix structurally cannot cover. The daemon reserves each local node's loopback port and commits it into that node's consumers' dial list at plan time — before the producing node process has bound the port — and the node then discards its own bind-verification result. If the reserved port is lost in the reserve→bind window, the producer runs listener-less, its consumers hold a hardcoded dead endpoint, and multicast fallback has been explicitly disabled for them, so there is no recovery.

Why #1858's fix does not protect this path

The #1858 fix works because the daemon does bind-then-advertise: open_zenoh_session_with_listen opens the session, confirms the requested endpoint is in info().locators(), and only then returns effective_listen_endpoint = Some(ep) for the daemon to inject (libraries/core/src/topics.rs ~330). The per-node path from #2716 inverts this to advertise-then-bind:

1. plan_zenoh_peering reserves each local node's port with reserve_loopback_zenoh_endpoint() (binds 127.0.0.1:0, reads the port, drops the socket immediately — libraries/core/src/topics.rs:401) and records it in binaries/daemon/src/spawn/spawner.rs:163.
2. The plan is committed into dataflow.zenoh_peering and each consumer's DORA_ZENOH_CONNECT list already contains the producer's tcp/127.0.0.1:P (maybe_inject_zenoh_connect, spawner.rs ~258) — this happens as the nodes are spawned, i.e. before the producer binds P.
3. The producer node opens its session via open_zenoh_session(None) → open_zenoh_session_with_listen(None, None, None), which picks up DORA_ZENOH_LISTEN from the env, inserts listen/endpoints: tcp/127.0.0.1:P, and sets listen/exit_on_failure: false (topics.rs ~228–260). A failed bind therefore does not fail zenoh::open.
4. open_zenoh_session discards the returned effective_listen_endpoint (let (session, _) = …, topics.rs:46), so even the node's own knowledge that its listener didn't bind is thrown away and never propagated back to the daemon or the consumers.

There is no feedback loop from "did the node actually bind its reserved port?" back to the consumers that were already told to dial it. #1858's verification guards only the daemon's own listener, not these per-node ones.

The race (silent partition)

1. reserve_loopback_zenoh_endpoint() reserves 127.0.0.1:P, drops the socket.
2. Tiny window: another process grabs P.
3. Producer node inserts listen/endpoints: tcp/127.0.0.1:P, zenoh::open returns Ok anyway (exit_on_failure: false), bind silently failed. The node logs "zenoh session opened but listener for tcp/127.0.0.1:P did not bind … spawned nodes will use multicast scouting only" and keeps running (topics.rs ~336).
4. Consumers dial tcp/127.0.0.1:P from DORA_ZENOH_CONNECT → connection refused (or a connection to the unrelated thief process). Because those consumers set connect/endpoints, they also set scouting/multicast/enabled: false (topics.rs ~197), so there is no multicast fallback. The producer, listener-less, cannot be dialed either.
5. That producer→consumer edge is permanently, silently dead — the exact "large-payload data path partitioned" outcome zenoh listener bind race: open Ok ≠ actually bound (follow-up to #1855 / #1856) #1858 describes.

Under #2716 this reserve-and-drop pattern is now applied to every local node (|nodes| reservations per dataflow instead of one daemon reservation), widening the aggregate exposure, and on a path with no fallback rather than the daemon path's verify-and-fall-back-to-multicast.

Secondary: the diagnostic is actively misleading

The warning at topics.rs ~336 says "spawned nodes will use multicast scouting only", but for a node that has DORA_ZENOH_CONNECT set, multicast scouting was just disabled (topics.rs ~197). So when this race does fire, the log points a debugger in the wrong direction.

Affected code paths

• libraries/core/src/topics.rs:401 — reserve_zenoh_endpoint / reserve_loopback_zenoh_endpoint: bind-read-drop reservation (unheld).
• binaries/daemon/src/spawn/spawner.rs:163 — per-node reservation in plan_zenoh_peering.
• binaries/daemon/src/spawn/spawner.rs ~258 — maybe_inject_zenoh_connect: injects the producer's reserved port into consumers before bind.
• binaries/daemon/src/lib.rs (spawn path) — dataflow.zenoh_peering = Arc::new(plan_zenoh_peering(...)) computed before nodes spawn.
• libraries/core/src/topics.rs:46 — open_zenoh_session discards effective_listen_endpoint.
• libraries/core/src/topics.rs ~228–260 / ~336 — node picks up DORA_ZENOH_LISTEN, sets exit_on_failure: false, only warns on bind failure.

Suggested fixes (increasing effort)

1. Hold the reservation until the node binds. Have plan_zenoh_peering hold each TcpListener (return a guard from reserve_*) and pass the fd/socket to the child, or at least keep it bound until the child process is confirmed up, collapsing the window to OS port-handoff. This is fix strategy (2) from zenoh listener bind race: open Ok ≠ actually bound (follow-up to #1855 / #1856) #1858 applied to the node path.
2. Verify per-node bind and re-plan on failure. Propagate each node's effective_listen_endpoint (stop discarding it in open_zenoh_session) back to the daemon; if a node's listener didn't bind, either restart it with a fresh reserved port and update consumers, or keep multicast enabled for that node so a fallback path exists.
3. At minimum, fix the misleading "multicast scouting only" warning for the connect-configured node case, and document on plan_zenoh_peering that the per-node endpoints are advertised before bind and are not covered by the zenoh listener bind race: open Ok ≠ actually bound (follow-up to #1855 / #1856) #1858 verification — so future readers don't assume they are.

References

• Introduced by: commit fd8af08 / PR fix(zenoh): restore reliable node links and large cross-host messages under zenoh 1.9 #2716 (2026-07-16).
• Previously-fixed sibling: issue zenoh listener bind race: open Ok ≠ actually bound (follow-up to #1855 / #1856) #1858 (closed completed) — same race class, daemon listener only.
• Related: PR fix(daemon): bind zenoh listener on a routable address for multi-machine setups #2722 (--zenoh-listen, daemon bind verification) shares the reserve_zenoh_endpoint helper.

[phil-opp]

Triaging this as part of an automated pass over open issues.

The reasoning holds up as a code-path analysis: the per-node listener path from #2716 does advertise-then-bind (reserve_loopback_zenoh_endpoint() binds 127.0.0.1:0, reads the port, and drops the socket at libraries/core/src/topics.rs:401; the consumer's DORA_ZENOH_CONNECT is committed before the producer binds; open_zenoh_session discards the effective_listen_endpoint; and scouting/multicast/enabled is disabled for those consumers), so #1858's bind-then-advertise guard structurally cannot cover it.

I'm not opening a fix PR here, for two reasons:

1. It needs an architectural decision, not a local patch. The reserve→bind gap spans a process boundary (the daemon reserves the port; the node binds it later), so the natural fixes each have real trade-offs a maintainer should choose between: keep the reserved socket alive and hand the bound listener to the node (changes the spawn handshake); have the node verify its own bind via session.info().locators() and report failure back so consumers re-enable multicast or re-dial (adds a feedback channel that doesn't exist today); or retry-with-fresh-port on bind failure. This is the kind of silent-data-loss path where a wrong fix is worse than the documented low-probability race.

2. It overlaps with in-flight zenoh work — fix(node): prove zenoh startup routes with a marker/ack handshake, switching paths producer-side #2806 (zenoh startup route markers), fix(core): fall back to non-multicast zenoh discovery when scouting bind fails #2850 (non-multicast fallback), and the broader topology rework tracked in zenoh 1.9: dora is one flat peer region — remaining topology gaps after #2716 #2721 — which may change or subsume this path.

Flagging for maintainer triage. Once the intended approach is chosen I'm happy to implement it (including a bind-verification feedback loop mirroring #1858 on the per-node path). The issue notes probability is low; impact (silent partition of a producer→consumer edge) is high, so it seems worth keeping open until the #2716 peering path settles.

---

Generated by Claude Code