zenoh 1.9: dora is one flat peer region — remaining topology gaps after #2716

[phil-opp]

#2715 bundled four defects. #2716 fixes three outright (dead routing/peer insert, lowlatency losing >64 KiB cross-host messages, inverted SHM comment) and the fourth — peers no longer relaying — for the case that actually bites: same-machine links between statically-spawned nodes, now established by construction from the dataflow graph.

Closing #2715 as superseded rather than editing it again: its framing predates the evidence below, and it still carries a mis-sourced quote. This issue tracks only what #2716 leaves open.

Root cause, stated properly this time

#2715 said "zenoh 1.9 changed behavior under us." That's true but understates it. The precise statement, from DEFAULT_CONFIG.json5 @ 1.9.0:

The auto preset mode puts peers and clients south of routers and client south of peers. This is the only available preset.

Dora never sets a session mode anywhere in the repo, so every session — daemon, coordinator, node — is a default peer. No routers, no clients. Under auto that means no gateway boundary materializes anywhere: the whole dataflow is a single flat peer region, and per the deployment docs:

In peer to peer regions, Zenoh nodes must be deployed with peer mode and must also be deployed in a clique topology where every node is connected to all other nodes of the region.

So this isn't tuning we lost — it's a topology model dora no longer fits. linkstate was our exemption from the clique requirement, and 1.9 deleted the exemption without auto offering a replacement. The 5/20 route failures measured in #2716 are the predicted behavior of an incomplete clique, not bad luck.

zenoh's 1.9 migration guidance names our case directly: "Users are expected to migrate non-clique peer-to-peer deployments from relying on router failover brokering to instead use multiple peer subregions in the router."

#2716 satisfies the clique requirement the cheap way — over exactly the |edges| pairs the graph needs, rather than all N². That is the correct fix for local static nodes and should not be replaced by a relay-based shape: any gateway hop would undo the direct zero-copy path from #1787. The two gaps below are the ones that approach structurally cannot reach.

1. daemon↔daemon across networks

inter_daemon_peer is an ordinary peer that won a bind race and serves as a gossip hub. It provides discovery only — and since peers don't relay, discovery isn't connectivity. Two daemons behind NAT can find each other and still exchange nothing. This is exactly the use case the old linkstate comment claimed to serve ("connect two daemons on different network through a public daemon"), and it is the one case that is now entirely unserved.

Not urgent-in-practice: multi-machine users already inject their own config via ZENOH_CONFIG, which bypasses dora's tuning wholesale. But we ship no working default, and the recipe we do ship is now wrong (see below).

Measured against zenoh 1.9.0

NAT simulated by giving leaves no listen endpoints (an unlistening session is undialable — exactly a NAT'd daemon's property). Both leaves dial the hub outbound; multicast off, gossip default. 5 messages, leaf A → leaf B:

hub	leaves	gateway	delivered
peer	peer	none — dora today	0/5
router	peer	none	0/5
peer	client	none	5/5
peer	peer	2 subregions by region_names	5/5
router	peer	2 subregions	5/5
router	peer	1 subregion (both leaves)	0/5

Three conclusions:

1. A router does not fix this. Peers under a router, in one region, relay nothing — because 1.9 removed peer failover brokering, which is what made this work in 1.8. So examples/multiple-daemons/README.md's "run a zenohd and point daemons at it" is wrong for the NAT case it's offered to solve. It still works on a LAN, where the router only supplies discovery and daemons dial each other directly. This doc needs fixing regardless of what else we do.
2. A peer can be a gateway — no router-mode session needed anywhere.
3. The discriminator is the subregion split, not the hub's mode. Same router, same peers: two subregions relay, one does not. This is exactly what zenoh's migration guidance prescribes.

Confirmed end-to-end in dora's real shape (node → daemon → hub → daemon → node, two hops, nodes dialing only their own daemon): all-peers-no-regions delivers 0/5; hub splitting daemons by site + each daemon gatewaying its local nodes delivers 5/5, with nodes staying peers so #2716's dial lists and #1787's zero-copy are untouched.

Config footgun found while testing: insert_json5("gateway/south", ..) is rejected as unknown key — gateway must be set wholesale, as insert_json5("gateway", "{ south: .. }"). Also Config::default() reports gateway = {south: null} and mode = null, where the shipped DEFAULT_CONFIG.json5 says south: "auto" / mode: "peer". We use Config::default(), not the file — the same file-vs-default() divergence that made #2715's SHM comment wrong.

A mesh VPN is probably the better answer

The relay requirement exists only because NAT'd daemons cannot dial each other. Put the machines on a tailnet (Tailscale/WireGuard/Nebula) and they become mutually dialable, so the clique 1.9 requires is achievable and no relay is needed at all. Measured, leaves dialable, 10 rounds each: gossip-only 10/10, explicit mesh 10/10.

That is strictly better than a gateway: direct peer-to-peer data path, no hub hop, no hub as chokepoint or SPOF, and no static subregion enumeration. Caveats: a tailnet has no multicast (so --zenoh-peer or explicit endpoints are still required — we have both); the daemon clique is still quadratic, which large fleets will feel; and gossip convergence is still best-effort, so the #2716 remedy (dial explicitly — we know the addresses from cluster.yml) applies one tier up.

Suggested order:

1. Fix the wrong zenohd advice in examples/multiple-daemons/README.md, and document the mesh-VPN recipe as the recommended cross-network deployment. No code, unblocks users, correct today.
2. Consider explicit daemon↔daemon dialing from cluster.yml — fix(zenoh): restore reliable node links and large cross-host messages under zenoh 1.9 #2716's approach one tier up, removes the gossip race for the VPN case.
3. Regions only if we need to serve fleets too large for a daemon clique, or users who can't run a VPN. Open design problem if so: gateway.south is static at session open and must enumerate one subregion per site, but inter_daemon_peer's hub is whoever-binds-first. Giving every daemon the same south list is tempting but hazardous — two same-LAN daemons would each match the other's south filter and each consider the other south of it. The hub would need explicit designation.

2. Dynamic nodes are still gossip-raced

#2716's peering plan is computed from the spawn set, so dynamic nodes — which join at arbitrary times and aren't in it — get daemon-endpoint-only plus gossip autoconnect. That's the same best-effort race, with the same failure mode: when it loses, delivery for that pair is permanently zero, not degraded.

Masked today, because #2666's readiness barrier keeps the producer on the daemon-forwarded path when the direct route never forms. Silent slow path, and the #1787 optimization never engages.

Regions give the structural answer: put dynamic nodes in a south subregion of their daemon. One link, established by construction — no clique, no gossip, nothing to race.

The cheap version needs no custom gateway config at all: open dynamic-node sessions with mode: "client", since auto already puts "client south of peers," making the daemon their gateway automatically.

Cost is a relay hop — but that is what these nodes already get on the #2666 fallback path, so the trade is an application-level workaround for a transport-level guarantee, and a silent permanent-zero route for a slower-but-correct one. That looks like a clear win for this class of node specifically. Wants measurement against the current fallback before committing.

3. Upstream: put() returns Ok for a frame the peer must reject

No longer affects us — #2716 drops lowlatency. Recording it so it isn't lost: lowlatency/link.rs @ 1.9.0 writes a 4-byte u32 length prefix with no TX-side size check, while the receiver's buffer is bounded by the u16 batch_size and rejects the frame with Batch len is invalid. The publisher believes it succeeded. Should be filed against eclipse-zenoh/zenoh; no dora action.

Minor, separable

coordinator_addr at topics.rs:203-213 does a second insert_json5("connect/endpoints", ...) that replaces the list built ~75 lines earlier, while scouting/multicast/enabled: false — set on the basis of the first insert — stays disabled. That combination yields a session with no path to its peers.

Unreachable today: all three call sites pass None (daemon/src/lib.rs, apis/rust/node/src/node/mod.rs, coordinator/src/ws_control.rs). But it's the only place the router: ["tcp/[::]:7447"] / peer: 5456 endpoints from docs/architecture.md:686 appear in code — i.e. the architecture doc describes a router path that nothing invokes and that would be broken if it did. Delete the branch, or fix it and give it a caller.

Possible follow-up: network_id → region_name

network_id is hardcoded to "default" in topics.rs and documented as isolating separate dora clusters, but it's only a topic-key prefix today. region_name is an optional UTF-8 string of ≤32 bytes. Mapping one onto the other would turn an existing conceptual boundary into a real topology boundary. Speculative; noting it while the context is fresh.

Sourcing note

#2715 got burned by a quote that turned out not to exist, so: every behavioral claim above is measured against the zenoh 1.9.0 crate we actually depend on, not quoted. That includes the two claims #2716 withdrew as unverified — the clique requirement and the removal of peer failover brokering — both of which reproduce (a router hub relays 0/5 for non-clique peers, and 5/5 once they are split into subregions).

Config/source facts were read from raw GitHub and docs.rs at tag/version 1.9.0. The release-blog quotes (migration guidance, gossip note) came via the rendered page; their substance is confirmed by the measurements, but re-check the exact wording before quoting them anywhere.

Versions: zenoh 1.9.0 (Cargo.lock). Supersedes #2715. Follows #2716, #2666, #1787.

[phil-opp]

Triage: this is a design-tracking issue that requires maintainer decisions, so I'm not opening a fix PR for it as a whole. The two remaining gaps (daemon↔daemon across networks, and dynamic-node gossip races) each hinge on an architectural choice the issue itself flags as open — mesh-VPN recipe vs. explicit dial vs. zenoh subregions, and the noted footgun that a shared gateway.south list would make same-LAN daemons treat each other as mutually "south". Those need a human call before implementation.

A few sub-parts look separable and could be split into their own scoped PRs if desired:

1. Doc fix (no code, correct today): examples/multiple-daemons/README.md's "run a zenohd and point daemons at it" advice is wrong for the NAT case (a router relays 0/5 for non-clique peers under zenoh 1.9); the mesh-VPN recipe is the recommended cross-network deployment.
2. Dead branch removal: the second insert_json5("connect/endpoints", ...) in libraries/core/src/topics.rs:203-213 (coordinator_addr path) is unreachable — all three call sites pass None — and would produce a session with no path to its peers if invoked. It's also the only place the router endpoints from docs/architecture.md appear in code.

Happy to take on either (1) or (2) as a standalone PR if a maintainer confirms the intended direction. The larger topology work (subregions / client-mode dynamic nodes) should be scoped as its own design first.

---

Generated by Claude Code

[phil-opp]

Triaging this as part of an automated pass over open issues.

This is a design / architecture tracking issue (the zenoh 1.9 "single flat peer region" model and the topology gaps that remain after #2716), not a discrete, self-contained bug with a clear localized fix. Deciding the intended end-state here — whether dora should introduce an explicit router/gateway boundary, formalize the clique invariant for peer regions, rely on the per-node DORA_ZENOH_LISTEN/DORA_ZENOH_CONNECT planning from #2716, or something else — is a maintainer-level architectural call that shouldn't be pre-empted by an automated patch.

Leaving it for maintainer direction rather than opening a speculative PR. Happy to help implement once the target topology model is decided. Note there is related in-flight work touching the same area (e.g. #2806 zenoh startup route markers, #2850 non-multicast fallback) that likely informs the direction.

---

Generated by Claude Code