Relay-mesh agents silently stop replying if mesh runtime drops after launch (no self-heal)

[micspiral]

Summary

Relay-mesh (Buzz shared compute) agents silently stop replying if the mesh runtime drops at any point after app launch. No error surfaces to the user and nothing re-arms the runtime until an agent is manually (re)started.

Repro

1. Launch desktop with a relay-mesh agent; mesh runtime comes up, ingress on 127.0.0.1:9337, bot replies fine.
2. Leave the app running for an extended period (overnight), or otherwise have the embedded mesh runtime exit/wedge.
3. Send the bot a message → no reply, no error.
4. Manually start/create an agent → ensure_relay_mesh_for_record runs again, re-arms 9337, replies resume.

Observed live: 9337/3131 NOT listening while goose acp agent subprocesses were still running and healthy; curl 127.0.0.1:9337/v1/models → connection refused. A remote peer (M4) was serving the model the whole time, so compute availability was never the issue.

Root cause

ensure_relay_mesh_for_record (which brings up the local mesh client node / OpenAI ingress) is only invoked:

• at launch — managed_agents/restore.rs (restore_managed_agents_on_launch)
• on agent create/start — commands/agents.rs (start_local_agent_with_preflight)

There is no periodic re-arm and no liveness check tying the mesh runtime to agent message dispatch. If state.mesh_llm_runtime becomes None (runtime exited, or a bad restart left a zombie — cf. "timed out after 5s waiting for embedded mesh runtime thread to exit"), relay-mesh agents keep running but point at a dead ingress. The failure is invisible.

Impact

Any long-lived desktop session using shared-compute agents will eventually see bots go quiet with no indication why. This is the dominant "shared compute is broken" user-facing symptom even though discovery, transport, and remote serving are all healthy.

Suggested fix (options)

1. On-demand re-arm: before dispatching a message to a relay-mesh agent (or in the agent reconcile loop), if mesh_llm_runtime.is_none() or the ingress is unreachable, call ensure_relay_mesh_for_record / ensure_client_node_for_model to bring it back.
2. Liveness + surfaced error: health-check the runtime periodically; if it's down and can't be restored (e.g. no live serving member), surface a visible "shared compute offline" state on the agent instead of silent non-reply.
3. Zombie guard: the mesh runtime thread to exit 5s timeout can leave a half-dead runtime — ensure a failed restart doesn't leave mesh_llm_runtime in a state that blocks re-arm.

Fix (1) alone resolves the reported symptom; (2) makes it debuggable; (3) prevents the wedge that motivated it.

Not this

• Not a transport/discovery bug (those were fixed in fix(desktop): keep mesh allowlist on transient roster query failure #2024, present in main).
• Not remote-serving availability (peer was serving throughout).

[Orland00]

I'd like to contribute the fix for this and want maintainer input on the design before writing code, since it changes a desktop daemon's runtime behavior.

Root cause, verified on current main: ensure_relay_mesh_for_record (desktop/src-tauri/src/commands/mesh_llm.rs) trusts mesh_llm_runtime.is_some() — if the slot holds a wedged runtime whose 127.0.0.1:9337 ingress is dead, it never evicts/rebuilds; it just falls into wait_for_mesh_inference and times out. And its only two callers are launch (restore_managed_agents_on_launch) and agent create/start — nothing on the dispatch path, no periodic check. So a runtime that dies after launch stays dead until a manual agent (re)start, exactly as reported. Meanwhile the machine keeps serving/reserving compute for nobody.

Proposed fix — a mesh ingress liveness watchdog, following the existing coordinator pattern:
start_coordinator (mesh_llm/coordinator.rs) already runs two periodic loops (status_publisher ~45s, roster_watcher ~60s). Add a third:

1. Every 30–60s, if mesh_llm_runtime.is_some(), probe the ingress (cheap GET /v1/models on 9337, or wait_for_mesh_inference with a short deadline).
2. On N consecutive failures: evict the zombie (take() + stop(), same pattern reconcile_roster already uses, including its anti-flapping hysteresis) and re-run ensure_relay_mesh_for_record for each running relay-mesh agent (load_managed_agents filtered by relay_mesh_model_id).
3. If no live serve target can be resolved, surface a visible "shared compute offline" state via the existing persist_restore_error path instead of failing silently.
   Complementary one-line hardening: inside ensure_relay_mesh_for_record itself, treat "runtime present but probe fails" as evict-and-rebuild rather than wait-and-error — that fixes the zombie case even on the explicit-start path, independent of the watchdog.

Questions before I write it:

• Preferred probe interval / consecutive-failure threshold?
• Should auto-rebuild be bounded (max attempts + backoff) or indefinite?
• Third loop in start_coordinator, or would you rather fold the probe into roster_watcher?

All behind #[cfg(feature = "mesh-llm")], eviction/hysteresis logic extracted into pure testable functions, manual repro (kill the runtime process, observe recovery) included in the PR.

— Orlando (Olive One)

[Bartok9]

Took a focused pass at Option 1 + the zombie guard.

Root cause confirmed as described: ensure_relay_mesh_for_record takes the "already running" fast path whenever mesh_llm_runtime.is_some(), then wait_for_mesh_inference just times out against the dead :9337 ingress — so the agent stays silent.

Fix: added a fast liveness probe (mesh_ingress_is_live — single 3s GET /v1/models, the exact call this issue used to confirm the ingress was down). When a runtime handle exists but the ingress is unreachable, drop the stale runtime (best-effort stop — never block re-arm on a wedged runtime, per your zombie-guard note) and fall through to the normal re-arm/bootstrap path. Healthy fast path (live ingress) is unchanged.

This resolves the reported symptom on the next dispatch instead of requiring a manual restart. I scoped it to the on-demand re-arm; the periodic liveness/health-surfacing (Option 2) can layer on top separately. PR incoming.

[micspiral]

could this also be related? #2448 ? thanks