perf(agents): instrument end-to-end mention-to-reply latency

[BradGroux]

Summary

Buzz has open work for several reply-reliability failures, but no issue or benchmark currently measures the full user-visible path from a published agent mention to the agent's published reply.

A small live test on 2026-07-22 produced 13–31 second response times for trivial acknowledgements, with a 22.5 second median. The sample is too small to establish a performance distribution, but it is sufficient evidence that we need stage-level timing before we can tell whether delay comes from relay delivery, harness admission/startup, ACP/model execution, or reply publication.

Live evidence

Environment:

• Hosted relay: digitalmeld.communities.buzz.xyz
• Channel: Welcome (aa75944d-9e11-4cb7-adc8-96ebcdb9f967)
• Thread root event: 9ce3aef3c21f558de5f809e45b11fbfd192f1aa26564d039631ed68e10512c0e
• Test date: 2026-07-22 UTC
• Prompts were sequential and explicitly requested only a short presence acknowledgement with no lookup.
• Times below are relay event created_at values, so resolution is one second.

Test	Mention event	Reply event	Mention time (UTC)	Reply time (UTC)	Elapsed
Fizz, first	9ce3aef3c21f558de5f809e45b11fbfd192f1aa26564d039631ed68e10512c0e	18c404f0a4311bd0832f8e0b953379748017dc98a1dd1b667a77569f74f22803	16:11:23	16:11:43	20s
Bumble	25a3a8c3c24e5a8d2928df4ca05e17295bcf1e26810a7ea20bcde6cfbb55b169	661fd0526bb9da1df1ba67d529e500332823be99a2b8d7acc0b2a870668b788b	16:12:24	16:12:55	31s
Honey	369df72e8f409e06cc944b7c917d4695d191db3853d04c1388273af6e39a38a1	24a6d15e54149f04f1c650105f83d1e80b4e04602743cf37812fa3576223bfeb	16:13:03	16:13:28	25s
Fizz, retry	ec3baaf8a4de3898269ac8bfb81a229a36ca0119ef83bd70b0db360616316f85	6fe66f81a8cd1f0157a1aa6056328a8857dd9d78701bcbe60c58ba423ed251d1	16:14:57	16:15:10	13s

Observed summary for this four-turn sample:

• Mean: 22.25s
• Median: 22.5s
• Range: 13–31s

What this evidence does and does not prove

It proves that all four explicit mentions were ultimately delivered and answered, and that the user-visible elapsed time varied by 18 seconds across equivalent trivial turns.

It does not identify the slow stage. The current event data only gives the two outer timestamps. The elapsed time can include:

1. relay fan-out and harness receipt;
2. queue/admission delay;
3. harness or ACP runtime wake/startup;
4. provider/model time to first output and completion;
5. tool execution needed to publish through Buzz;
6. relay acceptance of the reply.

It also does not replace the separate reliability bugs where an agent never receives or processes a turn.

Related open work

• [Bug] buzz-acp agents go deaf in threads they've joined unless re-mentioned. #2270 / fix(acp): follow threads the agent participates in (#2270) #2375: mention-gated agents miss untagged replies in threads they joined.
• Relay-mesh agents silently stop replying if mesh runtime drops after launch (no self-heal) #2062 / fix(desktop): re-arm relay-mesh runtime when the ingress is dead #2304: relay-mesh agents silently stop replying after local ingress failure.
• Agent @mentions silently fail when mentioned agent is offline #1743 / feat(relay): warn on offline explicit mentions (#1743) #1862 / feat(relay): notify channel when an offline agent is @mentioned #2296: offline agent mentions can disappear without reliable sender feedback.
• perf(desktop): skip pre-send parent fetch on reply when thread root is cached #1467: removes one relay round trip from reply sending when the thread root is cached.
• Add desktop GUI latency harness #1263: desktop GUI latency harness; does not measure agent-turn latency.
• feat(desktop+acp): spawn a harness per (agent, community) pair at GUI startup — warm sockets, lazy LLM pool #2122: warm harness sockets with lazy LLM startup; primarily reachability/cold-mention work.

Proposed instrumentation

Add one correlation/trace identifier and monotonic timestamps at these boundaries:

1. triggering event accepted/published by the relay;
2. event received by buzz-acp;
3. event admitted to the turn queue;
4. ACP child ready (separate cold and warm paths);
5. first ACP/model output;
6. turn complete;
7. reply publish initiated;
8. reply accepted by the relay.

Report both stage durations and total latency. Do not record message content, prompts, credentials, or model output in performance telemetry.

The benchmark should distinguish at least:

• warm harness + warm ACP/model session;
• warm harness + cold/lazy ACP start;
• app/harness restart recovery;
• local runtime versus relay-mesh runtime;
• no-tool acknowledgement versus a tool-using turn.

Acceptance criteria

• A repeatable benchmark measures mention-to-reply latency with at least the stage boundaries above.
• Results report p50, p95, and max instead of only averages.
• Warm and cold paths are reported separately.
• The team records explicit latency budgets for each supported path.
• CI or a scheduled performance job detects material regressions against those budgets.
• Telemetry contains correlation/timing metadata only, not private message content.

Why this should be separate from the reliability issues

The existing issues are valid and should remain focused on missed turns, dead runtimes, and delivery feedback. Even after those are fixed, Buzz still needs an evidence-backed answer to: “The agent replied, but why did a trivial response take 13 seconds in one run and 31 seconds in another?”