test(evals): webchat leg of the game matrix — night-collection scenario + single-conversation Werewolf

[Poytr1]

What

The webchat leg of the collaboration game matrix — the live single-conversation Werewolf topology (one multi-agent webchat conversation; role delivery and night actions as postless toAgent + needsReply calls; referee-MEDIATED wolf coordination; a model-shaped referee), reproduced as evals in two stages. The referee is a scripted subject agent acting through the real tool surface (sendMessage/needsReply over the MCP control socket) — never the trusted deliverRefereeEvent control path — so eval composition == live composition except the referee's brain is deterministic.

Stage 1 — the night-collection scenario

evals/games/night-collection.ts + evals/test/webchat-night-collection.test.ts (added to eval:collab:contracts): the host posts night-start; the referee's conversation session issues THREE concurrent needsReply postless calls (wolf-A propose / seer inspect / doctor protect) while players post public filler; wolf-A's proposal is relayed to wolf-B. Scripted cells pin, on current main:

• a child reply sent correctly (sendMessage {sessionId}) wakes the referee exactly once — as its own turn or coalesced into an in-flight turn whose input still carries it; both shapes distinguished and recorded (own-turn / coalesced), with a forced-coalesce cell that holds the referee's turn open while a reply lands;
• a child that answers in prose (headless, no tool call) is a LOST reply — the fix(daemon): scope the #800 SendMessage-collision fix to the parent-report directive #905 validation cell, pinned lost because that IS current main;
• the referee-mediated relay leg round-trips end to end.

Real-model variant (webchat-night-collection-real.test.ts, env-gated; real local Claude Code children, scripted referee via a puppet ACP adapter running the same brain): 3 trials, 9/9 owed night replies LOST — every child explored (ListAgents in all nine cases) and then answered its delegation as an ordinary assistant response (a correctly formatted line, delivered to nobody). Referee woken zero times; the wolf-B relay never fired. At/above the historical delegate-and-forward baseline (2/5–3/5), and exactly the live night-1 failure shape.

Stage 2 — single-conversation webchat Werewolf

evals/games/webchat-werewolf{,-runner}.ts + evals/test/webchat-werewolf.test.ts (in the gate): the full game — role delivery + night actions private via needsReply, mediated night kill (propose → agree/counter), public sequential day + votes carried by the #906 continuation, a reactive human HOST answering the referee's "open night N" asks (which also keeps each round inside the webchat hop budget). Scripted 5-player and multi-round 6-player games play to a winner deterministically; canaries never surface in the conversation. Rules/roles/win logic shared with the Slack game via evals/games/werewolf-rules.ts; the Slack-shaped Werewolf is untouched and keeps pinning the other composition.

Real run (seed 1, 5 real players, scripted referee): an honest stall at night 1 — awaiting protect→player-5 — which is the expected pre-#905 baseline, with unusually rich evidence:

• 8 of 9 needsReply calls answered correctly, including the full mediated kill leg ("We kill player-4 tonight." → relay → "I agree.") and the seer's inspection;
• the one prose loss (the doctor: "I protect player-5 tonight.", no tool call) stalled the entire game;
• the doctor's earlier ROLE turn had first attempted the runtime's built-in SendMessage three times (the Claude Code's built-in SendMessage tool collides with the AgentConnect messaging tool and silently swallows parent reports #800 collision), recovered via ToolSearch — and still lost the night reply to prose in the same session;
• stage 2 lost 1/9 where stage 1 lost 9/9: the role-ack leg seeds each pairwise session with one successful report-back, so session precedent strongly mitigates the cold-call loss.

What the webchat composition surfaced that the Slack-shaped one hid

1. fix(daemon): post an agent wake's inbound message to the live webchat view #926 makes needsReply reports room-visible: a child's {sessionId} report into a conversation-origin parent is posted live into the conversation and fanned to the roster — the night kill statement is readable by every player (pinned privateReportsPostedPublicly > 0; canaryLeaks: 0 still holds since canaries ride only the never-posted role calls).
2. Sibling pairwise sessions share one transcript: every postless child of one caller lives on a2a:<caller> + one thread, so a child's context refresh shows the referee's calls to its siblings (role assignments included) and their replies.
3. A {sessionId} reply carries no sender label in the delivered text — a referee holding several concurrent needsReply calls cannot attribute a direct reply wake from the text; coalesced replies (context rows) ARE attributable.
4. Turn-final regeneration replaces the prompt with the delta and silently discards the drafted answer — scripted policies (like models) must not latch on their own drafts; the players carry an explicit regeneration escape hatch.

All four are recorded in docs/designs/collaboration-arena-baseline.md §3.5 (the new webchat leg of the game matrix, what each composition validates) with the explicit note that the stage-1 real loss rate is the pre-#905 baseline. PR #905 itself stays parked and untouched.

Verification

• pnpm eval:collab:contracts — 17 files / 119 tests (was 15/115)
• pnpm eval:parity 23, pnpm eval:contracts 46 — green
• pnpm typecheck, pnpm lint, evals tsc — green
• daemon package tests: 3372 passed; failures only in the known local-environmental files (acp-matrix, daemon-agent-mention-routing, shim-cancellation, shim-exec-handler, workspace-git-runner-seam) — this branch touches nothing under packages/daemon (git diff origin/main -- packages/daemon is empty)
• Real-run artifacts (events JSONL, per-trial records, game result, referee prompts) preserved off-repo on the measurement host

Refs #905 (parked; this coverage is its gate), #906, #926, #800.

🤖 Generated with Claude Code

[agentconnect-md-test]

Requesting changes for one blocking eval-validity issue. The new night-collection scorer cannot distinguish the intended needsReply wake from ordinary webchat transcript fan-out: under the current #926 behavior, a child's report is committed as a conversation post and the fixture immediately fans a context copy back to every other seat, including the referee. A later unrelated wake can therefore expose the marker as a bracketed context row even if the reply's own queued wake was dropped. Because the scorer classifies any such sighting as coalesced, the scripted assertions—and the full-game accounting that absorbs the same context rows—can pass while the regression this PR is intended to gate still exists. Please correlate the report with evidence for its accepted/started parent wake (or otherwise instrument the coalesced queue path) instead of using generic transcript visibility as delivery proof.

I inspected the exact current head 1519db8f0b7ab2c683c48ab3ff925b6fc8970ac4 through read-only GitHub data because the local worktree was not at the trusted revision. At review time, all nine GitHub build/test jobs had passed; only the review check remained pending.

_{sent by review-bot (Codex · gpt-5.6-sol) · open in session}

[agentconnect-md-test]

This produces a false coalesced result for an unrelated context refresh. WebchatArena.fanOut() sends every committed child report back to the referee as context, independently of whether that report's queued needsReply wake starts or is coalesced into a turn. If that wake is dropped but another child/public wake later refreshes the referee, contextRowSightings > 0 still makes this test pass. The ownTurnStarts <= 1 assertion does not close the gap. Please require correlation to the report's actual parent wake (including a positively identified coalesced wake), rather than treating any later transcript row as delivery evidence.

[Poytr1]

Fixed in e0ec4b8 — you were right that transcript visibility was doing delivery's job. Verdicts now bind to the daemon's own wake-admission evidence (agentReplyWakeEvidence in evals/games/webchat-fixture.ts): a {sessionId} reply wake is discriminable in the evaluation events (turn.accepted with source:'agent' and a bare-UUID deliveryId turn-id suffix; continuation wakes carry postId#target ids, host turns are source:'user'). own-turn requires the delivered form in a turn started on an admitted reply wake; coalesced requires visible content and an unconsumed coalesced reply wake (turn.cancelled / coalesced_into_turn on an admitted id); visibility with no admitted wake — your masking case — now scores lost. The full-game accounting gets the same treatment: the runner reports replyWakesAccepted, and both CI games assert answered rows == admitted reply wakes, so a #926 context echo of a dropped wake cannot pass as an answer. Discriminator validated against the real-run artifacts: the stage-2 game shows exactly 8 accepted reply wakes for its 8 answered calls (0 coalesced), the all-lost night-collection trials show 0.

[agentconnect-md-test]

Confirmed on e0ec4b83. The admission/lifecycle correlation closes the transcript-visibility false positive, and I’ve approved the revised head.

_{sent by review-bot (Codex · gpt-5.6-sol) · open in session}

[agentconnect-md-test]

Approved. The prior blocking issue is resolved at e0ec4b833becae65f166e60b1a6562345e413dd3: reply verdicts now require daemon-side wake-admission evidence, own-turn delivery is tied to the matching admitted wake's turn.started input, and coalesced delivery requires a matching coalesced_into_turn lifecycle event rather than transcript visibility alone. The full Werewolf results also cross-check answered calls against the number of admitted reply wakes, so a #926 context echo cannot independently satisfy the gate.

I inspected the exact synchronized revision through read-only GitHub data because Git cannot initialize in the provided sandbox. Build, Check, image, sandbox, and integration jobs were green at review time; Unit Test was still running. I found no remaining blocking correctness issue in the revision.

_{sent by review-bot (Codex · gpt-5.6-sol) · open in session}