diff --git a/docs/designs/collaboration-arena-baseline.md b/docs/designs/collaboration-arena-baseline.md index e239bbed3..3f69c6540 100644 --- a/docs/designs/collaboration-arena-baseline.md +++ b/docs/designs/collaboration-arena-baseline.md @@ -200,6 +200,142 @@ failure. Measured leaks: **0**. See §6.4. The game is present and its referee semantics are covered, but the milestone cannot complete against the landed `sendMessage`. +### 3.5 The webchat leg of the game matrix + +Werewolf now runs in a SECOND composition — +`evals/games/webchat-werewolf.ts` + `webchat-werewolf-runner.ts`, gated by +`evals/test/webchat-werewolf.test.ts` — mapped onto ONE multi-agent **webchat +conversation** at the daemon seam (the same `handleRelayMsg` seam PR #906's +tests and the parity webchat leg drive), per the live single-conversation +runbook topology: + +| Piece | Slack-shaped (§3.3) | Webchat leg | +| ------------------------ | ------------------------------------------ | ------------------------------------------------------------------------- | +| Public day speech, votes | Channel posts + platform echo | Ordinary conversation posts, carried by the #906 continuation | +| Role delivery | Trusted referee DM (`deliverRefereeEvent`) | Postless `toAgent + needsReply` calls from the referee's session | +| Night actions | Player's own referee DM | `needsReply` calls; child reports back via `sendMessage {sessionId}` | +| Wolf coordination | REAL private den room | Referee-MEDIATED relay (propose → agree/counter), no den exists | +| The referee | Code referee on the trusted control path | SCRIPTED SUBJECT AGENT acting through the real `sendMessage` tool surface | +| Phase pacing | Referee waves | A reactive human HOST answering the referee's public "open night N" asks | + +**What each composition validates.** The Slack-shaped game pins room-membership +isolation, the platform echo, and the loop-guard/turn-budget bounds (§6.1–§6.5). +The webchat leg pins the OTHER half of the product: the `needsReply` delegation +round-trip into a live conversation session (exactly-once wakes, coalescing into +in-flight turns), the pre-addressed §5.2 fan-out, and the webchat hop budget — +with the referee composed exactly as a live model referee would be (its calls +run the trusted-session-context MCP path; only its brain is deterministic, and +for real-subject runs the same TypeScript brain drives a puppet ACP adapter, +`evals/games/puppet-acp-agent.mjs` + `puppet.ts`). Rules, roles and win logic +are shared (`evals/games/werewolf-rules.ts`) so the two compositions cannot +drift on the game itself. + +Upstream of the full game sits the **night-collection scenario** +(`evals/games/night-collection.ts`, `evals/test/webchat-night-collection.test.ts`) +— the live night-1 failure shape in isolation: one host night-start, THREE +concurrent `needsReply` postless calls (wolf-A propose / seer inspect / doctor +protect) with public filler interleaved, plus the mediated wolf-B relay. Its +scripted cells pin, on current `main`: + +- a child reply sent correctly (`sendMessage {sessionId}`) wakes the referee's + session **exactly once**, whether as its own turn or coalesced into an + in-flight turn whose input then still carries it (both shapes are recorded). + Verdicts are bound to the daemon's own wake-admission evidence + (`agentReplyWakeEvidence`): under #926 a reply's public copy can surface in + a later unrelated context refresh, so content visibility alone never counts + as delivery — a visible marker with no admitted reply wake scores LOST; +- a child that answers its delegation in **prose** (headless, no tool call) is + a **lost** reply — the #905 validation cell, pinned `lost` because that IS + current main; +- the referee-mediated relay leg round-trips end to end. + +**What the webchat composition surfaced that the Slack-shaped one hid** (all +measured, recorded in the run artifacts, and — where deterministic — pinned): + +- **A needsReply report is not private in a conversation-origin parent.** Since + #926, a child's `sendMessage {sessionId}` report into a session whose origin + is a webchat conversation is ALSO posted live into that conversation view (and + fans §5.2 context copies to the whole roster). Night statements — "We kill + player-1 tonight." — are visible to every player. The Slack composition's + private den/DM rooms structurally hide this class. Pinned as + `privateReportsPostedPublicly > 0`. +- **Sibling pairwise sessions share one transcript.** Every postless `toAgent` + child of one caller lives on the same synthetic `a2a:` channel + + thread, so a child's context refresh shows the referee's calls to its + SIBLINGS — including role assignments — and their replies. Role secrecy in + the webchat topology therefore depends on each child ignoring context rows + addressed to others; the scripted players enforce that discipline explicitly. +- **A raw reply carries no sender label.** `messageAgent` deliveries render as + `From : …`, but a `{sessionId}` reply is injected verbatim — a referee + holding several concurrent `needsReply` calls cannot attribute a direct reply + wake from the text alone. Coalesced replies, ironically, ARE attributable + (they surface as `[] …` context rows). +- **The turn-final regeneration fence replaces the prompt with the delta** and + silently discards the drafted answer; a policy that latches "I already acted" + on its own draft loses actions. The scripted players carry the same + observed-state discipline the Slack scripted host documents, plus an explicit + regeneration escape hatch. + +**Stage-1 real-model baseline — measured (2026-08-14).** The real-model +night-collection variant (`evals/test/webchat-night-collection-real.test.ts`; +scripted referee via the puppet adapter, real local Claude Code children over +ACP — the §4.1 recipe: node-launched claude-agent-acp, model pinned sonnet, +`permissionMode: default`, memory off) ran three trials. With PR #905 +deliberately parked pending exactly this coverage, the loss rate measured here +**is the pre-#905 baseline** (the delegate-and-forward measurements ran +2/5–3/5 lost): + +| Trial | Scoreable | needsReply replies owed (wolf-A / seer / doctor) | Lost | Wolf-B relay reached | Referee wakes | +| ----- | --------- | ------------------------------------------------ | ------- | --------------------- | ------------- | +| 1 | yes | 3 | **3/3** | never (proposal lost) | 0 | +| 2 | yes | 3 | **3/3** | never | 0 | +| 3 | yes | 3 | **3/3** | never | 0 | + +Every loss has the same shape, and it is exactly the live night-1 failure: the +child session runs, usually explores first (`ListAgents` in all nine cases, +once `ToolSearch`, once a terminal probe), and then answers the delegation as +its ORDINARY ASSISTANT RESPONSE — a correctly formatted single line +("`DOCTOR-REPORT: …`", "`SEER-REPORT: …`", "`WOLF-PROPOSAL: …`") that a +headless child delivers to nobody. Zero `sendMessage` calls, zero permission +requests, zero failed turns; the referee's session was never woken again, so +the mediated wolf-B relay never fired. 9/9 owed replies lost across the three +trials — at or above the historical baseline; the marker-formatted instruction +("answer with a single line starting …") plausibly biases the child further +toward prose than the free-form delegations of the earlier measurements. + +**Stage-2 real run — measured (2026-08-14, seed 1, 5 players).** One full +webchat Werewolf game, real local Claude Code players, scripted puppet +referee. Result: an HONEST STALL at night 1 — `stalledAt: phase=night round=1 +awaiting protect→player-5` — which is a valid result and precisely the +pre-#905 baseline the run exists to demonstrate: + +- **8 of 9 `needsReply` calls were answered correctly** via + `sendMessage {sessionId}`: all five role acks, the wolf lead's kill + proposal ("We kill player-4 tonight."), the mediated wolf-B relay + ("I agree."), and the seer's inspection. The referee-mediated night-kill + leg worked END TO END with real models. +- **The one lost reply stalled the whole game.** The doctor answered its + night call in PROSE ("I protect player-5 tonight." — a correct answer, + delivered to nobody), the referee was never woken again, and night 1 could + not resolve. One headless prose reply is all it takes. +- **The #800 tool collision appeared on the way.** The doctor's ROLE turn + first attempted the runtime's built-in `SendMessage` three times (delivered + nowhere), recovered via `ToolSearch` to `mcp__agentconnect__sendMessage`, + and delivered its ack — then still lost the night reply to prose in the + SAME session two turns later. +- **Why stage 2 lost 1/9 where stage 1 lost 9/9:** the role-delivery leg ran + first, so each night call landed in a pairwise session whose context + already contained one successful report-back. Session precedent strongly + mitigates the cold-call prose loss — and the loss still concentrates + exactly where the live game stalled. +- Leak posture: `canaryLeaks: 0`; `privateReportsPostedPublicly: 8` — every + answered "private" report, the night kill included, was visible in the + conversation view (the #926 surface above). + +A stalled run is reported, never failed: the runner records `stalledAt` and +the unanswered `needsReply` rows (`replyLoss`), and the artifacts carry the +full event stream and referee prompts. + ## 4. Reproducing every result Node 24 (`.nvmrc`) and pnpm 11. No model credentials required. @@ -209,7 +345,7 @@ pnpm install pnpm build # protocol must be built before typecheck # the whole arena gate -pnpm eval:collab:contracts # 15 files, 115 tests +pnpm eval:collab:contracts # 17 files, 119 tests # the routing acceptance cases alone pnpm eval:collab:routing # routing-acceptance + connection-surface @@ -218,6 +354,8 @@ pnpm eval:collab:routing # routing-acceptance + connection-surface npx vitest run evals/test/counting.test.ts npx vitest run evals/test/quota-counting.test.ts npx vitest run evals/test/werewolf.test.ts +npx vitest run evals/test/webchat-night-collection.test.ts +npx vitest run evals/test/webchat-werewolf.test.ts npx vitest run evals/test/cross-room-counting.test.ts npx vitest run evals/test/game-runner.test.ts npx vitest run evals/test/routing-acceptance.test.ts @@ -237,6 +375,8 @@ Full gate, measured on this branch: | `evals/test/routing-acceptance.test.ts` | 8 | | `evals/test/game-runner.test.ts` | 5 | | `evals/test/werewolf.test.ts` | 18 | +| `evals/test/webchat-night-collection.test.ts` | 2 | +| `evals/test/webchat-werewolf.test.ts` | 2 | | `evals/test/quota-counting.test.ts` | 8 | | `evals/test/cross-room-counting.test.ts` | 6 | | `evals/test/counting.test.ts` | 11 | @@ -249,7 +389,7 @@ Full gate, measured on this branch: | `evals/test/virtual-connections.test.ts` | 4 | | `packages/daemon/test/evaluation-game-ingress.test.ts` | 5 | | `packages/daemon/test/evaluation-game-tools.test.ts` | 3 | -| **Total** | **115** | +| **Total** | **119** | **0 expected-fail.** Every pin is an ordinary assertion. diff --git a/evals/games/night-collection.ts b/evals/games/night-collection.ts new file mode 100644 index 000000000..96bb4f967 --- /dev/null +++ b/evals/games/night-collection.ts @@ -0,0 +1,293 @@ +/** + * The "night collection" scenario — the live Werewolf night-1 failure shape, + * reproduced webchat-shaped (see the runbook topology this mirrors): + * + * ONE multi-agent webchat conversation (referee + players + the human host). + * The host posts a night-start message; the REFEREE's conversation session + * issues THREE concurrent `needsReply` postless calls (wolf-A: propose; + * seer: inspect; doctor: protect) while other players post public filler + * ("Waiting.") into the conversation. Children that reply correctly + * (`sendMessage {sessionId}`) must wake the referee's session exactly once + * each, regardless of interleaved public posts; a child that answers in + * PROSE (headless, no tool call) is a LOST reply — the known headless-child + * prose-reply loss that stalled the live game (PR #905 is parked on exactly + * this eval coverage). The referee-mediated wolf relay (wolf-A's proposal + * forwarded to wolf-B, wolf-B's verdict back) is the end-to-end leg. + * + * The referee is a SCRIPTED subject agent acting through the REAL tool + * surface (`sendMessage` over the daemon's MCP control socket) — never the + * trusted referee control path. Scripted and real-model variants share this + * module; only who plays the CHILDREN differs. + */ +import type { EvaluationEvent } from '../../packages/daemon/src/evaluation/index.js' +import type { RdWebchatPost } from '../../packages/protocol/src/index.js' +import { NO_RESPONSE, agentReplyWakeEvidence, type WebchatSeat } from './webchat-fixture.js' +import type { BrainCallOutcome, BrainTurn, ScriptedBrain } from './webchat-referee.js' + +export const NIGHT_ALIASES = ['referee', 'wolf-a', 'wolf-b', 'seer', 'doctor', 'villager'] as const + +/** The reply markers each child is instructed to lead its report with. The + * referee (and the scoring) recognize replies by marker, never by uuid. */ +export const MARKERS = { + proposal: 'WOLF-PROPOSAL:', + seer: 'SEER-REPORT:', + doctor: 'DOCTOR-REPORT:', + verdict: 'WOLF-B-VERDICT:' +} as const + +export type NightMarker = keyof typeof MARKERS + +export const NIGHT_START_TEXT = + 'NIGHT 1 begins. Referee: collect the night actions privately now — contact wolf-a for the kill proposal, ' + + 'the seer for an inspection, and the doctor for a protection, all at the same time. Everyone else: you may ' + + 'chat here while the referee works.' + +/** One needsReply call the referee issued, with its delivery verdict. */ +export interface IssuedCall { + to: string + purpose: NightMarker + needsReply: boolean + delivered: boolean + targetSession?: string + error?: string +} + +export interface NightCollectionRefereeConfig { + wolfA: WebchatSeat + wolfB: WebchatSeat + seer: WebchatSeat + doctor: WebchatSeat +} + +const instruction = (task: string, marker: string): string => + `${task} Answer with a single line that starts exactly with \`${marker}\` — nothing before it. ` + + `Do not contact anyone else and do not post anywhere.` + +/** + * The deterministic referee brain. Wake-driven only (a model referee has no + * timers either): the night calls go out on the host's night-start wake, the + * wolf-B relay goes out on the wolf-A proposal wake, and the closing public + * post rides the wolf-B verdict wake. Idempotent under coalesced/regenerated + * prompts — every trigger fires at most once. + */ +export class NightCollectionReferee implements ScriptedBrain { + readonly issued: IssuedCall[] = [] + /** Marker → number of onPrompt() calls whose text contained the reply. */ + readonly markerSightings = new Map() + private nightIssued = false + private relayIssued = false + private closed = false + private pendingPurposes: NightMarker[] = [] + + constructor(private readonly cfg: NightCollectionRefereeConfig) {} + + onPrompt(text: string): BrainTurn { + const calls: BrainTurn['calls'] = [] + let reply = NO_RESPONSE + for (const marker of Object.keys(MARKERS) as NightMarker[]) { + if (text.includes(MARKERS[marker])) { + this.markerSightings.set(marker, (this.markerSightings.get(marker) ?? 0) + 1) + } + } + if (!this.nightIssued && /NIGHT 1 begins/.test(text)) { + this.nightIssued = true + this.pendingPurposes = ['proposal', 'seer', 'doctor'] + calls.push( + this.needsReplyCall( + this.cfg.wolfA, + 'proposal', + instruction('Night 1: as werewolf lead, state your kill proposal for tonight.', MARKERS.proposal) + ), + this.needsReplyCall( + this.cfg.seer, + 'seer', + instruction('Night 1: name the one player you inspect tonight.', MARKERS.seer) + ), + this.needsReplyCall( + this.cfg.doctor, + 'doctor', + instruction('Night 1: name the one player you protect tonight.', MARKERS.doctor) + ) + ) + } + if (!this.relayIssued && text.includes(MARKERS.proposal)) { + this.relayIssued = true + const line = text.split('\n').find((candidate) => candidate.includes(MARKERS.proposal)) ?? MARKERS.proposal + calls.push( + this.needsReplyCall( + this.cfg.wolfB, + 'verdict', + instruction(`Your fellow wolf proposes: "${line.trim()}". Do you agree, or counter?`, MARKERS.verdict) + ) + ) + } + if (!this.closed && text.includes(MARKERS.verdict)) { + this.closed = true + reply = 'The night is resolved.' + } + return { calls, reply } + } + + onCallResult(outcome: BrainCallOutcome): void { + const toAgentId = (outcome.args.toAgent as { agentId?: string } | undefined)?.agentId + const row = this.issued.find((candidate) => candidate.to === toAgentId && !this.settled.has(candidate)) + if (!row) return + this.settled.add(row) + const parsed = parseToolResult(outcome.result) + row.delivered = outcome.ok && parsed?.delivered !== false + if (parsed?.targetSession !== undefined) row.targetSession = String(parsed.targetSession) + if (!outcome.ok && outcome.error !== undefined) row.error = outcome.error + } + + private readonly settled = new Set() + + private needsReplyCall(seat: WebchatSeat, purpose: NightMarker, message: string): BrainTurn['calls'][number] { + this.issued.push({ to: seat.agentId, purpose, needsReply: true, delivered: false }) + return { + tool: 'sendMessage', + args: { toAgent: { agentId: seat.agentId, needsReply: true }, message } + } + } +} + +function parseToolResult(result: unknown): Record | undefined { + if (result && typeof result === 'object' && !Array.isArray(result)) { + const content = (result as { content?: { type?: string; text?: string }[] }).content + if (Array.isArray(content)) { + const text = content.find((block) => block?.type === 'text')?.text + if (typeof text === 'string') { + try { + return JSON.parse(text) as Record + } catch { + return undefined + } + } + } + return result as Record + } + if (typeof result === 'string') { + try { + return JSON.parse(result) as Record + } catch { + return undefined + } + } + return undefined +} + +// ── scoring ──────────────────────────────────────────────────────────────── + +export type ReplyMode = 'own-turn' | 'coalesced' | 'lost' + +/** + * Classification is grounded in the DAEMON'S OWN wake evidence + * (`agentReplyWakeEvidence`), never in content visibility alone: under #926 a + * child's report is also committed as a conversation post whose `context` copy + * fans back to the referee, so the marker can surface in a LATER, unrelated + * referee prompt even if the reply's own queued wake was dropped. Visibility + * without an admitted reply wake therefore scores LOST. + * + * The two shapes an ADMITTED reply then takes inside a referee prompt: + * + * - DELIVERED form — the reply body as the wake's own message (a raw line + * starting with the marker, or `From : MARKER …`) in the input of a + * turn that STARTED on an admitted reply wake; + * - CONTEXT-ROW form — `[] MARKER …`: how a reply whose admitted + * wake was COALESCED into an in-flight turn is represented (the regenerated + * turn input carries the row). Later catch-up context re-shows the same row + * shape, which is exactly why a `coalesced` verdict additionally requires an + * unconsumed coalesced reply wake in the evidence. + */ +export interface ReplyOutcome { + child: string + marker: NightMarker + /** 'own-turn': a referee turn started on the delivered reply; 'coalesced': + * no turn started on it, but a referee turn's input carried it (context + * row of a coalesced wake); 'lost': the referee never saw it at all — the + * headless prose-reply loss. */ + mode: ReplyMode + /** Turns STARTED on an admitted reply wake whose input carries the + * DELIVERED form. Must be ≤ 1. */ + ownTurnStarts: number + /** Referee prompt deliveries (incl. regenerations) with the delivered form. */ + deliveredPromptSightings: number + /** Referee prompt deliveries carrying the context-row form (observational — + * never delivery proof by itself). */ + contextRowSightings: number + /** Whether the reply body surfaced as a committed conversation post — the + * #926 agent-wake inbound live post. Recorded, since it means a "private" + * needsReply report is visible to the whole conversation on current main. */ + postedPublicly: boolean +} + +export interface NightCollectionScore { + replies: ReplyOutcome[] + lost: string[] + /** Admitted `sendMessage {sessionId}` reply wakes at the referee — the + * daemon-side ground truth the per-marker verdicts are bound to. */ + acceptedReplyWakes: number + /** Of those, wakes coalesced into an in-flight referee turn. */ + coalescedReplyWakes: number + /** Public filler posts observed during the night. */ + fillerPosts: number + /** The referee's closing public post landed. */ + nightResolvedPosted: boolean +} + +export interface ScoreInputs { + events: readonly EvaluationEvent[] + /** Every prompt text delivered to the referee host/puppet, in order. */ + refereePrompts: readonly string[] + posts: readonly RdWebchatPost[] + refereeAgentId: string + children: { alias: string; marker: NightMarker }[] +} + +const escapeRegExp = (value: string): string => value.replace(/[.*+?^${}()|[\]\\]/g, '\\$&') + +/** The delivered-wake form: the reply body as its own message. */ +export function deliveredFormPattern(marker: NightMarker): RegExp { + return new RegExp(`^(?:From [^:\\n]+: ?)?${escapeRegExp(MARKERS[marker])}`, 'm') +} + +/** The context-row form: a shared-transcript row shown by a context refresh. */ +export function contextRowPattern(marker: NightMarker): RegExp { + return new RegExp(`^\\[[^\\]\\n]+\\] ?${escapeRegExp(MARKERS[marker])}`, 'm') +} + +export function scoreNightCollection(inputs: ScoreInputs): NightCollectionScore { + const evidence = agentReplyWakeEvidence(inputs.events, inputs.refereeAgentId) + const replyWakeTurnInputs = [...evidence.startedInputs.values()] + // Coalesced wakes are a budget consumed one per marker: a `coalesced` + // verdict needs BOTH visible content and an unconsumed coalesced reply wake + // (content alone can be a #926 public-copy echo of a dropped wake). + let coalescedBudget = evidence.coalesced.size + const replies: ReplyOutcome[] = inputs.children.map(({ alias, marker }) => { + const token = MARKERS[marker] + const delivered = deliveredFormPattern(marker) + const contextRow = contextRowPattern(marker) + const ownTurnStarts = replyWakeTurnInputs.filter((input) => delivered.test(input)).length + const deliveredPromptSightings = inputs.refereePrompts.filter((text) => delivered.test(text)).length + const contextRowSightings = inputs.refereePrompts.filter((text) => contextRow.test(text)).length + const contentVisible = deliveredPromptSightings + contextRowSightings > 0 + let mode: ReplyMode = 'lost' + if (ownTurnStarts > 0) { + mode = 'own-turn' + } else if (contentVisible && coalescedBudget > 0) { + coalescedBudget -= 1 + mode = 'coalesced' + } + const postedPublicly = inputs.posts.some((post) => post.agentId !== 'host' && post.post.text.includes(token)) + return { child: alias, marker, mode, ownTurnStarts, deliveredPromptSightings, contextRowSightings, postedPublicly } + }) + return { + replies, + lost: replies.filter((reply) => reply.mode === 'lost').map((reply) => reply.child), + acceptedReplyWakes: evidence.accepted.size, + coalescedReplyWakes: evidence.coalesced.size, + fillerPosts: inputs.posts.filter((post) => post.agentId !== 'host' && /^Waiting\./.test(post.post.text)).length, + nightResolvedPosted: inputs.posts.some( + (post) => post.agentId === inputs.refereeAgentId && post.post.text.includes('The night is resolved.') + ) + } +} diff --git a/evals/games/puppet-acp-agent.mjs b/evals/games/puppet-acp-agent.mjs new file mode 100644 index 000000000..c6fe4fe6a --- /dev/null +++ b/evals/games/puppet-acp-agent.mjs @@ -0,0 +1,113 @@ +#!/usr/bin/env node +// The PUPPET ACP adapter — how a real-subject run gets a SCRIPTED referee that +// still acts through the real tool surface. +// +// The daemon launches this like any other ACP runtime (config.json `runtimes` +// entry). It is a hand-rolled JSON-RPC 2.0 peer over newline-delimited JSON on +// stdio (like packages/daemon/test/fixtures/scriptable-acp-agent.mjs), but it +// holds NO policy at all: every `session/prompt` is forwarded over a local +// socket (`AC_PUPPET_ENDPOINT`) to the evaluation driver, which runs the +// deterministic brain (TypeScript, shared with the scripted CI variant) and +// performs the brain's `sendMessage` calls itself against the daemon's MCP +// control socket, using the per-session binding this adapter captured at +// `session/new` (AC_MCP_ENDPOINT / AC_MCP_TOKEN in the injected server env — +// the exact material the mcp-bridge subprocess would use). +// +// The result: eval composition == live composition — the referee is an +// ordinary agent whose calls run the production trusted-session-context path — +// while its brain stays deterministic and fully typed in the driver process. +import net from 'node:net' +import { createInterface } from 'node:readline' + +const endpoint = process.env.AC_PUPPET_ENDPOINT +if (!endpoint) { + process.stderr.write('puppet-acp-agent: AC_PUPPET_ENDPOINT must be set\n') + process.exit(1) +} + +// ── driver link (newline-delimited JSON over the unix socket) ────────────── +const driver = net.connect(endpoint) +driver.setEncoding('utf8') +let driverBuffer = '' +let requestId = 0 +const pendingDriver = new Map() +driver.on('data', (chunk) => { + driverBuffer += chunk + let newline + while ((newline = driverBuffer.indexOf('\n')) !== -1) { + const line = driverBuffer.slice(0, newline) + driverBuffer = driverBuffer.slice(newline + 1) + if (!line.trim()) continue + let message + try { + message = JSON.parse(line) + } catch { + continue + } + const resolve = pendingDriver.get(message.id) + if (resolve) { + pendingDriver.delete(message.id) + resolve(message) + } + } +}) +driver.on('error', (error) => { + process.stderr.write(`puppet-acp-agent: driver socket error: ${error.message}\n`) + process.exit(1) +}) +function askDriver(payload) { + const id = ++requestId + return new Promise((resolve) => { + pendingDriver.set(id, resolve) + driver.write(`${JSON.stringify({ id, ...payload })}\n`) + }) +} + +/** Extract the daemon control-socket binding from `session/new`'s mcpServers. */ +function bindingOf(mcpServers) { + if (!Array.isArray(mcpServers)) return undefined + for (const server of mcpServers) { + const env = server?.env + if (!Array.isArray(env)) continue + const endpointVar = env.find((entry) => entry?.name === 'AC_MCP_ENDPOINT')?.value + const token = env.find((entry) => entry?.name === 'AC_MCP_TOKEN')?.value + if (typeof endpointVar === 'string' && typeof token === 'string') return { endpoint: endpointVar, token } + } + return undefined +} + +// ── the ACP stdio side ───────────────────────────────────────────────────── +const rl = createInterface({ input: process.stdin }) +const send = (obj) => process.stdout.write(JSON.stringify(obj) + '\n') +const update = (sessionId, u) => send({ jsonrpc: '2.0', method: 'session/update', params: { sessionId, update: u } }) +let sessionCounter = 0 + +rl.on('line', async (line) => { + if (!line.trim()) return + let msg + try { + msg = JSON.parse(line) + } catch { + return + } + const { id, method, params } = msg + if (method === 'initialize') { + send({ jsonrpc: '2.0', id, result: { protocolVersion: 1, agentCapabilities: {} } }) + } else if (method === 'session/new') { + const sessionId = `puppet-${++sessionCounter}` + await askDriver({ op: 'new', sessionId, binding: bindingOf(params?.mcpServers) ?? null }) + send({ jsonrpc: '2.0', id, result: { sessionId } }) + } else if (method === 'session/prompt') { + const text = (params?.prompt ?? []).map((block) => block?.text ?? '').join('\n') + const answer = await askDriver({ op: 'prompt', sessionId: params?.sessionId, text }) + update(params?.sessionId, { + sessionUpdate: 'agent_message_chunk', + content: { type: 'text', text: typeof answer.reply === 'string' ? answer.reply : 'AC_NO_RESPONSE' } + }) + send({ jsonrpc: '2.0', id, result: { stopReason: 'end_turn' } }) + } else if (method === 'session/cancel') { + if (id !== undefined) send({ jsonrpc: '2.0', id, result: null }) + } else if (id !== undefined) { + send({ jsonrpc: '2.0', id, result: null }) + } +}) diff --git a/evals/games/puppet.ts b/evals/games/puppet.ts new file mode 100644 index 000000000..8621170f6 --- /dev/null +++ b/evals/games/puppet.ts @@ -0,0 +1,102 @@ +/** + * Driver side of the puppet ACP adapter (`puppet-acp-agent.mjs`): a local + * socket server the adapter forwards every `session/new` binding and + * `session/prompt` text to. The driver runs the deterministic brain — the + * same `ScriptedBrain` the scripted CI variant runs in-process — and performs + * the brain's tool calls against the daemon's MCP control socket with the + * forwarded per-session binding, so every call still runs the production + * trusted-session-context path attributed to the puppet agent's session. + */ +import { randomUUID } from 'node:crypto' +import net from 'node:net' +import { tmpdir } from 'node:os' +import { join } from 'node:path' +import type { DaemonMcpBinding } from './mcp-client.js' +import { executeBrainTurn, type ScriptedBrain } from './webchat-referee.js' + +export interface PuppetPromptLogEntry { + sessionId: string + text: string +} + +export class PuppetDriver { + readonly endpoint: string + /** Every prompt the puppet agent received, in order (incl. regenerations) — + * the real-subject replacement for the in-process prompt log. */ + readonly promptLog: PuppetPromptLogEntry[] = [] + private readonly server: net.Server + private readonly bindings = new Map() + private brain: ScriptedBrain | undefined + private startedAt?: Promise + + constructor() { + this.endpoint = join(tmpdir(), `ac-puppet-${randomUUID().slice(0, 8)}.sock`) + this.server = net.createServer((socket) => this.serve(socket)) + } + + /** Install the brain for this run (one brain, shared by every session the + * puppet agent opens — sessions are distinguished inside the prompt text). */ + useBrain(brain: ScriptedBrain): void { + this.brain = brain + } + + async start(): Promise { + this.startedAt ??= new Promise((resolve, reject) => { + this.server.once('error', reject) + this.server.listen(this.endpoint, () => resolve()) + }) + await this.startedAt + } + + private serve(socket: net.Socket): void { + socket.setEncoding('utf8') + let buffer = '' + socket.on('data', (chunk: string) => { + buffer += chunk + let newline + while ((newline = buffer.indexOf('\n')) !== -1) { + const line = buffer.slice(0, newline) + buffer = buffer.slice(newline + 1) + if (line.trim()) void this.handle(socket, line) + } + }) + socket.on('error', () => {}) + } + + private async handle(socket: net.Socket, line: string): Promise { + let message: { id?: number; op?: string; sessionId?: string; binding?: DaemonMcpBinding | null; text?: string } + try { + message = JSON.parse(line) + } catch { + return + } + const answer = (payload: Record) => { + if (!socket.destroyed) socket.write(`${JSON.stringify({ id: message.id, ...payload })}\n`) + } + if (message.op === 'new') { + this.bindings.set(message.sessionId ?? '', message.binding ?? undefined) + answer({}) + return + } + if (message.op === 'prompt') { + const text = message.text ?? '' + this.promptLog.push({ sessionId: message.sessionId ?? '', text }) + if (!this.brain) { + answer({ reply: 'AC_NO_RESPONSE' }) + return + } + try { + const { reply } = await executeBrainTurn(this.brain, this.bindings.get(message.sessionId ?? ''), text) + answer({ reply }) + } catch (error) { + answer({ reply: `puppet brain error: ${(error as Error).message}` }) + } + return + } + answer({}) + } + + async stop(): Promise { + await new Promise((resolve) => this.server.close(() => resolve())) + } +} diff --git a/evals/games/webchat-fixture.ts b/evals/games/webchat-fixture.ts new file mode 100644 index 000000000..c48126c39 --- /dev/null +++ b/evals/games/webchat-fixture.ts @@ -0,0 +1,502 @@ +/** + * The webchat conversation fixture — ONE multi-agent webchat conversation at + * the daemon seam, for the night-collection scenario and webchat Werewolf. + * + * This is the same seam PR #906's tests and the parity webchat leg drive + * (`packages/daemon/test/webchat-continuation-fixture.ts`): `handleRelayMsg` + * fed the relay's pre-addressed `turn` frames, with the §5.2 roster fan-out of + * committed posts played by the fixture — no relay process, no platform SDKs, + * no credentials. What this fixture ADDS over the continuation fixture: + * + * - **UUID agent ids** (with an alias map for readability), because several + * production branches key on `UUID_RE.test(msg.sender.id)` — most notably + * #926's live posting of an agent-initiated wake's INBOUND message. The + * live composition uses UUID agent ids, so a faithful eval must too. + * - the **evaluation observer** (`turn.*` / `acp.update` events), which is + * how a REAL-subject run — where there is no in-process prompt log — still + * yields per-turn input evidence for reply-loss accounting. + * - subject preparation for both compositions: a scripted root (in-process + * hosts, the CI gate) and a real root (template runtimes for the players + + * the puppet ACP adapter for the scripted referee). + * + * The Slack-shaped games keep `evals/games/{world,topology}.ts`; nothing here + * touches them. This fixture is deliberately conversation-shaped instead of + * room-shaped: webchat has ONE conversation, no channels, and its private legs + * are postless `toAgent` calls rather than private rooms. + */ +import { randomUUID } from 'node:crypto' +import { mkdtempSync, mkdirSync, writeFileSync, rmSync, readFileSync, existsSync, lstatSync } from 'node:fs' +import { tmpdir } from 'node:os' +import { join, resolve } from 'node:path' +import { Daemon } from '../../packages/daemon/src/daemon.js' +import { + EvaluationEventCollector, + collectObjectSecrets, + compositeEvaluationObserver, + environmentSecrets +} from '../../packages/daemon/src/evaluation/index.js' +import { transcriptChannelKey } from '../../packages/daemon/src/store/local-store.js' +import { selectTurnTargets } from '../../packages/relay/src/relay-browser-connection.js' +import type { EvaluationEvent } from '../../packages/daemon/src/evaluation/index.js' +import type { RdMsgWebchat, RdWebchatPost, WebchatPost } from '../../packages/protocol/src/index.js' + +/** The standing response-choice sentinel (product-conventions §No-response). */ +export const NO_RESPONSE = 'AC_NO_RESPONSE' + +export interface WebchatSeat { + /** Human-readable seat name (referee / player-1 / …) — used in transcripts, + * policies, and reports. */ + alias: string + /** UUID agent id, minted per run. The daemon only ever sees this. */ + agentId: string +} + +/** Mint the roster: UUID agent ids behind stable aliases. */ +export function mintSeats(aliases: readonly string[]): WebchatSeat[] { + return aliases.map((alias) => ({ alias, agentId: randomUUID() })) +} + +export interface ScriptedWebchatRootOptions { + /** Extra config.json fields merged at the top level. */ + config?: Record +} + +/** Scaffold a control-plane-less daemon root with one stub agent per seat. + * Mirrors the continuation fixture's scaffold, with the turn-final refresh + * feature ON (the #906 behavior the scenario depends on). */ +export function prepareScriptedWebchatRoot(seats: readonly WebchatSeat[], options: ScriptedWebchatRootOptions = {}) { + const root = mkdtempSync(join(tmpdir(), 'ac-webchat-arena-')) + writeFileSync( + join(root, 'config.json'), + JSON.stringify({ + version: 1, + controlPlane: { enabled: false }, + features: { turnFinalContextRefresh: true }, + runtimes: { scripted: { command: 'node', args: ['unused'] } }, + ...(options.config ?? {}) + }) + ) + for (const seat of seats) { + const agentDir = join(root, 'agents', seat.agentId) + mkdirSync(agentDir, { recursive: true, mode: 0o700 }) + writeFileSync( + join(agentDir, 'agent.json'), + JSON.stringify({ + id: seat.agentId, + name: seat.alias, + status: 'active', + runtime: 'scripted', + workspace: { mode: 'from-scratch', path: join(agentDir, 'workspace') }, + integrations: [], + output: { mode: 'low', showFooter: false, showStatusBar: false } + }) + ) + } + return { root, secrets: [] as string[], cleanup: () => rmSync(root, { recursive: true, force: true }) } +} + +export interface RealWebchatRootOptions { + /** Template root: config.json (explicit runtimes) + agents//agent.json. */ + subjectRoot: string + /** Template agent ids mapped onto the PLAYER seats in order (broadcast when + * a single id is given) — the referee seat never consumes one. */ + templateAgentIds: string[] + /** The seat played by the scripted referee (puppet ACP adapter). */ + refereeAlias: string + /** Unix-socket endpoint of the puppet driver (see `puppet.ts`). */ + puppetEndpoint: string +} + +function assertNotSymlink(path: string, label: string): void { + if (lstatSync(path).isSymbolicLink()) throw new Error(`${label} may not be a symbolic link`) +} + +/** + * Disposable REAL webchat subject: player seats materialized from the + * operator's template (real runtimes, real provider credentials — the + * collaboration-arena-baseline §4.1 recipe), the referee seat bound to the + * puppet ACP adapter so its brain stays deterministic while acting through the + * REAL tool surface. Control plane, relays, crons, integrations stripped; + * memory off; from-scratch workspaces; template secrets harvested for + * redaction. Mirrors `prepareRealSubject` (evals/games/subject.ts) minus the + * compiled Slack topology, which webchat does not have. + */ +export function prepareRealWebchatRoot(seats: readonly WebchatSeat[], options: RealWebchatRootOptions) { + const sourceRoot = resolve(options.subjectRoot) + const configPath = join(sourceRoot, 'config.json') + if (!existsSync(configPath)) throw new Error(`webchat subject template is missing ${configPath}`) + if (options.templateAgentIds.length === 0) throw new Error('real webchat subject requires a templateAgentId') + const root = mkdtempSync(join(tmpdir(), 'ac-webchat-real-')) + const cleanup = () => rmSync(root, { recursive: true, force: true }) + try { + assertNotSymlink(configPath, 'webchat subject config') + const config = JSON.parse(readFileSync(configPath, 'utf8')) as Record + const secrets = collectObjectSecrets(config) + const runtimes = + config.runtimes && typeof config.runtimes === 'object' && !Array.isArray(config.runtimes) + ? (config.runtimes as Record) + : {} + writeFileSync( + join(root, 'config.json'), + `${JSON.stringify( + { + ...config, + daemonId: undefined, + agentsDir: join(root, 'agents'), + controlPlane: { enabled: false }, + relays: [], + features: { turnFinalContextRefresh: true }, + security: { + ...(config.security && typeof config.security === 'object' ? (config.security as object) : {}), + isolateAccountApps: true + }, + runtimes: { + ...runtimes, + 'ac-puppet': { + command: process.execPath, + args: [resolve(process.cwd(), 'evals', 'games', 'puppet-acp-agent.mjs')], + env: [{ name: 'AC_PUPPET_ENDPOINT', value: options.puppetEndpoint }] + } + } + }, + null, + 2 + )}\n`, + { mode: 0o600 } + ) + const players = seats.filter((seat) => seat.alias !== options.refereeAlias) + for (const [index, seat] of players.entries()) { + const templateId = options.templateAgentIds[index % options.templateAgentIds.length]! + if (templateId === '.' || templateId === '..' || /[/\\\0]/.test(templateId)) { + throw new Error(`webchat subject template agent id is not a safe path segment: ${JSON.stringify(templateId)}`) + } + const sourceAgentPath = join(sourceRoot, 'agents', templateId, 'agent.json') + if (!existsSync(sourceAgentPath)) throw new Error(`webchat subject template has no agent "${templateId}"`) + const template = JSON.parse(readFileSync(sourceAgentPath, 'utf8')) as Record + collectObjectSecrets(template, '', secrets) + if (typeof template.runtime !== 'string' || !Object.prototype.hasOwnProperty.call(runtimes, template.runtime)) { + throw new Error(`webchat subject agent "${templateId}" requires an explicit runtime definition in config.json`) + } + writeSeatAgent(root, seat, { ...template }) + } + const referee = seats.find((seat) => seat.alias === options.refereeAlias) + if (!referee) throw new Error(`no seat named "${options.refereeAlias}" for the referee`) + // `none` memory needs a runtime-verified off-switch, which the puppet + // adapter (an unregistered runtime) cannot offer — `managed` is our own + // no-op-for-the-adapter store and the evaluation profile keeps memory off. + writeSeatAgent(root, referee, { runtime: 'ac-puppet' }, 'managed') + return { + root, + secrets: [...new Set([...secrets, ...environmentSecrets()].filter((secret) => secret.length >= 4))], + cleanup + } + } catch (error) { + cleanup() + throw error + } +} + +function writeSeatAgent( + root: string, + seat: WebchatSeat, + template: Record, + memoryProvider: 'none' | 'managed' = 'none' +): void { + const agentDir = join(root, 'agents', seat.agentId) + mkdirSync(agentDir, { recursive: true, mode: 0o700 }) + const workspacePath = join(agentDir, 'workspace') + mkdirSync(workspacePath, { recursive: true, mode: 0o700 }) + const prepared = { + ...template, + id: seat.agentId, + name: seat.alias, + displayName: seat.alias, + status: 'active', + pause: false, + integrations: [], + crons: [], + mcpServers: [], + memory: { provider: memoryProvider }, + workspace: { mode: 'from-scratch', path: workspacePath, gitBranch: 'main', pullOnNewSession: true, skills: [] }, + output: { mode: 'low', showFooter: false, showStatusBar: false } + } + writeFileSync(join(agentDir, 'agent.json'), `${JSON.stringify(prepared, null, 2)}\n`, { mode: 0o600 }) +} + +export interface WebchatArenaOptions { + root: string + seats: readonly WebchatSeat[] + conversationId?: string + hostFactory?: NonNullable[0]>['hostFactory'] + runId?: string + secrets?: readonly string[] +} + +/** The webchat user who plays the HOST in the conversation. */ +export const HOST_USER = 'owner' + +/** + * One booted daemon + the fixture playing the relay for ONE conversation: + * pre-addressed `turn` frames for the host's posts, and the §5.2 fan-out of + * every committed `rd/webchat-post` (a `context` copy to each other roster + * member). `posts` is the browser's view of the conversation. + */ +export class WebchatArena { + readonly daemon: Daemon + readonly conversationId: string + readonly posts: RdWebchatPost[] = [] + private readonly collector: EvaluationEventCollector + private readonly seatByAlias = new Map() + private readonly seatById = new Map() + private turnSeq = 0 + private started = false + + constructor(private readonly options: WebchatArenaOptions) { + this.conversationId = options.conversationId ?? randomUUID() + this.collector = new EvaluationEventCollector(options.secrets) + for (const seat of options.seats) { + this.seatByAlias.set(seat.alias, seat) + this.seatById.set(seat.agentId, seat) + } + this.daemon = new Daemon({ + root: options.root, + evaluation: { + observer: compositeEvaluationObserver(this.collector), + runId: options.runId ?? randomUUID(), + capabilityProfile: { memory: 'off' } + }, + probeRuntimes: async () => [], + ...(options.hostFactory ? { hostFactory: options.hostFactory } : {}) + }) + } + + seat(alias: string): WebchatSeat { + const seat = this.seatByAlias.get(alias) + if (!seat) throw new Error(`no webchat seat named "${alias}"`) + return seat + } + + aliasOf(agentId: string): string { + return this.seatById.get(agentId)?.alias ?? agentId + } + + async start(): Promise { + if (this.started) return + await this.daemon.start() + this.started = true + // The flat org directory: every roster pair may call every other (the + // `admits()` check runs per continuation edge and per agent call). + ;(this.daemon as any).cpCollab.replace({ + generation: 1, + channels: [], + agents: this.options.seats.map((seat) => ({ + agentId: seat.agentId, + orgId: 'org-webchat-arena', + callPolicy: 'all', + allowedCallerAgentIds: [], + outboundPolicy: 'all', + allowedTargetAgentIds: [] + })) + }) + ;(this.daemon as any).relays = { sendWebchatPost: (post: RdWebchatPost) => this.fanOut(post), stop: async () => {} } + } + + /** Play the relay: record the committed post and fan a pre-addressed + * `context` copy to every OTHER roster member (webchat-multi-agents §5.2). */ + private ctxSeq = 0 + fanOut(post: RdWebchatPost): void { + this.posts.push(post) + for (const seat of this.options.seats) { + if (seat.agentId === post.agentId) continue + ;(this.daemon as any).handleRelayMsg( + this.rd({ op: 'context', post: post.post }, { agentId: seat.agentId, msgId: `ctx-${this.ctxSeq++}` }), + () => {} + ) + } + } + + private rd(payload: RdMsgWebchat['payload'], over: Partial = {}): RdMsgWebchat { + return { + source: 'webchat', + agentId: over.agentId ?? this.options.seats[0]!.agentId, + sessionKey: this.conversationId, + msgId: over.msgId ?? `m-${this.turnSeq}`, + chatId: this.conversationId, + payload, + ...over + } + } + + /** + * The HOST posts into the conversation: the relay's §5.2 user-turn fan-out + * for the target set the PRODUCTION choice (`selectTurnTargets`) computes — + * a pre-addressed `turn` frame per target, a transcript-only user `context` + * copy to every other roster member. `mentions` narrows by seat alias. + */ + postHost(text: string, options: { mentions?: string[] } = {}): { postId: string } { + const postId = randomUUID() + const at = Date.now() + const seq = ++this.turnSeq + const roster = this.options.seats.map((seat) => seat.agentId) + const mentionIds = options.mentions?.map((alias) => this.seat(alias).agentId) + const chosen = selectTurnTargets(roster, mentionIds ? { mentions: mentionIds } : {}) + if (chosen.invalid.length > 0) throw new Error(`postHost: invalid mention targets ${chosen.invalid.join(', ')}`) + for (const [index, agentId] of chosen.valid.entries()) { + const result = (this.daemon as any).handleRelayMsg( + this.rd( + { + op: 'turn', + text, + user: HOST_USER, + turnId: postId, + ...(mentionIds !== undefined ? { mentions: mentionIds } : {}), + post: { postId, at } + }, + { agentId, msgId: `turn-${seq}-t${index}` } + ), + () => {}, + (post: RdWebchatPost) => this.fanOut(post) + ) + if (!result || result.accepted !== true) { + throw new Error(`postHost: turn frame for ${this.aliasOf(agentId)} was not accepted`) + } + } + const userPost: WebchatPost = { + postId, + conversationId: this.conversationId, + author: { kind: 'user', user: HOST_USER }, + text, + at + } + for (const [index, agentId] of roster.filter((id) => !chosen.valid.includes(id)).entries()) { + ;(this.daemon as any).handleRelayMsg( + this.rd({ op: 'context', post: userPost }, { agentId, msgId: `turn-${seq}-c${index}` }), + () => {} + ) + } + // The host's own post is part of the conversation view too. + this.posts.push({ conversationId: this.conversationId, agentId: 'host', post: userPost } as RdWebchatPost) + return { postId } + } + + events(): readonly EvaluationEvent[] { + return this.collector.events() + } + + eventCollector(): EvaluationEventCollector { + return this.collector + } + + /** Committed conversation posts authored by agents (the browser's view minus + * the host's own posts). */ + agentPosts(): RdWebchatPost[] { + return this.posts.filter((post) => post.agentId !== 'host') + } + + /** Shared-conversation transcript rows (what a peer's context refresh reads). */ + transcriptRows(): { sender: string; text: string }[] { + return (this.daemon as any).store.transcriptSince( + transcriptChannelKey(this.conversationId, undefined), + `webchat:${this.conversationId}`, + null + ) as { sender: string; text: string }[] + } + + /** + * Settle: wait until the daemon reports evaluation idleness AND no new + * evaluation events have landed for `quietMs`. Cascades (fan-out wakes, MCP + * calls mid-turn, reply wakes) all surface as events, so a quiet window on + * top of the idle barrier is what "the night has fully drained" means here. + */ + async settle(options: { quietMs?: number; timeoutMs?: number } = {}): Promise { + const quietMs = options.quietMs ?? 700 + const timeoutMs = options.timeoutMs ?? 60_000 + const deadline = Date.now() + timeoutMs + let lastCount = -1 + let quietSince = Date.now() + while (Date.now() < deadline) { + await this.daemon.waitForEvaluationIdle(Math.max(1_000, deadline - Date.now())) + const count = this.collector.events().length + if (count !== lastCount) { + lastCount = count + quietSince = Date.now() + } else if (Date.now() - quietSince >= quietMs) { + return + } + await new Promise((resolveSleep) => setTimeout(resolveSleep, 100)) + } + throw new Error(`webchat arena did not settle within ${timeoutMs}ms`) + } + + /** Like {@link settle} but never throws — a real-model run that stalls is a + * RESULT, not an error. Returns whether it settled. */ + async settleOrStall(options: { quietMs?: number; timeoutMs?: number } = {}): Promise { + try { + await this.settle(options) + return true + } catch { + return false + } + } + + async stop(): Promise { + if (!this.started) return + this.started = false + await this.daemon.stop() + } +} + +// ── reply-wake evidence ───────────────────────────────────────────────────── + +const BARE_UUID = /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/ + +/** + * The daemon-side admission evidence for `sendMessage {sessionId}` reply wakes + * into one agent's sessions — what lets a scorer tell "the reply's wake was + * admitted (and either started a turn or was coalesced into one)" apart from + * "the reply's body merely became VISIBLE somewhere" (which, under #926, any + * committed public copy can do via a later, unrelated context refresh). + * + * The discriminator: a reply wake's evaluation `turnId` is + * `:` with a BARE-UUID deliveryId and `source: 'agent'` + * at admission. The other webchat wake shapes cannot collide — a §5.2a + * continuation wake's id is `#` (never a bare UUID), a host + * user turn is `source: 'user'`, and a `messageAgent` call's deliveryId is a + * monotonic timestamp. Validated against real-run artifacts: the stage-2 + * Werewolf game shows exactly one accepted id per answered needsReply call, + * and the all-lost night-collection trials show zero. + */ +export interface ReplyWakeEvidence { + /** Reply-wake deliveryIds ADMITTED into the agent's sessions. */ + accepted: Set + /** Turn inputs of reply wakes that STARTED their own turn, by deliveryId. */ + startedInputs: Map + /** Reply-wake deliveryIds whose queued activation was COALESCED into an + * in-flight turn (the turn's refreshed input represents the reply). */ + coalesced: Set +} + +export function agentReplyWakeEvidence(events: readonly EvaluationEvent[], agentId: string): ReplyWakeEvidence { + const suffixOf = (event: EvaluationEvent): string | undefined => { + if (event.agentId !== agentId || typeof event.turnId !== 'string') return undefined + const prefix = `${agentId}:` + if (!event.turnId.startsWith(prefix)) return undefined + const suffix = event.turnId.slice(prefix.length) + return BARE_UUID.test(suffix) ? suffix : undefined + } + const accepted = new Set() + for (const event of events) { + if (event.type !== 'turn.accepted' || event.data.source !== 'agent') continue + const suffix = suffixOf(event) + if (suffix !== undefined) accepted.add(suffix) + } + const startedInputs = new Map() + const coalesced = new Set() + for (const event of events) { + const suffix = suffixOf(event) + if (suffix === undefined || !accepted.has(suffix)) continue + if (event.type === 'turn.started') startedInputs.set(suffix, String(event.data.input ?? '')) + if (event.type === 'turn.cancelled' && event.data.reason === 'coalesced_into_turn') coalesced.add(suffix) + } + return { accepted, startedInputs, coalesced } +} diff --git a/evals/games/webchat-referee.ts b/evals/games/webchat-referee.ts new file mode 100644 index 000000000..57da67466 --- /dev/null +++ b/evals/games/webchat-referee.ts @@ -0,0 +1,157 @@ +/** + * The scripted-subject seam for webchat scenarios: a deterministic BRAIN acting + * through the REAL tool surface. + * + * The point of the webchat leg (night-collection + webchat Werewolf) is that + * the referee is an ordinary subject agent — its calls are `sendMessage` + * (`toAgent`+`needsReply`, `sessionId` replies) through the daemon's MCP + * control socket, never the trusted `deliverRefereeEvent` control path the + * Slack-shaped games use. Eval composition == live composition, except the + * referee's brain is deterministic. + * + * One brain drives BOTH subject kinds: + * - scripted CI: an in-process ACP host (`brainHostEntry`) wraps the brain; + * - real runs: the puppet ACP adapter (`puppet-acp-agent.mjs`) forwards each + * prompt to the driver (`puppet.ts`), which runs the same brain and makes + * the same control-socket calls. + */ +import { callDaemonTool, daemonMcpBinding, type DaemonMcpBinding } from './mcp-client.js' + +/** One product tool call the brain wants executed from its CURRENT session. */ +export interface BrainCall { + tool: string + args: Record +} + +export interface BrainCallOutcome extends BrainCall { + ok: boolean + /** The tool result (JSON text payload, parsed when possible). */ + result?: unknown + error?: string +} + +/** What one prompt produced: tool calls to execute (in order, awaited), then a + * single reply chunk. Reply `AC_NO_RESPONSE` to stay silent. */ +export interface BrainTurn { + calls: BrainCall[] + reply: string +} + +/** A deterministic scripted subject: pure decision over (prompt text, own + * state). Implementations keep their own state — one brain instance per run. */ +export interface ScriptedBrain { + onPrompt(text: string): BrainTurn + /** Called with each executed call's outcome, in order. */ + onCallResult?(outcome: BrainCallOutcome): void +} + +export type CallTool = ( + binding: DaemonMcpBinding, + tool: string, + args: Record +) => Promise<{ ok: boolean; result?: unknown; error?: string }> + +/** Execute one brain turn against a session's captured control-socket binding. */ +export async function executeBrainTurn( + brain: ScriptedBrain, + binding: DaemonMcpBinding | undefined, + text: string, + callTool: CallTool = callDaemonTool +): Promise<{ reply: string; outcomes: BrainCallOutcome[] }> { + const turn = brain.onPrompt(text) + const outcomes: BrainCallOutcome[] = [] + for (const call of turn.calls) { + if (!binding) { + const outcome: BrainCallOutcome = { ...call, ok: false, error: 'session has no daemon tool binding' } + outcomes.push(outcome) + brain.onCallResult?.(outcome) + continue + } + const result = await callTool(binding, call.tool, call.args) + const outcome: BrainCallOutcome = { + ...call, + ok: result.ok, + ...(result.result !== undefined ? { result: result.result } : {}), + ...(result.error !== undefined ? { error: result.error } : {}) + } + outcomes.push(outcome) + brain.onCallResult?.(outcome) + } + return { reply: turn.reply, outcomes } +} + +/** Parse the parent-session reply target out of a needsReply child's prompt + * (the standing report-back directive carries the exact JSON to send). */ +export function parentSessionIdOf(text: string): string | undefined { + const meta = /^- Parent session: (\S+)$/m.exec(text) + if (meta) return meta[1] + // The standing collaboration guidance carries a PLACEHOLDER example + // (`{"sessionId":"", …}`); only the report-back directive + // carries the real id. Skip placeholder-shaped values. + for (const match of text.matchAll(/"sessionId":"([^"]+)"/g)) { + if (!match[1]!.startsWith('<')) return match[1] + } + return undefined +} + +/** Prompt log entry recorded by the in-process hosts: which agent, which ACP + * session, and the full prompt text of one `session/prompt` call (a turn may + * log several — one per regeneration). */ +export interface PromptLogEntry { + agentId: string + sessionId: string + text: string +} + +export interface InProcessHost { + factory: (agent: { id: string; name?: string }, onUpdate: (sessionId: string, update: unknown) => void) => unknown +} + +export type ScriptedSessionHandler = (input: { + agentId: string + sessionId: string + text: string + binding: DaemonMcpBinding | undefined + chunk: (text: string) => void +}) => Promise | string | undefined + +/** + * Build a per-agent in-process ACP host from a session handler. The handler + * returns the reply text (or undefined ⇒ AC_NO_RESPONSE) and may await real + * control-socket calls; every prompt is appended to `log` first. + */ +export function scriptedWebchatHostFactory( + handlers: Map, + log: PromptLogEntry[] +): NonNullable { + return (agent, onUpdate) => { + let sessions = 0 + const bindings = new Map() + const handler = handlers.get(agent.id) + return { + start: async () => {}, + newSession: async (_cwd: string, mcpServers?: unknown) => { + const sessionId = `scripted-${agent.id.slice(0, 8)}-${(sessions += 1)}` + const binding = daemonMcpBinding(mcpServers) + if (binding) bindings.set(sessionId, binding) + return sessionId + }, + hasSession: () => true, + modelOptions: () => ({ current: 'scripted-webchat', models: ['scripted-webchat'] }), + prompt: async (sessionId: string, blocks: { text?: string }[]) => { + const text = blocks.map((block) => block.text ?? '').join('\n') + log.push({ agentId: agent.id, sessionId, text }) + const chunk = (value: string) => + onUpdate(sessionId, { sessionUpdate: 'agent_message_chunk', content: { type: 'text', text: value } }) + let reply: string | undefined + if (handler) { + reply = await handler({ agentId: agent.id, sessionId, text, binding: bindings.get(sessionId), chunk }) + } + chunk(reply ?? 'AC_NO_RESPONSE') + return { stopReason: 'end_turn' } + }, + cancel: async () => {}, + stop: async () => {} + } + } +} diff --git a/evals/games/webchat-werewolf-runner.ts b/evals/games/webchat-werewolf-runner.ts new file mode 100644 index 000000000..895433742 --- /dev/null +++ b/evals/games/webchat-werewolf-runner.ts @@ -0,0 +1,254 @@ +/** + * Runner for webchat Werewolf (`webchat-werewolf.ts`): boots the webchat + * arena, plays the human HOST (kickoff + the night cues the referee asks + * for), lets the wake-driven referee brain and the players carry everything + * else, and assembles the verdict from the daemon's own records. + * + * The host loop is REACTIVE, mirroring the live runbook: the host posts the + * kickoff, then only ever answers the referee's explicit public asks + * ("HOST: please open night N."). Each cue is a trusted human turn, which is + * also what keeps every round's continuation chain inside the webchat hop + * budget (the same reason the live game's host paces the phases). + * + * A run that stops advancing is an HONEST STALL, not an error: with PR #905 + * parked, a real child answering its needsReply delegation in prose never + * wakes the referee again — the runner records where the game stood + * (`stalledAt`) and which needsReply calls never came back (`replyLoss`). + */ +import { mkdirSync } from 'node:fs' +import { join } from 'node:path' +import { atomicWrite, redactEvaluationValue } from '../../packages/daemon/src/evaluation/index.js' +import { callDaemonTool } from './mcp-client.js' +import { PuppetDriver } from './puppet.js' +import { preflightRealSubject } from './subject.js' +import { + WebchatArena, + agentReplyWakeEvidence, + mintSeats, + prepareRealWebchatRoot, + prepareScriptedWebchatRoot, + type WebchatSeat +} from './webchat-fixture.js' +import { executeBrainTurn, parentSessionIdOf, scriptedWebchatHostFactory } from './webchat-referee.js' +import type { PromptLogEntry, ScriptedSessionHandler } from './webchat-referee.js' +import { + GAME_OVER_PATTERN, + HOST_CUE_PATTERN, + ROUND_LIMIT_PATTERN, + WebchatWerewolfReferee, + hostKickoffText, + hostNightCueText, + scriptedWebchatPlayer, + type NeedsReplyLogRow, + type WebchatDayRecord, + type WebchatNightRecord +} from './webchat-werewolf.js' +import type { WerewolfRole } from './werewolf-rules.js' + +export interface WebchatWerewolfRunOptions { + seed: number + playerCount?: number + maxRounds?: number + subject?: { kind: 'scripted' } | { kind: 'real'; subjectRoot: string; templateAgentIds: string[] } + /** Whole-run budget. Scripted default 180s; real default 30min. */ + budgetMs?: number + /** Quiet window the settle barrier uses between host-loop passes. */ + quietMs?: number + artifactDir?: string +} + +export interface WebchatWerewolfRunResult { + terminalReason: 'completed' | 'round_limit' | 'stalled' | 'budget_exhausted' + winner?: 'village' | 'werewolves' + rounds: number + roles: Record + survivors: string[] + nights: WebchatNightRecord[] + days: WebchatDayRecord[] + /** Every needsReply call the referee issued, with whether an answer ever + * came back — the pending rows of a finished run ARE the reply losses. */ + replyLoss: (Omit & { to: string })[] + /** Admitted `sendMessage {sessionId}` reply wakes at the referee — the + * daemon-side ground truth for `answered` rows. The brain also absorbs + * #926 public-copy CONTEXT rows (a model referee reads the room too), so + * the CI gate asserts this count equals the answered rows: an `answered` + * verdict fed only by a context echo of a DROPPED wake cannot pass. */ + replyWakesAccepted: number + /** Of those, wakes coalesced into an in-flight referee turn. */ + replyWakesCoalesced: number + /** Canary strings observed in the shared conversation (posts or transcript). + * Must be zero: canaries ride only the private role calls. */ + canaryLeaks: number + /** Committed conversation posts whose text is a private needsReply REPORT + * body (role acks / night statements) — the #926 surface: on current main + * a child's reply into a conversation-origin parent session is posted live + * into the conversation view, so webchat "private" night traffic is + * visible to the whole room. Measured, not failed. */ + privateReportsPostedPublicly: number + stalledAt?: string + posts: { author: string; text: string }[] +} + +const NIGHT_REPORT_SHAPES = [ + /^ROLE-ACK: player-\d+/, + /\bkill player-\d+\b|\bkill\b.*\btonight\b/i, + /^I agree\./, + /\bI inspect player-\d+/i, + /\bI protect player-\d+/i +] + +export async function runWebchatWerewolf(options: WebchatWerewolfRunOptions): Promise { + const playerCount = options.playerCount ?? 5 + const subjectSpec = options.subject ?? { kind: 'scripted' as const } + const budgetMs = options.budgetMs ?? (subjectSpec.kind === 'scripted' ? 180_000 : 30 * 60_000) + const quietMs = options.quietMs ?? (subjectSpec.kind === 'scripted' ? 900 : 15_000) + const playerAliases = Array.from({ length: playerCount }, (_, index) => `player-${index + 1}`) + const seats = mintSeats(['referee', ...playerAliases]) + const refereeSeat = seats[0]! + const playerSeats = seats.slice(1) + const aliasOf = (agentId: string) => seats.find((seat) => seat.agentId === agentId)?.alias ?? agentId + const brain = new WebchatWerewolfReferee({ + seed: options.seed, + players: playerSeats, + ...(options.maxRounds !== undefined ? { maxRounds: options.maxRounds } : {}) + }) + + const log: PromptLogEntry[] = [] + let driver: PuppetDriver | undefined + let subject: { root: string; secrets: string[]; cleanup: () => void } + let hostFactory: ReturnType | undefined + if (subjectSpec.kind === 'scripted') { + const handlers = new Map() + handlers.set(refereeSeat.agentId, async ({ text, binding }) => { + const { reply } = await executeBrainTurn(brain, binding, text) + return reply + }) + for (const seat of playerSeats) { + const player = scriptedWebchatPlayer({ + alias: seat.alias, + callTool: callDaemonTool, + parentSessionIdOf + }) + handlers.set(seat.agentId, ({ sessionId, text, binding }) => player({ sessionId, text, binding })) + } + hostFactory = scriptedWebchatHostFactory(handlers, log) + subject = prepareScriptedWebchatRoot(seats) + } else { + driver = new PuppetDriver() + await driver.start() + driver.useBrain(brain) + subject = prepareRealWebchatRoot(seats, { + subjectRoot: subjectSpec.subjectRoot, + templateAgentIds: subjectSpec.templateAgentIds, + refereeAlias: 'referee', + puppetEndpoint: driver.endpoint + }) + } + + const arena = new WebchatArena({ + root: subject.root, + seats, + secrets: subject.secrets, + ...(hostFactory ? { hostFactory: hostFactory as never } : {}) + }) + const deadline = Date.now() + budgetMs + let terminalReason: WebchatWerewolfRunResult['terminalReason'] = 'stalled' + let transcriptTexts: string[] = [] + try { + if (subjectSpec.kind === 'real') await preflightRealSubject(subject.root) + await arena.start() + arena.postHost(hostKickoffText(), { mentions: ['referee'] }) + + const answeredCues = new Set() + let lastProgressPosts = -1 + while (Date.now() < deadline) { + const settled = await arena.settleOrStall({ + quietMs, + timeoutMs: Math.max(5_000, Math.min(deadline - Date.now(), budgetMs)) + }) + // React to the referee's public asks, newest state first. + const refereePosts = arena.posts.filter((post) => post.agentId === refereeSeat.agentId) + const over = refereePosts.some( + (post) => GAME_OVER_PATTERN.test(post.post.text) || ROUND_LIMIT_PATTERN.test(post.post.text) + ) + if (over || brain.phase === 'done') { + terminalReason = brain.terminalReason ?? 'stalled' + break + } + let acted = false + for (const post of refereePosts) { + const cue = HOST_CUE_PATTERN.exec(post.post.text) + if (cue) { + const round = Number(cue[1]) + if (!answeredCues.has(round)) { + answeredCues.add(round) + arena.postHost(hostNightCueText(round), { mentions: ['referee'] }) + acted = true + } + } + } + if (acted) continue + if (settled) { + // Fully drained, nothing to answer, game not done: the honest stall. + if (arena.posts.length === lastProgressPosts) break + lastProgressPosts = arena.posts.length + // One grace pass: a wave may have landed between settle and the scan. + continue + } + } + if (brain.phase !== 'done' && terminalReason === 'stalled' && Date.now() >= deadline) { + terminalReason = 'budget_exhausted' + } + transcriptTexts = arena.transcriptRows().map((row) => row.text) + } finally { + await arena.stop().catch(() => {}) + await driver?.stop().catch(() => {}) + } + + const posts = arena.posts.map((post) => ({ author: aliasOf(post.agentId), text: post.post.text })) + const wakeEvidence = agentReplyWakeEvidence(arena.events(), refereeSeat.agentId) + const canaryLeaks = [...posts.map((post) => post.text), ...transcriptTexts].filter( + (text) => text.includes(brain.canaries.wolf) || text.includes(brain.canaries.seer) + ).length + const privateReportsPostedPublicly = posts.filter( + (post) => post.author.startsWith('player-') && NIGHT_REPORT_SHAPES.some((pattern) => pattern.test(post.text.trim())) + ).length + + const result: WebchatWerewolfRunResult = { + terminalReason, + ...(brain.winner !== undefined ? { winner: brain.winner } : {}), + rounds: brain.round, + roles: Object.fromEntries(brain.roles), + survivors: brain.aliveAliases(), + nights: brain.nights, + days: brain.days, + replyLoss: brain.needsReplyLog.map((row) => ({ ...row, to: aliasOf(row.to) })), + replyWakesAccepted: wakeEvidence.accepted.size, + replyWakesCoalesced: wakeEvidence.coalesced.size, + canaryLeaks, + privateReportsPostedPublicly, + ...(terminalReason === 'stalled' || terminalReason === 'budget_exhausted' ? { stalledAt: brain.stallState() } : {}), + posts + } + + if (options.artifactDir) { + mkdirSync(options.artifactDir, { recursive: true, mode: 0o700 }) + arena.eventCollector().writeJsonl(join(options.artifactDir, 'events.jsonl')) + atomicWrite( + join(options.artifactDir, 'game-result.json'), + `${JSON.stringify(redactEvaluationValue(result, subject.secrets), null, 2)}\n` + ) + const refereePrompts = + subjectSpec.kind === 'scripted' + ? log.filter((entry) => entry.agentId === refereeSeat.agentId).map((entry) => entry.text) + : (driver?.promptLog.map((entry) => entry.text) ?? []) + atomicWrite( + join(options.artifactDir, 'referee-prompts.json'), + `${JSON.stringify(redactEvaluationValue(refereePrompts, subject.secrets), null, 2)}\n` + ) + } + subject.cleanup() + return result +} + +export type { WebchatSeat } diff --git a/evals/games/webchat-werewolf.ts b/evals/games/webchat-werewolf.ts new file mode 100644 index 000000000..50fce362c --- /dev/null +++ b/evals/games/webchat-werewolf.ts @@ -0,0 +1,775 @@ +/** + * Webchat Werewolf — the whole game mapped onto ONE multi-agent webchat + * conversation plus postless `needsReply` private legs, per the live runbook + * topology: + * + * - public day speech and votes are ordinary conversation posts (the #906 + * continuation carries the sequential speaking order); + * - role delivery and night actions are postless `toAgent + needsReply` + * calls from the referee's CONVERSATION session; + * - the night kill is referee-MEDIATED: the referee relays the wolf lead's + * proposal to the second wolf privately (no wolf den room exists); + * - the referee is a SCRIPTED subject agent acting through the REAL tool + * surface (`sendMessage`/`needsReply`) — never the trusted + * `deliverRefereeEvent` control path. Eval composition == live composition + * except the referee's brain is deterministic. + * + * The Slack-shaped Werewolf (`werewolf.ts`) stays untouched — it pins the + * other composition. Rules, roles, and win logic are shared via + * `werewolf-rules.ts`. + * + * The referee brain is WAKE-DRIVEN only (like a model referee, it has no + * timers): every transition rides a wake — the host's kickoff/night cues, + * needsReply child replies into its session, and the public continuation + * wakes of player posts. With PR #905 parked, a real child that answers its + * needsReply delegation in PROSE never wakes the referee again and the night + * stalls — an HONEST STALL is a valid result and exactly the pre-#905 + * baseline; the runner records where it happened instead of failing. + */ +import { HOST_USER, NO_RESPONSE, type WebchatSeat } from './webchat-fixture.js' +import { + assignWerewolfRoles, + parseStatedTarget, + werewolfCanaries, + werewolfWinner, + type WerewolfRole +} from './werewolf-rules.js' +import type { BrainCallOutcome, BrainTurn, ScriptedBrain } from './webchat-referee.js' + +export type WebchatWerewolfPhase = 'setup' | 'awaiting-night-cue' | 'night' | 'day-discussion' | 'day-vote' | 'done' + +export interface NeedsReplyLogRow { + round: number + purpose: 'role' | 'kill-proposal' | 'kill-verdict' | 'inspect' | 'protect' + to: string + delivered: boolean + /** Whether ANY wake ever brought this call's answer back. Pending rows on a + * finished run are the measured reply losses. */ + answered: boolean + error?: string +} + +export interface WebchatNightRecord { + round: number + wolfLead?: string + proposal?: string + verdict?: 'agreed' | 'countered' | 'unparseable' + kill?: string + protect?: string + inspect?: string + inspectResult?: 'werewolf' | 'not-werewolf' + death?: string + saved: boolean +} + +export interface WebchatDayRecord { + round: number + order: string[] + spoke: string[] + votes: Record + lynched?: string + revealed?: WerewolfRole +} + +/** The public asks the runner (playing the human HOST) reacts to. */ +export const HOST_CUE_PATTERN = /HOST: please open night (\d+)\./ +export const GAME_OVER_PATTERN = /The game is over: the (village|werewolves) win\./ +export const ROUND_LIMIT_PATTERN = /Round limit reached\./ + +export interface WebchatWerewolfRefereeOptions { + seed: number + players: readonly WebchatSeat[] + maxRounds?: number +} + +const BRACKET_LINE = /^\[([^\]\n]+)\]\s?(.*)$/ +const UUID_RE = /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/ + +export class WebchatWerewolfReferee implements ScriptedBrain { + readonly roles: Map + readonly canaries: { wolf: string; seer: string } + readonly needsReplyLog: NeedsReplyLogRow[] = [] + readonly nights: WebchatNightRecord[] = [] + readonly days: WebchatDayRecord[] = [] + phase: WebchatWerewolfPhase = 'setup' + round = 0 + winner: 'village' | 'werewolves' | undefined + terminalReason: 'completed' | 'round_limit' | undefined + private readonly alive = new Set() + private readonly aliasById = new Map() + private readonly seatByAlias = new Map() + private readonly maxRounds: number + private readonly roleAcks = new Set() + private night: WebchatNightRecord | undefined + private day: WebchatDayRecord | undefined + private pendingVerdict = false + /** Rows of the log still awaiting an answer, matched by purpose. */ + private readonly awaiting = new Map() + + constructor(private readonly options: WebchatWerewolfRefereeOptions) { + const aliases = options.players.map((seat) => seat.alias) + this.roles = assignWerewolfRoles(aliases, options.seed) + this.canaries = werewolfCanaries(options.seed) + this.maxRounds = options.maxRounds ?? 6 + for (const seat of options.players) { + this.alive.add(seat.alias) + this.aliasById.set(seat.agentId, seat.alias) + this.seatByAlias.set(seat.alias, seat) + } + } + + aliveAliases(): string[] { + return this.options.players.map((seat) => seat.alias).filter((alias) => this.alive.has(alias)) + } + + roleOf(alias: string): WerewolfRole | undefined { + return this.roles.get(alias) + } + + /** Where a stalled run stood — the honest-stall report. */ + stallState(): string { + const pending = [...this.awaiting.values()].map((row) => `${row.purpose}→${this.aliasById.get(row.to) ?? row.to}`) + return `phase=${this.phase} round=${this.round}${pending.length ? ` awaiting ${pending.join(', ')}` : ''}${ + this.day ? ` spoke=${this.day.spoke.join(',')} votes=${Object.keys(this.day.votes).join(',')}` : '' + }` + } + + onPrompt(text: string): BrainTurn { + const calls: BrainTurn['calls'] = [] + const replies: string[] = [] + + // ── needsReply answers. A reply that wakes the referee directly is + // DELIVERED as raw, unbracketed message text (with no sender label — the + // parent cannot attribute a raw reply). A reply whose wake was COALESCED + // into an in-flight referee turn is represented as a bracketed context row + // instead (`[] …`, the #926 public copy) — which the brain must + // absorb too, or a coalesced night answer is silently missed; those rows + // do carry the sender. + const privateItems: { sender?: string; content: string }[] = [] + for (const line of text.split('\n')) { + const bracketed = BRACKET_LINE.exec(line) + if (bracketed) { + const sender = bracketed[1]! + if (UUID_RE.test(sender)) { + const alias = this.aliasById.get(sender) + if (alias !== undefined) privateItems.push({ sender: alias, content: bracketed[2]! }) + } + } else if (!line.startsWith('- ') && !line.startsWith('#') && !line.startsWith('(') && !line.startsWith('<')) { + privateItems.push({ content: line }) + } + } + this.absorbRoleAcks(privateItems) + if (this.phase === 'night') this.absorbNightAnswers(privateItems, calls, replies) + + // ── public conversation content: host cues, speeches, votes ── + for (const line of text.split('\n')) { + const bracketed = BRACKET_LINE.exec(line) + const sender = bracketed?.[1] + const content = bracketed ? bracketed[2]! : line + const uuidSender = sender !== undefined && UUID_RE.test(sender) + const senderAlias = uuidSender ? this.aliasById.get(sender) : undefined + if (uuidSender && senderAlias === undefined) continue // our own echoed post + // Human/host content: the raw delivered user text, or a `[]` + // conversation row (never a uuid sender). + const humanLine = !uuidSender + if (humanLine && this.phase === 'setup' && /begin the werewolf game/i.test(content)) { + this.beginGame(calls, replies) + continue + } + const nightCue = /(?:^|\s)NIGHT (\d+) begins\./.exec(content) + if (nightCue && humanLine && this.phase === 'awaiting-night-cue') { + this.openNight(Number(nightCue[1]), calls) + continue + } + if (!senderAlias || !this.alive.has(senderAlias)) continue + if (this.phase === 'day-discussion' && this.day) { + // Only a SELF-PREFIXED post counts as speech ("player-2: …", the form + // the day instructions mandate, with the prefix matching the verified + // sender). Content-agnostic counting would mis-read the #926 public + // copies of night replies — which surface as conversation rows from + // the same players — as day speeches. + const prefixed = /^([a-z0-9-]+):\s/.exec(content) + if (prefixed?.[1] === senderAlias && !this.day.spoke.includes(senderAlias)) { + this.day.spoke.push(senderAlias) + } + if (this.day.order.every((alias) => this.day!.spoke.includes(alias))) { + this.day.spoke = [...this.day.spoke] + this.phase = 'day-vote' + replies.push( + `VOTE ${this.round}. Discussion is closed. Living players: ${this.aliveAliases().join(', ')}. ` + + `Every living player now says their vote out loud in this conversation, exactly once — for example ` + + `"${this.aliveAliases()[0]}: I vote for player-2". Name exactly one living player.` + ) + } + continue + } + if (this.phase === 'day-vote' && this.day) { + const intent = parseStatedTarget(content, 'vote') + if (intent.kind === 'target' && this.day.votes[senderAlias] === undefined && this.alive.has(intent.target)) { + this.day.votes[senderAlias] = intent.target + } + if (this.aliveAliases().every((alias) => this.day!.votes[alias] !== undefined)) { + this.resolveDay(replies) + } + } + } + + const reply = replies.length > 0 ? replies.join('\n\n') : NO_RESPONSE + return { calls, reply } + } + + onCallResult(outcome: BrainCallOutcome): void { + const toAgentId = + typeof outcome.args.toAgent === 'string' + ? outcome.args.toAgent + : (outcome.args.toAgent as { agentId?: string } | undefined)?.agentId + if (!toAgentId) return + const row = this.needsReplyLog.find( + (candidate) => candidate.to === toAgentId && !candidate.delivered && !candidate.error + ) + if (!row) return + const parsed = parseDelivered(outcome) + row.delivered = outcome.ok && parsed !== false + if (!outcome.ok && outcome.error !== undefined) row.error = outcome.error + } + + // ── phase transitions ───────────────────────────────────────────────────── + + private beginGame(calls: BrainTurn['calls'], replies: string[]): void { + this.phase = 'awaiting-night-cue' + for (const seat of this.options.players) { + calls.push(this.needsReply(seat, 'role', 0, this.roleMessage(seat.alias))) + } + replies.push( + `Werewolf begins with ${this.options.players.length} players: ${[...this.roles.keys()].join(', ')}. ` + + `Roles arrive privately — never reveal private referee content in this conversation. Say nothing here ` + + `until I open a day and give you the speaking order. HOST: please open night 1.` + ) + } + + private roleMessage(alias: string): string { + const role = this.roles.get(alias)! + const ack = + `First, report back to my session with a single line: "ROLE-ACK: ${alias}". ` + + `Then wait — I will contact you again here when I need your night action.` + const base = `Werewolf role assignment. Your alias: ${alias}. Your role: ${role}.` + if (role === 'werewolf') { + const partner = [...this.roles.entries()].find(([other, r]) => r === 'werewolf' && other !== alias)?.[0] + return ( + `${base} Your fellow wolf: ${partner ?? 'none'}. I mediate the pack privately — never coordinate in ` + + `the conversation. (private canary: ${this.canaries.wolf} — never repeat it anywhere.) ${ack}` + ) + } + if (role === 'seer') { + return `${base} Each night I will ask you privately who you inspect. (private canary: ${this.canaries.seer} — never repeat it anywhere.) ${ack}` + } + if (role === 'doctor') return `${base} Each night I will ask you privately who you protect. ${ack}` + return `${base} Sleep at night, discuss and vote by day. ${ack}` + } + + private openNight(round: number, calls: BrainTurn['calls']): void { + if (round !== this.round + 1) return + this.round = round + if (this.round > this.maxRounds) { + this.phase = 'done' + this.terminalReason = 'round_limit' + return + } + this.phase = 'night' + this.pendingVerdict = false + const living = this.aliveAliases() + const wolves = living.filter((alias) => this.roles.get(alias) === 'werewolf') + const wolfTargets = living.filter((alias) => this.roles.get(alias) !== 'werewolf') + this.night = { round: this.round, saved: false } + if (wolves.length > 0) { + const lead = wolves[0]! + this.night.wolfLead = lead + calls.push( + this.needsReply( + this.seatByAlias.get(lead)!, + 'kill-proposal', + this.round, + `NIGHT ${this.round}. You are the pack lead tonight. Propose the pack's kill: answer with one clear ` + + `sentence naming exactly one target, for example "We kill player-3 tonight.". ` + + `Targets: ${wolfTargets.join(', ')}. I will relay your proposal to your fellow wolf for agreement.` + ) + ) + } + const seer = living.find((alias) => this.roles.get(alias) === 'seer') + if (seer) { + calls.push( + this.needsReply( + this.seatByAlias.get(seer)!, + 'inspect', + this.round, + `NIGHT ${this.round}. Name the ONE living player you inspect tonight, for example "I inspect player-3.". ` + + `Living: ${living.filter((alias) => alias !== seer).join(', ')}.` + ) + ) + } + const doctor = living.find((alias) => this.roles.get(alias) === 'doctor') + if (doctor) { + calls.push( + this.needsReply( + this.seatByAlias.get(doctor)!, + 'protect', + this.round, + `NIGHT ${this.round}. Name the ONE living player you protect tonight, for example "I protect player-3.". ` + + `Living: ${living.join(', ')}.` + ) + ) + } + } + + private absorbRoleAcks(items: { sender?: string; content: string }[]): void { + for (const item of items) { + const match = /ROLE-ACK: (player-\d+)/.exec(item.content) + if (!match) continue + const alias = match[1]! + if (this.roleAcks.has(alias)) continue + this.roleAcks.add(alias) + const row = this.needsReplyLog.find( + (candidate) => candidate.purpose === 'role' && candidate.to === this.seatByAlias.get(alias)?.agentId + ) + if (row) row.answered = true + } + } + + private absorbNightAnswers( + items: { sender?: string; content: string }[], + calls: BrainTurn['calls'], + replies: string[] + ): void { + const night = this.night + if (!night) return + const seer = this.aliveAliases().find((alias) => this.roles.get(alias) === 'seer') + const doctor = this.aliveAliases().find((alias) => this.roles.get(alias) === 'doctor') + for (const item of items) { + const { sender, content } = item + // Verdict first: the second wolf's answer may itself contain a kill verb. + if (this.pendingVerdict && night.kill === undefined && (sender === undefined || sender === this.verdictWolf)) { + const counter = parseStatedTarget(content, 'kill') + if (/\bagree/i.test(content)) { + night.verdict = 'agreed' + night.kill = night.proposal + } else if ( + counter.kind === 'target' && + this.roles.get(counter.target) !== 'werewolf' && + this.alive.has(counter.target) + ) { + night.verdict = 'countered' + night.kill = counter.target + } else if (counter.kind !== 'none' || /\bcounter/i.test(content)) { + night.verdict = 'unparseable' + night.kill = night.proposal + } + if (night.kill !== undefined) { + this.pendingVerdict = false + this.settleAwaiting('kill-verdict') + continue + } + } + if ( + !this.pendingVerdict && + night.proposal === undefined && + night.kill === undefined && + (sender === undefined || sender === night.wolfLead) + ) { + const proposal = parseStatedTarget(content, 'kill') + if ( + proposal.kind === 'target' && + this.roles.get(proposal.target) !== 'werewolf' && + this.alive.has(proposal.target) + ) { + night.proposal = proposal.target + this.settleAwaiting('kill-proposal') + const wolves = this.aliveAliases().filter((alias) => this.roles.get(alias) === 'werewolf') + const partner = wolves.find((alias) => alias !== night.wolfLead) + if (partner) { + this.pendingVerdict = true + this.verdictWolf = partner + calls.push( + this.needsReply( + this.seatByAlias.get(partner)!, + 'kill-verdict', + this.round, + `NIGHT ${this.round}. Your fellow wolf ${night.wolfLead} proposes to kill ${night.proposal} tonight. ` + + `Do you agree, or counter? Answer with "I agree." or one clear counter sentence naming exactly one ` + + `target, for example "Counter: we kill player-4 tonight.".` + ) + ) + } else { + night.kill = night.proposal + } + continue + } + } + if (night.inspect === undefined && (sender === undefined || sender === seer)) { + const inspect = parseStatedTarget(content, 'inspect') + if (inspect.kind === 'target' && this.alive.has(inspect.target)) { + night.inspect = inspect.target + this.settleAwaiting('inspect') + continue + } + } + if (night.protect === undefined && (sender === undefined || sender === doctor)) { + const protect = parseStatedTarget(content, 'protect') + if (protect.kind === 'target' && this.alive.has(protect.target)) { + night.protect = protect.target + this.settleAwaiting('protect') + } + } + } + this.maybeResolveNight(calls, replies) + } + + private verdictWolf: string | undefined + + private maybeResolveNight(calls: BrainTurn['calls'], replies: string[]): void { + const night = this.night + if (!night || this.phase !== 'night') return + const living = this.aliveAliases() + const needsKill = living.some((alias) => this.roles.get(alias) === 'werewolf') + const needsInspect = living.some((alias) => this.roles.get(alias) === 'seer') + const needsProtect = living.some((alias) => this.roles.get(alias) === 'doctor') + if (needsKill && night.kill === undefined) return + if (needsInspect && night.inspect === undefined) return + if (needsProtect && night.protect === undefined) return + + const saved = night.kill !== undefined && night.protect === night.kill + night.saved = saved + let deathLine = 'No one died last night.' + if (night.kill !== undefined && !saved) { + night.death = night.kill + this.alive.delete(night.kill) + deathLine = `${night.kill} was killed last night.` + } else if (saved) { + deathLine = 'The doctor saved a life last night — no one died.' + } + if (night.inspect !== undefined) { + night.inspectResult = this.roles.get(night.inspect) === 'werewolf' ? 'werewolf' : 'not-werewolf' + const seer = this.aliveAliases().find((alias) => this.roles.get(alias) === 'seer') + if (seer) { + // Fire-and-forget private result — a postless wake into the seer's + // pairwise session; nothing returns and nothing is posted. + calls.push({ + tool: 'sendMessage', + args: { + toAgent: this.seatByAlias.get(seer)!.agentId, + message: `Inspection result: ${night.inspect} is ${night.inspectResult === 'werewolf' ? 'a werewolf' : 'not a werewolf'}. Keep it private until you can use it.` + } + }) + } + } + this.nights.push(night) + this.night = undefined + const winner = werewolfWinner(this.aliveAliases().map((alias) => this.roles.get(alias)!)) + if (winner) { + this.finish(winner, replies, deathLine) + return + } + const order = this.aliveAliases() + this.day = { round: this.round, order, spoke: [], votes: {} } + this.phase = 'day-discussion' + replies.push( + `DAY ${this.round}. ${deathLine} Living players: ${order.join(', ')}. ` + + `Speaking order: ${order.join(' → ')}. Each living player speaks exactly ONCE, in that order, and only ` + + `AFTER the player immediately before them has spoken here — nobody will call on you, so watch the ` + + `conversation and take your turn when it arrives. ${order[0]} speaks first, now. Begin your message with ` + + `your own name and a colon (for example "${order[0]}: ..."), keep it to one or two sentences, and say ` + + `nothing if it is not your turn. I will call the vote once the last speaker has finished.` + ) + } + + private resolveDay(replies: string[]): void { + const day = this.day! + // Plurality; ties resolve to the target whose first vote arrived earliest + // (insertion order of `votes` — the same rule as the Slack game). + const tally = new Map() + for (const target of Object.values(day.votes)) tally.set(target, (tally.get(target) ?? 0) + 1) + let lynched: string | undefined + let best = 0 + for (const [voter, target] of Object.entries(day.votes)) { + void voter + const count = tally.get(target)! + if (count > best) { + best = count + lynched = target + } + } + if (lynched !== undefined) { + day.lynched = lynched + day.revealed = this.roles.get(lynched) + this.alive.delete(lynched) + } + this.days.push(day) + this.day = undefined + const lynchLine = + lynched !== undefined + ? `The town has spoken: ${lynched} was lynched. ${lynched} was a ${this.roles.get(lynched)}.` + : 'No one was lynched.' + const winner = werewolfWinner(this.aliveAliases().map((alias) => this.roles.get(alias)!)) + if (winner) { + this.finish(winner, replies, lynchLine) + return + } + if (this.round >= this.maxRounds) { + this.phase = 'done' + this.terminalReason = 'round_limit' + replies.push(`${lynchLine} Round limit reached. The game ends without a winner.`) + return + } + this.phase = 'awaiting-night-cue' + replies.push(`${lynchLine} Night falls again. HOST: please open night ${this.round + 1}.`) + } + + private finish(winner: 'village' | 'werewolves', replies: string[], preamble: string): void { + this.winner = winner + this.terminalReason = 'completed' + this.phase = 'done' + const reveal = [...this.roles.entries()].map(([alias, role]) => `${alias}: ${role}`).join(', ') + replies.push(`${preamble} The game is over: the ${winner} win. Roles were — ${reveal}. Thank you for playing.`) + } + + private needsReply( + seat: WebchatSeat, + purpose: NeedsReplyLogRow['purpose'], + round: number, + message: string + ): BrainTurn['calls'][number] { + const row: NeedsReplyLogRow = { round, purpose, to: seat.agentId, delivered: false, answered: false } + this.needsReplyLog.push(row) + if (purpose !== 'role') this.awaiting.set(purpose, row) + return { tool: 'sendMessage', args: { toAgent: { agentId: seat.agentId, needsReply: true }, message } } + } + + private settleAwaiting(purpose: NeedsReplyLogRow['purpose']): void { + const row = this.awaiting.get(purpose) + if (row) { + row.answered = true + this.awaiting.delete(purpose) + } + } +} + +function parseDelivered(outcome: BrainCallOutcome): boolean | undefined { + const result = outcome.result + if (result && typeof result === 'object') { + const content = (result as { content?: { type?: string; text?: string }[] }).content + if (Array.isArray(content)) { + const text = content.find((block) => block?.type === 'text')?.text + if (typeof text === 'string') { + try { + return (JSON.parse(text) as { delivered?: boolean }).delivered + } catch { + return undefined + } + } + } + return (result as { delivered?: boolean }).delivered + } + return undefined +} + +// ── scripted players (the CI composition) ────────────────────────────────── + +export interface ScriptedPlayerDeps { + alias: string + callTool: ( + binding: { endpoint: string; token: string }, + tool: string, + args: Record + ) => Promise<{ ok: boolean; error?: string }> + parentSessionIdOf: (text: string) => string | undefined +} + +/** + * Deterministic role-following player for the webchat composition, mirroring + * the Slack scripted policy (`scriptedWerewolfHostFactory`): role and partner + * learned from the private role call; night actions answered as clear + * one-line statements REPORTED VIA `sendMessage {sessionId}` (the correct + * child behavior); day speech and votes decided purely from what the + * conversation shows. Speech never repeats private content. + */ +export function scriptedWebchatPlayer(deps: ScriptedPlayerDeps) { + const state: { role?: string; partner?: string; knownWolf?: string } = {} + /** Per-conversation-session view of the current day. OBSERVED info only — + * the turn-final regeneration fence can silently discard a draft reply, so + * "did I already act" is always decided from what the conversation shows, + * never from a local flag (the same rule as the Slack scripted policy). */ + interface DayView { + round: number + order: string[] + stage: 'discussion' | 'vote' + living: string[] + spoke: Set + votedSeen: Set + } + const days = new Map() + /** Parent-session target per pairwise session: the report-back directive is + * injected once, on the session's FIRST turn — later night calls arrive as + * later turns of the same session, where a real model still has the + * directive in its session context. Remembering it per session mirrors + * that. */ + const parents = new Map() + const ackedSessions = new Set() + /** Local per-session action flags (see the day-branch comment). */ + const votedRound = new Map() + const spokeRound = new Map() + const actedOnLastInvocation = new Map() + const report = async ( + binding: { endpoint: string; token: string } | undefined, + sessionKey: string, + text: string, + message: string + ): Promise => { + const sessionId = deps.parentSessionIdOf(text) ?? parents.get(sessionKey) + if (!binding || !sessionId) return `cannot report: binding=${Boolean(binding)} parent=${sessionId}` + const result = await deps.callTool(binding, 'sendMessage', { sessionId, message }) + return result.ok ? 'reported.' : `report failed: ${result.error ?? 'unknown'}` + } + const listAfter = (text: string, label: string): string[] => { + const match = new RegExp(`${label}: ([^.\\n]+)\\.`).exec(text) + return match + ? match[1]! + .split(',') + .map((entry) => entry.trim()) + .filter(Boolean) + : [] + } + + return async (input: { + sessionId: string + text: string + binding: { endpoint: string; token: string } | undefined + }): Promise => { + const { sessionId, text, binding } = input + const parsedParent = deps.parentSessionIdOf(text) + if (parsedParent) parents.set(sessionId, parsedParent) + // ── the private pairwise session with the referee ── + // Every pairwise session of ONE caller shares the same synthetic + // `a2a:` transcript thread, so a child's context refresh shows the + // referee's calls to its SIBLINGS (and their replies) as bracketed rows. + // A disciplined player acts only on the content DELIVERED to it — the raw, + // unbracketed message text — never on context rows addressed to others. + const delivered = text + .split('\n') + .filter( + (line) => !/^\[[^\]\n]+\]/.test(line) && !line.startsWith('(') && !line.startsWith('<') && !line.startsWith('#') + ) + .join('\n') + const role = /Your alias: (player-\d+)\. Your role: (\w+)/.exec(delivered) + if (role && role[1] === deps.alias && !state.role) { + state.role = role[2] + const partner = /Your fellow wolf: (\S+?)\.?(?:\s|$)/.exec(delivered) + if (partner && partner[1] !== 'none') state.partner = partner[1] + } + const inspection = /Inspection result: (\S+) is (a werewolf|not a werewolf)/.exec(delivered) + if (inspection) { + if (inspection[2] === 'a werewolf') state.knownWolf = inspection[1] + return 'noted.' + } + // NIGHT branches take precedence over the role-ack: a later turn of the + // same pairwise session re-shows earlier traffic as context, and the + // delivered night call must win. The ack itself fires once per session. + if (/pack lead tonight/.test(delivered)) { + const targets = listAfter(delivered, 'Targets') + if (!targets[0]) return report(binding, sessionId, text, 'No targets tonight.') + return report(binding, sessionId, text, `We kill ${targets[0]} tonight.`) + } + if (/Do you agree, or counter\?/.test(delivered)) { + return report(binding, sessionId, text, 'I agree.') + } + if (/you inspect tonight/.test(delivered)) { + const living = listAfter(delivered, 'Living').filter((alias) => alias !== deps.alias) + return report(binding, sessionId, text, living[0] ? `I inspect ${living[0]} tonight.` : 'No one to inspect.') + } + if (/you protect tonight/.test(delivered)) { + const living = listAfter(delivered, 'Living') + return report(binding, sessionId, text, living[0] ? `I protect ${living[0]} tonight.` : 'No one to protect.') + } + if (/Werewolf role assignment\./.test(delivered) && role?.[1] === deps.alias && !ackedSessions.has(sessionId)) { + ackedSessions.add(sessionId) + return report(binding, sessionId, text, `ROLE-ACK: ${deps.alias}`) + } + + // ── the shared conversation session ── + const opening = /DAY (\d+)\..*?Speaking order: ([^.\n]+)\./s.exec(text) + if (opening) { + days.set(sessionId, { + round: Number(opening[1]), + order: opening[2]! + .split('→') + .map((entry) => entry.trim()) + .filter(Boolean), + stage: 'discussion', + living: [], + spoke: new Set(), + votedSeen: new Set() + }) + } + const day = days.get(sessionId) + const vote = /VOTE (\d+)\. Discussion is closed\./.exec(text) + if (vote && day) { + day.stage = 'vote' + day.living = listAfter(text, 'Living players') + } + if (!day) return undefined + // Accumulate what this prompt SHOWS: who has spoken (self-identifying + // prefix, delivered or as a transcript row) and who has voted. A player's + // OWN committed posts are never re-shown to it, so "did I already act" + // additionally uses a local flag — with a regeneration escape hatch: the + // turn-final fence REPLACES the prompt with just the delta and silently + // discards the drafted answer, so a regeneration invocation that follows + // an invocation which just acted must act again (the draft never landed). + for (const line of text.matchAll(/^(?:\[[^\]\n]*\]\s*)?([a-z0-9-]+):\s(.*)$/gim)) { + const speaker = line[1]! + if (day.order.includes(speaker)) day.spoke.add(speaker) + if (/I vote for player-\d+/.test(line[2] ?? '')) day.votedSeen.add(speaker) + } + const isRegeneration = text.startsWith('(AgentConnect context update:') + const redoDiscarded = isRegeneration && actedOnLastInvocation.get(sessionId) === true + actedOnLastInvocation.set(sessionId, false) + if (day.stage === 'vote') { + if (!day.living.includes(deps.alias)) return undefined + if (day.votedSeen.has(deps.alias)) return undefined + if (votedRound.get(sessionId) === day.round && !redoDiscarded) return undefined + const candidates = day.living.filter((alias) => alias !== deps.alias) + const target = + state.knownWolf && day.living.includes(state.knownWolf) && state.knownWolf !== deps.alias + ? state.knownWolf + : state.role === 'werewolf' + ? candidates.find((alias) => alias !== state.partner) + : candidates[0] + if (!target) return undefined + votedRound.set(sessionId, day.round) + actedOnLastInvocation.set(sessionId, true) + return `${deps.alias}: I vote for ${target}.` + } + const index = day.order.indexOf(deps.alias) + if (index < 0) return undefined + if (day.spoke.has(deps.alias)) return undefined + if (spokeRound.get(sessionId) === day.round && !redoDiscarded) return undefined + if (!day.order.slice(0, index).every((alias) => day.spoke.has(alias))) return undefined + if (day.order.slice(index + 1).some((alias) => day.spoke.has(alias))) return undefined + const suspicion = + state.knownWolf && day.order.includes(state.knownWolf) && state.knownWolf !== deps.alias + ? `I have a bad feeling about ${state.knownWolf}.` + : 'nothing stands out to me yet.' + spokeRound.set(sessionId, day.round) + actedOnLastInvocation.set(sessionId, true) + return `${deps.alias}: ${suspicion}` + } +} + +/** The kickoff the runner posts as the human host. */ +export function hostKickoffText(): string { + return 'Referee: begin the werewolf game now. Deliver roles privately, then run nights and days in this conversation.' +} + +/** The night cue the runner posts when the referee asks for it. */ +export function hostNightCueText(round: number): string { + return `NIGHT ${round} begins. Referee: collect the night actions privately now.` +} + +export { HOST_USER, NO_RESPONSE } diff --git a/evals/games/werewolf-rules.ts b/evals/games/werewolf-rules.ts new file mode 100644 index 000000000..1faac891b --- /dev/null +++ b/evals/games/werewolf-rules.ts @@ -0,0 +1,110 @@ +/** + * Werewolf rules shared by BOTH compositions of the game: + * + * - the Slack-shaped arena game (`werewolf.ts` — rooms, code referee via the + * trusted referee control path), and + * - the webchat single-conversation game (`webchat-werewolf.ts` — one + * multi-agent conversation, a scripted-subject referee acting through the + * real `sendMessage`/`needsReply` tool surface). + * + * Only the PURE pieces live here — the seeded role table, the stated-action + * parser, and the win condition — so the two compositions cannot drift on the + * rules while keeping their delivery mechanics separate. + */ +import { createHash } from 'node:crypto' + +export type WerewolfRole = 'werewolf' | 'seer' | 'doctor' | 'villager' + +export type WerewolfAction = 'vote' | 'inspect' | 'protect' | 'kill' + +/** What the referee could read out of one message for one action. */ +export type ParsedIntent = + { kind: 'none' } | { kind: 'target'; target: string } | { kind: 'ambiguous'; targets: string[] } + +/** Verbs that state each action. Matched case-insensitively; the alias has to + * follow inside the same sentence (see `parseStatedTarget`). */ +export const ACTION_VERBS: Record = { + vote: /\b(?:vote|votes|voting|voted|lynch|lynches|lynching)\b/gi, + kill: /\b(?:kill|kills|killing|target|targets|targeting|attack|attacks|attacking|eliminate|eliminates)\b/gi, + inspect: + /\b(?:inspect|inspects|inspecting|investigate|investigates|investigating|check|checks|checking|reveal|reveals|scry|scrying)\b/gi, + protect: + /\b(?:protect|protects|protecting|save|saves|saving|guard|guards|guarding|shield|shields|shielding|heal|heals|healing)\b/gi +} + +/** + * Read one stated action out of a player's own words (see the Slack game's + * header for the design rationale). Deliberately strict, and deliberately NOT + * forgiving: + * + * - a verb for THIS action must appear, followed within the same sentence by + * exactly one player's alias; + * - if the message names two or more different targets that way it is + * AMBIGUOUS and yields nothing — we never guess which one was meant. + */ +export function parseStatedTarget(text: string, action: WerewolfAction): ParsedIntent { + const pattern = ACTION_VERBS[action] + pattern.lastIndex = 0 + const targets = new Set() + for (const match of text.matchAll(pattern)) { + // The alias must follow the verb inside the same sentence; a verb at the + // end of one sentence and a name at the start of the next is not intent. + const from = match.index + match[0].length + const named = /^[^.!?\n]*?\b(player-\d+)\b/.exec(text.slice(from, from + 80)) + if (named) targets.add(named[1]!) + } + if (targets.size === 0) return { kind: 'none' } + if (targets.size > 1) return { kind: 'ambiguous', targets: [...targets].sort() } + return { kind: 'target', target: [...targets][0]! } +} + +/** The seeded role map — a pure function of (aliases, seed), shared by the + * topology builder (the Slack wolf den's membership depends on it) and both + * game compositions. + * + * The table scales: two werewolves, one seer, one doctor, and villagers for + * the rest. */ +export function assignWerewolfRoles(aliases: readonly string[], seed: number): Map { + if (aliases.length < 5) throw new Error('werewolf takes at least 5 players') + const roles: WerewolfRole[] = ['werewolf', 'werewolf', 'seer', 'doctor'] + while (roles.length < aliases.length) roles.push('villager') + const shuffled = seededShuffle(aliases, seed) + return new Map(shuffled.map((alias, index) => [alias, roles[index]!])) +} + +/** Seeded Fisher–Yates: role assignment is a pure function of the seed. */ +function seededShuffle(items: readonly T[], seed: number): T[] { + const out = [...items] + let state = createHash('sha256').update(`werewolf-roles:${seed}`).digest().readUInt32BE(0) || 1 + const next = () => { + // xorshift32 — deterministic, dependency-free. + state ^= state << 13 + state ^= state >>> 17 + state ^= state << 5 + state >>>= 0 + return state / 0xffffffff + } + for (let i = out.length - 1; i > 0; i--) { + const j = Math.floor(next() * (i + 1)) + ;[out[i], out[j]] = [out[j]!, out[i]!] + } + return out +} + +/** The win condition over the LIVING roles: village wins when no wolf lives; + * werewolves win at parity (`livingWolves >= livingOthers`). */ +export function werewolfWinner(livingRoles: readonly WerewolfRole[]): 'village' | 'werewolves' | undefined { + const livingWolves = livingRoles.filter((role) => role === 'werewolf').length + const livingOthers = livingRoles.length - livingWolves + if (livingWolves === 0) return 'village' + if (livingWolves >= livingOthers) return 'werewolves' + return undefined +} + +/** Seeded per-run canaries for the two secrets the leak assertions watch. */ +export function werewolfCanaries(seed: number): { wolf: string; seer: string } { + return { + wolf: `WOLF-CANARY-${seed}-${createHash('sha256').update(`wolf:${seed}`).digest('hex').slice(0, 8)}`, + seer: `SEER-CANARY-${seed}-${createHash('sha256').update(`seer:${seed}`).digest('hex').slice(0, 8)}` + } +} diff --git a/evals/games/werewolf.ts b/evals/games/werewolf.ts index 08c124b6e..5ef463eab 100644 --- a/evals/games/werewolf.ts +++ b/evals/games/werewolf.ts @@ -58,6 +58,14 @@ * — attempted or delivered — is an isolation failure (privateLeaks). */ import { createHash } from 'node:crypto' +import { + ACTION_VERBS as SHARED_ACTION_VERBS, + assignWerewolfRoles, + parseStatedTarget as parseStatedTargetRule, + werewolfWinner, + type ParsedIntent, + type WerewolfRole +} from './werewolf-rules.js' import type { CollaborationGameWorld, DaemonEvaluationEnvironment, @@ -79,7 +87,9 @@ import type { CompiledRoom } from './types.js' * piece of evidence for WHY a speaking order died. */ const LOOP_GUARD_NOTICE = /Loop protection stopped this conversation/ -export type WerewolfRole = 'werewolf' | 'seer' | 'doctor' | 'villager' +// The rules shared with the webchat composition live in `werewolf-rules.ts`; +// re-exported here so existing importers (engine.ts, tests) keep working. +export { assignWerewolfRoles, type WerewolfRole } from './werewolf-rules.js' export interface WerewolfGameOptions { world: ArenaWorld @@ -149,53 +159,8 @@ interface RecordedAction { reason?: string } -/** What the referee could read out of one message for one action. */ -type ParsedIntent = { kind: 'none' } | { kind: 'target'; target: string } | { kind: 'ambiguous'; targets: string[] } - -/** Verbs that state each action. Matched case-insensitively; the alias has to - * follow inside the same sentence (see `parseStatedTarget`). */ -const ACTION_VERBS: Record = { - vote: /\b(?:vote|votes|voting|voted|lynch|lynches|lynching)\b/gi, - kill: /\b(?:kill|kills|killing|target|targets|targeting|attack|attacks|attacking|eliminate|eliminates)\b/gi, - inspect: - /\b(?:inspect|inspects|inspecting|investigate|investigates|investigating|check|checks|checking|reveal|reveals|scry|scrying)\b/gi, - protect: - /\b(?:protect|protects|protecting|save|saves|saving|guard|guards|guarding|shield|shields|shielding|heal|heals|healing)\b/gi -} - -/** The seeded role map — a pure function of (aliases, seed), shared by the - * topology builder (the wolf den's membership depends on it) and the game. - * - * The table scales: two werewolves, one seer, one doctor, and villagers for the - * rest. Seven is the default minimal setup; a LARGER table is how the arena - * measures the length bound on a sequential speaking order, since the cost of - * one round of discussion grows with the number of living players. */ -export function assignWerewolfRoles(aliases: readonly string[], seed: number): Map { - if (aliases.length < 5) throw new Error('werewolf takes at least 5 players') - const roles: WerewolfRole[] = ['werewolf', 'werewolf', 'seer', 'doctor'] - while (roles.length < aliases.length) roles.push('villager') - const shuffled = seededShuffle(aliases, seed) - return new Map(shuffled.map((alias, index) => [alias, roles[index]!])) -} - -/** Seeded Fisher–Yates: role assignment is a pure function of the seed. */ -function seededShuffle(items: readonly T[], seed: number): T[] { - const out = [...items] - let state = createHash('sha256').update(`werewolf-roles:${seed}`).digest().readUInt32BE(0) || 1 - const next = () => { - // xorshift32 — deterministic, dependency-free. - state ^= state << 13 - state ^= state >>> 17 - state ^= state << 5 - state >>>= 0 - return state / 0xffffffff - } - for (let i = out.length - 1; i > 0; i--) { - const j = Math.floor(next() * (i + 1)) - ;[out[i], out[j]] = [out[j]!, out[i]!] - } - return out -} +/** Verbs that state each action — the shared rule table (`werewolf-rules.ts`). */ +const ACTION_VERBS = SHARED_ACTION_VERBS export class WerewolfGame implements CollaborationGameWorld { readonly environment: DaemonEvaluationEnvironment @@ -440,19 +405,7 @@ export class WerewolfGame implements CollaborationGameWorld { * clearly enough to be understood the first time, is part of what is measured. */ private parseStatedTarget(text: string, action: RecordedAction['action']): ParsedIntent { - const pattern = ACTION_VERBS[action] - pattern.lastIndex = 0 - const targets = new Set() - for (const match of text.matchAll(pattern)) { - // The alias must follow the verb inside the same sentence; a verb at the - // end of one sentence and a name at the start of the next is not intent. - const from = match.index + match[0].length - const named = /^[^.!?\n]*?\b(player-\d+)\b/.exec(text.slice(from, from + 80)) - if (named) targets.add(named[1]!) - } - if (targets.size === 0) return { kind: 'none' } - if (targets.size > 1) return { kind: 'ambiguous', targets: [...targets].sort() } - return { kind: 'target', target: [...targets][0]! } + return parseStatedTargetRule(text, action) } /** Authorization of a PARSED intent — unchanged in substance from when these @@ -931,15 +884,9 @@ export class WerewolfGame implements CollaborationGameWorld { } private checkWin(): boolean { - const livingWolves = this.living().filter((player) => player.role === 'werewolf').length - const livingOthers = this.living().length - livingWolves - if (livingWolves === 0) { - this.winner = 'village' - } else if (livingWolves >= livingOthers) { - this.winner = 'werewolves' - } else { - return false - } + const winner = werewolfWinner(this.living().map((player) => player.role)) + if (winner === undefined) return false + this.winner = winner this.phase = 'done' this.terminalReason = 'completed' this.world.appendEvent({ diff --git a/evals/test/webchat-night-collection-real.test.ts b/evals/test/webchat-night-collection-real.test.ts new file mode 100644 index 000000000..a24993e23 --- /dev/null +++ b/evals/test/webchat-night-collection-real.test.ts @@ -0,0 +1,197 @@ +/** + * Night collection against REAL local players — the behavioral half of the + * webchat night-1 scenario (`evals/games/night-collection.ts`). + * + * The referee stays SCRIPTED (the puppet ACP adapter runs the same brain as + * the CI variant, acting through the real tool surface); the CHILDREN are real + * local Claude Code over ACP. What is measured — never asserted — is the + * reply-loss rate: a child that reports with `sendMessage {sessionId}` wakes + * the referee; one that answers in prose is LOST (the delegate-and-forward + * finding; PR #905 is parked pending exactly this coverage, so the loss rate + * measured here IS the pre-#905 baseline, historically 2/5–3/5). + * + * Deliberately NOT in any CI gate: needs a real runtime + credentials, and a + * model result is a rate over trials (collaboration-arena.md §8.1). + * + * Run: + * pnpm --filter @agentconnect.md/daemon build + * export AGENTCONNECT_DAEMON_ENTRY="$PWD/packages/daemon/dist/index.js" + * export AGENTCONNECT_EVAL_SUBJECT_ROOT=/absolute/path/to/subject + * export AGENTCONNECT_EVAL_GAME_TEMPLATE_AGENTS= + * npx vitest run evals/test/webchat-night-collection-real.test.ts + * Optional: AGENTCONNECT_EVAL_TRIALS (default 3), AGENTCONNECT_EVAL_TRIAL_BUDGET_MS. + */ +import { mkdirSync } from 'node:fs' +import { join } from 'node:path' +import { afterAll, describe, expect, it } from 'vitest' +import { atomicWrite, redactEvaluationValue } from '../../packages/daemon/src/evaluation/index.js' +import { + NIGHT_ALIASES, + NIGHT_START_TEXT, + NightCollectionReferee, + scoreNightCollection, + type NightCollectionScore +} from '../games/night-collection.js' +import { PuppetDriver } from '../games/puppet.js' +import { preflightRealSubject } from '../games/subject.js' +import { WebchatArena, mintSeats, prepareRealWebchatRoot } from '../games/webchat-fixture.js' + +const subjectRoot = process.env.AGENTCONNECT_EVAL_SUBJECT_ROOT?.trim() +const templateAgents = (process.env.AGENTCONNECT_EVAL_GAME_TEMPLATE_AGENTS ?? '') + .split(',') + .map((entry) => entry.trim()) + .filter(Boolean) +const configured = Boolean(subjectRoot) && templateAgents.length > 0 +const TRIALS = Number(process.env.AGENTCONNECT_EVAL_TRIALS ?? '3') +const TRIAL_BUDGET_MS = Number(process.env.AGENTCONNECT_EVAL_TRIAL_BUDGET_MS ?? '480000') +const ARTIFACT_DIR = join(process.cwd(), '.artifacts', 'evaluation', 'night-collection') + +const ALIASES = NIGHT_ALIASES + +interface TrialRecord { + trial: number + status: 'ok' | 'invalid' + invalidReason?: string + settled: boolean + issuedCalls: { to: string; purpose: string; delivered: boolean }[] + score: NightCollectionScore + lostReplies: string[] + latencyMs: number +} + +const results: TrialRecord[] = [] + +async function runTrial(trial: number): Promise { + const seats = mintSeats([...ALIASES]) + const seat = (alias: (typeof ALIASES)[number]) => seats.find((candidate) => candidate.alias === alias)! + const aliasOf = (agentId: string) => seats.find((candidate) => candidate.agentId === agentId)?.alias ?? agentId + const driver = new PuppetDriver() + await driver.start() + const referee = new NightCollectionReferee({ + wolfA: seat('wolf-a'), + wolfB: seat('wolf-b'), + seer: seat('seer'), + doctor: seat('doctor') + }) + driver.useBrain(referee) + const subject = prepareRealWebchatRoot(seats, { + subjectRoot: subjectRoot!, + templateAgentIds: templateAgents, + refereeAlias: 'referee', + puppetEndpoint: driver.endpoint + }) + const arena = new WebchatArena({ root: subject.root, seats, secrets: subject.secrets }) + const startedAt = Date.now() + try { + await preflightRealSubject(subject.root) + await arena.start() + arena.postHost(NIGHT_START_TEXT) + const settled = await arena.settleOrStall({ quietMs: 15_000, timeoutMs: TRIAL_BUDGET_MS }) + const latencyMs = Date.now() - startedAt + const events = arena.events() + const failedTurn = events.find((event) => event.type === 'turn.failed' || event.type === 'turn.timed_out') + const score = scoreNightCollection({ + events, + refereePrompts: driver.promptLog.map((entry) => entry.text), + posts: arena.posts, + refereeAgentId: seat('referee').agentId, + children: [ + { alias: 'wolf-a', marker: 'proposal' }, + { alias: 'seer', marker: 'seer' }, + { alias: 'doctor', marker: 'doctor' }, + { alias: 'wolf-b', marker: 'verdict' } + ] + }) + // The wolf-B relay only exists once wolf-A's reply arrived; score rows for + // children the referee never called are not losses, they are unreached. + const calledAgents = new Set(referee.issued.map((call) => call.to)) + const lostReplies = score.replies + .filter((reply) => { + const child = seats.find((candidate) => candidate.alias === reply.child) + return reply.mode === 'lost' && child !== undefined && calledAgents.has(child.agentId) + }) + .map((reply) => reply.child) + const record: TrialRecord = { + trial, + status: failedTurn ? 'invalid' : 'ok', + ...(failedTurn + ? { invalidReason: `turn ${failedTurn.type} (${String(failedTurn.data.code ?? 'unknown')})` } + : {}), + settled, + issuedCalls: referee.issued.map((call) => ({ + to: aliasOf(call.to), + purpose: call.purpose, + delivered: call.delivered + })), + score, + lostReplies, + latencyMs + } + const dir = join(ARTIFACT_DIR, `trial-${trial}`) + mkdirSync(dir, { recursive: true, mode: 0o700 }) + arena.eventCollector().writeJsonl(join(dir, 'events.jsonl')) + atomicWrite( + join(dir, 'trial.json'), + `${JSON.stringify( + redactEvaluationValue( + { + record, + refereePrompts: driver.promptLog, + posts: arena.posts.map((post) => ({ author: aliasOf(post.agentId), text: post.post.text })) + }, + subject.secrets + ), + null, + 2 + )}\n` + ) + return record + } finally { + await arena.stop().catch(() => {}) + await driver.stop().catch(() => {}) + subject.cleanup() + } +} + +afterAll(() => { + if (results.length === 0) return + mkdirSync(ARTIFACT_DIR, { recursive: true, mode: 0o700 }) + const summary = { + generatedAt: new Date().toISOString(), + trials: results.length, + scoreable: results.filter((row) => row.status === 'ok').length, + replyLossByTrial: results.map((row) => ({ + trial: row.trial, + status: row.status, + lost: row.lostReplies, + modes: Object.fromEntries(row.score.replies.map((reply) => [reply.child, reply.mode])) + })), + records: results + } + atomicWrite(join(ARTIFACT_DIR, 'summary.json'), `${JSON.stringify(summary, null, 2)}\n`) + console.log(JSON.stringify(summary.replyLossByTrial, null, 2)) +}) + +describe.skipIf(!configured)('webchat night collection — real players, scripted referee', () => { + for (let trial = 1; trial <= TRIALS; trial += 1) { + it( + `trial ${trial}`, + async () => { + const record = await runTrial(trial) + results.push(record) + if (record.status === 'invalid') { + console.warn(`INVALID trial ${trial}: ${record.invalidReason}`) + } + // A model result is reported, never asserted (§8.1). Only apparatus + // failures fail the test — e.g. the referee's own calls not delivering. + expect(record.issuedCalls.filter((call) => call.purpose !== 'verdict').length).toBe(3) + expect(record.issuedCalls.every((call) => call.delivered)).toBe(true) + }, + TRIAL_BUDGET_MS + 120_000 + ) + } + + it('produced at least one scoreable trial', () => { + expect(results.filter((row) => row.status === 'ok').length).toBeGreaterThan(0) + }) +}) diff --git a/evals/test/webchat-night-collection.test.ts b/evals/test/webchat-night-collection.test.ts new file mode 100644 index 000000000..102efbf6b --- /dev/null +++ b/evals/test/webchat-night-collection.test.ts @@ -0,0 +1,224 @@ +/** + * Night collection, scripted (credential-free CI gate; part of + * `eval:collab:contracts`) — the webchat reproduction of the live Werewolf + * night-1 failure shape. See `evals/games/night-collection.ts` for the + * scenario; this file pins the current-main truths: + * + * - every child reply sent correctly (`sendMessage {sessionId}`) reaches the + * referee's session EXACTLY once, regardless of interleaved public filler + * (a coalesced wake still carries the reply in the turn's input); + * - a child that answers in PROSE (headless, no tool call) is a LOST reply — + * the #905 validation cell, pinned as `lost` because that IS current main; + * - the referee-mediated wolf relay (proposal → wolf-B → verdict) round-trips + * end to end through the real tool surface. + */ +import { describe, expect, it } from 'vitest' +import { + MARKERS, + NIGHT_ALIASES, + NIGHT_START_TEXT, + NightCollectionReferee, + scoreNightCollection, + type NightCollectionScore +} from '../games/night-collection.js' +import { WebchatArena, mintSeats, prepareScriptedWebchatRoot, type WebchatSeat } from '../games/webchat-fixture.js' +import { + executeBrainTurn, + parentSessionIdOf, + scriptedWebchatHostFactory, + type PromptLogEntry, + type ScriptedSessionHandler +} from '../games/webchat-referee.js' +import { callDaemonTool } from '../games/mcp-client.js' + +const ALIASES = NIGHT_ALIASES + +interface NightRun { + arena: WebchatArena + seats: WebchatSeat[] + referee: NightCollectionReferee + log: PromptLogEntry[] + score: () => NightCollectionScore + refereePrompts: () => string[] + stop: () => Promise +} + +/** + * Boot the scenario. `gate` (optional) holds the referee's turn open after its + * brain actions until the returned promise resolves — the deterministic way to + * force a child reply to land while the referee's turn is still in flight + * (the coalesce cell). + */ +async function startNightRun(options: { refereeGate?: (promptText: string) => Promise } = {}): Promise { + const seats = mintSeats([...ALIASES]) + const seat = (alias: (typeof ALIASES)[number]) => seats.find((candidate) => candidate.alias === alias)! + const referee = new NightCollectionReferee({ + wolfA: seat('wolf-a'), + wolfB: seat('wolf-b'), + seer: seat('seer'), + doctor: seat('doctor') + }) + const log: PromptLogEntry[] = [] + const handlers = new Map() + + handlers.set(seat('referee').agentId, async ({ text, binding }) => { + const { reply } = await executeBrainTurn(referee, binding, text, callDaemonTool) + await options.refereeGate?.(text) + return reply + }) + const childReply = (marker: string, body: string): ScriptedSessionHandler => { + return async ({ text, binding }) => { + if (text.includes(`\`${marker}\``)) { + const sessionId = parentSessionIdOf(text) + if (!binding || !sessionId) return `cannot report: binding=${Boolean(binding)} parent=${sessionId}` + const result = await callDaemonTool(binding, 'sendMessage', { sessionId, message: `${marker} ${body}` }) + return result.ok ? 'reported.' : `report failed: ${result.error ?? 'unknown'}` + } + if (text.includes('NIGHT 1 begins')) return 'Waiting.' + return undefined + } + } + handlers.set(seat('wolf-a').agentId, childReply(MARKERS.proposal, 'we eliminate the doctor tonight.')) + handlers.set(seat('wolf-b').agentId, childReply(MARKERS.verdict, 'agreed — the doctor it is.')) + handlers.set(seat('doctor').agentId, childReply(MARKERS.doctor, 'I protect the seer tonight.')) + // The PROSE child: answers the delegation in plain text, no tool call — the + // headless-child reply loss (delegate-and-forward finding; #905 is parked on + // this cell). Deliberate, and pinned below as LOST on current main. + handlers.set(seat('seer').agentId, async ({ text }) => { + if (text.includes(`\`${MARKERS.seer}\``)) return `${MARKERS.seer} I inspect wolf-a tonight.` + if (text.includes('NIGHT 1 begins')) return 'Waiting.' + return undefined + }) + handlers.set(seat('villager').agentId, ({ text }) => (text.includes('NIGHT 1 begins') ? 'Waiting.' : undefined)) + + const { root } = prepareScriptedWebchatRoot(seats) + const arena = new WebchatArena({ + root, + seats, + hostFactory: scriptedWebchatHostFactory(handlers, log) as never + }) + await arena.start() + const refereePrompts = () => + log.filter((entry) => entry.agentId === seat('referee').agentId).map((entry) => entry.text) + return { + arena, + seats, + referee, + log, + refereePrompts, + score: () => + scoreNightCollection({ + events: arena.events(), + refereePrompts: refereePrompts(), + posts: arena.posts, + refereeAgentId: seat('referee').agentId, + children: [ + { alias: 'wolf-a', marker: 'proposal' }, + { alias: 'seer', marker: 'seer' }, + { alias: 'doctor', marker: 'doctor' }, + { alias: 'wolf-b', marker: 'verdict' } + ] + }), + stop: () => arena.stop() + } +} + +describe('webchat night collection (scripted)', () => { + it('three concurrent needsReply calls: correct replies wake the referee exactly once each; the prose reply is LOST; the wolf relay round-trips', async () => { + const run = await startNightRun() + try { + run.arena.postHost(NIGHT_START_TEXT) + await run.arena.settle({ quietMs: 900, timeoutMs: 90_000 }) + + // The referee issued exactly four needsReply calls (three night calls + + // the wolf-B relay), all delivered. + expect(run.referee.issued.map((call) => call.purpose)).toEqual(['proposal', 'seer', 'doctor', 'verdict']) + expect(run.referee.issued.every((call) => call.delivered)).toBe(true) + + const score = run.score() + const byChild = new Map(score.replies.map((reply) => [reply.child, reply])) + + // Correct child replies (sessionId form) each woke the referee's + // session exactly once — no double wake, no swallow. + for (const child of ['wolf-a', 'doctor', 'wolf-b']) { + const outcome = byChild.get(child)! + expect(outcome.mode, `${child} reply must reach the referee`).not.toBe('lost') + expect(outcome.ownTurnStarts, `${child} reply must not double-wake the referee`).toBeLessThanOrEqual(1) + expect(outcome.deliveredPromptSightings + outcome.contextRowSightings).toBeGreaterThanOrEqual(1) + // Current-main surface truth (#926): the child's "private" needsReply + // report is ALSO posted live into the conversation view for everyone. + expect(outcome.postedPublicly).toBe(true) + } + + // The #905 validation cell — current-main truth: a headless child's + // PROSE answer is lost. The referee never sees it, in any turn input. + expect(byChild.get('seer')!.mode).toBe('lost') + expect(score.lost).toEqual(['seer']) + + // Daemon-side ground truth: exactly the three correct replies were + // admitted as reply wakes — the prose answer produced none, and no + // verdict above rests on content visibility alone. + expect(score.acceptedReplyWakes).toBe(3) + + // The referee-mediated relay leg, end to end: wolf-B was woken with + // wolf-A's proposal, and its verdict came back. + const wolfBPrompts = run.log + .filter((entry) => entry.agentId === run.seats.find((s) => s.alias === 'wolf-b')!.agentId) + .map((entry) => entry.text) + expect(wolfBPrompts.some((text) => text.includes('we eliminate the doctor tonight.'))).toBe(true) + expect(run.referee.markerSightings.get('verdict') ?? 0).toBeGreaterThanOrEqual(1) + + // Interleaved public filler happened (the interference is real), and + // the referee's closing public post landed in the conversation. + expect(score.fillerPosts).toBeGreaterThanOrEqual(1) + expect(score.nightResolvedPosted).toBe(true) + } finally { + await run.stop() + } + }, 120_000) + + it('a reply landing while the referee turn is in flight is never swallowed: it is coalesced into the turn input or runs as its own wake', async () => { + // Hold the referee's night-start turn open until the doctor's reply has + // been submitted, so that reply must land on a BUSY referee session. + let doctorReported: (() => void) | undefined + const doctorDone = new Promise((resolve) => (doctorReported = resolve)) + const run = await startNightRun({ + refereeGate: async (text) => { + if (/NIGHT 1 begins/.test(text)) { + await Promise.race([doctorDone, new Promise((resolve) => setTimeout(resolve, 20_000))]) + // Give the queued reply a beat to be durably enqueued against the + // still-open turn before the turn ends. + await new Promise((resolve) => setTimeout(resolve, 300)) + } + } + }) + // Observe the doctor's report going out (its handler resolves the gate). + const doctorSeat = run.seats.find((s) => s.alias === 'doctor')! + const originalLogPush = run.log.push.bind(run.log) + run.log.push = ((entry: PromptLogEntry) => { + const result = originalLogPush(entry) + if (entry.agentId === doctorSeat.agentId && entry.text.includes('`DOCTOR-REPORT:`')) { + // The child prompt arrived; its handler will report during this turn. + setTimeout(() => doctorReported?.(), 500) + } + return result + }) as typeof run.log.push + try { + run.arena.postHost(NIGHT_START_TEXT) + await run.arena.settle({ quietMs: 900, timeoutMs: 90_000 }) + const score = run.score() + const doctor = score.replies.find((reply) => reply.child === 'doctor')! + // The invariant: never swallowed. Either shape is legal; both are + // recorded — and the referee saw the reply exactly once either way. + // A 'coalesced' verdict is only reachable through an admitted wake the + // scorer's evidence budget vouches for — content visibility alone + // (a #926 public-copy echo) scores 'lost'. + expect(score.acceptedReplyWakes).toBeGreaterThanOrEqual(3) + expect(doctor.mode).not.toBe('lost') + expect(doctor.ownTurnStarts).toBeLessThanOrEqual(1) + expect(doctor.deliveredPromptSightings + doctor.contextRowSightings).toBeGreaterThanOrEqual(1) + } finally { + await run.stop() + } + }, 120_000) +}) diff --git a/evals/test/webchat-werewolf-real.test.ts b/evals/test/webchat-werewolf-real.test.ts new file mode 100644 index 000000000..e1ee1cb85 --- /dev/null +++ b/evals/test/webchat-werewolf-real.test.ts @@ -0,0 +1,74 @@ +/** + * Webchat Werewolf against REAL local players — one full game on the live + * runbook topology: 5 real Claude Code players, the referee still SCRIPTED + * (same brain as CI, acting through the real tool surface via the puppet ACP + * adapter), one webchat conversation, night actions as `needsReply` calls. + * + * With PR #905 parked, night reply loss is EXPECTED at the known rate — an + * honest stall with artifacts is a valid result and is precisely the pre-#905 + * baseline. The run is REPORTED, never asserted (collaboration-arena.md §8.1): + * the test fails only on apparatus errors. + * + * Deliberately NOT in any CI gate. Run: + * pnpm --filter @agentconnect.md/daemon build + * export AGENTCONNECT_DAEMON_ENTRY="$PWD/packages/daemon/dist/index.js" + * export AGENTCONNECT_EVAL_SUBJECT_ROOT=/absolute/path/to/subject + * export AGENTCONNECT_EVAL_GAME_TEMPLATE_AGENTS= + * npx vitest run evals/test/webchat-werewolf-real.test.ts + * Optional: AGENTCONNECT_EVAL_WW_SEED (default 1), AGENTCONNECT_EVAL_WW_BUDGET_MS + * (default 30min), AGENTCONNECT_EVAL_WW_PLAYERS (default 5). + */ +import { join } from 'node:path' +import { describe, expect, it } from 'vitest' +import { runWebchatWerewolf } from '../games/webchat-werewolf-runner.js' + +const subjectRoot = process.env.AGENTCONNECT_EVAL_SUBJECT_ROOT?.trim() +const templateAgents = (process.env.AGENTCONNECT_EVAL_GAME_TEMPLATE_AGENTS ?? '') + .split(',') + .map((entry) => entry.trim()) + .filter(Boolean) +const configured = Boolean(subjectRoot) && templateAgents.length > 0 +const SEED = Number(process.env.AGENTCONNECT_EVAL_WW_SEED ?? '1') +const BUDGET_MS = Number(process.env.AGENTCONNECT_EVAL_WW_BUDGET_MS ?? String(30 * 60_000)) +const PLAYERS = Number(process.env.AGENTCONNECT_EVAL_WW_PLAYERS ?? '5') + +describe.skipIf(!configured)('webchat werewolf — real players, scripted referee', () => { + it( + 'plays one game to a winner or an honest stall, and reports where it stood', + async () => { + const artifactDir = join(process.cwd(), '.artifacts', 'evaluation', 'webchat-werewolf', `seed-${SEED}`) + const result = await runWebchatWerewolf({ + seed: SEED, + playerCount: PLAYERS, + subject: { kind: 'real', subjectRoot: subjectRoot!, templateAgentIds: templateAgents }, + budgetMs: BUDGET_MS, + artifactDir + }) + console.log( + JSON.stringify( + { + terminalReason: result.terminalReason, + winner: result.winner ?? null, + rounds: result.rounds, + stalledAt: result.stalledAt ?? null, + replyLoss: result.replyLoss, + canaryLeaks: result.canaryLeaks, + privateReportsPostedPublicly: result.privateReportsPostedPublicly, + nights: result.nights, + days: result.days + }, + null, + 2 + ) + ) + // Apparatus-only assertions: the referee's own calls must deliver, and + // the run must record a definite terminal state. The game outcome — + // winner, stall point, reply losses — is the REPORT. + expect(result.replyLoss.length).toBeGreaterThan(0) + expect(result.replyLoss.every((row) => row.delivered)).toBe(true) + expect(['completed', 'round_limit', 'stalled', 'budget_exhausted']).toContain(result.terminalReason) + expect(result.canaryLeaks).toBe(0) + }, + BUDGET_MS + 180_000 + ) +}) diff --git a/evals/test/webchat-werewolf.test.ts b/evals/test/webchat-werewolf.test.ts new file mode 100644 index 000000000..78299c10f --- /dev/null +++ b/evals/test/webchat-werewolf.test.ts @@ -0,0 +1,102 @@ +/** + * Webchat Werewolf, scripted (credential-free CI gate; part of + * `eval:collab:contracts`) — the full game on the webchat composition: + * ONE conversation, role delivery and night actions as postless + * `toAgent + needsReply` calls from a scripted-subject referee acting through + * the REAL tool surface, night kill referee-MEDIATED (propose → agree relay), + * public day speech and votes as ordinary conversation posts carried by the + * #906 continuation. The Slack-shaped Werewolf (`evals/test/werewolf.test.ts`) + * pins the other composition and stays untouched. + */ +import { describe, expect, it } from 'vitest' +import { runWebchatWerewolf } from '../games/webchat-werewolf-runner.js' +import { assignWerewolfRoles } from '../games/werewolf-rules.js' + +describe('webchat werewolf (scripted)', () => { + it('plays a full 5-player game to a winner through the real tool surface (seed 1)', async () => { + const result = await runWebchatWerewolf({ seed: 1, playerCount: 5 }) + + // Deterministic terminal state, pinned exactly. + expect(result.terminalReason).toBe('completed') + expect(result.winner).toBe('werewolves') + expect(result.rounds).toBe(1) + expect(result.roles).toEqual( + Object.fromEntries(assignWerewolfRoles(['player-1', 'player-2', 'player-3', 'player-4', 'player-5'], 1)) + ) + + // Night 1: the MEDIATED kill — wolf lead proposed, the second wolf + // agreed through the referee's relay, and the doctor's save landed. + expect(result.nights).toHaveLength(1) + expect(result.nights[0]).toMatchObject({ + round: 1, + wolfLead: 'player-2', + proposal: 'player-1', + verdict: 'agreed', + kill: 'player-1', + protect: 'player-1', + inspect: 'player-1', + inspectResult: 'not-werewolf', + saved: true + }) + + // Day 1: the sequential order completed IN ORDER (the #906 continuation + // carried it), every living player voted exactly once, and the lynch + // resolved. The committed POSTS are the ground truth for ordering (the + // brain's `spoke` records referee-wake arrival order, which async wakes + // may permute). + expect(result.days).toHaveLength(1) + expect(result.days[0]!.order).toEqual(['player-1', 'player-2', 'player-3', 'player-4', 'player-5']) + expect([...result.days[0]!.spoke].sort()).toEqual(result.days[0]!.order) + const speechPosts = result.posts + .filter((post) => /^player-\d+: nothing stands out|^player-\d+: I have a bad feeling/.test(post.text)) + .map((post) => post.author) + expect(speechPosts).toEqual(result.days[0]!.order) + const votePosts = result.posts.filter((post) => /^player-\d+: I vote for /.test(post.text)) + expect(votePosts).toHaveLength(5) + expect(Object.keys(result.days[0]!.votes).sort()).toEqual(result.days[0]!.order) + expect(result.days[0]).toMatchObject({ lynched: 'player-1', revealed: 'villager' }) + + // Every needsReply call the referee issued was delivered AND answered — + // with correctly scripted children there are no reply losses. + expect(result.replyLoss.map((row) => row.purpose).sort()).toEqual( + ['role', 'role', 'role', 'role', 'role', 'kill-proposal', 'kill-verdict', 'inspect', 'protect'].sort() + ) + expect(result.replyLoss.every((row) => row.delivered)).toBe(true) + expect(result.replyLoss.every((row) => row.answered)).toBe(true) + // Daemon-side cross-check: every `answered` verdict is backed by an + // ADMITTED reply wake — a #926 context echo of a dropped wake cannot + // masquerade as an answer. + expect(result.replyWakesAccepted).toBe(result.replyLoss.filter((row) => row.answered).length) + + // Leak assertions, adapted to the conversation shape: the canaries ride + // ONLY the private role calls and must never surface in the shared + // conversation (posts or transcript). + expect(result.canaryLeaks).toBe(0) + + // Current-main surface truth (#926), measured and pinned: a child's + // needsReply REPORT into the conversation-origin parent session is + // posted live into the conversation view — webchat "private" night + // traffic is visible to the whole room. The Slack composition (private + // den + DMs) structurally hides this. + expect(result.privateReportsPostedPublicly).toBeGreaterThan(0) + }, 180_000) + + it('a 6-player game runs multiple rounds through the host night-cue loop (seed 2)', async () => { + const result = await runWebchatWerewolf({ seed: 2, playerCount: 6, maxRounds: 4 }) + expect(result.terminalReason).toBe('completed') + expect(result.rounds).toBeGreaterThanOrEqual(2) + expect(result.winner).toBeDefined() + expect(result.canaryLeaks).toBe(0) + // Multi-round means at least two night cue round-trips through the host. + expect(result.nights.length).toBeGreaterThanOrEqual(2) + // Every night's kill was mediated: a proposal preceded the kill. + for (const night of result.nights) { + if (night.kill !== undefined) expect(night.proposal).toBeDefined() + } + expect(result.replyLoss.every((row) => row.answered)).toBe(true) + // Daemon-side cross-check: every `answered` verdict is backed by an + // ADMITTED reply wake — a #926 context echo of a dropped wake cannot + // masquerade as an answer. + expect(result.replyWakesAccepted).toBe(result.replyLoss.filter((row) => row.answered).length) + }, 180_000) +}) diff --git a/package.json b/package.json index 716ae793e..62eeab841 100644 --- a/package.json +++ b/package.json @@ -11,7 +11,7 @@ "eval:addons": "pnpm --filter @agentconnect.md/daemon build && node evals/run-addons.mjs", "eval:addons:view": "promptfoo view -n", "eval:collab": "pnpm --filter @agentconnect.md/daemon build && node evals/run-collaboration.mjs", - "eval:collab:contracts": "vitest run evals/test/routing-acceptance.test.ts evals/test/connection-surface.test.ts evals/test/virtual-connections.test.ts evals/test/world-authorization.test.ts evals/test/topology.test.ts evals/test/counting.test.ts evals/test/quota-counting.test.ts evals/test/cross-room-counting.test.ts evals/test/werewolf.test.ts evals/test/game-runner.test.ts evals/test/game-subject.test.ts evals/test/collaboration-game-provider.test.ts evals/test/game-result-assertion.test.ts packages/daemon/test/evaluation-game-ingress.test.ts packages/daemon/test/evaluation-game-tools.test.ts", + "eval:collab:contracts": "vitest run evals/test/routing-acceptance.test.ts evals/test/connection-surface.test.ts evals/test/virtual-connections.test.ts evals/test/world-authorization.test.ts evals/test/topology.test.ts evals/test/counting.test.ts evals/test/quota-counting.test.ts evals/test/cross-room-counting.test.ts evals/test/werewolf.test.ts evals/test/webchat-night-collection.test.ts evals/test/webchat-werewolf.test.ts evals/test/game-runner.test.ts evals/test/game-subject.test.ts evals/test/collaboration-game-provider.test.ts evals/test/game-result-assertion.test.ts packages/daemon/test/evaluation-game-ingress.test.ts packages/daemon/test/evaluation-game-tools.test.ts", "eval:collab:routing": "vitest run evals/test/routing-acceptance.test.ts evals/test/connection-surface.test.ts", "eval:collab:view": "promptfoo view -n", "eval:contracts": "vitest run packages/daemon/test/evaluation-events.test.ts packages/daemon/test/evaluation-atif.test.ts packages/daemon/test/evaluation-permission.test.ts packages/daemon/test/evaluation-runner.test.ts packages/daemon/test/daemon-evaluation.test.ts evals/test/outcome.test.ts evals/test/provider.test.ts evals/test/paired-summary.test.ts",