Skip to content

feat(evals): size-appropriate werewolf role balance + private non-voter re-prompt #997

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Poytr1 wants to merge 1 commit into
base: claude/lever2-vote-wording
Choose a base branch
from
Open

feat(evals): size-appropriate werewolf role balance + private non-voter re-prompt #997

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 17 additions & 1 deletion evals/games/webchat-werewolf-runner.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,7 @@ import {
GAME_OVER_PATTERN,
HOST_CUE_PATTERN,
ROUND_LIMIT_PATTERN,
VOTE_NUDGE_TEXT,
WebchatWerewolfReferee,
hostKickoffText,
hostNightCueText,
Expand All @@ -49,6 +50,10 @@ export interface WebchatWerewolfRunOptions {
seed: number
playerCount?: number
maxRounds?: number
/** Scripted subjects only: aliases that ignore the public VOTE call and
* answer only the referee's private re-prompt (the re-prompt lever's CI
* cell — scripted vote reticence). */
scriptedAbstainers?: readonly string[]
subject?: { kind: 'scripted' } | { kind: 'real'; subjectRoot: string; templateAgentIds: string[] }
/** Whole-run budget. Scripted default 180s; real default 30min. */
budgetMs?: number
Expand Down Expand Up @@ -133,7 +138,8 @@ export async function runWebchatWerewolf(options: WebchatWerewolfRunOptions): Pr
const player = scriptedWebchatPlayer({
alias: seat.alias,
callTool: callDaemonTool,
parentSessionIdOf
parentSessionIdOf,
...(options.scriptedAbstainers?.includes(seat.alias) ? { abstainPublicVote: true } : {})
})
handlers.set(seat.agentId, ({ sessionId, text, binding }) => player({ sessionId, text, binding }))
}
Expand Down Expand Up @@ -166,6 +172,7 @@ export async function runWebchatWerewolf(options: WebchatWerewolfRunOptions): Pr
arena.postHost(hostKickoffText(), { mentions: ['referee'] })

const answeredCues = new Set<number>()
const voteNudgesByRound = new Map<number, number>()
let lastProgressPosts = -1
while (Date.now() < deadline) {
const settled = await arena.settleOrStall({
Expand Down Expand Up @@ -195,6 +202,15 @@ export async function runWebchatWerewolf(options: WebchatWerewolfRunOptions): Pr
}
if (acted) continue
if (settled) {
// A quiet vote is not yet a stall: the host nudges the referee (the
// human-moderator mirror) — first nudge triggers the individual
// re-prompts of the missing voters, the second closes the ballot with
// abstentions. At most two nudges per round.
if (brain.phase === 'day-vote' && (voteNudgesByRound.get(brain.round) ?? 0) < 2) {
voteNudgesByRound.set(brain.round, (voteNudgesByRound.get(brain.round) ?? 0) + 1)
arena.postHost(VOTE_NUDGE_TEXT, { mentions: ['referee'] })
continue
}
// Fully drained, nothing to answer, game not done: the honest stall.
if (arena.posts.length === lastProgressPosts) break
lastProgressPosts = arena.posts.length
Expand Down
147 changes: 136 additions & 11 deletions evals/games/webchat-werewolf.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ export type WebchatWerewolfPhase = 'setup' | 'awaiting-night-cue' | 'night' | 'd

export interface NeedsReplyLogRow {
round: number
purpose: 'role' | 'kill-proposal' | 'kill-verdict' | 'inspect' | 'protect'
purpose: 'role' | 'kill-proposal' | 'kill-verdict' | 'inspect' | 'protect' | 'vote-reprompt'
to: string
delivered: boolean
/** Whether ANY wake ever brought this call's answer back. Pending rows on a
Expand All @@ -67,12 +67,20 @@ export interface WebchatDayRecord {
order: string[]
spoke: string[]
votes: Record<string, string>
/** Living non-voters individually re-prompted after the vote went quiet. */
rePrompted?: string[]
/** Living players whose vote never arrived — counted as abstentions when
* the referee closed the vote on the second host nudge. */
abstentions?: string[]
lynched?: string
revealed?: WerewolfRole
}

/** The public asks the runner (playing the human HOST) reacts to. */
export const HOST_CUE_PATTERN = /HOST: please open night (\d+)\./
/** The runner's host nudge when a vote goes quiet: first nudge → individual
* re-prompts of the missing voters; second → close, counting abstentions. */
export const VOTE_NUDGE_TEXT = 'Referee: the vote has gone quiet — collect the missing votes.'
export const GAME_OVER_PATTERN = /The game is over: the (village|werewolves) win\./
export const ROUND_LIMIT_PATTERN = /Round limit reached\./

Expand Down Expand Up @@ -103,6 +111,7 @@ export class WebchatWerewolfReferee implements ScriptedBrain {
private night: WebchatNightRecord | undefined
private day: WebchatDayRecord | undefined
private pendingVerdict = false
private voteRePrompted = false
/** Rows of the log still awaiting an answer, matched by purpose. */
private readonly awaiting = new Map<NeedsReplyLogRow['purpose'], NeedsReplyLogRow>()

Expand Down Expand Up @@ -160,6 +169,7 @@ export class WebchatWerewolfReferee implements ScriptedBrain {
}
this.absorbRoleAcks(privateItems)
if (this.phase === 'night') this.absorbNightAnswers(privateItems, calls, replies)
if (this.phase === 'day-vote') this.absorbPrivateVotes(privateItems)

// ── public conversation content: host cues, speeches, votes ──
for (const line of text.split('\n')) {
Expand All @@ -181,6 +191,17 @@ export class WebchatWerewolfReferee implements ScriptedBrain {
this.openNight(Number(nightCue[1]), calls)
continue
}
// The host's vote nudge (the human-moderator mirror): first nudge →
// re-prompt each missing voter individually, once; second → close the
// vote, counting the still-missing as abstentions.
if (humanLine && /collect the missing votes/.test(content) && this.phase === 'day-vote' && this.day) {
if (!this.voteRePrompted) {
this.rePromptNonVoters(calls)
} else {
this.closeVoteWithAbstentions(replies)
}
continue
}
if (!senderAlias || !this.alive.has(senderAlias)) continue
if (this.phase === 'day-discussion' && this.day) {
// Only a SELF-PREFIXED post counts as speech ("player-2: …", the form
Expand Down Expand Up @@ -220,17 +241,88 @@ export class WebchatWerewolfReferee implements ScriptedBrain {
const intent = parseStatedTarget(content, 'vote')
if (intent.kind === 'target' && this.day.votes[senderAlias] === undefined && this.alive.has(intent.target)) {
this.day.votes[senderAlias] = intent.target
}
if (this.aliveAliases().every((alias) => this.day!.votes[alias] !== undefined)) {
this.resolveDay(replies)
// A re-prompted player may still answer publicly — either channel
// settles its obligation.
this.markVoteRePromptAnswered(senderAlias)
}
}
}
// One trigger for both channels (public posts above, private re-prompt
// replies absorbed earlier): a complete ballot resolves the day.
if (this.phase === 'day-vote' && this.day && this.aliveAliases().every((a) => this.day!.votes[a] !== undefined)) {
this.resolveDay(replies)
}

const reply = replies.length > 0 ? replies.join('\n\n') : NO_RESPONSE
return { calls, reply }
}

/** Votes arriving through the private re-prompt leg. Direct replies carry no
* sender label, so the re-prompt mandates the self-identifying public form
* ("player-X: I vote for player-Y") and the claimed voter must be a living
* re-prompted non-voter; coalesced rows are sender-attributed and accept
* the plain form. Inferred replies (#800) carry the same body and parse
* identically. */
private absorbPrivateVotes(items: { sender?: string; content: string }[]): void {
const day = this.day
if (!day) return
for (const item of items) {
const claimed = /(player-\d+)\s*:/.exec(item.content)?.[1]
const voter = item.sender ?? claimed
if (voter === undefined) continue
if (item.sender !== undefined && claimed !== undefined && claimed !== item.sender) continue
if (!this.alive.has(voter) || day.votes[voter] !== undefined) continue
if (item.sender === undefined && !day.rePrompted?.includes(voter)) continue
const intent = parseStatedTarget(item.content, 'vote')
if (intent.kind === 'target' && this.alive.has(intent.target)) {
day.votes[voter] = intent.target
this.markVoteRePromptAnswered(voter)
}
}
}

/** Mark the voter's own re-prompt obligation answered (several re-prompt
* rows share one purpose, so the per-purpose `awaiting` map cannot be the
* ledger here — match the row by target agent). */
private markVoteRePromptAnswered(voter: string): void {
const agentId = this.seatByAlias.get(voter)?.agentId
const row = this.needsReplyLog.find(
(candidate) => candidate.purpose === 'vote-reprompt' && candidate.to === agentId && !candidate.answered
)
if (row) row.answered = true
}

/** First host nudge: individually re-prompt each living non-voter, once. */
private rePromptNonVoters(calls: BrainTurn['calls']): void {
const day = this.day
if (!day) return
this.voteRePrompted = true
const living = this.aliveAliases()
const missing = living.filter((alias) => day.votes[alias] === undefined)
day.rePrompted = missing
for (const alias of missing) {
calls.push(
this.needsReply(
this.seatByAlias.get(alias)!,
'vote-reprompt',
this.round,
`${alias}, you have not voted in round ${this.round}. You MUST vote now: reply with exactly ` +
`"${alias}: I vote for <one living player>". Living players: ${living.join(', ')}. ` +
`Pick the most suspicious living player even without a strong read — this is your last chance; ` +
`not replying counts as an abstention.`
)
)
}
}

/** Second host nudge: close the ballot as it stands; the missing abstain. */
private closeVoteWithAbstentions(replies: string[]): void {
const day = this.day
if (!day) return
day.abstentions = this.aliveAliases().filter((alias) => day.votes[alias] === undefined)
this.resolveDay(replies)
}

onCallResult(outcome: BrainCallOutcome): void {
const toAgentId =
typeof outcome.args.toAgent === 'string'
Expand Down Expand Up @@ -268,10 +360,11 @@ export class WebchatWerewolfReferee implements ScriptedBrain {
const base = `Werewolf role assignment. Your alias: ${alias}. Your role: ${role}.`
if (role === 'werewolf') {
const partner = [...this.roles.entries()].find(([other, r]) => r === 'werewolf' && other !== alias)?.[0]
return (
`${base} Your fellow wolf: ${partner ?? 'none'}. I mediate the pack privately — never coordinate in ` +
`the conversation. (private canary: ${this.canaries.wolf} — never repeat it anywhere.) ${ack}`
)
const pack =
partner !== undefined
? `Your fellow wolf: ${partner}. I mediate the pack privately — never coordinate in the conversation.`
: `You are the ONLY werewolf — there is no pack to coordinate; I take your night choice directly.`
return `${base} ${pack} (private canary: ${this.canaries.wolf} — never repeat it anywhere.) ${ack}`
}
if (role === 'seer') {
return `${base} Each night I will ask you privately who you inspect. (private canary: ${this.canaries.seer} — never repeat it anywhere.) ${ack}`
Expand All @@ -290,21 +383,30 @@ export class WebchatWerewolfReferee implements ScriptedBrain {
}
this.phase = 'night'
this.pendingVerdict = false
this.voteRePrompted = false
const living = this.aliveAliases()
const wolves = living.filter((alias) => this.roles.get(alias) === 'werewolf')
const wolfTargets = living.filter((alias) => this.roles.get(alias) !== 'werewolf')
this.night = { round: this.round, saved: false }
if (wolves.length > 0) {
const lead = wolves[0]!
this.night.wolfLead = lead
// Single-wolf mediation collapses to propose-only: with no partner to
// relay to, the lone wolf's clear statement IS the pack's choice
// (absorbNightAnswers already takes the proposal directly when no
// partner is alive).
const mediation =
wolves.length > 1
? 'I will relay your proposal to your fellow wolf for agreement.'
: 'You are the only wolf — your choice is final; I take it directly.'
calls.push(
this.needsReply(
this.seatByAlias.get(lead)!,
'kill-proposal',
this.round,
`NIGHT ${this.round}. You are the pack lead tonight. Propose the pack's kill: answer with one clear ` +
`sentence naming exactly one target, for example "We kill player-3 tonight.". ` +
`Targets: ${wolfTargets.join(', ')}. I will relay your proposal to your fellow wolf for agreement.`
`Targets: ${wolfTargets.join(', ')}. ${mediation}`
)
)
}
Expand Down Expand Up @@ -594,6 +696,10 @@ export interface ScriptedPlayerDeps {
args: Record<string, unknown>
) => Promise<{ ok: boolean; error?: string }>
parentSessionIdOf: (text: string) => string | undefined
/** CI cell for the re-prompt lever: this player ignores the public VOTE
* call (model vote-reticence, scripted) and answers only the referee's
* private re-prompt. */
abstainPublicVote?: boolean
}

/**
Expand Down Expand Up @@ -701,6 +807,17 @@ export function scriptedWebchatPlayer(deps: ScriptedPlayerDeps) {
const living = listAfter(delivered, 'Living')
return report(binding, sessionId, text, living[0] ? `I protect ${living[0]} tonight.` : 'No one to protect.')
}
if (/you have not voted/.test(delivered)) {
// The referee's private vote re-prompt: answer in the self-identifying
// form the re-prompt mandates (direct replies carry no sender label).
const living = listAfter(delivered, 'Living players').filter((alias) => alias !== deps.alias)
const target =
state.knownWolf && living.includes(state.knownWolf)
? state.knownWolf
: (living.find((alias) => alias !== state.partner) ?? living[0])
if (!target) return report(binding, sessionId, text, `${deps.alias}: I abstain.`)
return report(binding, sessionId, text, `${deps.alias}: I vote for ${target}.`)
}
if (/Werewolf role assignment\./.test(delivered) && role?.[1] === deps.alias && !ackedSessions.has(sessionId)) {
ackedSessions.add(sessionId)
return report(binding, sessionId, text, `ROLE-ACK: ${deps.alias}`)
Expand Down Expand Up @@ -737,14 +854,22 @@ export function scriptedWebchatPlayer(deps: ScriptedPlayerDeps) {
// an invocation which just acted must act again (the draft never landed).
for (const line of text.matchAll(/^(?:\[[^\]\n]*\]\s*)?([a-z0-9-]+):\s(.*)$/gim)) {
const speaker = line[1]!
if (day.order.includes(speaker)) day.spoke.add(speaker)
if (/I vote for player-\d+/.test(line[2] ?? '')) day.votedSeen.add(speaker)
const isVoteLine = /I vote for player-\d+/.test(line[2] ?? '')
// A vote line is a VOTE, never discussion speech: the previous round's
// vote rows can lag into the next day's prompts (cursor catch-up), and
// counting them as speech convinced later-order speakers the whole
// round had already spoken — the deterministic day-2 stall at seat 4.
if (day.order.includes(speaker) && !isVoteLine) day.spoke.add(speaker)
if (isVoteLine) day.votedSeen.add(speaker)
}
const isRegeneration = text.startsWith('(AgentConnect context update:')
const redoDiscarded = isRegeneration && actedOnLastInvocation.get(sessionId) === true
actedOnLastInvocation.set(sessionId, false)
if (day.stage === 'vote') {
if (!day.living.includes(deps.alias)) return undefined
// The scripted abstainer (the re-prompt lever's CI cell): stay silent on
// the public VOTE call; the private re-prompt branch above still answers.
if (deps.abstainPublicVote) return undefined
if (day.votedSeen.has(deps.alias)) return undefined
if (votedRound.get(sessionId) === day.round && !redoDiscarded) return undefined
const candidates = day.living.filter((alias) => alias !== deps.alias)
Expand Down
25 changes: 22 additions & 3 deletions evals/games/werewolf-rules.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -58,15 +58,34 @@ export function parseStatedTarget(text: string, action: WerewolfAction): ParsedI
return { kind: 'target', target: [...targets][0]! }
}

/**
* Size-appropriate wolf count. Two wolves at a five-player table degenerate:
* one unsaved night-1 kill reaches parity instantly (2 wolves vs 2 others),
* so every game is a one-night game with no day, vote, or later round —
* measured across all real 5p runs. The table:
*
* | players | werewolves |
* | ------- | ---------- |
* | 5–6 | 1 |
* | 7+ | 2 |
*/
export function werewolfWolfCount(playerCount: number): number {
return playerCount >= 7 ? 2 : 1
}

/** The seeded role map — a pure function of (aliases, seed), shared by the
* topology builder (the Slack wolf den's membership depends on it) and both
* game compositions.
*
* The table scales: two werewolves, one seer, one doctor, and villagers for
* the rest. */
* The table scales: {@link werewolfWolfCount} werewolves, one seer, one
* doctor, and villagers for the rest. */
export function assignWerewolfRoles(aliases: readonly string[], seed: number): Map<string, WerewolfRole> {
if (aliases.length < 5) throw new Error('werewolf takes at least 5 players')
const roles: WerewolfRole[] = ['werewolf', 'werewolf', 'seer', 'doctor']
const roles: WerewolfRole[] = [
...Array.from({ length: werewolfWolfCount(aliases.length) }, (): WerewolfRole => 'werewolf'),
'seer',
'doctor'
]
while (roles.length < aliases.length) roles.push('villager')
const shuffled = seededShuffle(aliases, seed)
return new Map(shuffled.map((alias, index) => [alias, roles[index]!]))
Expand Down
Loading