Skip to content

Add multi-turn eval support to test decision probe behavior #70

Description

@kenjudy

Context

Decision probes are injected into phase prompts to prompt the human (via the model) to make decisions before a phase begins.

The correct model behavior is to present the probe as a question to the human and wait for a response before generating the phase output.

Problem

The current eval harness is single-turn: it sends a system prompt + scenario input and expects a complete phase response. The model sees the probe as an instruction to resolve itself, runs exploratory analysis, and the phase output looks like 1a (discovery) rather than 1b (structured plan).

Current workaround

executor.py strips decision probe blocks from the system prompt before sending to the API (PR #69). This tests core planning behavior in isolation but does not test whether the model correctly presents probes to the human.

What is needed

Multi-turn eval support:

  • Turn 1: send phase prompt + scenario; model output should include probe question(s) to the human
  • Turn 2: send simulated human response; model output should be the full phase deliverable
  • Rubric for turn 1: did the model ask the probe? Did it withhold the deliverable until the human responded?
  • Rubric for turn 2: existing single-turn rubric applied to the final output

Acceptance criteria

  • run_phase supports an optional turns parameter (list of human messages)
  • At least one scenario per probe-bearing phase tests probe-asking behavior in turn 1
  • Existing single-turn scenarios continue to pass
  • executor.py probe-stripping can be removed once multi-turn scenarios cover probe behavior

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions