Context
Decision probes are injected into phase prompts to prompt the human (via the model) to make decisions before a phase begins.
The correct model behavior is to present the probe as a question to the human and wait for a response before generating the phase output.
Problem
The current eval harness is single-turn: it sends a system prompt + scenario input and expects a complete phase response. The model sees the probe as an instruction to resolve itself, runs exploratory analysis, and the phase output looks like 1a (discovery) rather than 1b (structured plan).
Current workaround
executor.py strips decision probe blocks from the system prompt before sending to the API (PR #69). This tests core planning behavior in isolation but does not test whether the model correctly presents probes to the human.
What is needed
Multi-turn eval support:
- Turn 1: send phase prompt + scenario; model output should include probe question(s) to the human
- Turn 2: send simulated human response; model output should be the full phase deliverable
- Rubric for turn 1: did the model ask the probe? Did it withhold the deliverable until the human responded?
- Rubric for turn 2: existing single-turn rubric applied to the final output
Acceptance criteria
- run_phase supports an optional turns parameter (list of human messages)
- At least one scenario per probe-bearing phase tests probe-asking behavior in turn 1
- Existing single-turn scenarios continue to pass
- executor.py probe-stripping can be removed once multi-turn scenarios cover probe behavior
Context
Decision probes are injected into phase prompts to prompt the human (via the model) to make decisions before a phase begins.
The correct model behavior is to present the probe as a question to the human and wait for a response before generating the phase output.
Problem
The current eval harness is single-turn: it sends a system prompt + scenario input and expects a complete phase response. The model sees the probe as an instruction to resolve itself, runs exploratory analysis, and the phase output looks like 1a (discovery) rather than 1b (structured plan).
Current workaround
executor.py strips decision probe blocks from the system prompt before sending to the API (PR #69). This tests core planning behavior in isolation but does not test whether the model correctly presents probes to the human.
What is needed
Multi-turn eval support:
Acceptance criteria