Background
Users want to define common expect checks once in eval.yaml instead of repeating the same block in every evals/cases/*.yaml file.
Today, expect is case-only:
config.EvalConfig has no Expect field.
config.CaseDefaults currently supports defaults such as timeout_seconds, max_turns, and collect_artifacts, but not expect.
config.CaseConfig has Expect Expect yaml:"expect".
defaultEvaluator.runExpectPreCheck() calls resolveExpectConfig(&caseCfg.Expect), so only per-case expect is evaluated.
User Need
Many eval suites have invariant zero-cost checks that should apply to every case, for example:
exit_code: 0
- output must not contain generic failure text or placeholders;
- generated files must exist for every case;
- a common safety/format marker must be present or absent.
Repeating those checks in every case file is noisy and easy to drift. Since expect is the cheap pre-judge gate, making common checks reusable would reduce boilerplate while preserving the existing fast-fail behavior.
Possible Design
Prefer adding expect under cases.defaults, because it matches the existing case-defaults inheritance model:
# evals/eval.yaml
cases:
files:
- evals/cases/*.yaml
defaults:
timeout_seconds: 180
max_turns: 1
expect:
exit_code: 0
must_not_contain:
- "TODO"
- "I cannot"
files_exist:
- "result.json"
Each case could still add or override checks:
# evals/cases/special.yaml
id: special
input:
prompt: Run the special task
expect:
must_contain:
- "SPECIAL_OK"
An alternative is a top-level expect: in eval.yaml, but cases.defaults.expect may be less ambiguous because expect is evaluated per case, after each case's agent run.
Merge Semantics To Define
The implementation should explicitly define how defaults and per-case expect combine. A reasonable starting point:
- If no case-level
expect is set, use cases.defaults.expect.
- Slice fields such as
must_contain, must_not_contain, files_exist, files_not_exist, and file_contains are appended/de-duplicated with defaults first.
- Scalar fields such as
exit_code and golden_file can be overridden by the case when set.
- Existing case-only behavior remains backward compatible.
expect still runs before judge, and a failing merged expect still skips judge execution.
Acceptance Criteria
CaseDefaults or EvalConfig supports a documented eval-level way to define default expect checks.
- The evaluator applies the effective merged
expect for every case.
- Per-case
expect can add to or override defaults using documented rules.
- Existing evals with only case-level
expect behave unchanged.
- Unit tests cover config loading, merge behavior, scalar override behavior, and expect pre-check execution.
- Documentation and templates are updated in
docs/guide/writing-evals.md, docs/zh/guide/writing-evals.md, and skills/skill-upper references/templates.
Background
Users want to define common
expectchecks once ineval.yamlinstead of repeating the same block in everyevals/cases/*.yamlfile.Today,
expectis case-only:config.EvalConfighas noExpectfield.config.CaseDefaultscurrently supports defaults such astimeout_seconds,max_turns, andcollect_artifacts, but notexpect.config.CaseConfighasExpect Expect yaml:"expect".defaultEvaluator.runExpectPreCheck()callsresolveExpectConfig(&caseCfg.Expect), so only per-caseexpectis evaluated.User Need
Many eval suites have invariant zero-cost checks that should apply to every case, for example:
exit_code: 0Repeating those checks in every case file is noisy and easy to drift. Since
expectis the cheap pre-judge gate, making common checks reusable would reduce boilerplate while preserving the existing fast-fail behavior.Possible Design
Prefer adding
expectundercases.defaults, because it matches the existing case-defaults inheritance model:Each case could still add or override checks:
An alternative is a top-level
expect:ineval.yaml, butcases.defaults.expectmay be less ambiguous becauseexpectis evaluated per case, after each case's agent run.Merge Semantics To Define
The implementation should explicitly define how defaults and per-case
expectcombine. A reasonable starting point:expectis set, usecases.defaults.expect.must_contain,must_not_contain,files_exist,files_not_exist, andfile_containsare appended/de-duplicated with defaults first.exit_codeandgolden_filecan be overridden by the case when set.expectstill runs beforejudge, and a failing merged expect still skips judge execution.Acceptance Criteria
CaseDefaultsorEvalConfigsupports a documented eval-level way to define defaultexpectchecks.expectfor every case.expectcan add to or override defaults using documented rules.expectbehave unchanged.docs/guide/writing-evals.md,docs/zh/guide/writing-evals.md, andskills/skill-upperreferences/templates.