Skip to content

feat(config): support eval-level default expect checks #135

Description

@roark47

Background

Users want to define common expect checks once in eval.yaml instead of repeating the same block in every evals/cases/*.yaml file.

Today, expect is case-only:

  • config.EvalConfig has no Expect field.
  • config.CaseDefaults currently supports defaults such as timeout_seconds, max_turns, and collect_artifacts, but not expect.
  • config.CaseConfig has Expect Expect yaml:"expect".
  • defaultEvaluator.runExpectPreCheck() calls resolveExpectConfig(&caseCfg.Expect), so only per-case expect is evaluated.

User Need

Many eval suites have invariant zero-cost checks that should apply to every case, for example:

  • exit_code: 0
  • output must not contain generic failure text or placeholders;
  • generated files must exist for every case;
  • a common safety/format marker must be present or absent.

Repeating those checks in every case file is noisy and easy to drift. Since expect is the cheap pre-judge gate, making common checks reusable would reduce boilerplate while preserving the existing fast-fail behavior.

Possible Design

Prefer adding expect under cases.defaults, because it matches the existing case-defaults inheritance model:

# evals/eval.yaml
cases:
  files:
    - evals/cases/*.yaml
  defaults:
    timeout_seconds: 180
    max_turns: 1
    expect:
      exit_code: 0
      must_not_contain:
        - "TODO"
        - "I cannot"
      files_exist:
        - "result.json"

Each case could still add or override checks:

# evals/cases/special.yaml
id: special
input:
  prompt: Run the special task
expect:
  must_contain:
    - "SPECIAL_OK"

An alternative is a top-level expect: in eval.yaml, but cases.defaults.expect may be less ambiguous because expect is evaluated per case, after each case's agent run.

Merge Semantics To Define

The implementation should explicitly define how defaults and per-case expect combine. A reasonable starting point:

  • If no case-level expect is set, use cases.defaults.expect.
  • Slice fields such as must_contain, must_not_contain, files_exist, files_not_exist, and file_contains are appended/de-duplicated with defaults first.
  • Scalar fields such as exit_code and golden_file can be overridden by the case when set.
  • Existing case-only behavior remains backward compatible.
  • expect still runs before judge, and a failing merged expect still skips judge execution.

Acceptance Criteria

  • CaseDefaults or EvalConfig supports a documented eval-level way to define default expect checks.
  • The evaluator applies the effective merged expect for every case.
  • Per-case expect can add to or override defaults using documented rules.
  • Existing evals with only case-level expect behave unchanged.
  • Unit tests cover config loading, merge behavior, scalar override behavior, and expect pre-check execution.
  • Documentation and templates are updated in docs/guide/writing-evals.md, docs/zh/guide/writing-evals.md, and skills/skill-upper references/templates.

Metadata

Metadata

Labels

No labels
No labels

Fields

No fields configured for Feature.

Projects

Status
In progress

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions