Skip to content

Improve eval coverage: score distribution, baseline comparison, and trigger accuracy #37

Description

@kenjudy

Context

The existing LLM-as-judge eval pipeline tests process compliance well but has three measurable gaps identified by comparing against the skill-creator eval methodology.

Gaps to Close

Gap 1: Score distribution + flakiness detection (Low effort)

The 3-shot retry already runs; the reporter discards the individual shot scores. Adding mean +/- stddev per scenario surfaces which scenarios are inherently flaky vs. consistently passing. A scenario with sigma > 0.2 is a signal, not a green check.

Gap 2: Baseline comparison -- with vs. without skill (High value)

Every scenario currently runs only with the skill prompt injected. We have no signal on whether the prompt is causing the behaviors the rubrics measure, or whether Claude would do them anyway. Adding a without_skill mode to the executor and a delta column to the report closes this.

Gap 3: Trigger accuracy -- delegate to skill-creator (Documentation + scenario quality)

Trigger accuracy validation is owned by the skill-creator skill, not this repo. eval/trigger-eval.json already exists as the scenario input for skill-creator runs. However:

  • The current 20 scenarios are too obvious (explicit "PDCA" mentions vs. completely unrelated questions), which likely explains why the optimization plateaued at 0% recall on positive cases
  • Agent.md needs a section documenting that trigger accuracy is validated by running skill-creator, not by this test suite
  • Scenario quality needs improvement: add ambiguous cases where PDCA should apply but the user does not say so explicitly

Out of Scope

  • Building a bespoke trigger_judge.py -- skill-creator owns this and will evolve independently
  • Blind A/B comparison between prompt versions -- deferred until a motivating regression exists

Acceptance Criteria

  • Reporter produces mean and stddev per scenario result when retried
  • Scenarios with sigma > 0.2 flagged in analyst notes section of report
  • executor.py run_phase supports include_skill_prompt=False mode
  • TestBaselineComparison class asserts positive delta for >= 10 of 15 scenarios
  • Reporter delta column shows with minus without score per scenario
  • eval/trigger-eval.json improved with ambiguous/realistic trigger scenarios (>= 10 new cases)
  • Agent.md documents skill-creator as the trigger accuracy validation tool
  • All existing tests continue to pass

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions