Context
The existing LLM-as-judge eval pipeline tests process compliance well but has three measurable gaps identified by comparing against the skill-creator eval methodology.
Gaps to Close
Gap 1: Score distribution + flakiness detection (Low effort)
The 3-shot retry already runs; the reporter discards the individual shot scores. Adding mean +/- stddev per scenario surfaces which scenarios are inherently flaky vs. consistently passing. A scenario with sigma > 0.2 is a signal, not a green check.
Gap 2: Baseline comparison -- with vs. without skill (High value)
Every scenario currently runs only with the skill prompt injected. We have no signal on whether the prompt is causing the behaviors the rubrics measure, or whether Claude would do them anyway. Adding a without_skill mode to the executor and a delta column to the report closes this.
Gap 3: Trigger accuracy -- delegate to skill-creator (Documentation + scenario quality)
Trigger accuracy validation is owned by the skill-creator skill, not this repo. eval/trigger-eval.json already exists as the scenario input for skill-creator runs. However:
- The current 20 scenarios are too obvious (explicit "PDCA" mentions vs. completely unrelated questions), which likely explains why the optimization plateaued at 0% recall on positive cases
- Agent.md needs a section documenting that trigger accuracy is validated by running skill-creator, not by this test suite
- Scenario quality needs improvement: add ambiguous cases where PDCA should apply but the user does not say so explicitly
Out of Scope
- Building a bespoke trigger_judge.py -- skill-creator owns this and will evolve independently
- Blind A/B comparison between prompt versions -- deferred until a motivating regression exists
Acceptance Criteria
Context
The existing LLM-as-judge eval pipeline tests process compliance well but has three measurable gaps identified by comparing against the skill-creator eval methodology.
Gaps to Close
Gap 1: Score distribution + flakiness detection (Low effort)
The 3-shot retry already runs; the reporter discards the individual shot scores. Adding mean +/- stddev per scenario surfaces which scenarios are inherently flaky vs. consistently passing. A scenario with sigma > 0.2 is a signal, not a green check.
Gap 2: Baseline comparison -- with vs. without skill (High value)
Every scenario currently runs only with the skill prompt injected. We have no signal on whether the prompt is causing the behaviors the rubrics measure, or whether Claude would do them anyway. Adding a without_skill mode to the executor and a delta column to the report closes this.
Gap 3: Trigger accuracy -- delegate to skill-creator (Documentation + scenario quality)
Trigger accuracy validation is owned by the skill-creator skill, not this repo. eval/trigger-eval.json already exists as the scenario input for skill-creator runs. However:
Out of Scope
Acceptance Criteria