Add LLM-as-judge scorer, dispatcher validation, and eval aggregation

[argen]

Summary

Adds three major components to the evaluation and harness infrastructure:

1. LLM-as-judge scorer (evals/judge.py) — provider-agnostic scoring tool that reads synthesis outputs and scores them against the rubric using Claude, GPT-4, or Gemini. Complements human scoring with fast, repeatable LLM judgments.

2. Dispatcher validation (scripts/validate-dispatcher.sh) — structural checks ensuring routing fixtures are consistent with the catalog, descriptions are complete, and referenced skills have scenarios frontmatter. Integrated into smoke-check.sh as Step 12.

3. Enhanced eval aggregation — updated evals/score.sh to handle both human (score_human.json) and LLM (score_llm.json) scores, report deltas, and flag runs where |human − llm| ≥ 0.25 for calibration review.

Key Changes

• evals/judge.py (330 lines)

  • Abstract Judge base class with provider-specific implementations: AnthropicJudge, OpenAIJudge, GeminiJudge
  • Lazy SDK imports — only the chosen provider's SDK is required
  • Reads rubric.md + synthesis.md, calls LLM with strict JSON schema, validates response, writes score_llm.json with metadata (provider, model, timestamp, mean)
  • CLI with --dry-run, --provider, --model flags; respects EVAL_JUDGE_PROVIDER and EVAL_JUDGE_MODEL env vars
  • Strict score validation: only {0, 0.25, 0.5, 0.75, 1.0} allowed; all four dimensions + rationales required

• evals/score.sh (refactored)

  • Renamed human scoring output to score_human.json (backward-compatible: reads score.json if score_human.json absent)
  • Added --judge flag to delegate to evals/judge.py
  • Aggregation now reads both human and LLM scores, computes delta, flags |delta| ≥ 0.25 with ← review marker
  • Human score takes precedence for reported mean; LLM shown alongside for calibration

• scripts/validate-dispatcher.sh (141 lines)

  • Check 1: routes.tsv → CATALOG.md dead reference detection
  • Check 2: CATALOG.md description completeness (no empty "When to invoke" fields)
  • Check 3: scenarios frontmatter presence in all referenced skills
  • Integrated into smoke-check.sh Step 12

• tests/dispatcher/routes.tsv (53 lines)

  • Routing fixtures documenting expected dispatch for representative prompts
  • Covers slash commands, artifact intake signals, verb+shape triggers
  • Format: prompt<TAB>expected_skill<TAB>expected_role<TAB>notes

• hooks/session-start-check.sh (27 lines)

  • Lightweight structural guard on SessionStart
  • Verifies presence of CATALOG.md, DISPATCHER.md, chronicle/SCHEMA.md
  • Warns on stderr; does not block session

• Documentation

  • evals/README.md — updated with LLM judge workflow, provider table, recommended scoring sequence
  • examples/use-cases.md — six PM workflows showing multi-skill sequences (discovery interview cycle, strategy kernel review, launch readiness, finance review, weekly rhythm, async decision capture)
  • playbooks/harness-guide.md — determinism invariants, five layers (dispatcher, skill anatomy, chronicle, lifecycle hooks, build-time checks), verification suite
  • CONTRIBUTING.md — added dispatcher routing fixtures section, reference to harness-guide.md

Notable Implementation Details

• Provider abstraction: Each judge implements score(system, user) → str. SDK import failures are caught and reported with install

https://claude.ai/code/session_01JWU8bLomUJVpCL8qyMB3XT