Build the result-blind review replay evaluator and corpus contract

.gitignore

@@ -3,3 +3,4 @@ __pycache__/
 .DS_Store
 .venv/
 .tools/agentskills/
+review-suite/evals/artifacts/

CHANGELOG.md

@@ -4,10 +4,31 @@ summary: Chronological history of repository and skill changes.
 
 # Changelog
 
+## 2026-07-26 — Added the result-blind review replay evaluator
+
+- fix: skip the recipe-execution tests when `just` is absent
+- feat: record the evaluated skill closure with every run
+  (`b605051a7385dd310b0eff9dbf14c10dda87c633`)
+- docs: record the measured smoke evaluation and its variance
+  (`87ec303d949301c908c3a29cb220bed22d44c775`)
+- fix: evaluate the target skill's whole declared closure
+  (`62a9ed8fab166c7d380724e426449f0585714b07`)
+- docs: pin the recorded smoke evaluation to its run
+  (`f00ce2db80ed3a7bed6afb4962cf0bb5a68390fe`)
+- fix: complete the evaluated skill text and the audit ordering
+  (`67efd94339034674de6ca250f2b03e4a0213fc8b`)
+- fix: close the replay evaluator's review-gate gaps
+  (`6ef8e25ce2e0183ef270111549660461493da5f4`)
+- fix: stop misattributing review failures in the replay evaluator
+  (`e46184d6e856199fe0792d43e7f6e0c5a86e131f`)
+- feat: add the result-blind review replay evaluator
+  (`8f0e9d646ec4e959d7adc7448f5fc7a82f4334d8`)
+
 ## 2026-07-25 — Added coordinator-neutral delegated ticket execution
 
 - fix: pin CI to the established Ruff rule set so dependency drift cannot
   redefine the repository-wide lint gate
+  (`901dc3596207a88b6c8edcf548b5be3151ca7ab2`)
 - feat: add a versioned delegated-execution contract for `implement-ticket`
   (`b53efa674e929c181bdaac63ff0306cb756386db`)
 

README.md

@@ -54,8 +54,9 @@ causes the workflow to fail closed.
 ## Repository Layout
 
 - `skills/` — skill folders, each containing a `SKILL.md` and bundled resources
-- `review-suite/` — canonical code-review contracts, validators, and raw
-  evaluation fixtures shared by repository-owned review skills
+- `review-suite/` — canonical code-review contracts, validators, raw evaluation
+  fixtures, and the result-blind replay evaluator shared by repository-owned
+  review skills
 - `justfile` — common tasks for testing, validation, and formatting
 
 Current reusable agent skills:
@@ -175,6 +176,53 @@ observations with `--output-dir`. A Claude Code headless adapter is bundled:
 just eval-implement-ticket-claude
 ```
 
+### Result-blind review replay evaluation
+
+`review-suite/evals/` is the canonical evaluator that measures what the review
+skills actually do when a real agent runtime executes them repeatedly, rather
+than asserting quality through expected JSON. It changes no review behaviour.
+See [its README](review-suite/evals/README.md) for the protocol, the corpus
+contract, and the grading interface.
+
+Three commands cover it:
+
+```bash
+just test-review-suite                      # deterministic tests, no runtime
+just audit-review-corpus                    # corpus integrity, no runtime
+just eval-review-suite '<executor command>' # the only one that may cost money
+```
+
+`just test` includes the deterministic evaluator tests and never launches a paid
+runtime. `just eval-review-suite` is deliberately absent from `test`, `lint`,
+and `check`. The bundled real-runtime adapter needs the `claude` CLI on `PATH`:
+
+```bash
+just eval-review-suite "python3 review-suite/scripts/evals/claude_executor.py"
+```
+
+Each attempt starts a fresh process and receives one result-blind JSON request:
+the target skill text, the raw review packet, the review contracts, and public
+run identity. Expected findings, private grader labels, provenance, and even the
+case name stay out of the payload, and `just audit-review-corpus` proves it by
+inspecting the complete payload every case would produce. Spawn, timeout,
+runtime, oversized-output, malformed, and protocol failures are reported
+separately from a valid review and are never scored as clean.
+
+The bundled `fixture_executor.py` is a deterministic simulation of a compliant
+reviewer, not a model. Its runs are marked `simulation`, so no baseline report
+can be produced from them. Run the runner directly for repeated attempts,
+per-attempt records, and the aggregate report:
+
+```bash
+python3 review-suite/scripts/evals/runner.py \
+  --executor "python3 review-suite/scripts/evals/claude_executor.py" \
+  --runs 5 --report-out out/report.json
+```
+
+Corpus curation, grader calibration, and the frozen v1 baseline are separate
+follow-up work; this evaluator supplies only the synthetic cases needed to prove
+the protocol.
+
 ## Prerequisites
 
 - Python 3.11+

justfile

@@ -41,6 +41,18 @@ test: test-plugins
 test-review-suite:
   python3 -m unittest discover -s review-suite/scripts/tests -p 'test_*.py'
 
+# Validate the replay corpus: schemas, cross-field expectation semantics,
+# reviewer/private separation, provenance shape, outcome-revealing names, and
+# the complete executor payload. Never launches a model.
+audit-review-corpus:
+  python3 review-suite/scripts/evals/audit_corpus.py
+
+# Result-blind replay evaluation through an explicit real-runtime executor.
+# Deliberately excluded from `test`, `lint`, and `check`: this is the only
+# review-suite command that may spend money.
+eval-review-suite executor:
+  python3 review-suite/scripts/evals/runner.py --executor "{{executor}}"
+
 test-plugins:
   python3 -m unittest discover -s scripts/tests -p 'test_*.py'