feat: freeze the v1 review baseline configuration and calibrate the grader

CHANGELOG.md

@@ -4,9 +4,19 @@ summary: Chronological history of repository and skill changes.
 
 # Changelog
 
-## 2026-07-26 — Added the result-blind review replay evaluator
-
+## 2026-07-26 — Added the replay evaluator, then froze the v1 baseline configuration
+
+- fix: reconcile every recorded figure with its retained artifact
+- fix: attribute the pilot to a reproducible commit and correct the records
+  (`3a8388d42e355e4bc9731b98b6dcd42ffd13ff2f`)
+- feat: report the stratum a run evaluated
+  (`2ae0d23c18f247f49d3cc5e76f26d1cf9610c83e`)
+- fix: make the frozen baseline record auditable, and measure the envelope
+  (`f7787dcba681db1de079f57ce1f2f2941e0923b2`)
+- feat: add baseline strata, grader calibration, and the frozen v1 record
+  (`16b77e447dbcc844edd8f3fb58728d96826e177c`)
 - fix: skip the recipe-execution tests when `just` is absent
+  (`f544aa0c19d97dd4f1aabd7dfab3df08b2ee6a6b`)
 - feat: record the evaluated skill closure with every run
   (`b605051a7385dd310b0eff9dbf14c10dda87c633`)
 - docs: record the measured smoke evaluation and its variance

README.md

@@ -219,9 +219,28 @@ python3 review-suite/scripts/evals/runner.py \
   --runs 5 --report-out out/report.json
 ```
 
-Corpus curation, grader calibration, and the frozen v1 baseline are separate
-follow-up work; this evaluator supplies only the synthetic cases needed to prove
-the protocol.
+Cases are grouped into **strata** under `review-suite/evals/strata/`. A stratum
+is the unit of valid comparison — same target skill, same declared dependency
+closure, same runtime and model, same kind of ground truth — and each directory
+is a complete corpus declaring its own target. `just audit-review-corpus`
+discovers every one of them.
+
+The frozen v1 baseline record lives in `review-suite/evals/baseline/v1/`: the
+immutable configuration, the unscored pilot's per-stratum cost and latency
+envelope, a per-stratum cost-ceiling proposal built from those numbers, the
+grader calibration and adjudication record, a plan for satisfying the
+two-independent-adjudication gate, the ground-truth sourcing and sanitization
+record, and the baseline limitations.
+
+Two things are deliberately still outstanding, and the configuration record says
+so rather than implying otherwise: the scored strata are declared but not
+populated, and no scored run may launch until the repository owner preregisters
+a per-stratum cost ceiling and each private expectation has two independent
+adjudications. Read
+[the limitations record](review-suite/evals/baseline/v1/LIMITATIONS.md) before
+quoting any figure — in particular, **the connector stratum is deferred, not
+satisfied**, so connector-escape recall has never been measured and no
+human-review figure may be reported as a connector figure.
 
 ## Prerequisites
 

justfile

@@ -50,8 +50,14 @@ audit-review-corpus:
 # Result-blind replay evaluation through an explicit real-runtime executor.
 # Deliberately excluded from `test`, `lint`, and `check`: this is the only
 # review-suite command that may spend money.
-eval-review-suite executor:
-  python3 review-suite/scripts/evals/runner.py --executor "{{executor}}"
+#
+# Extra arguments are forwarded to the runner, because a stratum is not
+# reachable without them: `--corpus` defaults to the protocol-proof corpus and
+# `--runs` to 1, so a frozen per-stratum configuration cannot be executed by
+# naming an executor alone. The exact per-stratum invocations are recorded in
+# review-suite/evals/baseline/v1/frozen-configuration.json.
+eval-review-suite executor *args:
+  python3 review-suite/scripts/evals/runner.py --executor "{{executor}}" {{args}}
 
 test-plugins:
   python3 -m unittest discover -s scripts/tests -p 'test_*.py'

review-suite/evals/README.md

@@ -18,28 +18,76 @@ review-suite/evals/
 ├── contracts/                          versioned evaluator schemas
 │   ├── executor-request.schema.json    the complete result-blind payload
 │   ├── executor-response.schema.json   the single reply an executor returns
-│   ├── corpus.schema.json              versions, target closure, case list
+│   ├── corpus.schema.json              versions, target closure, stratum, cases
 │   ├── expectation.schema.json         private material root causes
-│   └── provenance.schema.json          origin and retention authority
-├── corpus/
+│   ├── provenance.schema.json          origin and retention authority
+│   └── calibration.schema.json         probe reviews and required classifications
+├── corpus/                             the protocol-proof corpus
 │   ├── corpus.json                     versions, target closure, case ids
 │   ├── reviewer/PROMPT.md              shared reviewer instructions
 │   ├── reviewer/<case>/packet.json     reviewer-visible artifacts
 │   ├── private/expectations/<case>.json
 │   └── private/provenance/<case>.json
+├── strata/                             one directory per baseline stratum
+│   ├── README.md                       what a stratum is, and the measured envelope
+│   └── <stratum>/                      a complete corpus with its own target
+├── calibration/<case_id>.json          private grader calibration probes
+├── baseline/v1/                        the frozen v1 baseline record
+│   ├── frozen-configuration.json       the immutable configuration
+│   ├── COST-CEILING-PROPOSAL.md        per-stratum ceiling, from pilot numbers
+│   ├── CALIBRATION.md                  what was calibrated, and from what
+│   ├── ADJUDICATION-PLAN.md            how two independent adjudications can hold
+│   ├── SOURCING.md                     ground truth, sanitization, batches
+│   ├── LIMITATIONS.md                  explicit inputs to interpretation
+│   └── pilot/<stratum>.report.json     the unscored pilot's compact reports
 └── artifacts/                          opt-in captured output, not in git
 
 review-suite/scripts/evals/
 ├── protocol.py           versioned request/response contract, failure taxonomy
-├── corpus.py             corpus loading, separation, and naming rules
+├── corpus.py             corpus loading, separation, naming, and discovery
 ├── grader.py             root-cause grading interface and reference grader
+├── calibration.py        calibration sets and the case index they grade against
 ├── report.py             per-attempt records and the aggregate report
 ├── runner.py             fresh-process replay driver
 ├── audit_corpus.py       `just audit-review-corpus`
 ├── fixture_executor.py   deterministic simulation, never a baseline
 └── claude_executor.py    documented real-runtime adapter
 ```
 
+## Strata
+
+A stratum is the unit of valid comparison: same target skill, same declared
+dependency closure, same runtime and model, same kind of ground truth. Each
+directory under `strata/` is a complete corpus declaring its own `target_skill`
+and a `stratum` block naming its id, its ground truth, whether it is scored, and
+what it is for. `just audit-review-corpus` discovers every one of them, so a
+stratum added later is gated without changing the recipe.
+
+Read [`strata/README.md`](strata/README.md) for the measured per-stratum cost
+and latency envelope, and
+[`baseline/v1/LIMITATIONS.md`](baseline/v1/LIMITATIONS.md) before quoting any
+figure. In particular: **the connector stratum is deferred, not satisfied.**
+Connector-escape recall has never been measured here, and no human-review figure
+may be reported as a connector figure.
+
+## Calibration
+
+An uncalibrated grader does not report a conservative score; it reports a
+meaningless one, because matching is containment on normalised text.
+Formulations are therefore calibrated against prose a real reviewer actually
+returned — from the unscored pilot only, never from scored output — and
+`review-suite/scripts/tests/test_eval_calibration.py` replays committed probe
+reviews through the real grader to assert the classification each must receive.
+Calibration probes every boundary: paraphrase, overlapping symptom, duplicate
+report, partial claim, plausible false positive, and accepted non-finding.
+
+See [`baseline/v1/CALIBRATION.md`](baseline/v1/CALIBRATION.md) for what was
+calibrated, what it measured before and after, and what remains un-adjudicated,
+and [`baseline/v1/ADJUDICATION-PLAN.md`](baseline/v1/ADJUDICATION-PLAN.md) for
+how the two-independent-adjudication gate can honestly be satisfied — including
+why a blind agent context sharing a model family with the evaluated reviewer is
+not a legitimate second adjudicator for whether a defect is materially real.
+
 ## Commands
 
 ```bash
@@ -247,8 +295,19 @@ freezing a cost envelope:
 ## Scope and known limitations
 
 This directory proves the protocol, the grading interface, the contamination
-controls, and the failure taxonomy. It deliberately does not curate a
-representative scored corpus, calibrate the grader, or capture a v1 baseline.
+controls, and the failure taxonomy. It now also carries the stratum layout, the
+per-stratum unscored pilot envelope, the grader calibration machinery, and the
+frozen v1 configuration record.
+
+It still does **not** carry a populated scored corpus or a captured v1 baseline.
+`baseline/v1/frozen-configuration.json` declares three scored strata in state
+`declared_unpopulated`, and its `status` is
+`incomplete_pending_owner_preregistration`: the per-stratum cost ceiling must be
+preregistered by the repository owner before any scored output is examined, and
+each private expectation needs two independent adjudications from genuinely
+separate parties. Neither can come from an implementing context.
+[`baseline/v1/SOURCING.md`](baseline/v1/SOURCING.md) records the population
+batches and the ground truth already identified for every required case class.
 
 ### Protocol smoke evaluation
 
@@ -290,25 +349,46 @@ result as a measurement only once the denominator supports it.
 
 ### Limitations for whoever curates the scored corpus
 
+The first two of these have since been measured rather than predicted; see
+[`baseline/v1/LIMITATIONS.md`](baseline/v1/LIMITATIONS.md) for what the pilot
+found and what it cost to fix.
+
 - Surface matching is file-level, because a private root cause names a function
   while a finding names a line. A finding in the right file that the grader does
   not recognize is therefore reported as a partial match needing adjudication,
-  not as a false positive. Calibration must decide how those are scored.
+  not as a false positive. Calibration must decide how those are scored. The
+  pilot showed the sharper form of this: a surface written as a path prefix
+  shares a token with almost every location in a packet, and made a deliberately
+  wrong gating finding an unfalsifiable partial. Write a surface as the smallest
+  identifying symbol.
+
 - The shipped formulations were written before any real run and were not tuned
   afterwards, which is why recall is 0.0 above. That is the conservative
   behaviour this interface is meant to have, and it means grader calibration is
   required before any recall number means anything.
+
+  The scale of it has since been measured, on this ticket's own pilot rather
+  than against the void smoke figure above, which cannot anchor a comparison.
+  Two cases in one 20-attempt batch, differing only in whether their
+  formulations had ever been confronted with real prose: the calibrated case
+  scored recall 1.0 over five attempts with zero false positives and zero
+  referrals, and the deliberately uncalibrated one scored recall 0.0 over five
+  attempts with ten referrals, while the reviewer gated the change every time in
+  both. An uncalibrated expectation reports a number about itself.
+
 - Completeness of the evaluated skill closure is load-bearing, not incidental.
   Earlier revisions omitted first the orchestration protocol and then the three
   lens skills that `review-code-change` requires; each omission changed observed
   behaviour on the incomplete-evidence case. Any change to what a payload
   carries is a change to what is being measured, and starts a new stratum. Every
   run records its closure's membership and digest so a stratum can state what it
   evaluated.
+
 - The choice of scored target, the strata to compare, and the cost envelope to
   preregister all follow from that closure and are deliberately left open. Size
   the envelope from a preregistered run of the chosen closure; do not
   extrapolate it from the protocol smoke run above.
+
 - `expectation.schema.json` requires a `severity` on every root cause that no
   metric currently consumes. Either score severity agreement or drop the
   requirement; do not assume it is being measured.