From 16b77e447dbcc844edd8f3fb58728d96826e177c Mon Sep 17 00:00:00 2001 From: Scott Haug Date: Sun, 26 Jul 2026 20:01:20 -0700 Subject: [PATCH 1/5] feat: add baseline strata, grader calibration, and the frozen v1 record MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Summary - Add `review-suite/evals/strata/`, one directory per baseline stratum, with an optional `stratum` block in the corpus index declaring the stratum id, the ground truth its expectations came from, whether it is scored, and its purpose. - Discover every shipped corpus in `just audit-review-corpus`, so a stratum added by a later population batch is gated without editing the recipe. - Add three unscored pilot corpora carrying one byte-identical minimized case, so the declared skill closure is the only variable, and record the measured per-stratum cost and latency envelope from a real-runtime run. - Add grader calibration: a versioned calibration contract, committed probe reviews, and a test that replays each probe through the real grader and asserts its classification, matched root causes, recall, false positives, and adjudication referrals. - Add the frozen v1 baseline record: the immutable configuration, a per-stratum cost-ceiling proposal built from the pilot numbers, the calibration and adjudication record, the ground-truth sourcing and sanitization record, and the baseline limitations. ## Why The evaluator could measure a review but could not yet say what it measured against. Three things were missing and each would have corrupted a baseline quietly. **A stratum had no identity.** The corpus index declares exactly one target skill, and the payload for the orchestrator ships eight documents against a single lens's two. Without a labelled stratum, two incomparable runs could be reported as one figure. Ground truth had no label either, which matters here more than usual: this repository has no pull-request review history at all, so the connector stratum is deferred rather than satisfied, and a human-review figure must never be reported as a connector figure. A test now asserts no shipped corpus claims connector ground truth. **The grader was uncalibrated, which is not the same as conservative.** Matching is containment on normalised text, so formulations written before any real prose existed recognised nothing: the pilot found the real root cause on all three runs and scored recall 0.0. Calibrating against the observed prose — pilot output only, never scored output — moved three fresh runs to recall 1.0 with zero false positives. Calibration also found that a `surface` written as a path prefix shares a token with almost every location in a packet, which made a deliberately wrong gating finding an unfalsifiable partial and left the false-positive rate unmeasurable until the surface was narrowed to a symbol. **The cost envelope could not be inherited.** The earlier smoke figure predates the usage-accounting fix and rests on a denominator of one. Measured fresh, the orchestrator stratum costs about 1.8 times a lens stratum per attempt, and the first attempt of any stratum costs three to four times the ones after it because it pays prompt-cache creation. A ceiling built from a mean would be exceeded by any run whose cache does not stay warm, so the proposal is built from the cold figure. ## Preserved behaviour Review behaviour and the v1 packet and result contracts are unchanged. No file under `skills/` is modified, `review-suite/CONTRACT.md`, both v1 schemas, `validate.py`, the lens prompts, and `review-suite/fixtures/` are untouched, and `just sync-contracts` leaves the bundled mirrors byte-identical. The existing protocol-proof corpus is unchanged. `just eval-review-suite` remains the only command that may spend money and remains unreachable from `test`, `lint`, and `check`. ## Deliberately not done The scored strata are declared and **not populated**, and no baseline figure exists. Two inputs cannot come from an implementing context: a per-stratum cost ceiling preregistered by the repository owner before any scored output is examined, and two independent adjudications per private expectation from genuinely separate parties. The configuration record states this as its status rather than implying a freeze that has not happened, and the sourcing record names every required case class together with the real adjudicated review already identified for it, so nothing is silently omitted. --- CHANGELOG.md | 5 + README.md | 24 +- review-suite/evals/README.md | 80 ++++- review-suite/evals/baseline/v1/CALIBRATION.md | 126 ++++++++ .../baseline/v1/COST-CEILING-PROPOSAL.md | 91 ++++++ review-suite/evals/baseline/v1/LIMITATIONS.md | 139 +++++++++ review-suite/evals/baseline/v1/SOURCING.md | 143 +++++++++ .../baseline/v1/frozen-configuration.json | 138 +++++++++ .../pilot/pilot-code-simplicity.report.json | 100 ++++++ .../v1/pilot/pilot-orchestrator.report.json | 114 +++++++ .../pilot-solution-simplicity.report.json | 100 ++++++ .../calibration/rollback-guidance-render.json | 290 ++++++++++++++++++ .../evals/contracts/calibration.schema.json | 88 ++++++ .../evals/contracts/corpus.schema.json | 23 ++ review-suite/evals/strata/README.md | 94 ++++++ .../strata/pilot-code-simplicity/corpus.json | 16 + .../rollback-guidance-render.json | 44 +++ .../provenance/rollback-guidance-render.json | 9 + .../pilot-code-simplicity/reviewer/PROMPT.md | 7 + .../rollback-guidance-render/packet.json | 85 +++++ .../strata/pilot-orchestrator/corpus.json | 20 ++ .../rollback-guidance-render.json | 44 +++ .../provenance/rollback-guidance-render.json | 9 + .../pilot-orchestrator/reviewer/PROMPT.md | 7 + .../rollback-guidance-render/packet.json | 85 +++++ .../pilot-solution-simplicity/corpus.json | 16 + .../rollback-guidance-render.json | 44 +++ .../provenance/rollback-guidance-render.json | 9 + .../reviewer/PROMPT.md | 7 + .../rollback-guidance-render/packet.json | 85 +++++ review-suite/scripts/evals/audit_corpus.py | 18 +- review-suite/scripts/evals/calibration.py | 126 ++++++++ review-suite/scripts/evals/corpus.py | 41 ++- .../scripts/tests/test_eval_calibration.py | 181 +++++++++++ 34 files changed, 2393 insertions(+), 15 deletions(-) create mode 100644 review-suite/evals/baseline/v1/CALIBRATION.md create mode 100644 review-suite/evals/baseline/v1/COST-CEILING-PROPOSAL.md create mode 100644 review-suite/evals/baseline/v1/LIMITATIONS.md create mode 100644 review-suite/evals/baseline/v1/SOURCING.md create mode 100644 review-suite/evals/baseline/v1/frozen-configuration.json create mode 100644 review-suite/evals/baseline/v1/pilot/pilot-code-simplicity.report.json create mode 100644 review-suite/evals/baseline/v1/pilot/pilot-orchestrator.report.json create mode 100644 review-suite/evals/baseline/v1/pilot/pilot-solution-simplicity.report.json create mode 100644 review-suite/evals/calibration/rollback-guidance-render.json create mode 100644 review-suite/evals/contracts/calibration.schema.json create mode 100644 review-suite/evals/strata/README.md create mode 100644 review-suite/evals/strata/pilot-code-simplicity/corpus.json create mode 100644 review-suite/evals/strata/pilot-code-simplicity/private/expectations/rollback-guidance-render.json create mode 100644 review-suite/evals/strata/pilot-code-simplicity/private/provenance/rollback-guidance-render.json create mode 100644 review-suite/evals/strata/pilot-code-simplicity/reviewer/PROMPT.md create mode 100644 review-suite/evals/strata/pilot-code-simplicity/reviewer/rollback-guidance-render/packet.json create mode 100644 review-suite/evals/strata/pilot-orchestrator/corpus.json create mode 100644 review-suite/evals/strata/pilot-orchestrator/private/expectations/rollback-guidance-render.json create mode 100644 review-suite/evals/strata/pilot-orchestrator/private/provenance/rollback-guidance-render.json create mode 100644 review-suite/evals/strata/pilot-orchestrator/reviewer/PROMPT.md create mode 100644 review-suite/evals/strata/pilot-orchestrator/reviewer/rollback-guidance-render/packet.json create mode 100644 review-suite/evals/strata/pilot-solution-simplicity/corpus.json create mode 100644 review-suite/evals/strata/pilot-solution-simplicity/private/expectations/rollback-guidance-render.json create mode 100644 review-suite/evals/strata/pilot-solution-simplicity/private/provenance/rollback-guidance-render.json create mode 100644 review-suite/evals/strata/pilot-solution-simplicity/reviewer/PROMPT.md create mode 100644 review-suite/evals/strata/pilot-solution-simplicity/reviewer/rollback-guidance-render/packet.json create mode 100644 review-suite/scripts/evals/calibration.py create mode 100644 review-suite/scripts/tests/test_eval_calibration.py diff --git a/CHANGELOG.md b/CHANGELOG.md index a899a8b..ba24f6e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,9 +4,14 @@ summary: Chronological history of repository and skill changes. # Changelog +## 2026-07-27 — Froze the v1 baseline configuration and calibrated the grader + +- feat: add baseline strata, grader calibration, and the frozen v1 record + ## 2026-07-26 — Added the result-blind review replay evaluator - fix: skip the recipe-execution tests when `just` is absent + (`f544aa0c19d97dd4f1aabd7dfab3df08b2ee6a6b`) - feat: record the evaluated skill closure with every run (`b605051a7385dd310b0eff9dbf14c10dda87c633`) - docs: record the measured smoke evaluation and its variance diff --git a/README.md b/README.md index e009a63..30ba921 100644 --- a/README.md +++ b/README.md @@ -219,9 +219,27 @@ python3 review-suite/scripts/evals/runner.py \ --runs 5 --report-out out/report.json ``` -Corpus curation, grader calibration, and the frozen v1 baseline are separate -follow-up work; this evaluator supplies only the synthetic cases needed to prove -the protocol. +Cases are grouped into **strata** under `review-suite/evals/strata/`. A stratum +is the unit of valid comparison — same target skill, same declared dependency +closure, same runtime and model, same kind of ground truth — and each directory +is a complete corpus declaring its own target. `just audit-review-corpus` +discovers every one of them. + +The frozen v1 baseline record lives in `review-suite/evals/baseline/v1/`: the +immutable configuration, the unscored pilot's per-stratum cost and latency +envelope, a per-stratum cost-ceiling proposal built from those numbers, the +grader calibration and adjudication record, the ground-truth sourcing and +sanitization record, and the baseline limitations. + +Two things are deliberately still outstanding, and the configuration record says +so rather than implying otherwise: the scored strata are declared but not +populated, and no scored run may launch until the repository owner preregisters +a per-stratum cost ceiling and each private expectation has two independent +adjudications. Read +[the limitations record](review-suite/evals/baseline/v1/LIMITATIONS.md) before +quoting any figure — in particular, **the connector stratum is deferred, not +satisfied**, so connector-escape recall has never been measured and no +human-review figure may be reported as a connector figure. ## Prerequisites diff --git a/review-suite/evals/README.md b/review-suite/evals/README.md index 844bfa8..59e2983 100644 --- a/review-suite/evals/README.md +++ b/review-suite/evals/README.md @@ -18,21 +18,34 @@ review-suite/evals/ ├── contracts/ versioned evaluator schemas │ ├── executor-request.schema.json the complete result-blind payload │ ├── executor-response.schema.json the single reply an executor returns -│ ├── corpus.schema.json versions, target closure, case list +│ ├── corpus.schema.json versions, target closure, stratum, cases │ ├── expectation.schema.json private material root causes -│ └── provenance.schema.json origin and retention authority -├── corpus/ +│ ├── provenance.schema.json origin and retention authority +│ └── calibration.schema.json probe reviews and required classifications +├── corpus/ the protocol-proof corpus │ ├── corpus.json versions, target closure, case ids │ ├── reviewer/PROMPT.md shared reviewer instructions │ ├── reviewer//packet.json reviewer-visible artifacts │ ├── private/expectations/.json │ └── private/provenance/.json +├── strata/ one directory per baseline stratum +│ ├── README.md what a stratum is, and the measured envelope +│ └── / a complete corpus with its own target +├── calibration/.json private grader calibration probes +├── baseline/v1/ the frozen v1 baseline record +│ ├── frozen-configuration.json the immutable configuration +│ ├── COST-CEILING-PROPOSAL.md per-stratum ceiling, from pilot numbers +│ ├── CALIBRATION.md what was calibrated, and from what +│ ├── SOURCING.md ground truth, sanitization, batches +│ ├── LIMITATIONS.md explicit inputs to interpretation +│ └── pilot/.report.json the unscored pilot's compact reports └── artifacts/ opt-in captured output, not in git review-suite/scripts/evals/ ├── protocol.py versioned request/response contract, failure taxonomy -├── corpus.py corpus loading, separation, and naming rules +├── corpus.py corpus loading, separation, naming, and discovery ├── grader.py root-cause grading interface and reference grader +├── calibration.py calibration sets and the case index they grade against ├── report.py per-attempt records and the aggregate report ├── runner.py fresh-process replay driver ├── audit_corpus.py `just audit-review-corpus` @@ -40,6 +53,36 @@ review-suite/scripts/evals/ └── claude_executor.py documented real-runtime adapter ``` +## Strata + +A stratum is the unit of valid comparison: same target skill, same declared +dependency closure, same runtime and model, same kind of ground truth. Each +directory under `strata/` is a complete corpus declaring its own `target_skill` +and a `stratum` block naming its id, its ground truth, whether it is scored, and +what it is for. `just audit-review-corpus` discovers every one of them, so a +stratum added later is gated without changing the recipe. + +Read [`strata/README.md`](strata/README.md) for the measured per-stratum cost +and latency envelope, and +[`baseline/v1/LIMITATIONS.md`](baseline/v1/LIMITATIONS.md) before quoting any +figure. In particular: **the connector stratum is deferred, not satisfied.** +Connector-escape recall has never been measured here, and no human-review figure +may be reported as a connector figure. + +## Calibration + +An uncalibrated grader does not report a conservative score; it reports a +meaningless one, because matching is containment on normalised text. +Formulations are therefore calibrated against prose a real reviewer actually +returned — from the unscored pilot only, never from scored output — and +`review-suite/scripts/tests/test_eval_calibration.py` replays committed probe +reviews through the real grader to assert the classification each must receive. +Calibration probes every boundary: paraphrase, overlapping symptom, duplicate +report, partial claim, plausible false positive, and accepted non-finding. + +See [`baseline/v1/CALIBRATION.md`](baseline/v1/CALIBRATION.md) for what was +calibrated, what it measured before and after, and what remains un-adjudicated. + ## Commands ```bash @@ -247,8 +290,19 @@ freezing a cost envelope: ## Scope and known limitations This directory proves the protocol, the grading interface, the contamination -controls, and the failure taxonomy. It deliberately does not curate a -representative scored corpus, calibrate the grader, or capture a v1 baseline. +controls, and the failure taxonomy. It now also carries the stratum layout, the +per-stratum unscored pilot envelope, the grader calibration machinery, and the +frozen v1 configuration record. + +It still does **not** carry a populated scored corpus or a captured v1 baseline. +`baseline/v1/frozen-configuration.json` declares three scored strata in state +`declared_unpopulated`, and its `status` is +`incomplete_pending_owner_preregistration`: the per-stratum cost ceiling must be +preregistered by the repository owner before any scored output is examined, and +each private expectation needs two independent adjudications from genuinely +separate parties. Neither can come from an implementing context. +[`baseline/v1/SOURCING.md`](baseline/v1/SOURCING.md) records the population +batches and the ground truth already identified for every required case class. ### Protocol smoke evaluation @@ -290,14 +344,24 @@ result as a measurement only once the denominator supports it. ### Limitations for whoever curates the scored corpus +The first two of these have since been measured rather than predicted; see +[`baseline/v1/LIMITATIONS.md`](baseline/v1/LIMITATIONS.md) for what the pilot +found and what it cost to fix. + - Surface matching is file-level, because a private root cause names a function while a finding names a line. A finding in the right file that the grader does not recognize is therefore reported as a partial match needing adjudication, - not as a false positive. Calibration must decide how those are scored. + not as a false positive. Calibration must decide how those are scored. The + pilot showed the sharper form of this: a surface written as a path prefix + shares a token with almost every location in a packet, and made a deliberately + wrong gating finding an unfalsifiable partial. Write a surface as the smallest + identifying symbol. - The shipped formulations were written before any real run and were not tuned afterwards, which is why recall is 0.0 above. That is the conservative behaviour this interface is meant to have, and it means grader calibration is - required before any recall number means anything. + required before any recall number means anything. Calibrated against the + pilot's observed prose, the same interface scored recall 1.0 with zero false + positives on three fresh runs. - Completeness of the evaluated skill closure is load-bearing, not incidental. Earlier revisions omitted first the orchestration protocol and then the three lens skills that `review-code-change` requires; each omission changed observed diff --git a/review-suite/evals/baseline/v1/CALIBRATION.md b/review-suite/evals/baseline/v1/CALIBRATION.md new file mode 100644 index 0000000..9f31b92 --- /dev/null +++ b/review-suite/evals/baseline/v1/CALIBRATION.md @@ -0,0 +1,126 @@ +# Grader calibration and adjudication record + +## What calibration is, and why it cannot be skipped + +The reference grader matches an observed finding to a private root cause when +the finding names the affected surface *and* its prose contains one of the +shipped formulations, after normalisation. Containment is exact, so a +formulation written before any real reviewer prose existed can be a perfectly +reasonable sentence and still recognise nothing. An uncalibrated grader does not +report a conservative score; it reports a meaningless one. + +The evaluator that produced this interface said so explicitly: its formulations +were written before any run and were never tuned, which is why its recall was +0.0 while its verdicts were mostly right. + +## Calibration is derived from pilot output only + +Tuning a formulation after seeing scored output is fitting the grader to the +answer. That is why the pilot corpus is disjoint from every scored stratum — a +test asserts it — and why calibration happened before any scored case exists at +all. + +## What was calibrated, from what + +Case `rollback-guidance-render`, carried identically by all three pilot strata. + +**Source run:** three attempts at corpus version `1.0-pilot-orchestrator`, +target `review-code-change`, closure digest `9b2805f14cdd6158`, model +`claude-opus-4-6[1m]`, suite commit `16560d807c66076fcbf3f00d3a87f543c6ae2458`. +Raw output retained outside git at +`review-suite/evals/artifacts/pilot-orchestrator/`. Spend: 0.2857 USD. + +**Observed before calibration:** all three attempts returned `changes_required` +and all three identified the real root cause. All three scored `partial` — +recall 0.0 — because none of the four shipped formulations appeared in the prose +the reviewer actually wrote. + +**Two changes, both recorded in the corpus at version `1.1-pilot-*`:** + +1. `equivalent_formulations` for `rc.unsupported-emitted-subcommand` were + replaced with four phrases drawn verbatim from the observed prose, each + naming the offending subcommand explicitly so a finding about a different + command cannot match: + - `` `storectl export` does not exist in the installed CLI `` + - `` `storectl export`, which is not a registered subcommand `` + - `` there is no `export` subcommand `` + - `` the `export` subcommand does not exist `` +2. `surface` was narrowed from `storectl/guidance.py render_rollback_guidance` + to `render_rollback_guidance`. Surface matching is token-level, so the path + component `storectl` shared a token with essentially every location in the + packet, and a deliberately wrong gating finding at an unrelated file scored + `partial` instead of being charged as a false positive. With a broad surface + the false-positive rate is not measurable at all. + +Formulations for `anf.test-asserts-shape-only` were likewise drawn from observed +prose. Those for `anf.unquoted-store-root` were **not** observed in any run and +remain unvalidated; they are retained because a competent reviewer plausibly +raises the point. + +## Validation run + +Three fresh attempts per stratum at the calibrated version `1.1-pilot-*`, nine +attempts, zero evaluation failures, 0.6031 USD. On the orchestrator stratum the +calibrated formulations scored **recall 1.0, false-positive rate 0.0, zero +adjudication referrals, verdict stability 1.0 over three attempts** against +prose generated after calibration — so the calibration generalised beyond the +exact sentences it was drawn from. It is not proven to generalise indefinitely; +see limitation 3 in [LIMITATIONS.md](LIMITATIONS.md). + +## Calibration cases and how they are enforced + +`review-suite/evals/calibration/.json` holds the probe reviews. +`review-suite/scripts/tests/test_eval_calibration.py` replays each probe through +the real grader against the real shipped expectation and asserts the exact +classification, matched root causes, recall, false-positive ids, and +adjudication referrals. It runs under `just test-review-suite` and `just test`, +and launches nothing. + +The calibration set must probe every grading boundary; a set that omits one +fails the test rather than passing quietly. Current probes for +`rollback-guidance-render`: + +| probe | kind | asserts | +| ----------------------------------- | -------------------------- | -------------------------------------------------------------------------------------------------------- | +| `probe.observed-root-cause` | `observed` | the prose a real reviewer returned is `matched`, recall 1.0 | +| `probe.paraphrase` | `paraphrase` | a differently worded equivalent claim is also `matched` | +| `probe.overlapping-symptom` | `overlapping_symptom` | the same symptom without the cause is `partial`, earns no recall, and is not charged as a false positive | +| `probe.duplicate-report` | `duplicate_report` | one root cause reported twice earns recall once; the second is `duplicate` | +| `probe.partial-claim-wrong-surface` | `partial_claim` | the right cause at the wrong surface is `partial`, neither credited nor punished | +| `probe.plausible-false-positive` | `plausible_false_positive` | a plausible but unevidenced gating finding is `unexpected` and **is** charged as a false positive | +| `probe.accepted-non-finding` | `accepted_non_finding` | a real but non-material observation is `accepted`, not a false positive | + +## Independent adjudication: not satisfied + +Every expectation shipped here was authored by **one** context. The requirement +is two independent adjudications for each material root cause, accepted +non-finding, severity, and allowed formulation, from genuinely separate parties, +with disagreements resolved in a recorded note. + +One context generating both sides is not independence, and nothing here pretends +otherwise. No scored run may launch until this is satisfied. + +What an adjudicator needs to decide, per root cause: + +1. Is the stated root cause the material one, or a symptom of a different one? +2. Is the severity right? +3. Is each accepted formulation genuinely equivalent, and is any of them loose + enough to match a wrong finding? +4. Is each accepted non-finding genuinely non-material, rather than a second + root cause being tolerated away? +5. Is the surface the smallest identifying symbol? + +Disagreements are recorded here. A disagreement that cannot be resolved excludes +the case or marks it unscorable; it is never recorded as a reviewer miss. + +### Adjudication status + +| case | root cause | adjudications | status | +| -------------------------- | ----------------------------------- | ------------- | ------------------------------------------------------------------------------ | +| `rollback-guidance-render` | `rc.unsupported-emitted-subcommand` | 1 | **awaiting second party** | +| `rollback-guidance-render` | `anf.test-asserts-shape-only` | 1 | **awaiting second party** | +| `rollback-guidance-render` | `anf.unquoted-store-root` | 1 | **awaiting second party**; formulations unvalidated against any observed prose | + +This case is a pilot case and is never scored, so its adjudication gap does not +affect a baseline figure. It is listed because the same gap will apply to every +scored case, and the procedure above is the one the scored strata will use. diff --git a/review-suite/evals/baseline/v1/COST-CEILING-PROPOSAL.md b/review-suite/evals/baseline/v1/COST-CEILING-PROPOSAL.md new file mode 100644 index 0000000..01e4d31 --- /dev/null +++ b/review-suite/evals/baseline/v1/COST-CEILING-PROPOSAL.md @@ -0,0 +1,91 @@ +# Per-stratum cost ceiling: proposal + +This is a **proposal**, not a ceiling. A scored run spends real money, so the +ceiling has to be preregistered by the repository owner *before* any scored +output is examined; otherwise a run that looks disappointing can be quietly +extended, or a run that looks good can be quietly stopped. Every number below is +derived from this ticket's own unscored pilot against the fixed usage +accounting. Nothing here is extrapolated from any earlier figure. + +## Why an earlier figure could not be used + +The evaluator's protocol smoke run reported a cost, and that figure is void for +this purpose twice over. Its usage accounting dropped cache-creation and +cache-read tokens, understating input by orders of magnitude, and its verdicts +rested on a denominator of one. The envelope below was therefore measured fresh, +per stratum, at the frozen configuration. + +## Measured pilot input + +Three runs per stratum, one case per stratum, nine attempts, zero evaluation +failures. Suite commit `16560d807c66076fcbf3f00d3a87f543c6ae2458`, model +`claude-opus-4-6[1m]`, timeout 300 s, no retries. + +| stratum | closure docs | input tokens / attempt | cost / attempt observed | latency max | +| --------------------------- | ------------ | ---------------------- | -------------------------- | ----------- | +| `pilot-orchestrator` | 8 | 32,573 | 0.1774, 0.0563, 0.0542 USD | 37.2 s | +| `pilot-solution-simplicity` | 2 | 25,901 | 0.1089, 0.0240, 0.0228 USD | 14.2 s | +| `pilot-code-simplicity` | 2 | 25,691 | 0.1110, 0.0242, 0.0245 USD | 15.0 s | + +## The one non-obvious thing the pilot measured + +Per-attempt cost is not uniform. In every stratum the **first** attempt cost +three to four times the ones after it, because it pays prompt-cache creation +while later attempts read the cache. The whole payload — skill closure, +contracts, reviewer prompt — is identical across attempts of a stratum, so +almost all of it is cacheable. + +A ceiling built from the mean would therefore be exceeded by any real run whose +cache does not stay warm: a stratum executed in short bursts, strata +interleaved, a cache eviction, or simply a longer wall-clock run. The proposal +below is built from the **cold** per-attempt cost, so the ceiling holds in the +worst case rather than the lucky one. + +## Proposal + +Scored composition and run count as frozen: 5 runs per case; 7 cases in the +orchestrator-targeted correctness stratum, 4 in each lens stratum. + +| stratum | cases | runs | attempts | cold cost / attempt | worst-case spend | +25% headroom | **proposed ceiling** | +| ----------------------------- | ----- | ---- | -------- | ------------------- | ---------------- | ------------- | -------------------- | +| `s1-correctness-orchestrator` | 7 | 5 | 35 | 0.1774 USD | 6.21 USD | 7.76 USD | **8.00 USD** | +| `s2-solution-simplicity-lens` | 4 | 5 | 20 | 0.1089 USD | 2.18 USD | 2.72 USD | **3.00 USD** | +| `s3-code-simplicity-lens` | 4 | 5 | 20 | 0.1110 USD | 2.22 USD | 2.78 USD | **3.00 USD** | +| all three | 15 | 5 | 75 | | 10.61 USD | 13.26 USD | **14.00 USD** | + +Expected spend if caching behaves as observed, using the warm figures, is +materially lower: about 3.4 USD for `s1`, 1.0 USD for each lens stratum, roughly +**5.5 USD total**. The ceiling is deliberately the pessimistic number; the +expectation is deliberately not. + +The headroom covers reported-cost variance, a slightly longer answer, and the +possibility that a scored packet is larger than the pilot packet. It does not +cover a model change, a closure change, or a run-count change: each of those is +a new stratum and needs its own measured envelope. + +## Ceilings are per stratum, not blended + +The orchestrator stratum costs roughly 1.8 times a lens stratum per attempt +because it ships eight documents rather than two. One blended figure would +over-fund the lens strata and under-fund the orchestrator, and the first stratum +to run would silently consume another's budget. Track and enforce each +separately. + +## What to do when a ceiling is reached + +Stop further runs in that stratum and record an incomplete baseline for it. +Never reduce repetitions after outputs are visible: choosing a smaller +denominator once results are known is how a stability figure becomes a claim +about the model rather than a measurement. + +## Costs this proposal does not cover + +- The scored run itself is the only paid step; `just test`, `just lint`, + `just check`, and `just audit-review-corpus` never launch a runtime. +- The pilot already spent **0.8889 USD** in total: 0.6031 USD over the nine + attempts recorded here, plus 0.2857 USD over three earlier orchestrator + attempts at corpus version `1.0-pilot-orchestrator`, whose observed prose the + shipped formulations were calibrated against. +- Re-running a stratum after a formulation change is not free. Calibration is + done against pilot output only, so a scored stratum should need no re-run for + grading reasons. diff --git a/review-suite/evals/baseline/v1/LIMITATIONS.md b/review-suite/evals/baseline/v1/LIMITATIONS.md new file mode 100644 index 0000000..983c6c6 --- /dev/null +++ b/review-suite/evals/baseline/v1/LIMITATIONS.md @@ -0,0 +1,139 @@ +# Baseline limitations + +These are explicit inputs to baseline interpretation. Read them before quoting +any figure this directory produces or will produce. + +## 1. The connector stratum is deferred, not satisfied + +**Connector-escape recall has never been measured.** Nothing in this baseline +says anything about whether the review suite catches defects that a connector +review missed. + +This repository has no pull-request review history at all: zero reviews, zero +review threads, and zero comments across every one of its pull requests. There +was no in-repository connector material to curate. A private repository that +does carry connector review history was identified and deliberately excluded on +third-party authority and disclosure grounds; it is not named here, was not +read, and nothing here derives from it. Fabricating escapes was rejected +outright — a baseline frozen against invented cases would reproduce the exact +defect this work exists to correct, and would then miscalibrate every +preregistered gate downstream. + +Ground truth therefore comes from real adjudicated **human** review and from +this suite's own delivery history. Consequences: + +- every stratum is labelled `human-review` or `repository-history`, never + `connector`, in `corpus.json` and in every report; +- a test asserts no shipped corpus claims `connector-review` ground truth, so + the label cannot drift by accident; and +- **a human-review figure must never be reported as a connector-escape figure, + and the two must never be blended.** + +Human review and connector review are not interchangeable ground truth. They +have different miss profiles, different adjudication standards, and different +incentives. Treat the absence of a connector stratum as an open measurement gap, +not as a gap that the human-review stratum partially fills. + +## 2. No aggregate verdict-accuracy rate exists + +The per-attempt grade records `verdict_match`, but the aggregate report has no +verdict-accuracy figure. Worse for this purpose: only `review_result` attempts +are graded at all, so a case whose expected verdict is `blocked` enters no +quality denominator — when the reviewer correctly refuses, the attempt is +classified `blocked` and never graded. This was raised twice during the +evaluator's own review and deliberately deferred there. + +No scored stratum declared in `frozen-configuration.json` expects a `blocked` +verdict, so the gap does not silently drop a scored case from the baseline. That +is a composition decision, and it has a cost: **the baseline will not measure +refusal behaviour on incomplete evidence**, which is one of the behaviours the +surrounding work most wants to know about. Adding such a case class requires the +metric to exist first. + +## 3. Grader matching is textual containment, and does not generalise on its own + +The reference grader matches an observed finding to a private root cause when +the finding names the affected surface *and* its prose contains one of the +shipped formulations, after normalisation. Containment is exact. The pilot +demonstrated both halves of what that means: + +- **Before calibration**, the shipped formulations recognised nothing. Every run + found the real root cause and every run scored `partial`, because the + formulations had been written before any real prose existed. Recall was 0.0 + while the reviewer was, in fact, correct on all three runs. +- **After calibration** against the observed prose, three fresh runs at the + frozen version scored recall 1.0 with zero false positives and zero + adjudication referrals. + +The generalisation limit is real: a future run that phrases the same finding a +fifth way — "export is not among the registered subcommands" — matches none of +the four shipped formulations and would be referred for adjudication. Read +`adjudication_required` as a first-class part of any result, not as noise. A +semantic matcher, or a standing adjudication queue, is a candidate mechanism for +whoever owns v2. + +## 4. A root cause's `surface` must name a symbol, not a path prefix + +Surface matching is token-level and file-level. Calibration found that a surface +written as `storectl/guidance.py render_rollback_guidance` shares the token +`storectl` with essentially every location in the packet, which made a +deliberately wrong gating finding at an unrelated file score `partial` instead +of being charged as a false positive. **With a broad surface, the false-positive +rate is not measurable**, because nearly every wrong finding is referred instead +of counted. + +Narrowing the surface to `render_rollback_guidance` restored the boundary. Any +stratum populated later must write surfaces as the smallest identifying symbol, +and its calibration set must include a `plausible_false_positive` probe that +actually classifies as `unexpected`. + +## 5. An expectation is target-specific + +The three pilot strata carry one byte-identical case so that the declared skill +closure is the only variable. The case's expectation is calibrated for the +orchestrator target, and the two lens strata consequently graded it as a miss — +correctly, since a correctness root cause is not a defect a code-simplicity lens +is contracted to report. Their graded output is therefore **not** a quality +signal; only their payload size, latency, cost, and protocol outcomes are. + +The rule this yields for the scored strata: expectations must be authored for +the lens the stratum targets. Sharing one case list across strata with different +targets would grade contract-faithful reviewers as wrong and invalidate the +affected stratum. + +## 6. Severity is required but not measured + +`expectation.schema.json` requires a `severity` on every root cause, and no +metric consumes it. Severity agreement is not measured. Either score it or drop +the requirement; do not assume it is being measured. + +## 7. The scored corpus is not yet populated + +`frozen-configuration.json` declares three scored strata in state +`declared_unpopulated`. No scored case exists yet, so **no baseline figure +exists yet**. The corpus-population batches, their sourced ground truth, and the +case classes each will carry are recorded in [SOURCING.md](SOURCING.md). Every +case class required of the corpus is named there; none has been silently +omitted. + +## 8. Independent adjudication is outstanding + +Every private expectation currently shipped was authored by one context. That +does not satisfy the requirement for two independent adjudications per material +root cause, accepted non-finding, severity, and allowed formulation, and it is +not presented as satisfying it. See [CALIBRATION.md](CALIBRATION.md) for exactly +what has and has not been adjudicated. + +## 9. Cost figures are runtime-reported, and cache-sensitive + +Cost comes from what the runtime reports, not from an independent meter. The +pilot also showed per-attempt cost varying three- to fourfold within a stratum +purely from prompt-cache state. Any cost figure is a report about one run's +caching behaviour as much as about the work done. + +## 10. A stratum boundary is not a rounding difference + +A change of runtime, runtime version, model, target skill, dependency closure, +or run count creates a new stratum. Comparing across one of those boundaries is +an invalid comparison, not a noisier one. Every report records its closure +membership and digest so a stratum can always state what it evaluated. diff --git a/review-suite/evals/baseline/v1/SOURCING.md b/review-suite/evals/baseline/v1/SOURCING.md new file mode 100644 index 0000000..be2d994 --- /dev/null +++ b/review-suite/evals/baseline/v1/SOURCING.md @@ -0,0 +1,143 @@ +# Ground-truth sourcing, sanitization, and the population batches + +Every case in this corpus must come from a review that really happened and +really adjudicated the finding. Nothing may be invented: a corpus of fabricated +escapes would reproduce the exact defect this work exists to correct, and would +then miscalibrate every gate that reads the baseline. + +## Sources used + +| source | visibility | what it supplies | +| --------------------- | ---------- | ---------------------------------------------------------------------------------------------------------------------------------- | +| `shaug/atelier` | **public** | 899 pull-request review comments, all authored by the repository owner, across ~354 pull requests. | +| `shaug/agent-scripts` | **public** | This suite's own delivery history, including a defect that survived an aggregate `clean` review verdict and was then caught by CI. | + +In `shaug/atelier` the reviewer is the repository owner and the candidates are +in their own repository. This is human review of a real candidate with a real +adjudication trail — the review comment states the concern and the required +outcome, and acceptance is evidenced by a follow-up reply naming the +implementing commit. It is *not* two-party review, and it is not connector +review. Both facts are stated in every stratum label and in the limitations +record. + +## Sources deliberately not used + +### The private connector-bearing repository + +A private repository that does carry genuine connector review history was +identified and **deliberately excluded** on third-party authority and disclosure +grounds. It is not named here, it was not read, and nothing here derives from +it. The consequence — the connector stratum is deferred, not satisfied — is +limitation 1 in [LIMITATIONS.md](LIMITATIONS.md). + +### `shaug/eldritchdark` + +`shaug/eldritchdark` was available as a source (roughly 361 reviews) and was +**not read at all**. Only its visibility was checked: it is **private**, while +this repository is public. + +This was a deliberate choice, not an oversight. Retention requires that a case +carry no business logic, domain identifier, customer context, credential, +secret, or hidden reasoning, and when a case cannot be minimized without losing +the behaviour it demonstrates it must be dropped rather than have the rule +weakened. `shaug/atelier` is public and supplies every required case class with +clean provenance, so reading a private repository would have added disclosure +risk for no coverage gain. When in doubt, exclude. + +**No content from any private repository reached any artifact in this +repository.** No review comment, diff, or file from a private repository was +opened. + +## Sanitization rule applied to every case + +Even with public sources, no case reproduces source text. Each case is rewritten +from scratch against a fictional subject that preserves only the *failure +shape*: the requirement, the triggering condition, the surface kind, and the +material consequence. No identifier, path, symbol name, prose, or diff from the +source is copied. Provenance records the real source; the retained artifact does +not carry it. + +## Case record + +### `rollback-guidance-render` — pilot, unscored + +| field | value | +| ---------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| strata | `pilot-orchestrator`, `pilot-solution-simplicity`, `pilot-code-simplicity` (byte-identical) | +| source | `shaug/atelier` pull request 492, review comment 2882160198, authored by the repository owner. Public. | +| adjudication | The reviewer required the change; the follow-up reply (comment 2882224395) names the implementing commit and the regression tests added. Accepted, not deferred. | +| failure shape retained | Rendered operator guidance emits a subcommand the tool does not register, so the documented procedure fails at its first step. The accompanying test asserts the rendered shape rather than command validity, which is why a green suite did not catch it. | +| origin | `minimized_reproduction` | +| sanitization | Rewritten against a fictional `storectl` CLI. No source identifier, path, symbol, prose, or diff copied. No business logic, domain identifier, customer context, credential, or hidden reasoning. | +| retention authority | Public repository, owner-authored review, no third-party or customer material. | + +## Cases excluded, and why + +| candidate | class it would have served | why excluded | +| ----------------------------------------------------------- | -------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Every candidate in `shaug/eldritchdark` | any | Private repository, not read. Public sources cover every required class, so the disclosure risk bought nothing. See above. | +| Every candidate in the private connector-bearing repository | connector-escape stratum | Excluded on third-party authority and disclosure grounds. Not named, not read, not derived from. | +| The evaluator's own protocol smoke run | none | Its verdicts rest on a denominator of one and its cost accounting predates the usage-accounting fix. It is void as a capability, quality, or cost signal and is not used as a prior, a sanity check, or an estimate. | + +No case class has been silently omitted. Every class the corpus must carry is +listed below with the ground truth already identified for it. + +## Population batches + +The scored corpus is populated in evidence-preserving batches rather than in one +change. This batch delivers the strata layout, the per-stratum pilot envelope, +the grader calibration machinery and its first calibrated case, the +contamination audit over every corpus, the frozen configuration, and the +cost-ceiling proposal. The scored case population follows, one stratum per +batch, because each scored case needs individual sourcing, minimization, and +independent adjudication judgement, and fifteen of those in one change would not +be reviewable. + +The candidate ground truth below was identified while sourcing this batch and is +recorded so the evidence is not lost. Each still requires the adjudication trail +to be re-verified at the source and the reproduction to be minimized fresh. + +### Batch 2 — `s1-correctness-orchestrator`, 7 cases + +| class | candidate ground truth | +| ----------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| multi-file contract or untouched-consumer propagation failure | `shaug/atelier` PR 335, comment 2867594627: a strictness flag was added to one call site while a sibling call site kept calling the same helper without it, so a closed dependency could still be treated as finalized through non-strict signals. Alternate: PR 350, comments 2867972061 and 2867993101 — the same coupling defect in two separate files, both recovery paths skipping external-state reconciliation. | +| concurrency, retry, idempotency, transaction, or data-integrity failure | `shaug/atelier` PR 373, comment 2869270760: a stale-snapshot cleanup guard compared assignees only when the stale assignee was non-null, so a claim taken between collect and apply could be cleared. Alternate: PR 674, comment 2937093107 — one atomic note-plus-status write split into two, so a later failure leaves a fresh blocked reason on a changeset that is not blocked. | +| validation gap where passing tests did not exercise the changed risk | `shaug/agent-scripts` commit `f544aa0`: a probe for an optional executable checked only the return code, but a missing executable makes the call raise, so the intended skip never happened and the whole suite errored. It **survived an aggregate `clean` review verdict at head `b605051`** and was caught by CI. Complete provenance, this repository, no retention question. Alternate: `shaug/atelier` PR 318, comment 2867098097 — a tautological self-ancestor success when two branch references point at the same branch. | +| clean correctness control | `shaug/atelier` PR 335: the hardening hunk in isolation, which the reviewer explicitly adjudicated as correct in the same thread that raised the sibling gap. | +| clean correctness control | `shaug/atelier` PR 417, comment 2870710594: a typed outcome replacing a boolean, implemented and accepted, with the reviewer's stated requirement met. | +| adjudicated rejected or declined finding as negative control | `shaug/atelier` PR 279, comment 2862037362: a reviewer-suggested fallback regression path was **declined on the merits** after the parser was intentionally narrowed to one canonical shape. | +| adjudicated speculative or polish-only finding as negative control | `shaug/atelier` PR 333, comment 2867594626 (a scope question plus "if exclusion is intentional, a brief rationale would help") or PR 356, comment 2868540467 ("possible edge case to validate ... or add a regression test to prove current behaviour is intentional"). Both are real observations that identify no defect. | + +At least three of these require reasoning across multiple files or an untouched +downstream surface: the PR 335 and PR 350 propagation cases and the PR 674 +split-write case all do. + +### Batch 3 — `s2-solution-simplicity-lens`, 4 cases + +| class | candidate ground truth | +| --------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| whole-solution over-engineering | `shaug/atelier` PR 160, comment 2848776499: a service tier given an injected abstraction purely to make calls it could make directly, with the reviewer asking why any abstraction is needed. | +| whole-solution over-engineering | `shaug/atelier` PR 410, comment 2870262209: three overlapping client concepts threading the same root and working directory through every method, to be converged on one already-bound client. | +| requirement-justified near-miss control | `shaug/atelier` PR 417, comment 2870710594: replacing a boolean with a typed outcome looks like extra machinery and is required, because the caller must distinguish an intentional fail-closed block from a genuine failure. | +| requirement-justified near-miss control | `shaug/atelier` PR 277, comments 2861848880 and 2861868165: retry plus fail-closed auto-close after a two-step create looks like defensive scaffolding and is required by deferred-by-default semantics. | + +### Batch 4 — `s3-code-simplicity-lens`, 4 cases + +| class | candidate ground truth | +| -------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| local code-complexity or reuse | `shaug/agent-scripts` commit `9351619`: **the code-simplicity lens under evaluation itself** flagged the last inline copy of a policy predicate on PR 27; all three agreement sites now share one predicate. Adjudicated by this repository's own history. | +| local code-complexity or reuse | `shaug/atelier` PR 410, comment 2870262209, read at the local level: the duplicated client concepts and the repeated argument threading at every call site. | +| behaviour-clarifying near-miss control | `shaug/atelier` PR 630, comment 2906875752: compatibility accessors retained at the true downstream edge where untouched callers still need them — apparent duplication that is justified by the migration boundary. | +| non-material near-miss control | `shaug/atelier` PR 443, comment 2880556753: per-item bullet blocks chosen over a table for formatter stability and diff-friendliness — apparent verbosity that is justified. | + +## Guarantees this record makes + +- Every retained case is sourced from a real adjudicated review; none is + invented. +- No case is labelled connector, and no connector figure is implied. +- No content from any private repository reached any artifact here. +- Every excluded candidate is listed above with its reason. +- Retention authority is recorded per case, and a case whose authority could not + be established would be excluded before scoring rather than minimized into the + corpus. diff --git a/review-suite/evals/baseline/v1/frozen-configuration.json b/review-suite/evals/baseline/v1/frozen-configuration.json new file mode 100644 index 0000000..638f179 --- /dev/null +++ b/review-suite/evals/baseline/v1/frozen-configuration.json @@ -0,0 +1,138 @@ +{ + "record_version": "1.0", + "status": "incomplete_pending_owner_preregistration", + "status_detail": "Every value an implementing run can fix is fixed below. Two values cannot come from an implementing context and are therefore null: the per-stratum cost ceiling, which must be preregistered by the repository owner before any scored output is examined because a scored run spends real money, and the independent adjudication of each private expectation, which one context cannot supply both sides of. No scored run may launch until both are present. Freezing everything else now is what makes the remaining inputs a decision rather than a negotiation.", + "suite_commit": "16560d807c66076fcbf3f00d3a87f543c6ae2458", + "protocol_version": "1.0", + "grader_version": "1.0", + "calibration_version": "1.0", + "runtime": { + "adapter": "review-suite/scripts/evals/claude_executor.py", + "adapter_version": "1.0", + "executor_command": "python3 review-suite/scripts/evals/claude_executor.py", + "runtime": "claude", + "runtime_cli_version": "2.1.92", + "model": "claude-opus-4-6[1m]", + "model_identity_source": "resolved from the runtime's reported modelUsage on every attempt; an attempt that cannot name its model is recorded as a runtime failure rather than a result", + "stratum_rule": "a change of runtime, runtime version, or model creates a new stratum and must never be compared with this one" + }, + "execution": { + "runs_per_case": 5, + "runs_per_case_rationale": "A denominator of one cannot measure a stochastic reviewer, and this suite has already observed the same configuration return `blocked` on one attempt and a merge verdict on the next for the case that tests refusal on incomplete evidence. Five is the smallest count that makes a per-case verdict-stability figure mean anything while keeping the worst-case spend inside the proposed ceiling.", + "timeout_seconds": 300, + "timeout_rationale": "Worst observed pilot latency was 52.7 s on the orchestrator stratum. 300 s is roughly six times that, which absorbs a slow attempt without letting a hung process run unbounded.", + "retry_policy": "none", + "retry_policy_rationale": "A failed attempt is recorded as an evaluation failure and never retried. Retrying would silently replace an unstable attempt with a luckier one and would bias every stability figure toward agreement.", + "max_output_bytes": 4000000, + "process_isolation": "one fresh executor process per attempt", + "cost_ceiling_exhaustion_policy": "Exceeding a stratum's preregistered ceiling stops further runs in that stratum and records an incomplete baseline for it. Repetitions are never reduced after outputs are visible." + }, + "strata": [ + { + "id": "s1-correctness-orchestrator", + "state": "declared_unpopulated", + "target_skill": "review-code-change", + "target_skill_dependencies": [ + "review-solution-simplicity", + "review-correctness", + "review-code-simplicity" + ], + "closure_documents": 8, + "closure_digest_at_this_commit": "9b2805f14cdd6158", + "ground_truth": "human-review and repository-history", + "planned_cases": 7, + "planned_case_classes": [ + "multi-file contract or untouched-consumer propagation failure", + "concurrency, retry, idempotency, transaction, or data-integrity failure", + "validation gap where passing tests did not exercise the changed risk", + "two clean correctness controls", + "two adjudicated rejected, deferred, speculative, or polish-only findings as negative controls" + ], + "envelope_source": "pilot-orchestrator", + "cost_ceiling_usd": null, + "cost_ceiling_proposal_usd": 8.0 + }, + { + "id": "s2-solution-simplicity-lens", + "state": "declared_unpopulated", + "target_skill": "review-solution-simplicity", + "target_skill_dependencies": [], + "closure_documents": 2, + "closure_digest_at_this_commit": "6257ee4448b15874", + "ground_truth": "human-review", + "planned_cases": 4, + "planned_case_classes": [ + "two demonstrated whole-solution over-engineering cases", + "two requirement-justified near-miss controls" + ], + "envelope_source": "pilot-solution-simplicity", + "cost_ceiling_usd": null, + "cost_ceiling_proposal_usd": 3.0 + }, + { + "id": "s3-code-simplicity-lens", + "state": "declared_unpopulated", + "target_skill": "review-code-simplicity", + "target_skill_dependencies": [], + "closure_documents": 2, + "closure_digest_at_this_commit": "a6187d8971eaef24", + "ground_truth": "human-review", + "planned_cases": 4, + "planned_case_classes": [ + "two demonstrated local code-complexity or reuse cases", + "two behaviour-clarifying or non-material near-miss controls" + ], + "envelope_source": "pilot-code-simplicity", + "cost_ceiling_usd": null, + "cost_ceiling_proposal_usd": 3.0 + }, + { + "id": "connector-escape", + "state": "deferred_not_satisfied", + "target_skill": null, + "ground_truth": "connector-review", + "deferral_reason": "No connector review history is available to this repository under acceptable third-party authority and disclosure terms. Connector-escape recall is not measured by this baseline and must not be inferred from any human-review figure.", + "cost_ceiling_usd": null, + "cost_ceiling_proposal_usd": null + } + ], + "metrics_to_report": [ + "per-case and aggregate material-finding recall, with denominators", + "false-clean rate, with denominator", + "false-positive rate, with denominator", + "false-alarm rate, with denominator", + "unique finding contribution per case", + "per-case and mean verdict stability, with denominators", + "per-case and mean finding stability, with denominators", + "every failure-status rate: spawn_failure, timeout, runtime_failure, output_too_large, malformed_output, protocol_mismatch, blocked", + "latency count, mean, p50, min, max", + "reported usage: input tokens, output tokens, cost in USD, with the number of attempts reporting each", + "adjudication referrals, per case and per run", + "confidence and uncertainty appropriate to the observed run count; no point estimate is presented as exact model capability" + ], + "metrics_not_available": [ + { + "metric": "aggregate verdict-accuracy rate", + "detail": "The per-attempt grade records `verdict_match`, but the aggregate report has no verdict-accuracy figure, and only `review_result` attempts are graded at all. A case whose expected verdict is `blocked` therefore enters no quality denominator: when the reviewer correctly refuses, the attempt is classified `blocked` and never graded. This was raised twice during the evaluator's own review and deliberately deferred.", + "consequence_for_this_baseline": "No scored stratum declared above expects a `blocked` verdict, so the gap does not silently drop a scored case. That is a composition decision, not a repair: adding a refusal-on-incomplete-evidence case class to any scored stratum requires the metric first." + }, + { + "metric": "severity agreement", + "detail": "`expectation.schema.json` requires a `severity` on every root cause that no metric consumes. Severity agreement is not measured and must not be assumed." + } + ], + "pending_owner_inputs": [ + "A per-stratum cost ceiling, preregistered before any scored output is examined. Proposals are in COST-CEILING-PROPOSAL.md.", + "Two independent adjudications for each material root cause, accepted non-finding, severity, and allowed equivalent formulation, from genuinely separate parties, with disagreements resolved in a recorded adjudication note. Current state is in CALIBRATION.md." + ], + "pilot_reports": [ + "pilot/pilot-orchestrator.report.json", + "pilot/pilot-solution-simplicity.report.json", + "pilot/pilot-code-simplicity.report.json" + ], + "controlled_artifacts": { + "location": "review-suite/evals/artifacts//.run-.stdout.json", + "in_git": false, + "detail": "Raw executor output is retained outside git by the repository's existing ignore rule. The committed compact reports carry every metric, denominator, and identity a consumer needs; the raw transcripts are referenced by that path identity for anyone re-deriving a figure." + } +} diff --git a/review-suite/evals/baseline/v1/pilot/pilot-code-simplicity.report.json b/review-suite/evals/baseline/v1/pilot/pilot-code-simplicity.report.json new file mode 100644 index 0000000..58c9347 --- /dev/null +++ b/review-suite/evals/baseline/v1/pilot/pilot-code-simplicity.report.json @@ -0,0 +1,100 @@ +{ + "adjudication_required": [], + "attempts": 3, + "baseline_eligible": true, + "configuration": { + "artifacts_retained": true, + "cases": 1, + "corpus_version": "1.1-pilot-code-simplicity", + "executor": "python3 review-suite/scripts/evals/claude_executor.py", + "grader_version": "1.0", + "max_output_bytes": 4000000, + "runs_per_case": 3, + "suite_commit": "16560d807c66076fcbf3f00d3a87f543c6ae2458", + "target_skill": "review-code-simplicity", + "target_skill_dependencies": [], + "target_skill_digest": "a6187d8971eaef24", + "target_skill_documents": [ + "review-code-simplicity/SKILL.md", + "review-code-simplicity/references/code-simplicity-rubric.md" + ], + "timeout_seconds": 300.0 + }, + "failures": { + "blocked_rate": 0.0, + "malformed_output_rate": 0.0, + "output_too_large_rate": 0.0, + "protocol_mismatch_rate": 0.0, + "runtime_failure_rate": 0.0, + "spawn_failure_rate": 0.0, + "timeout_rate": 0.0 + }, + "graded_attempts": 3, + "grader_version": "1.0", + "latency": { + "count": 3, + "max_seconds": 14.965433417353779, + "mean_seconds": 13.427382861574491, + "min_seconds": 12.56160541716963, + "p50_seconds": 12.755109750200063 + }, + "per_case": [ + { + "attempts": 3, + "case_id": "rollback-guidance-render", + "expected_root_cause_ids": [ + "rc.unsupported-emitted-subcommand" + ], + "false_alarm_attempts": 0, + "false_clean_attempts": 3, + "false_positive_attempts": 0, + "finding_stability": 1.0, + "graded_attempts": 3, + "intersection_root_cause_ids": [], + "mean_recall": 0.0, + "stability_denominator": 3, + "statuses": { + "review_result": 3 + }, + "union_root_cause_ids": [], + "verdict_stability": 1.0 + } + ], + "protocol_version": "1.0", + "quality": { + "false_alarm_denominator": 0, + "false_alarm_rate": null, + "false_clean_denominator": 3, + "false_clean_rate": 1.0, + "false_positive_denominator": 3, + "false_positive_rate": 0.0, + "material_finding_recall": 0.0, + "recall_attempts": 3, + "unique_finding_contribution": [] + }, + "report_version": "1.0", + "simulation": false, + "stability": { + "mean_finding_stability": 1.0, + "mean_verdict_stability": 1.0, + "per_case_finding_stability": { + "rollback-guidance-render": 1.0 + }, + "per_case_stability_denominator": { + "rollback-guidance-render": 3 + }, + "per_case_verdict_stability": { + "rollback-guidance-render": 1.0 + }, + "stability_denominator": 3 + }, + "usage": { + "available": true, + "reporting_attempts_cost_usd": 3, + "reporting_attempts_input_tokens": 3, + "reporting_attempts_output_tokens": 3, + "total_cost_usd": 0.15965025, + "total_input_tokens": 77073.0, + "total_output_tokens": 1418.0 + } +} diff --git a/review-suite/evals/baseline/v1/pilot/pilot-orchestrator.report.json b/review-suite/evals/baseline/v1/pilot/pilot-orchestrator.report.json new file mode 100644 index 0000000..454c6b9 --- /dev/null +++ b/review-suite/evals/baseline/v1/pilot/pilot-orchestrator.report.json @@ -0,0 +1,114 @@ +{ + "adjudication_required": [], + "attempts": 3, + "baseline_eligible": true, + "configuration": { + "artifacts_retained": true, + "cases": 1, + "corpus_version": "1.1-pilot-orchestrator", + "executor": "python3 review-suite/scripts/evals/claude_executor.py", + "grader_version": "1.0", + "max_output_bytes": 4000000, + "runs_per_case": 3, + "suite_commit": "16560d807c66076fcbf3f00d3a87f543c6ae2458", + "target_skill": "review-code-change", + "target_skill_dependencies": [ + "review-solution-simplicity", + "review-correctness", + "review-code-simplicity" + ], + "target_skill_digest": "9b2805f14cdd6158", + "target_skill_documents": [ + "review-code-change/SKILL.md", + "review-code-change/references/orchestration-protocol.md", + "review-code-simplicity/SKILL.md", + "review-code-simplicity/references/code-simplicity-rubric.md", + "review-correctness/SKILL.md", + "review-correctness/references/correctness-rubric.md", + "review-solution-simplicity/SKILL.md", + "review-solution-simplicity/references/solution-simplicity-rubric.md" + ], + "timeout_seconds": 300.0 + }, + "failures": { + "blocked_rate": 0.0, + "malformed_output_rate": 0.0, + "output_too_large_rate": 0.0, + "protocol_mismatch_rate": 0.0, + "runtime_failure_rate": 0.0, + "spawn_failure_rate": 0.0, + "timeout_rate": 0.0 + }, + "graded_attempts": 3, + "grader_version": "1.0", + "latency": { + "count": 3, + "max_seconds": 37.20436308393255, + "mean_seconds": 34.76848513881365, + "min_seconds": 30.250969290733337, + "p50_seconds": 36.85012304177508 + }, + "per_case": [ + { + "attempts": 3, + "case_id": "rollback-guidance-render", + "expected_root_cause_ids": [ + "rc.unsupported-emitted-subcommand" + ], + "false_alarm_attempts": 0, + "false_clean_attempts": 0, + "false_positive_attempts": 0, + "finding_stability": 1.0, + "graded_attempts": 3, + "intersection_root_cause_ids": [ + "rc.unsupported-emitted-subcommand" + ], + "mean_recall": 1.0, + "stability_denominator": 3, + "statuses": { + "review_result": 3 + }, + "union_root_cause_ids": [ + "rc.unsupported-emitted-subcommand" + ], + "verdict_stability": 1.0 + } + ], + "protocol_version": "1.0", + "quality": { + "false_alarm_denominator": 0, + "false_alarm_rate": null, + "false_clean_denominator": 3, + "false_clean_rate": 0.0, + "false_positive_denominator": 3, + "false_positive_rate": 0.0, + "material_finding_recall": 1.0, + "recall_attempts": 3, + "unique_finding_contribution": [] + }, + "report_version": "1.0", + "simulation": false, + "stability": { + "mean_finding_stability": 1.0, + "mean_verdict_stability": 1.0, + "per_case_finding_stability": { + "rollback-guidance-render": 1.0 + }, + "per_case_stability_denominator": { + "rollback-guidance-render": 3 + }, + "per_case_verdict_stability": { + "rollback-guidance-render": 1.0 + }, + "stability_denominator": 3 + }, + "usage": { + "available": true, + "reporting_attempts_cost_usd": 3, + "reporting_attempts_input_tokens": 3, + "reporting_attempts_output_tokens": 3, + "total_cost_usd": 0.28784475, + "total_input_tokens": 97719.0, + "total_output_tokens": 4550.0 + } +} diff --git a/review-suite/evals/baseline/v1/pilot/pilot-solution-simplicity.report.json b/review-suite/evals/baseline/v1/pilot/pilot-solution-simplicity.report.json new file mode 100644 index 0000000..5471ef9 --- /dev/null +++ b/review-suite/evals/baseline/v1/pilot/pilot-solution-simplicity.report.json @@ -0,0 +1,100 @@ +{ + "adjudication_required": [], + "attempts": 3, + "baseline_eligible": true, + "configuration": { + "artifacts_retained": true, + "cases": 1, + "corpus_version": "1.1-pilot-solution-simplicity", + "executor": "python3 review-suite/scripts/evals/claude_executor.py", + "grader_version": "1.0", + "max_output_bytes": 4000000, + "runs_per_case": 3, + "suite_commit": "16560d807c66076fcbf3f00d3a87f543c6ae2458", + "target_skill": "review-solution-simplicity", + "target_skill_dependencies": [], + "target_skill_digest": "6257ee4448b15874", + "target_skill_documents": [ + "review-solution-simplicity/SKILL.md", + "review-solution-simplicity/references/solution-simplicity-rubric.md" + ], + "timeout_seconds": 300.0 + }, + "failures": { + "blocked_rate": 0.0, + "malformed_output_rate": 0.0, + "output_too_large_rate": 0.0, + "protocol_mismatch_rate": 0.0, + "runtime_failure_rate": 0.0, + "spawn_failure_rate": 0.0, + "timeout_rate": 0.0 + }, + "graded_attempts": 3, + "grader_version": "1.0", + "latency": { + "count": 3, + "max_seconds": 14.16039741691202, + "mean_seconds": 13.247394027964523, + "min_seconds": 12.726900917012244, + "p50_seconds": 12.854883749969304 + }, + "per_case": [ + { + "attempts": 3, + "case_id": "rollback-guidance-render", + "expected_root_cause_ids": [ + "rc.unsupported-emitted-subcommand" + ], + "false_alarm_attempts": 0, + "false_clean_attempts": 3, + "false_positive_attempts": 0, + "finding_stability": 1.0, + "graded_attempts": 3, + "intersection_root_cause_ids": [], + "mean_recall": 0.0, + "stability_denominator": 3, + "statuses": { + "review_result": 3 + }, + "union_root_cause_ids": [], + "verdict_stability": 1.0 + } + ], + "protocol_version": "1.0", + "quality": { + "false_alarm_denominator": 0, + "false_alarm_rate": null, + "false_clean_denominator": 3, + "false_clean_rate": 1.0, + "false_positive_denominator": 3, + "false_positive_rate": 0.0, + "material_finding_recall": 0.0, + "recall_attempts": 3, + "unique_finding_contribution": [] + }, + "report_version": "1.0", + "simulation": false, + "stability": { + "mean_finding_stability": 1.0, + "mean_verdict_stability": 1.0, + "per_case_finding_stability": { + "rollback-guidance-render": 1.0 + }, + "per_case_stability_denominator": { + "rollback-guidance-render": 3 + }, + "per_case_verdict_stability": { + "rollback-guidance-render": 1.0 + }, + "stability_denominator": 3 + }, + "usage": { + "available": true, + "reporting_attempts_cost_usd": 3, + "reporting_attempts_input_tokens": 3, + "reporting_attempts_output_tokens": 3, + "total_cost_usd": 0.15564775, + "total_input_tokens": 77703.0, + "total_output_tokens": 1197.0 + } +} diff --git a/review-suite/evals/calibration/rollback-guidance-render.json b/review-suite/evals/calibration/rollback-guidance-render.json new file mode 100644 index 0000000..bd54ae5 --- /dev/null +++ b/review-suite/evals/calibration/rollback-guidance-render.json @@ -0,0 +1,290 @@ +{ + "calibration_version": "1.0", + "case_id": "rollback-guidance-render", + "grader_version": "1.0", + "source": "Probes 1 and 7 are the verbatim findings returned by the unscored pilot at corpus version 1.0-pilot-orchestrator, runs 1-3, model claude-opus-4-6[1m], retained as controlled artifacts under review-suite/evals/artifacts/pilot-orchestrator/ and excluded from git. The remaining probes are constructed variants that probe one grading boundary each. No scored output was observed.", + "probes": [ + { + "id": "probe.observed-root-cause", + "kind": "observed", + "note": "The finding the pilot actually returned on every run. Before calibration this scored `partial`, because the shipped formulations were written before any real prose existed.", + "verdict": "changes_required", + "findings": [ + { + "id": "corr.nonexistent-export-subcommand", + "lens": "correctness", + "severity": "blocking", + "confidence": "high", + "rule": "Every emitted command exists in the installed `storectl` interface.", + "evidence": [ + { + "location": "storectl/guidance.py:render_rollback_guidance", + "detail": "The first emitted command is `storectl export --all > {store_root}.backup.json`." + }, + { + "location": "review packet context.data", + "detail": "`storectl/cli.py` registers exactly these subcommands: `apply`, `dump`, `import`, `info`, `migrate`, `status`. There is no `export` subcommand." + } + ], + "concern": "The rendered rollback guidance emits `storectl export`, which does not exist in the installed CLI. An operator copying this command into a shell during an incident will get a command-not-found error.", + "impact": "The documented rollback fails at its first step, so no backup exists before the destructive step runs.", + "proposed_change": "Emit `storectl dump` for the backup step, or render the backup step from the registered command surface.", + "expected_effect": "The rendered guidance runs as written against the installed CLI." + } + ], + "expect": { + "classifications": { + "corr.nonexistent-export-subcommand": "matched" + }, + "matched_root_cause_ids": [ + "rc.unsupported-emitted-subcommand" + ], + "recall": 1.0, + "false_positive_finding_ids": [], + "adjudication_finding_ids": [] + } + }, + { + "id": "probe.paraphrase", + "kind": "paraphrase", + "note": "Same root cause, different wording, different finding id and surface spelling. A grader that only recognises the observed sentence would miss this.", + "verdict": "changes_required", + "findings": [ + { + "id": "corr.rollback-first-step-unrunnable", + "lens": "correctness", + "severity": "blocking", + "confidence": "high", + "rule": "Emitted operator commands must be runnable against the installed CLI.", + "evidence": [ + { + "location": "render_rollback_guidance, returned list element 0", + "detail": "Comparing the rendered command against the registered command surface in the packet context: there is no `export` subcommand." + } + ], + "concern": "The backup step of the rollback procedure names a subcommand the tool never registers, so the sequence cannot be executed from the top.", + "impact": "An operator following the guidance during an incident proceeds to the destructive step with no backup taken.", + "proposed_change": "Render the backup step using a registered subcommand.", + "expected_effect": "Every rendered step is executable against the shipped CLI.", + "location": "storectl/guidance.py" + } + ], + "expect": { + "classifications": { + "corr.rollback-first-step-unrunnable": "matched" + }, + "matched_root_cause_ids": [ + "rc.unsupported-emitted-subcommand" + ], + "recall": 1.0, + "false_positive_finding_ids": [], + "adjudication_finding_ids": [] + } + }, + { + "id": "probe.overlapping-symptom", + "kind": "overlapping_symptom", + "note": "Describes the same symptom - the rollback does not complete - at the right surface without identifying the cause. It must not earn recall, and it must not be charged as a false positive either; it is exactly the case adjudication exists for.", + "verdict": "changes_required", + "findings": [ + { + "id": "corr.rollback-procedure-unreliable", + "lens": "correctness", + "severity": "blocking", + "confidence": "high", + "rule": "Operator guidance must describe a procedure that completes.", + "evidence": [ + { + "location": "storectl/guidance.py:render_rollback_guidance", + "detail": "The three rendered steps are treated as a sequence, but nothing in the change proves the sequence completes." + } + ], + "concern": "Operators cannot complete the documented rollback. The procedure is presented as authoritative without any evidence that it works end to end.", + "impact": "A rollback attempted from this guidance may leave the store in an unknown state.", + "proposed_change": "Prove the rendered procedure completes before presenting it to operators.", + "expected_effect": "The guidance is only emitted once it is known to work.", + "location": "storectl/guidance.py:render_rollback_guidance" + } + ], + "expect": { + "classifications": { + "corr.rollback-procedure-unreliable": "partial" + }, + "matched_root_cause_ids": [], + "recall": 0.0, + "false_positive_finding_ids": [], + "adjudication_finding_ids": [ + "corr.rollback-procedure-unreliable" + ] + } + }, + { + "id": "probe.duplicate-report", + "kind": "duplicate_report", + "note": "One root cause reported twice must earn recall once. Counting the second report again would inflate recall for a reviewer that merely repeats itself.", + "verdict": "changes_required", + "findings": [ + { + "id": "corr.nonexistent-export-subcommand", + "lens": "correctness", + "severity": "blocking", + "confidence": "high", + "rule": "Every emitted command exists in the installed `storectl` interface.", + "evidence": [ + { + "location": "storectl/guidance.py:render_rollback_guidance", + "detail": "The first emitted command is `storectl export --all > {store_root}.backup.json`." + }, + { + "location": "review packet context.data", + "detail": "`storectl/cli.py` registers exactly these subcommands: `apply`, `dump`, `import`, `info`, `migrate`, `status`. There is no `export` subcommand." + } + ], + "concern": "The rendered rollback guidance emits `storectl export`, which does not exist in the installed CLI. An operator copying this command into a shell during an incident will get a command-not-found error.", + "impact": "The documented rollback fails at its first step, so no backup exists before the destructive step runs.", + "proposed_change": "Emit `storectl dump` for the backup step, or render the backup step from the registered command surface.", + "expected_effect": "The rendered guidance runs as written against the installed CLI." + }, + { + "id": "corr.export-not-registered-again", + "lens": "correctness", + "severity": "blocking", + "confidence": "high", + "rule": "Every emitted command exists in the installed `storectl` interface.", + "evidence": [ + { + "location": "storectl/guidance.py:render_rollback_guidance", + "detail": "Restating the same defect from the returned list: the `export` subcommand does not exist." + } + ], + "concern": "The rendered backup command uses `export`, and the `export` subcommand does not exist in the registered surface.", + "impact": "Same consequence as the finding above: the rollback cannot start.", + "proposed_change": "Use a registered subcommand for the backup step.", + "expected_effect": "The backup step runs.", + "location": "storectl/guidance.py:render_rollback_guidance" + } + ], + "expect": { + "classifications": { + "corr.nonexistent-export-subcommand": "matched", + "corr.export-not-registered-again": "duplicate" + }, + "matched_root_cause_ids": [ + "rc.unsupported-emitted-subcommand" + ], + "recall": 1.0, + "false_positive_finding_ids": [], + "adjudication_finding_ids": [] + } + }, + { + "id": "probe.partial-claim-wrong-surface", + "kind": "partial_claim", + "note": "Names the root cause in prose but points at the command registry instead of the renderer, and asks for the opposite change. Partially correct, so it is referred rather than credited or punished.", + "verdict": "changes_required", + "findings": [ + { + "id": "corr.cli-surface-incomplete", + "lens": "correctness", + "severity": "strong_recommendation", + "confidence": "high", + "rule": "The registered command surface should cover documented operations.", + "evidence": [ + { + "location": "storectl/cli.py", + "detail": "The registry is the reason the guidance cannot run: the `export` subcommand does not exist." + } + ], + "concern": "The command registry is missing an export operation that operators are expected to have.", + "impact": "Operators have no export path at all.", + "proposed_change": "Register an export subcommand.", + "expected_effect": "Operators gain an export path.", + "location": "storectl/cli.py" + } + ], + "expect": { + "classifications": { + "corr.cli-surface-incomplete": "partial" + }, + "matched_root_cause_ids": [], + "recall": 0.0, + "false_positive_finding_ids": [], + "adjudication_finding_ids": [ + "corr.cli-surface-incomplete" + ] + } + }, + { + "id": "probe.plausible-false-positive", + "kind": "plausible_false_positive", + "note": "A gating finding about a real-sounding but unevidenced concern at an unrelated surface. This must be charged as a false positive. It is only reachable because the root cause's surface was narrowed to the renderer symbol: a surface written as a path prefix shares tokens with almost every location in the packet, and made this probe an unfalsifiable `partial`.", + "verdict": "changes_required", + "findings": [ + { + "id": "corr.cli-registry-hardcoded", + "lens": "correctness", + "severity": "blocking", + "confidence": "high", + "rule": "Command registration should not be duplicated across modules.", + "evidence": [ + { + "location": "storectl/cli.py", + "detail": "The registered subcommand list is enumerated in the packet context rather than derived at runtime." + } + ], + "concern": "The set of subcommands appears to be maintained by hand, so it can drift from what the tool actually dispatches.", + "impact": "A future subcommand may be dispatchable without appearing in the registry.", + "proposed_change": "Derive the registered surface from the dispatch table.", + "expected_effect": "The registry cannot drift from dispatch.", + "location": "storectl/cli.py" + } + ], + "expect": { + "classifications": { + "corr.cli-registry-hardcoded": "unexpected" + }, + "matched_root_cause_ids": [], + "recall": 0.0, + "false_positive_finding_ids": [ + "corr.cli-registry-hardcoded" + ], + "adjudication_finding_ids": [] + } + }, + { + "id": "probe.accepted-non-finding", + "kind": "accepted_non_finding", + "note": "The pilot's second observed finding. Real and worth saying, but it restates why the defect survived a green suite rather than adding a second root cause, so it is tolerated instead of counted as a false positive.", + "verdict": "changes_required", + "findings": [ + { + "id": "corr.test-does-not-validate-commands", + "lens": "correctness", + "severity": "strong_recommendation", + "confidence": "high", + "rule": "Tests must exercise the changed behavior.", + "evidence": [ + { + "location": "tests/test_guidance.py:test_render_rollback_guidance_emits_three_steps", + "detail": "Asserts only `len(commands) == 3` and that each command is non-blank. It does not verify that the emitted subcommands are valid." + } + ], + "concern": "The test checks structural properties rather than the behavioral requirement in the change contract.", + "impact": "A guidance change can regress the runnability requirement without failing the suite.", + "proposed_change": "Assert the emitted subcommands against the registered command surface.", + "expected_effect": "The suite fails when guidance names an unregistered subcommand.", + "location": "tests/test_guidance.py" + } + ], + "expect": { + "classifications": { + "corr.test-does-not-validate-commands": "accepted" + }, + "matched_root_cause_ids": [], + "recall": 0.0, + "false_positive_finding_ids": [], + "adjudication_finding_ids": [] + } + } + ] +} diff --git a/review-suite/evals/contracts/calibration.schema.json b/review-suite/evals/contracts/calibration.schema.json new file mode 100644 index 0000000..f745991 --- /dev/null +++ b/review-suite/evals/contracts/calibration.schema.json @@ -0,0 +1,88 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://github.com/shaug/agent-scripts/review-suite/evals/contracts/calibration.schema.json", + "title": "Grader calibration set for one corpus case", + "description": "Probe reviews graded against one case's private expectation, with the classification each probe must receive. Calibration proves the shipped formulations recognise a real reviewer's prose and its paraphrases while still refusing overlapping symptoms, partial claims, duplicate reports, and plausible false positives. Calibration is private grading evidence: it never enters an executor request.", + "type": "object", + "additionalProperties": false, + "required": [ + "calibration_version", + "case_id", + "grader_version", + "source", + "probes" + ], + "properties": { + "calibration_version": {"const": "1.0"}, + "case_id": {"type": "string", "pattern": "^[a-z0-9]+(-[a-z0-9]+)*$"}, + "grader_version": {"type": "string", "minLength": 1}, + "source": { + "description": "Where the probe prose came from. An observed probe must name the run that produced it, so a calibrated formulation can always be traced to prose a real reviewer actually wrote rather than to prose the corpus author guessed.", + "type": "string", + "minLength": 1 + }, + "probes": { + "type": "array", + "minItems": 1, + "items": { + "type": "object", + "additionalProperties": false, + "required": ["id", "kind", "note", "verdict", "findings", "expect"], + "properties": { + "id": {"type": "string", "pattern": "^probe\\.[a-z0-9][a-z0-9-]*$"}, + "kind": { + "enum": [ + "observed", + "paraphrase", + "overlapping_symptom", + "duplicate_report", + "partial_claim", + "plausible_false_positive", + "accepted_non_finding" + ] + }, + "note": {"type": "string", "minLength": 1}, + "verdict": {"enum": ["clean", "changes_required", "blocked"]}, + "findings": { + "description": "Findings exactly as a review result carries them. Each is validated against the v1 review-result contract, so a probe can never drift from the shape a real reviewer returns.", + "type": "array", + "minItems": 1, + "items": {"type": "object"} + }, + "expect": { + "type": "object", + "additionalProperties": false, + "required": [ + "classifications", + "matched_root_cause_ids", + "recall", + "false_positive_finding_ids", + "adjudication_finding_ids" + ], + "properties": { + "classifications": { + "description": "The classification the grader must give each finding, keyed by finding id.", + "type": "object" + }, + "matched_root_cause_ids": { + "type": "array", + "items": {"type": "string", "minLength": 1} + }, + "recall": { + "description": "The recall the grader must report for this probe, or null for a case that expects no material root cause. Left untyped because this repository's schema-subset validator has no numeric type; the calibration test compares it to the grader's value exactly." + }, + "false_positive_finding_ids": { + "type": "array", + "items": {"type": "string", "minLength": 1} + }, + "adjudication_finding_ids": { + "type": "array", + "items": {"type": "string", "minLength": 1} + } + } + } + } + } + } + } +} diff --git a/review-suite/evals/contracts/corpus.schema.json b/review-suite/evals/contracts/corpus.schema.json index 9211400..fe5c072 100644 --- a/review-suite/evals/contracts/corpus.schema.json +++ b/review-suite/evals/contracts/corpus.schema.json @@ -23,6 +23,29 @@ "type": "array", "items": {"type": "string", "minLength": 1} }, + "stratum": { + "description": "Which baseline stratum this corpus is. A stratum is the unit of valid comparison: two corpora whose target skill, dependency closure, runtime, or ground-truth kind differ are unlike strata and must never be reported as one figure. Absent on a corpus that predates stratum labelling.", + "type": "object", + "additionalProperties": false, + "required": ["id", "ground_truth", "scored", "purpose"], + "properties": { + "id": {"type": "string", "pattern": "^[a-z0-9]+(-[a-z0-9]+)*$"}, + "ground_truth": { + "description": "Where this stratum's expected outcomes actually came from. `human-review` is adjudicated human review of a real candidate. `repository-history` is this suite's own adjudicated delivery history, including a defect that survived a review verdict and was caught by a later gate. `connector-review` is adjudicated connector review output, which no shipped stratum currently has. `synthetic` is a hand-written case that no real review ever adjudicated, and can never stand in for any of the others.", + "enum": [ + "human-review", + "repository-history", + "connector-review", + "synthetic" + ] + }, + "scored": { + "description": "True when this corpus contributes to the reported baseline. A pilot corpus is false: it exists to size the envelope and must stay disjoint from the scored cases.", + "type": "boolean" + }, + "purpose": {"type": "string", "minLength": 1} + } + }, "cases": { "type": "array", "minItems": 1, diff --git a/review-suite/evals/strata/README.md b/review-suite/evals/strata/README.md new file mode 100644 index 0000000..8848d22 --- /dev/null +++ b/review-suite/evals/strata/README.md @@ -0,0 +1,94 @@ +# Baseline strata + +A stratum is the unit of valid comparison. Two corpora are the same stratum only +when they send the same payload composition to the same runtime: same target +skill, same declared dependency closure, same runtime and model, same kind of +ground truth. Anything else is a different stratum, and reporting two strata as +one figure is an invalid comparison rather than a rougher one. + +Each directory here is a complete, independently loadable corpus, because a +corpus index declares exactly one `target_skill`. `corpus.json` additionally +declares a `stratum` block naming the stratum id, the ground truth its +expectations came from, whether it is scored, and what it is for. + +## Why one directory per stratum rather than one corpus with mixed targets + +`review-code-change` requires `review-solution-simplicity`, +`review-correctness`, and `review-code-simplicity` to be readable and returns an +aggregate `blocked` result naming any that are missing. A stratum targeting the +orchestrator therefore ships eight documents; a stratum targeting one +self-sufficient lens ships two. The measured difference is not cosmetic — see +the pilot numbers below — so the payload has to be a property of the corpus, and +the corpus index is where a target can be swapped without a code change. + +## Ground truth, and the stratum label it forces + +This repository has no pull-request review history: zero reviews, zero review +threads, and zero comments across all of its pull requests. There is no +in-repository connector material to curate, and none may be invented, so ground +truth is sourced from real adjudicated **human** review elsewhere and from this +suite's own delivery history. Every shipped stratum is labelled `human-review` +or `repository-history` accordingly. + +The **connector stratum is deferred, not satisfied.** Connector-escape recall +has never been measured here. A test asserts that no shipped corpus claims +`connector-review` ground truth, so a human-review figure cannot be reported as +a connector figure by accident. See +[the baseline limitations record](../baseline/v1/LIMITATIONS.md). + +## Pilot strata + +The three `pilot-*` corpora are unscored. They exist to establish executor +compatibility, timeout behaviour, and a cost and latency envelope **per +stratum**, and they are disjoint from every scored case: a test asserts no case +id appears both in a scored corpus and in a pilot corpus, because calibrating a +formulation on a case that is also scored fits the grader to the answer. + +All three carry one byte-identical case, so the declared skill closure is the +only variable between them. That isolation is the point, and it has one +deliberate consequence: the case and its expectation are calibrated for the +orchestrator target, so the two lens strata grade as a miss and their graded +output is not a quality signal. Only their payload size, latency, cost, and +protocol outcomes are. + +That consequence is itself a corpus-design rule for the scored strata: **an +expectation is target-specific.** A correctness root cause is not a defect a +code-simplicity lens is contracted to report, so a scored simplicity stratum +must carry expectations authored for its own lens rather than a shared case +list. + +### Measured envelope, three runs per stratum + +Suite commit `16560d807c66076fcbf3f00d3a87f543c6ae2458`, model +`claude-opus-4-6[1m]`, corpus version `1.1-pilot-*`, grader version `1.0`, +timeout 300 s, no retries, nine attempts, zero evaluation failures. + +| stratum | target | docs | digest | input tokens / attempt | cost / attempt (mean) | cost / attempt (first, cold cache) | latency mean | latency max | +| --------------------------- | ---------------------------- | ---- | ------------------ | ---------------------- | --------------------- | ---------------------------------- | ------------ | ----------- | +| `pilot-orchestrator` | `review-code-change` | 8 | `9b2805f14cdd6158` | 32,573 | 0.0959 USD | 0.1774 USD | 34.8 s | 37.2 s | +| `pilot-solution-simplicity` | `review-solution-simplicity` | 2 | `6257ee4448b15874` | 25,901 | 0.0519 USD | 0.1089 USD | 13.2 s | 14.2 s | +| `pilot-code-simplicity` | `review-code-simplicity` | 2 | `a6187d8971eaef24` | 25,691 | 0.0532 USD | 0.1110 USD | 13.4 s | 15.0 s | + +The first attempt in every stratum cost roughly three to four times the +following ones, because it pays prompt-cache creation while later attempts read +the cache. A ceiling built from the mean would be exceeded by any run whose +cache does not stay warm, which is why +[the cost-ceiling proposal](../baseline/v1/COST-CEILING-PROPOSAL.md) is built +from the cold figure. + +Total pilot spend for the recorded run: **0.6031 USD over nine attempts**, plus +**0.2857 USD over three attempts** for the earlier calibration-source run at +corpus version `1.0-pilot-orchestrator`, whose observed prose the shipped +formulations were calibrated against. **0.8889 USD total.** + +## Adding a stratum + +1. Create `review-suite/evals/strata//` with `corpus.json`, + `reviewer/PROMPT.md`, `reviewer//packet.json`, and private + `expectations/` and `provenance/` records for every declared case. +2. Declare the `stratum` block, including honest `ground_truth`. +3. Add a calibration set under `review-suite/evals/calibration/` for every + scored case; the calibration test requires one and will fail without it. +4. Run `just audit-review-corpus`. It discovers every corpus here, so a new + stratum is gated without editing the recipe. +5. Run `just test-review-suite`. diff --git a/review-suite/evals/strata/pilot-code-simplicity/corpus.json b/review-suite/evals/strata/pilot-code-simplicity/corpus.json new file mode 100644 index 0000000..b72676a --- /dev/null +++ b/review-suite/evals/strata/pilot-code-simplicity/corpus.json @@ -0,0 +1,16 @@ +{ + "corpus_version": "1.1-pilot-code-simplicity", + "protocol_version": "1.0", + "grader_version": "1.0", + "target_skill": "review-code-simplicity", + "target_skill_dependencies": [], + "stratum": { + "id": "pilot-code-simplicity", + "ground_truth": "human-review", + "scored": false, + "purpose": "Size the cost and latency envelope for a stratum targeting the self-sufficient code-simplicity lens. Unscored." + }, + "cases": [ + "rollback-guidance-render" + ] +} diff --git a/review-suite/evals/strata/pilot-code-simplicity/private/expectations/rollback-guidance-render.json b/review-suite/evals/strata/pilot-code-simplicity/private/expectations/rollback-guidance-render.json new file mode 100644 index 0000000..fd268eb --- /dev/null +++ b/review-suite/evals/strata/pilot-code-simplicity/private/expectations/rollback-guidance-render.json @@ -0,0 +1,44 @@ +{ + "expectation_version": "1.0", + "case_id": "rollback-guidance-render", + "packet_valid": true, + "expected_verdict": "changes_required", + "material_root_causes": [ + { + "id": "rc.unsupported-emitted-subcommand", + "requirement": "Every emitted command exists in the installed `storectl` interface.", + "trigger": "An operator copies the first rendered line into a shell during a rollback.", + "surface": "render_rollback_guidance", + "consequence": "The documented rollback fails at its first step, because the registered command surface has no `export` subcommand, so the operator has no backup before running the destructive step.", + "severity": "blocking", + "equivalent_formulations": [ + "`storectl export` does not exist in the installed CLI", + "`storectl export`, which is not a registered subcommand", + "there is no `export` subcommand", + "the `export` subcommand does not exist" + ] + } + ], + "accepted_non_findings": [ + { + "id": "anf.unquoted-store-root", + "description": "The rendered commands interpolate the store root without shell quoting. Real, but the packet states the path is operator supplied at the shell they are already in, and the change contract does not claim shell-safe rendering.", + "equivalent_formulations": [ + "the store root is not shell quoted", + "path interpolation is unquoted", + "without shell quoting" + ] + }, + { + "id": "anf.test-asserts-shape-only", + "description": "The added test asserts the command count rather than command validity. Worth saying, and it is the reason the defect survived a green suite, but it restates the root cause rather than adding a second one.", + "equivalent_formulations": [ + "`len(commands) == 3`", + "does not verify that the emitted subcommands are valid", + "does not assert on the actual subcommand names", + "checks structural properties", + "checks count and non-emptiness" + ] + } + ] +} diff --git a/review-suite/evals/strata/pilot-code-simplicity/private/provenance/rollback-guidance-render.json b/review-suite/evals/strata/pilot-code-simplicity/private/provenance/rollback-guidance-render.json new file mode 100644 index 0000000..a50bb03 --- /dev/null +++ b/review-suite/evals/strata/pilot-code-simplicity/private/provenance/rollback-guidance-render.json @@ -0,0 +1,9 @@ +{ + "provenance_version": "1.0", + "case_id": "rollback-guidance-render", + "origin": "minimized_reproduction", + "retention_authority": "Minimized from an adjudicated human review finding in the public repository shaug/atelier, pull request 492, review comment 2882160198, authored by the repository owner. Public source, owner-authored review, no third-party or customer material.", + "sanitization": "Rewritten from scratch against a fictional `storectl` CLI. Retains only the failure shape: rendered operator guidance emits a subcommand the tool does not register, and the accompanying test asserts the rendered shape rather than command validity. Carries no business logic, no domain identifier, no customer context, no credential, and no hidden reasoning from the source repository. No source text was copied.", + "recorded_at": "2026-07-27", + "notes": "Pilot case. Deliberately disjoint from every scored stratum: it exists to size the per-stratum cost and latency envelope and is never graded into the baseline. The identical case is carried by every pilot corpus so the declared skill closure is the only variable between pilot strata." +} diff --git a/review-suite/evals/strata/pilot-code-simplicity/reviewer/PROMPT.md b/review-suite/evals/strata/pilot-code-simplicity/reviewer/PROMPT.md new file mode 100644 index 0000000..bba42bc --- /dev/null +++ b/review-suite/evals/strata/pilot-code-simplicity/reviewer/PROMPT.md @@ -0,0 +1,7 @@ +Review the supplied review packet as a read-only reviewer. Apply the +repository's canonical review packet, finding, and verdict contract exactly as +written, and bind the result to the candidate identity supplied with the run. + +Return only one conforming review result. Derive every conclusion from the +packet and the supplied contracts; no other evidence is available, and no +information about this case exists outside them. diff --git a/review-suite/evals/strata/pilot-code-simplicity/reviewer/rollback-guidance-render/packet.json b/review-suite/evals/strata/pilot-code-simplicity/reviewer/rollback-guidance-render/packet.json new file mode 100644 index 0000000..c7aff60 --- /dev/null +++ b/review-suite/evals/strata/pilot-code-simplicity/reviewer/rollback-guidance-render/packet.json @@ -0,0 +1,85 @@ +{ + "schema_version": "1.0", + "repository": { + "identity": "example/storectl", + "base_branch": "main" + }, + "candidate": { + "head_sha": "7c1d5e2a9b40f38617d4c5a2e08b96f31d7a4c02", + "comparison_base_sha": "2f9a6b18c3d47e05a1b8f26d904c7e3a5b81d6f4", + "diff": { + "format": "unified_diff", + "complete": true, + "content": "diff --git a/storectl/guidance.py b/storectl/guidance.py\n--- a/storectl/guidance.py\n+++ b/storectl/guidance.py\n@@ -14,6 +14,17 @@ def render_apply_plan(plan):\n return \"\\n\".join(lines)\n+\n+\n+def render_rollback_guidance(store_root):\n+ \"\"\"Return the operator commands that undo an applied migration.\"\"\"\n+ return [\n+ f\"storectl export --all > {store_root}.backup.json\",\n+ f\"storectl import {store_root}.backup.json\",\n+ f\"rm -f {store_root}.backup.json\",\n+ ]\ndiff --git a/tests/test_guidance.py b/tests/test_guidance.py\n--- a/tests/test_guidance.py\n+++ b/tests/test_guidance.py\n@@ -31,3 +31,9 @@ def test_render_apply_plan_lists_every_step():\n assert len(render_apply_plan(plan)) > 0\n+\n+\n+def test_render_rollback_guidance_emits_three_steps():\n+ commands = render_rollback_guidance(\"/srv/store\")\n+ assert len(commands) == 3\n+ assert all(command.strip() for command in commands)\n" + } + }, + "change_contract": { + "goal": "Give operators a copy-pasteable rollback procedure after an applied migration.", + "acceptance_criteria": [ + "The rendered guidance is a runnable sequence of `storectl` commands.", + "Every emitted command exists in the installed `storectl` interface.", + "The guidance names the store root the operator actually migrated." + ], + "non_goals": [ + "Perform the rollback automatically.", + "Change how the apply plan itself is rendered." + ], + "preserved_behaviors": [ + "`render_apply_plan` output is unchanged." + ] + }, + "sources": { + "repository_instructions": [ + { + "label": "Operator guidance rules", + "location": "AGENTS.md", + "summary": "Emitted operator commands must be runnable against the installed CLI; never document a subcommand the tool does not expose." + } + ], + "named_documents": [ + { + "label": "storectl CLI reference", + "location": "docs/storectl-cli.md" + } + ], + "nearby_patterns": [ + { + "label": "Apply-plan renderer", + "location": "storectl/guidance.py" + }, + { + "label": "Guidance tests", + "location": "tests/test_guidance.py" + }, + { + "label": "Command surface the CLI actually registers", + "location": "storectl/cli.py" + } + ] + }, + "validation": [ + { + "name": "guidance tests", + "command": "pytest tests/test_guidance.py", + "scope": "focused", + "status": "passed", + "result": "7 passed" + }, + { + "name": "full tests", + "command": "pytest", + "scope": "full", + "status": "passed", + "result": "212 passed" + } + ], + "context": { + "data": [ + "`storectl/cli.py` registers exactly these subcommands: `apply`, `dump`, `import`, `info`, `migrate`, `status`.", + "The store root is an operator-supplied filesystem path." + ], + "operational": [ + "Operators copy this guidance directly into a shell during an incident." + ] + } +} diff --git a/review-suite/evals/strata/pilot-orchestrator/corpus.json b/review-suite/evals/strata/pilot-orchestrator/corpus.json new file mode 100644 index 0000000..ab257cc --- /dev/null +++ b/review-suite/evals/strata/pilot-orchestrator/corpus.json @@ -0,0 +1,20 @@ +{ + "corpus_version": "1.1-pilot-orchestrator", + "protocol_version": "1.0", + "grader_version": "1.0", + "target_skill": "review-code-change", + "target_skill_dependencies": [ + "review-solution-simplicity", + "review-correctness", + "review-code-simplicity" + ], + "stratum": { + "id": "pilot-orchestrator", + "ground_truth": "human-review", + "scored": false, + "purpose": "Size the cost and latency envelope for a stratum whose target is the orchestrator, whose payload therefore carries its three required lens skills. Unscored." + }, + "cases": [ + "rollback-guidance-render" + ] +} diff --git a/review-suite/evals/strata/pilot-orchestrator/private/expectations/rollback-guidance-render.json b/review-suite/evals/strata/pilot-orchestrator/private/expectations/rollback-guidance-render.json new file mode 100644 index 0000000..fd268eb --- /dev/null +++ b/review-suite/evals/strata/pilot-orchestrator/private/expectations/rollback-guidance-render.json @@ -0,0 +1,44 @@ +{ + "expectation_version": "1.0", + "case_id": "rollback-guidance-render", + "packet_valid": true, + "expected_verdict": "changes_required", + "material_root_causes": [ + { + "id": "rc.unsupported-emitted-subcommand", + "requirement": "Every emitted command exists in the installed `storectl` interface.", + "trigger": "An operator copies the first rendered line into a shell during a rollback.", + "surface": "render_rollback_guidance", + "consequence": "The documented rollback fails at its first step, because the registered command surface has no `export` subcommand, so the operator has no backup before running the destructive step.", + "severity": "blocking", + "equivalent_formulations": [ + "`storectl export` does not exist in the installed CLI", + "`storectl export`, which is not a registered subcommand", + "there is no `export` subcommand", + "the `export` subcommand does not exist" + ] + } + ], + "accepted_non_findings": [ + { + "id": "anf.unquoted-store-root", + "description": "The rendered commands interpolate the store root without shell quoting. Real, but the packet states the path is operator supplied at the shell they are already in, and the change contract does not claim shell-safe rendering.", + "equivalent_formulations": [ + "the store root is not shell quoted", + "path interpolation is unquoted", + "without shell quoting" + ] + }, + { + "id": "anf.test-asserts-shape-only", + "description": "The added test asserts the command count rather than command validity. Worth saying, and it is the reason the defect survived a green suite, but it restates the root cause rather than adding a second one.", + "equivalent_formulations": [ + "`len(commands) == 3`", + "does not verify that the emitted subcommands are valid", + "does not assert on the actual subcommand names", + "checks structural properties", + "checks count and non-emptiness" + ] + } + ] +} diff --git a/review-suite/evals/strata/pilot-orchestrator/private/provenance/rollback-guidance-render.json b/review-suite/evals/strata/pilot-orchestrator/private/provenance/rollback-guidance-render.json new file mode 100644 index 0000000..a50bb03 --- /dev/null +++ b/review-suite/evals/strata/pilot-orchestrator/private/provenance/rollback-guidance-render.json @@ -0,0 +1,9 @@ +{ + "provenance_version": "1.0", + "case_id": "rollback-guidance-render", + "origin": "minimized_reproduction", + "retention_authority": "Minimized from an adjudicated human review finding in the public repository shaug/atelier, pull request 492, review comment 2882160198, authored by the repository owner. Public source, owner-authored review, no third-party or customer material.", + "sanitization": "Rewritten from scratch against a fictional `storectl` CLI. Retains only the failure shape: rendered operator guidance emits a subcommand the tool does not register, and the accompanying test asserts the rendered shape rather than command validity. Carries no business logic, no domain identifier, no customer context, no credential, and no hidden reasoning from the source repository. No source text was copied.", + "recorded_at": "2026-07-27", + "notes": "Pilot case. Deliberately disjoint from every scored stratum: it exists to size the per-stratum cost and latency envelope and is never graded into the baseline. The identical case is carried by every pilot corpus so the declared skill closure is the only variable between pilot strata." +} diff --git a/review-suite/evals/strata/pilot-orchestrator/reviewer/PROMPT.md b/review-suite/evals/strata/pilot-orchestrator/reviewer/PROMPT.md new file mode 100644 index 0000000..bba42bc --- /dev/null +++ b/review-suite/evals/strata/pilot-orchestrator/reviewer/PROMPT.md @@ -0,0 +1,7 @@ +Review the supplied review packet as a read-only reviewer. Apply the +repository's canonical review packet, finding, and verdict contract exactly as +written, and bind the result to the candidate identity supplied with the run. + +Return only one conforming review result. Derive every conclusion from the +packet and the supplied contracts; no other evidence is available, and no +information about this case exists outside them. diff --git a/review-suite/evals/strata/pilot-orchestrator/reviewer/rollback-guidance-render/packet.json b/review-suite/evals/strata/pilot-orchestrator/reviewer/rollback-guidance-render/packet.json new file mode 100644 index 0000000..c7aff60 --- /dev/null +++ b/review-suite/evals/strata/pilot-orchestrator/reviewer/rollback-guidance-render/packet.json @@ -0,0 +1,85 @@ +{ + "schema_version": "1.0", + "repository": { + "identity": "example/storectl", + "base_branch": "main" + }, + "candidate": { + "head_sha": "7c1d5e2a9b40f38617d4c5a2e08b96f31d7a4c02", + "comparison_base_sha": "2f9a6b18c3d47e05a1b8f26d904c7e3a5b81d6f4", + "diff": { + "format": "unified_diff", + "complete": true, + "content": "diff --git a/storectl/guidance.py b/storectl/guidance.py\n--- a/storectl/guidance.py\n+++ b/storectl/guidance.py\n@@ -14,6 +14,17 @@ def render_apply_plan(plan):\n return \"\\n\".join(lines)\n+\n+\n+def render_rollback_guidance(store_root):\n+ \"\"\"Return the operator commands that undo an applied migration.\"\"\"\n+ return [\n+ f\"storectl export --all > {store_root}.backup.json\",\n+ f\"storectl import {store_root}.backup.json\",\n+ f\"rm -f {store_root}.backup.json\",\n+ ]\ndiff --git a/tests/test_guidance.py b/tests/test_guidance.py\n--- a/tests/test_guidance.py\n+++ b/tests/test_guidance.py\n@@ -31,3 +31,9 @@ def test_render_apply_plan_lists_every_step():\n assert len(render_apply_plan(plan)) > 0\n+\n+\n+def test_render_rollback_guidance_emits_three_steps():\n+ commands = render_rollback_guidance(\"/srv/store\")\n+ assert len(commands) == 3\n+ assert all(command.strip() for command in commands)\n" + } + }, + "change_contract": { + "goal": "Give operators a copy-pasteable rollback procedure after an applied migration.", + "acceptance_criteria": [ + "The rendered guidance is a runnable sequence of `storectl` commands.", + "Every emitted command exists in the installed `storectl` interface.", + "The guidance names the store root the operator actually migrated." + ], + "non_goals": [ + "Perform the rollback automatically.", + "Change how the apply plan itself is rendered." + ], + "preserved_behaviors": [ + "`render_apply_plan` output is unchanged." + ] + }, + "sources": { + "repository_instructions": [ + { + "label": "Operator guidance rules", + "location": "AGENTS.md", + "summary": "Emitted operator commands must be runnable against the installed CLI; never document a subcommand the tool does not expose." + } + ], + "named_documents": [ + { + "label": "storectl CLI reference", + "location": "docs/storectl-cli.md" + } + ], + "nearby_patterns": [ + { + "label": "Apply-plan renderer", + "location": "storectl/guidance.py" + }, + { + "label": "Guidance tests", + "location": "tests/test_guidance.py" + }, + { + "label": "Command surface the CLI actually registers", + "location": "storectl/cli.py" + } + ] + }, + "validation": [ + { + "name": "guidance tests", + "command": "pytest tests/test_guidance.py", + "scope": "focused", + "status": "passed", + "result": "7 passed" + }, + { + "name": "full tests", + "command": "pytest", + "scope": "full", + "status": "passed", + "result": "212 passed" + } + ], + "context": { + "data": [ + "`storectl/cli.py` registers exactly these subcommands: `apply`, `dump`, `import`, `info`, `migrate`, `status`.", + "The store root is an operator-supplied filesystem path." + ], + "operational": [ + "Operators copy this guidance directly into a shell during an incident." + ] + } +} diff --git a/review-suite/evals/strata/pilot-solution-simplicity/corpus.json b/review-suite/evals/strata/pilot-solution-simplicity/corpus.json new file mode 100644 index 0000000..ba1a4bd --- /dev/null +++ b/review-suite/evals/strata/pilot-solution-simplicity/corpus.json @@ -0,0 +1,16 @@ +{ + "corpus_version": "1.1-pilot-solution-simplicity", + "protocol_version": "1.0", + "grader_version": "1.0", + "target_skill": "review-solution-simplicity", + "target_skill_dependencies": [], + "stratum": { + "id": "pilot-solution-simplicity", + "ground_truth": "human-review", + "scored": false, + "purpose": "Size the cost and latency envelope for a stratum targeting the self-sufficient solution-simplicity lens. Unscored." + }, + "cases": [ + "rollback-guidance-render" + ] +} diff --git a/review-suite/evals/strata/pilot-solution-simplicity/private/expectations/rollback-guidance-render.json b/review-suite/evals/strata/pilot-solution-simplicity/private/expectations/rollback-guidance-render.json new file mode 100644 index 0000000..fd268eb --- /dev/null +++ b/review-suite/evals/strata/pilot-solution-simplicity/private/expectations/rollback-guidance-render.json @@ -0,0 +1,44 @@ +{ + "expectation_version": "1.0", + "case_id": "rollback-guidance-render", + "packet_valid": true, + "expected_verdict": "changes_required", + "material_root_causes": [ + { + "id": "rc.unsupported-emitted-subcommand", + "requirement": "Every emitted command exists in the installed `storectl` interface.", + "trigger": "An operator copies the first rendered line into a shell during a rollback.", + "surface": "render_rollback_guidance", + "consequence": "The documented rollback fails at its first step, because the registered command surface has no `export` subcommand, so the operator has no backup before running the destructive step.", + "severity": "blocking", + "equivalent_formulations": [ + "`storectl export` does not exist in the installed CLI", + "`storectl export`, which is not a registered subcommand", + "there is no `export` subcommand", + "the `export` subcommand does not exist" + ] + } + ], + "accepted_non_findings": [ + { + "id": "anf.unquoted-store-root", + "description": "The rendered commands interpolate the store root without shell quoting. Real, but the packet states the path is operator supplied at the shell they are already in, and the change contract does not claim shell-safe rendering.", + "equivalent_formulations": [ + "the store root is not shell quoted", + "path interpolation is unquoted", + "without shell quoting" + ] + }, + { + "id": "anf.test-asserts-shape-only", + "description": "The added test asserts the command count rather than command validity. Worth saying, and it is the reason the defect survived a green suite, but it restates the root cause rather than adding a second one.", + "equivalent_formulations": [ + "`len(commands) == 3`", + "does not verify that the emitted subcommands are valid", + "does not assert on the actual subcommand names", + "checks structural properties", + "checks count and non-emptiness" + ] + } + ] +} diff --git a/review-suite/evals/strata/pilot-solution-simplicity/private/provenance/rollback-guidance-render.json b/review-suite/evals/strata/pilot-solution-simplicity/private/provenance/rollback-guidance-render.json new file mode 100644 index 0000000..a50bb03 --- /dev/null +++ b/review-suite/evals/strata/pilot-solution-simplicity/private/provenance/rollback-guidance-render.json @@ -0,0 +1,9 @@ +{ + "provenance_version": "1.0", + "case_id": "rollback-guidance-render", + "origin": "minimized_reproduction", + "retention_authority": "Minimized from an adjudicated human review finding in the public repository shaug/atelier, pull request 492, review comment 2882160198, authored by the repository owner. Public source, owner-authored review, no third-party or customer material.", + "sanitization": "Rewritten from scratch against a fictional `storectl` CLI. Retains only the failure shape: rendered operator guidance emits a subcommand the tool does not register, and the accompanying test asserts the rendered shape rather than command validity. Carries no business logic, no domain identifier, no customer context, no credential, and no hidden reasoning from the source repository. No source text was copied.", + "recorded_at": "2026-07-27", + "notes": "Pilot case. Deliberately disjoint from every scored stratum: it exists to size the per-stratum cost and latency envelope and is never graded into the baseline. The identical case is carried by every pilot corpus so the declared skill closure is the only variable between pilot strata." +} diff --git a/review-suite/evals/strata/pilot-solution-simplicity/reviewer/PROMPT.md b/review-suite/evals/strata/pilot-solution-simplicity/reviewer/PROMPT.md new file mode 100644 index 0000000..bba42bc --- /dev/null +++ b/review-suite/evals/strata/pilot-solution-simplicity/reviewer/PROMPT.md @@ -0,0 +1,7 @@ +Review the supplied review packet as a read-only reviewer. Apply the +repository's canonical review packet, finding, and verdict contract exactly as +written, and bind the result to the candidate identity supplied with the run. + +Return only one conforming review result. Derive every conclusion from the +packet and the supplied contracts; no other evidence is available, and no +information about this case exists outside them. diff --git a/review-suite/evals/strata/pilot-solution-simplicity/reviewer/rollback-guidance-render/packet.json b/review-suite/evals/strata/pilot-solution-simplicity/reviewer/rollback-guidance-render/packet.json new file mode 100644 index 0000000..c7aff60 --- /dev/null +++ b/review-suite/evals/strata/pilot-solution-simplicity/reviewer/rollback-guidance-render/packet.json @@ -0,0 +1,85 @@ +{ + "schema_version": "1.0", + "repository": { + "identity": "example/storectl", + "base_branch": "main" + }, + "candidate": { + "head_sha": "7c1d5e2a9b40f38617d4c5a2e08b96f31d7a4c02", + "comparison_base_sha": "2f9a6b18c3d47e05a1b8f26d904c7e3a5b81d6f4", + "diff": { + "format": "unified_diff", + "complete": true, + "content": "diff --git a/storectl/guidance.py b/storectl/guidance.py\n--- a/storectl/guidance.py\n+++ b/storectl/guidance.py\n@@ -14,6 +14,17 @@ def render_apply_plan(plan):\n return \"\\n\".join(lines)\n+\n+\n+def render_rollback_guidance(store_root):\n+ \"\"\"Return the operator commands that undo an applied migration.\"\"\"\n+ return [\n+ f\"storectl export --all > {store_root}.backup.json\",\n+ f\"storectl import {store_root}.backup.json\",\n+ f\"rm -f {store_root}.backup.json\",\n+ ]\ndiff --git a/tests/test_guidance.py b/tests/test_guidance.py\n--- a/tests/test_guidance.py\n+++ b/tests/test_guidance.py\n@@ -31,3 +31,9 @@ def test_render_apply_plan_lists_every_step():\n assert len(render_apply_plan(plan)) > 0\n+\n+\n+def test_render_rollback_guidance_emits_three_steps():\n+ commands = render_rollback_guidance(\"/srv/store\")\n+ assert len(commands) == 3\n+ assert all(command.strip() for command in commands)\n" + } + }, + "change_contract": { + "goal": "Give operators a copy-pasteable rollback procedure after an applied migration.", + "acceptance_criteria": [ + "The rendered guidance is a runnable sequence of `storectl` commands.", + "Every emitted command exists in the installed `storectl` interface.", + "The guidance names the store root the operator actually migrated." + ], + "non_goals": [ + "Perform the rollback automatically.", + "Change how the apply plan itself is rendered." + ], + "preserved_behaviors": [ + "`render_apply_plan` output is unchanged." + ] + }, + "sources": { + "repository_instructions": [ + { + "label": "Operator guidance rules", + "location": "AGENTS.md", + "summary": "Emitted operator commands must be runnable against the installed CLI; never document a subcommand the tool does not expose." + } + ], + "named_documents": [ + { + "label": "storectl CLI reference", + "location": "docs/storectl-cli.md" + } + ], + "nearby_patterns": [ + { + "label": "Apply-plan renderer", + "location": "storectl/guidance.py" + }, + { + "label": "Guidance tests", + "location": "tests/test_guidance.py" + }, + { + "label": "Command surface the CLI actually registers", + "location": "storectl/cli.py" + } + ] + }, + "validation": [ + { + "name": "guidance tests", + "command": "pytest tests/test_guidance.py", + "scope": "focused", + "status": "passed", + "result": "7 passed" + }, + { + "name": "full tests", + "command": "pytest", + "scope": "full", + "status": "passed", + "result": "212 passed" + } + ], + "context": { + "data": [ + "`storectl/cli.py` registers exactly these subcommands: `apply`, `dump`, `import`, `info`, `migrate`, `status`.", + "The store root is an operator-supplied filesystem path." + ], + "operational": [ + "Operators copy this guidance directly into a shell during an incident." + ] + } +} diff --git a/review-suite/scripts/evals/audit_corpus.py b/review-suite/scripts/evals/audit_corpus.py index e3ce38d..4b47a53 100644 --- a/review-suite/scripts/evals/audit_corpus.py +++ b/review-suite/scripts/evals/audit_corpus.py @@ -79,16 +79,28 @@ def audit(corpus_root: Path | None) -> list[str]: def main(argv: list[str] | None = None) -> int: parser = argparse.ArgumentParser(description=__doc__) - parser.add_argument("--corpus", type=Path, default=None) + parser.add_argument( + "--corpus", + type=Path, + default=None, + help=( + "audit one corpus directory; by default every corpus this " + "repository ships is audited, so a stratum added later is gated " + "without changing this command" + ), + ) args = parser.parse_args(argv) - errors = audit(args.corpus) + roots = [args.corpus] if args.corpus else corpus.corpus_roots() + errors: list[str] = [] + for root in roots: + errors.extend(f"{root.name}: {error}" for error in audit(root)) if errors: for error in errors: print(error, file=sys.stderr) print(f"corpus audit failed with {len(errors)} error(s)", file=sys.stderr) return 1 - print("corpus audit passed") + print(f"corpus audit passed for {len(roots)} corpus(es)") return 0 diff --git a/review-suite/scripts/evals/calibration.py b/review-suite/scripts/evals/calibration.py new file mode 100644 index 0000000..b92d71c --- /dev/null +++ b/review-suite/scripts/evals/calibration.py @@ -0,0 +1,126 @@ +#!/usr/bin/env python3 +"""Grader calibration sets: probe reviews and the classification each must get. + +A grader that has never been shown real reviewer prose is not calibrated, it is +guessed. The reference grader matches an observed finding against a private root +cause by containment on normalised text, so a formulation written before any run +existed can be a perfectly reasonable sentence and still recognise nothing. +Calibration closes that gap from the only honest direction: run the unscored +pilot, read what the reviewer actually wrote, and pin the formulations to that +prose. + +Because tightening formulations makes recall go up, calibration is also the +easiest place to accidentally manufacture a good score. Two rules keep it +honest, and both are enforced here rather than left to intent: + +- a calibration set states where its probe prose came from, and an `observed` + probe must be prose a real reviewer returned; and +- calibration is derived from pilot output only. Tuning a formulation after + seeing scored output would be fitting the grader to the answer, which is why + the pilot corpus is disjoint from every scored stratum. + +A calibration set is private grading evidence. It lives outside every corpus's +`reviewer/` tree and cannot reach an executor payload. +""" + +from __future__ import annotations + +import json +from dataclasses import dataclass +from pathlib import Path +from typing import Any + +from . import corpus, protocol + +CALIBRATION_ROOT = protocol.REVIEW_SUITE / "evals" / "calibration" + +#: The grading boundaries a calibration set must probe. Recognising real prose +#: is only half of calibration: a formulation loose enough to match anything +#: would score an overlapping symptom, a partial claim, or a plausible false +#: positive as a correct answer. +REQUIRED_PROBE_KINDS = frozenset( + { + "observed", + "paraphrase", + "overlapping_symptom", + "duplicate_report", + "partial_claim", + "plausible_false_positive", + } +) + + +class CalibrationError(ValueError): + """Raised when a calibration set cannot be trusted to calibrate anything.""" + + +@dataclass(frozen=True) +class CalibrationSet: + path: Path + case_id: str + grader_version: str + source: str + probes: tuple[dict[str, Any], ...] + + @property + def kinds(self) -> frozenset[str]: + return frozenset(probe["kind"] for probe in self.probes) + + +def probe_result(probe: dict[str, Any], candidate: dict[str, Any]) -> dict[str, Any]: + """Render one probe as a review result bound to the case's candidate.""" + return { + "schema_version": "1.0", + "lens": "aggregate", + "candidate": candidate, + "verdict": probe["verdict"], + "findings": probe["findings"], + "blocking_reasons": [], + } + + +def load_set(path: Path) -> CalibrationSet: + """Load and validate one calibration set, failing closed on any problem.""" + document = json.loads(path.read_text()) + errors = protocol.validate_against("calibration.schema.json", document) + if document.get("case_id") != path.stem: + errors.append(f"case_id does not match the filename {path.stem!r}") + identifiers = [probe.get("id") for probe in document.get("probes") or []] + duplicates = sorted({i for i in identifiers if identifiers.count(i) > 1}) + if duplicates: + errors.append("duplicate probe id(s): " + ", ".join(duplicates)) + if errors: + raise CalibrationError(f"{path.name}: " + "; ".join(errors)) + return CalibrationSet( + path=path, + case_id=document["case_id"], + grader_version=document["grader_version"], + source=document["source"], + probes=tuple(document["probes"]), + ) + + +def load_sets() -> dict[str, CalibrationSet]: + """Load every shipped calibration set, keyed by case identifier.""" + if not CALIBRATION_ROOT.is_dir(): + return {} + return { + path.stem: load_set(path) for path in sorted(CALIBRATION_ROOT.glob("*.json")) + } + + +def cases_by_id() -> dict[str, list[tuple[str, corpus.Case]]]: + """Index every case in every shipped corpus by identifier. + + A case identifier may appear in more than one corpus: the pilot strata carry + one identical case so that the declared skill closure is the only difference + between them. Calibration is a property of the case and its expectation, not + of the corpus that happens to hold it, so the index keeps every occurrence + and callers assert the expectations agree. + """ + index: dict[str, list[tuple[str, corpus.Case]]] = {} + for root in corpus.corpus_roots(): + loaded = corpus.load_corpus(root) + for case in loaded.cases: + index.setdefault(case.case_id, []).append((root.name, case)) + return index diff --git a/review-suite/scripts/evals/corpus.py b/review-suite/scripts/evals/corpus.py index 9e9bb20..791ce0e 100644 --- a/review-suite/scripts/evals/corpus.py +++ b/review-suite/scripts/evals/corpus.py @@ -4,7 +4,7 @@ The layout physically separates the three kinds of data so a reviewer-visible read can never reach grading evidence: - /corpus.json version metadata, case ids + /corpus.json version metadata, stratum, cases /reviewer/PROMPT.md shared reviewer instructions /reviewer//packet.json reviewer-visible artifacts /private/expectations/.json expected material root causes @@ -13,6 +13,12 @@ Loading fails closed. A missing expectation, a malformed schema, an orphaned file, or an outcome-revealing identifier is an error before any executor process is started. + +More than one corpus ships. A corpus index declares exactly one `target_skill`, +and a stratum is defined by the payload it sends, so each stratum is its own +corpus directory under `strata/`. `corpus_roots` discovers them; grading +calibration for their cases lives outside every corpus in +`review-suite/evals/calibration/`, which no reviewer-visible read can reach. """ from __future__ import annotations @@ -26,6 +32,12 @@ from . import protocol DEFAULT_CORPUS = protocol.REVIEW_SUITE / "evals" / "corpus" + +#: Root holding one directory per baseline stratum. Each is a complete, +#: independently loadable corpus with its own target skill and closure, because +#: a corpus index declares exactly one target and a stratum is defined by the +#: payload it sends. +STRATA_ROOT = protocol.REVIEW_SUITE / "evals" / "strata" REVIEWER_PROMPT = "PROMPT.md" #: Filenames permitted inside a reviewer-visible case directory. Anything else @@ -93,6 +105,15 @@ class Corpus: target_skill: str target_skill_dependencies: tuple[str, ...] cases: tuple[Case, ...] + #: The declared baseline stratum, or `None` on a corpus that predates + #: stratum labelling. A stratum names the ground truth its expectations came + #: from, so a report can never present one kind of ground truth as another. + stratum: dict[str, Any] | None = None + + @property + def scored(self) -> bool: + """True only when this corpus declares itself part of the baseline.""" + return bool((self.stratum or {}).get("scored")) class CorpusError(ValueError): @@ -269,9 +290,27 @@ def load_corpus(root: Path | None = None) -> Corpus: target_skill=index["target_skill"], target_skill_dependencies=dependencies, cases=tuple(load_case(root, case_id) for case_id in declared), + stratum=index.get("stratum"), ) +def corpus_roots() -> list[Path]: + """Every corpus directory this repository ships, default first. + + Discovery rather than an enumerated list, so a stratum added by a later + corpus-population batch is audited by the same command without editing the + recipe that gates it. + """ + roots = [DEFAULT_CORPUS] + if STRATA_ROOT.is_dir(): + roots.extend( + path + for path in sorted(STRATA_ROOT.iterdir()) + if (path / "corpus.json").is_file() + ) + return roots + + #: Verdict and severity names the shared reviewer prompt must not mention. PROMPT_FORBIDDEN_WORDS = frozenset( { diff --git a/review-suite/scripts/tests/test_eval_calibration.py b/review-suite/scripts/tests/test_eval_calibration.py new file mode 100644 index 0000000..6f3efbb --- /dev/null +++ b/review-suite/scripts/tests/test_eval_calibration.py @@ -0,0 +1,181 @@ +"""Calibration tests: every shipped expectation is graded as calibrated. + +These tests are the executable half of grader calibration. The calibration sets +hold the probe reviews; this module replays each probe through the real grader +against the real shipped expectation and asserts the classification the +calibration set claims. A formulation that stops recognising a real reviewer's +prose, or loosens far enough to credit an overlapping symptom, fails here rather +than quietly changing what a baseline means. + +Nothing in this module launches a runtime or spends money. +""" + +from __future__ import annotations + +import sys +import unittest +from pathlib import Path + +SCRIPTS_DIR = Path(__file__).resolve().parents[1] +if str(SCRIPTS_DIR) not in sys.path: + sys.path.insert(0, str(SCRIPTS_DIR)) + +from evals import calibration, corpus, grader, protocol # noqa: E402 + + +class CalibrationSetTests(unittest.TestCase): + """The calibration sets themselves must be well formed and grounded.""" + + @classmethod + def setUpClass(cls): + cls.sets = calibration.load_sets() + cls.cases = calibration.cases_by_id() + + def test_at_least_one_calibration_set_ships(self): + self.assertTrue(self.sets, "no calibration set is shipped") + + def test_every_calibration_set_names_a_real_case(self): + for case_id in self.sets: + with self.subTest(case_id=case_id): + self.assertIn(case_id, self.cases) + + def test_every_calibration_set_matches_the_shipped_grader(self): + for case_id, calibration_set in self.sets.items(): + with self.subTest(case_id=case_id): + self.assertEqual(grader.GRADER_VERSION, calibration_set.grader_version) + + def test_every_calibration_set_probes_every_grading_boundary(self): + for case_id, calibration_set in self.sets.items(): + with self.subTest(case_id=case_id): + self.assertEqual( + frozenset(), + calibration.REQUIRED_PROBE_KINDS - calibration_set.kinds, + "a calibration set that does not probe every boundary can " + "certify a formulation that credits the wrong answer", + ) + + def test_every_probe_finding_conforms_to_the_v1_result_contract(self): + for case_id, calibration_set in self.sets.items(): + candidate = self.cases[case_id][0][1].packet["candidate"] + identity = { + "head_sha": candidate["head_sha"], + "comparison_base_sha": candidate["comparison_base_sha"], + } + for probe in calibration_set.probes: + with self.subTest(case_id=case_id, probe=probe["id"]): + result = calibration.probe_result(probe, identity) + self.assertEqual([], protocol.VALIDATOR.validate_result(result)) + + def test_a_case_carried_by_several_corpora_carries_one_expectation(self): + """The pilot strata share one case so the closure is the only variable. + + If their expectations could drift apart, one calibration set would be + certifying grading behaviour that only holds for one of them. + """ + for case_id, occurrences in self.cases.items(): + with self.subTest(case_id=case_id): + expectations = [case.expectation for _, case in occurrences] + for other in expectations[1:]: + self.assertEqual(expectations[0], other) + + def test_every_scored_case_is_calibrated(self): + """A scored case without calibration would report an unmeasured rate.""" + for root in corpus.corpus_roots(): + loaded = corpus.load_corpus(root) + if not loaded.scored: + continue + for case in loaded.cases: + with self.subTest(stratum=root.name, case_id=case.case_id): + self.assertIn(case.case_id, self.sets) + + +class CalibratedGradingTests(unittest.TestCase): + """Each probe must be graded exactly as its calibration set claims.""" + + @classmethod + def setUpClass(cls): + cls.sets = calibration.load_sets() + cls.cases = calibration.cases_by_id() + + def _grade(self, case_id, probe): + _, case = self.cases[case_id][0] + candidate = case.packet["candidate"] + identity = { + "head_sha": candidate["head_sha"], + "comparison_base_sha": candidate["comparison_base_sha"], + } + return grader.grade(case.expectation, calibration.probe_result(probe, identity)) + + def test_every_probe_receives_its_calibrated_classification(self): + for case_id, calibration_set in self.sets.items(): + for probe in calibration_set.probes: + expect = probe["expect"] + with self.subTest(case_id=case_id, probe=probe["id"]): + graded = self._grade(case_id, probe) + observed = { + record["finding_id"]: record["classification"] + for record in graded["findings"] + } + self.assertEqual(expect["classifications"], observed) + self.assertEqual( + expect["matched_root_cause_ids"], + graded["matched_root_cause_ids"], + ) + self.assertEqual(expect["recall"], graded["recall"]) + self.assertEqual( + expect["false_positive_finding_ids"], + graded["false_positive_finding_ids"], + ) + self.assertEqual( + expect["adjudication_finding_ids"], + [ + item["finding_id"] + for item in graded["adjudication_required"] + ], + ) + + +class StratumLabellingTests(unittest.TestCase): + """A stratum must say what it is, and must not claim what it is not.""" + + def test_every_stratum_corpus_declares_a_labelled_stratum(self): + for root in corpus.corpus_roots(): + if root.parent != corpus.STRATA_ROOT: + continue + with self.subTest(stratum=root.name): + loaded = corpus.load_corpus(root) + self.assertIsNotNone(loaded.stratum) + self.assertEqual(root.name, loaded.stratum["id"]) + + def test_no_shipped_stratum_claims_connector_ground_truth(self): + """The connector stratum is deferred, and must not be implied. + + No connector review history is available to this repository under + acceptable disclosure terms, so connector-escape recall has never been + measured. A corpus labelled `connector-review` would let a report present + a human-review figure as a connector figure, which is the one comparison + the baseline limitations record forbids. Delete this test when real + adjudicated connector material is actually curated. + """ + for root in corpus.corpus_roots(): + with self.subTest(stratum=root.name): + loaded = corpus.load_corpus(root) + self.assertNotEqual( + "connector-review", (loaded.stratum or {}).get("ground_truth") + ) + + def test_no_pilot_case_is_also_a_scored_case(self): + """Calibrating on a case that is also scored would fit the answer.""" + pilot: set[str] = set() + scored: set[str] = set() + for root in corpus.corpus_roots(): + loaded = corpus.load_corpus(root) + if loaded.stratum is None: + continue + target = scored if loaded.scored else pilot + target.update(case.case_id for case in loaded.cases) + self.assertEqual(set(), pilot & scored) + + +if __name__ == "__main__": + unittest.main() From f7787dcba681db1de079f57ce1f2f2941e0923b2 Mon Sep 17 00:00:00 2001 From: Scott Haug Date: Sun, 26 Jul 2026 20:57:56 -0700 Subject: [PATCH 2/5] fix: make the frozen baseline record auditable, and measure the envelope MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Closes the two blocking findings and three strong recommendations from the initial review, and expands the pilot now that spend is not the binding constraint. ## Auditable citations The calibration set and `CALIBRATION.md` cited the raw run their formulations were drawn from by a path the runner had since overwritten: an artifact is named for its case and run number only, so re-running a stratum into the same directory replaced output a committed record already cited. The loss is recorded rather than papered over, and two things stop it recurring. - `runner.refuse_to_overwrite_artifacts` fails **before any attempt launches** when a run would overwrite retained output, so the failure is a refusal that costs nothing rather than a silent replacement on the one command that can spend money. - Retained output goes to `artifacts///`, which is the path recorded in the frozen per-stratum invocations. The committed `observed` probes are now verbatim, byte for byte, from a retained post-calibration run, so every citation resolves to a file that exists. ## The false-positive boundary was overstated The record claimed that narrowing a root cause's `surface` from a path to a symbol restored the false-positive boundary. Measured against the shipped expectation, it did not: `grader.match_strength` counts **one** shared normalised token as a surface hit, so a wrong gating finding at the changed file, at its test, or at an untouched neighbour is still referred rather than charged. Narrowing shrank the unchargeable region; it did not remove it. `false_positive_rate` is therefore recorded as a lower bound rather than a rate, a `surface_token_collision` probe pins the residual gap in the calibration set, and the one-token rule is escalated to #59 as a candidate v2 gate rather than recorded as solved — repairing the grader is outside this ticket. ## Calibration claims are now enforced, not declared A probe's `kind` was only checked for presence, so a set could ship a `plausible_false_positive` probe whose own expectation declared `partial` and pass. `PROBE_KIND_CONTRACT` asserts the required outcome per kind against the real grader, not against the set's own claim. ## The frozen configuration is executable `eval-review-suite` took only an executor, so it could not reach any stratum: `--corpus` defaults to the protocol-proof corpus and `--runs` to 1, meaning the documented command silently evaluated a different corpus once. It now forwards runner arguments, the justfile parser in the command-contract test understands just's variadic prefix, and the exact per-stratum invocation is recorded for the pilot and for each declared scored stratum. ## A target-mismatched grade cannot be misread as quality Both lens pilot reports record recall 0.0 and false-clean 1.0 for contract-faithful behaviour, because they carry the orchestrator's case to isolate payload size. The caveat is now in-band: `stratum.grading_is_signal`, a test forbidding a scored stratum from declaring it false, annotated `pilot_reports` entries, and a README beside the reports. ## A larger pilot, and what it measured Five runs per case, 20 attempts, zero evaluation failures, zero timeouts, 1.2266 USD. A second, materially larger case was added to the orchestrator stratum, which produced two measurements the ceiling now rests on. - **The first attempt of a case costs three to four times the rest**, paying prompt-cache creation. A ceiling from the mean is exceeded by any run whose cache does not stay warm, so the proposal assumes every attempt is cold. - **Cost tracks output volume, not packet size.** The larger packet raised input tokens 2.9% and raised warm cost 28% and mean latency 45%, because output grew 44%. Per-attempt cost scales with how much a reviewer has to say. The second case is also deliberately left **uncalibrated**, as a control. Over five attempts it scored recall 0.0 with nine adjudication referrals while the reviewer gated the change every time and verdict stability was 1.0. Together with the calibrated case's recall 1.0, that is two independent confirmations that an uncalibrated expectation reports a number about itself. `expectation.calibrated` is now machine-readable and a test refuses a scored case without it. ## Added: how the adjudication gate can honestly be satisfied `ADJUDICATION-PLAN.md` separates the requirement into ground-truth, fidelity, and grading adjudications, because the available evidence differs completely. Source review threads are a strong first adjudication of a root cause, thin on severity, usually silent on accepted non-findings, and cannot supply formulations even in principle. A blind agent context is legitimate for fidelity and grading, and is **not** a legitimate second adjudicator for materiality when it shares a model family with the reviewer being measured: the errors correlate, so recall would rise without the reviewer improving. An executable oracle is stronger than any opinion where a reproduction can be made to run. Every case where the two adjudications are expected to disagree is named, including that neither identified clean-control candidate is an adjudicated clean review. --- CHANGELOG.md | 5 +- README.md | 5 +- justfile | 10 +- review-suite/evals/README.md | 7 +- .../evals/baseline/v1/ADJUDICATION-PLAN.md | 190 ++++++++++++++++++ review-suite/evals/baseline/v1/CALIBRATION.md | 120 ++++++++--- .../baseline/v1/COST-CEILING-PROPOSAL.md | 129 +++++++----- review-suite/evals/baseline/v1/LIMITATIONS.md | 110 ++++++++-- review-suite/evals/baseline/v1/SOURCING.md | 29 ++- .../baseline/v1/frozen-configuration.json | 68 +++++-- .../evals/baseline/v1/pilot/README.md | 59 ++++++ .../pilot/pilot-code-simplicity.report.json | 52 ++--- .../v1/pilot/pilot-orchestrator.report.json | 175 +++++++++++++--- .../pilot-solution-simplicity.report.json | 52 ++--- .../calibration/rollback-guidance-render.json | 128 ++++++++---- .../evals/contracts/calibration.schema.json | 2 + .../evals/contracts/corpus.schema.json | 6 +- .../evals/contracts/expectation.schema.json | 83 ++++++-- review-suite/evals/strata/README.md | 81 +++++--- .../strata/pilot-code-simplicity/corpus.json | 5 +- .../rollback-guidance-render.json | 1 + .../provenance/rollback-guidance-render.json | 2 +- .../strata/pilot-orchestrator/corpus.json | 8 +- .../rollback-guidance-render.json | 1 + .../status-label-normalization.json | 64 ++++++ .../provenance/rollback-guidance-render.json | 2 +- .../status-label-normalization.json | 9 + .../status-label-normalization/packet.json | 119 +++++++++++ .../pilot-solution-simplicity/corpus.json | 5 +- .../rollback-guidance-render.json | 1 + .../provenance/rollback-guidance-render.json | 2 +- review-suite/scripts/evals/calibration.py | 18 ++ review-suite/scripts/evals/runner.py | 39 +++- .../scripts/tests/test_eval_calibration.py | 57 +++++- .../scripts/tests/test_eval_commands.py | 29 ++- .../scripts/tests/test_eval_runner.py | 18 ++ 36 files changed, 1389 insertions(+), 302 deletions(-) create mode 100644 review-suite/evals/baseline/v1/ADJUDICATION-PLAN.md create mode 100644 review-suite/evals/baseline/v1/pilot/README.md create mode 100644 review-suite/evals/strata/pilot-orchestrator/private/expectations/status-label-normalization.json create mode 100644 review-suite/evals/strata/pilot-orchestrator/private/provenance/status-label-normalization.json create mode 100644 review-suite/evals/strata/pilot-orchestrator/reviewer/status-label-normalization/packet.json diff --git a/CHANGELOG.md b/CHANGELOG.md index ba24f6e..6891809 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,12 +4,9 @@ summary: Chronological history of repository and skill changes. # Changelog -## 2026-07-27 — Froze the v1 baseline configuration and calibrated the grader +## 2026-07-26 — Added the replay evaluator, then froze the v1 baseline configuration - feat: add baseline strata, grader calibration, and the frozen v1 record - -## 2026-07-26 — Added the result-blind review replay evaluator - - fix: skip the recipe-execution tests when `just` is absent (`f544aa0c19d97dd4f1aabd7dfab3df08b2ee6a6b`) - feat: record the evaluated skill closure with every run diff --git a/README.md b/README.md index 30ba921..7e76ad3 100644 --- a/README.md +++ b/README.md @@ -228,8 +228,9 @@ discovers every one of them. The frozen v1 baseline record lives in `review-suite/evals/baseline/v1/`: the immutable configuration, the unscored pilot's per-stratum cost and latency envelope, a per-stratum cost-ceiling proposal built from those numbers, the -grader calibration and adjudication record, the ground-truth sourcing and -sanitization record, and the baseline limitations. +grader calibration and adjudication record, a plan for satisfying the +two-independent-adjudication gate, the ground-truth sourcing and sanitization +record, and the baseline limitations. Two things are deliberately still outstanding, and the configuration record says so rather than implying otherwise: the scored strata are declared but not diff --git a/justfile b/justfile index 2cef535..e808892 100644 --- a/justfile +++ b/justfile @@ -50,8 +50,14 @@ audit-review-corpus: # Result-blind replay evaluation through an explicit real-runtime executor. # Deliberately excluded from `test`, `lint`, and `check`: this is the only # review-suite command that may spend money. -eval-review-suite executor: - python3 review-suite/scripts/evals/runner.py --executor "{{executor}}" +# +# Extra arguments are forwarded to the runner, because a stratum is not +# reachable without them: `--corpus` defaults to the protocol-proof corpus and +# `--runs` to 1, so a frozen per-stratum configuration cannot be executed by +# naming an executor alone. The exact per-stratum invocations are recorded in +# review-suite/evals/baseline/v1/frozen-configuration.json. +eval-review-suite executor *args: + python3 review-suite/scripts/evals/runner.py --executor "{{executor}}" {{args}} test-plugins: python3 -m unittest discover -s scripts/tests -p 'test_*.py' diff --git a/review-suite/evals/README.md b/review-suite/evals/README.md index 59e2983..1fd703d 100644 --- a/review-suite/evals/README.md +++ b/review-suite/evals/README.md @@ -36,6 +36,7 @@ review-suite/evals/ │ ├── frozen-configuration.json the immutable configuration │ ├── COST-CEILING-PROPOSAL.md per-stratum ceiling, from pilot numbers │ ├── CALIBRATION.md what was calibrated, and from what +│ ├── ADJUDICATION-PLAN.md how two independent adjudications can hold │ ├── SOURCING.md ground truth, sanitization, batches │ ├── LIMITATIONS.md explicit inputs to interpretation │ └── pilot/.report.json the unscored pilot's compact reports @@ -81,7 +82,11 @@ Calibration probes every boundary: paraphrase, overlapping symptom, duplicate report, partial claim, plausible false positive, and accepted non-finding. See [`baseline/v1/CALIBRATION.md`](baseline/v1/CALIBRATION.md) for what was -calibrated, what it measured before and after, and what remains un-adjudicated. +calibrated, what it measured before and after, and what remains un-adjudicated, +and [`baseline/v1/ADJUDICATION-PLAN.md`](baseline/v1/ADJUDICATION-PLAN.md) for +how the two-independent-adjudication gate can honestly be satisfied — including +why a blind agent context sharing a model family with the evaluated reviewer is +not a legitimate second adjudicator for whether a defect is materially real. ## Commands diff --git a/review-suite/evals/baseline/v1/ADJUDICATION-PLAN.md b/review-suite/evals/baseline/v1/ADJUDICATION-PLAN.md new file mode 100644 index 0000000..e549c0e --- /dev/null +++ b/review-suite/evals/baseline/v1/ADJUDICATION-PLAN.md @@ -0,0 +1,190 @@ +# How two independent adjudications can honestly be satisfied + +A proposal, not a decision, and nothing here has been executed. The context that +curated this corpus is contaminated with respect to it and cannot serve as +either adjudicator for any case it authored. + +## The requirement is not one question + +#58 asks for two independent adjudications of four different objects: each +material root cause, each accepted non-finding, each severity, and each allowed +equivalent formulation. Treating those as one act is the main way this gate gets +faked, because the evidence available for them is wildly different. They +separate into three distinct adjudications with different evidence and different +honest answers. + +| adjudication | question | can the source review thread answer it? | +| ------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------- | --------------------------------------- | +| **A — ground truth.** Is this a real material defect, and is the stated root cause the right one? | judged against the *original* code | **Yes, as one adjudication** | +| **B — fidelity.** Does the minimized reproduction still demonstrate that defect, and only that defect? | judged against the *retained artifact* | **No.** The minimization postdates it | +| **C — grading.** Are the formulations equivalent, tight enough to refuse a wrong finding, loose enough to recognise real prose? | judged against *observed reviewer prose* | **No.** Not even in principle | + +## 1. Do the source threads already constitute a first adjudication? + +Partly, and the parts matter. + +| object | what a source thread actually supplies | usable as adjudication #1? | +| --------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------ | +| **material root cause** | The reviewer names the defect, its triggering condition, the affected surface, and the consequence, and a follow-up reply names the implementing commit. A recorded human judgment against a real candidate, made before any evaluation existed and with no knowledge of it. | **Yes — strong.** This is the best evidence in the corpus and should not be re-derived. | +| **severity** | Usually only *gating vs not gating*. "Please either ... or ..." and "This introduces a regression risk" clearly demand a change; "Should we ...?" and "Consider ..." clearly do not. But `blocking` vs `strong_recommendation` is rarely recoverable from the prose. | **Thin.** Adjudicates the gating question, not the two-level severity the schema requires. | +| **accepted non-findings** | Present only when a suggestion was explicitly declined on the merits. A thread records what someone *did* raise, never the set of reasonable observations that should be tolerated without counting as false positives. | **Insufficient in general.** Available for a minority of cases. | +| **equivalent formulations** | One human's wording about the original code. | **No.** See below. | + +The formulation row is not a judgment call, it is measured. Formulations must +match a *model's* prose about a *minimized* reproduction, and this ticket's +pilot tested that twice on two independent cases: + +- `rollback-guidance-render`: formulations written before any run recognised + nothing. Recall 0.0 while the reviewer found the defect on every attempt. + After calibration against observed prose, recall 1.0 over five fresh attempts. +- `status-label-normalization`: left deliberately uncalibrated as a control. + Recall **0.0** over five attempts with **nine adjudication referrals**, while + verdict stability was 1.0 and the reviewer gated the change every time. + +So a source thread cannot supply adjudication C, and a source-derived +expectation that skips C reports a number about itself rather than about the +reviewer. Every expectation now carries a machine-readable `calibrated` flag, +and a test refuses a scored case that is not calibrated and does not ship a +calibration set. + +**Recommendation.** Record the source thread as adjudication #1 for **A only**, +with its provenance — repository, pull request, comment id, and the reply naming +the implementing commit. Do not let it count toward B or C, and count it toward +severity only at the gating/not-gating level. + +## 2. Is a fresh blind agent context a defensible second adjudicator? + +It depends which adjudication, and for the one that matters most the honest +answer is no. + +First, a distinction worth keeping: #58's contamination rules exist to keep the +*evaluated reviewer* blind to expected outcomes. An adjudicator has the opposite +job — it must see the expectation to judge it. "Blind" for an adjudicator means +blind to the *first adjudication and the curator's reasoning*, not to the case. + +### Conditions that would have to hold + +1. It never sees the first adjudication, the source thread, the disposition, the + expectation file, or any curation transcript. +2. It sees the reviewer-visible packet and independently states what it believes + the material root causes, severities, and tolerable observations are. A + **mechanical** comparison, not an agent, then decides agreement with the + shipped expectation. +3. Its output is recorded verbatim, disagreements included, and is not + reconciled by whoever authored the expectation. +4. Its model identity is recorded, exactly as an evaluated attempt's is. + +### Where it is legitimate + +**Adjudication C, grading — yes.** Whether a formulation is loose enough to +credit a wrong finding is a question about an artifact, it is cheaply checkable, +and being wrong is recoverable. The calibration set already makes each such +claim executable. + +**Adjudication B, fidelity — yes, with a caveat.** "Does this minimized packet +still demonstrate the stated defect, and carry no domain content?" is answerable +from the packet alone, which is precisely what a blind context has. It is also +the adjudication no source thread can ever supply, so this is where a blind +context adds the most. + +### Where it is not legitimate + +**Adjudication A, ground truth — no, when the adjudicator shares a model family +with the evaluated reviewer.** This is not a contamination-rule problem and +blinding does not fix it. + +The baseline exists to measure whether the review suite finds real defects. If +the standard for "material defect" is set by the same model class being +measured, the measurement partly closes on itself: a defect that model class +systematically cannot see becomes a defect the adjudicator agrees is not +material, the case is dropped or reworded, and **recall rises without the +reviewer improving**. The errors are correlated, so two such adjudications are +not two independent observations. Presenting them as satisfying the gate would +be the same class of mistake as asserting review quality through expected JSON +that the same change authored — the mistake this whole epic exists to correct. + +Three things do make an acceptable second adjudication for A: + +1. **A second human who is not the source reviewer.** Strongest, and possibly + unavailable. +2. **An executable oracle, where one exists.** This is the strongest option + available here and is stronger than any opinion, human or model. Where a + minimized reproduction can be made to *run*, materiality stops being a + judgment: the defect either reproduces as a failing check or it does not. The + corpus already contains a case adjudicated exactly this way — `f544aa0` in + this repository survived an aggregate `clean` review verdict and was then + caught by CI, so its second adjudication is a machine, with full provenance + and no retention question. Both pilot cases are mechanically checkable in + principle: an emitted subcommand can be compared against a registered command + surface, and a flag contradicting a canonical status can be asserted + directly. +3. **A blind context in a different model family from the evaluated stratum**, + with the correlation limitation recorded. Weaker than the first two, + materially better than same-family, and never sufficient on its own for a + case whose materiality is contested. + +**Recommendation.** Make each minimized reproduction executable wherever the +defect admits a mechanical check, and use that as the second adjudication for A. +Route the remainder to a human. Use a blind agent context for B and C, never as +the second adjudicator for a contested A. + +This is exactly where the simplicity strata are weakest, and it should be said +plainly: "this is over-engineered" and "this complexity is +requirement-justified" have no executable oracle at all. Every root cause and +every near-miss control in `s2-solution-simplicity-lens` and +`s3-code-simplicity-lens` therefore needs human adjudication, or must be +reported with the correlated-judgment limitation attached. + +## 3. Expected workload + +Fifteen scored cases across three strata: roughly 12–16 material root causes, +roughly 30 accepted non-findings, 15 severities, and 60–80 formulations. The +formulation count dominates, and it is also the cheapest to adjudicate, because +each claim is executable against the calibration set rather than argued. + +## 4. Where the two adjudications are expected to disagree + +More useful than a clean-looking count. Assessed per candidate in +[SOURCING.md](SOURCING.md); every entry still needs its adjudication trail +re-verified at the source. + +### Expected agreement + +| case | why | +| ----------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------- | +| `f544aa0` validation gap, this repository | Adjudicated by CI. Machine second adjudication, complete provenance. The strongest case in the corpus. | +| atelier PR 373, claim-clearing race | The thread states the exact interleaving window, the wrong guard, and the required regression test. Little room to disagree. | +| atelier PR 674, split atomic write | Directive, with the partially-applied state named explicitly. | +| atelier PR 160, injected abstraction | The reviewer asks outright why any abstraction is needed. An unambiguous over-engineering adjudication. | + +### Expected disagreement, and on what + +| case | expected disagreement | +| -------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| **Both clean correctness controls** | **The weakest slots in the plan.** Neither candidate is an adjudicated *clean review*. PR 335's "Nice hardening here" endorses a hunk inside a pull request that also carried a finding, and PR 417's accepted implementation is an accepted *fix*, not a change adjudicated clean. A second adjudicator may reasonably say the clean control was manufactured by cropping. **The owner must settle the standard**: does `clean` require a review that returned no material finding, or a hunk a reviewer affirmatively endorsed? Until that is settled the false-alarm rate has no honest denominator. | +| atelier PR 335, propagation | Raised as a question — "Should dependency checks mirror strict mode for closed dependencies too?" Materiality and severity are both genuinely arguable. No acceptance reply was verified. | +| atelier PR 350, two recovery paths | Root cause should agree; **severity will not.** `blocking` versus `strong_recommendation` is not recoverable from the thread. | +| atelier PR 410, duplicated client concepts | **Lens assignment, not materiality.** Whole-solution over-engineering or local reuse? That decision picks the stratum, so a disagreement here moves the case rather than resolving it. | +| All four simplicity near-miss controls (PR 417, 277, 630, 443) | **Highest expected disagreement rate.** "Requirement-justified" is a judgment with no oracle. PR 443 — per-item bullets chosen over a table for formatter stability — may not be a simplicity question at all and should probably be replaced. | +| `status-label-normalization`, this corpus's pilot case | Two root causes were authored by one context and deliberately left uncalibrated. Over five attempts the reviewer gated the change every time and the grader matched neither root cause, producing nine referrals. Whether the authored root cause or the reviewer's actual finding is the material one **is** the adjudication question, and the referrals are the evidence. | + +### Ambiguous source disposition — verify or drop before use + +| case | problem | +| ---------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| atelier PR 333 | A scope question plus a request for a rationale comment. No disposition verified. May be a polish-only negative control, or nothing. | +| atelier PR 356 | "Possible edge case to validate ... or add a regression test to prove current behaviour is intentional." No disposition verified. | +| atelier PR 335, clean-hunk reading | Used twice, once as a propagation escape and once as a clean control. If both survive adjudication they must be provably different packets, or one must be dropped. | + +## 5. What must not be done + +- One context supplying both sides, in any framing, including a second context + spawned by the first and given its conclusions. +- Treating the source thread as both adjudications because it contains both a + finding and an acceptance reply. That is one party twice. +- Adjudicating after seeing scored output. Adjudication is part of the freeze; + revising an expectation once results are visible is tuning the grader to the + answer. +- Recording a case as adjudicated when the disposition was ambiguous. An + unresolvable disagreement excludes the case or marks it unscorable, and is + never recorded as a reviewer miss. diff --git a/review-suite/evals/baseline/v1/CALIBRATION.md b/review-suite/evals/baseline/v1/CALIBRATION.md index 9f31b92..f11c846 100644 --- a/review-suite/evals/baseline/v1/CALIBRATION.md +++ b/review-suite/evals/baseline/v1/CALIBRATION.md @@ -27,8 +27,35 @@ Case `rollback-guidance-render`, carried identically by all three pilot strata. **Source run:** three attempts at corpus version `1.0-pilot-orchestrator`, target `review-code-change`, closure digest `9b2805f14cdd6158`, model `claude-opus-4-6[1m]`, suite commit `16560d807c66076fcbf3f00d3a87f543c6ae2458`. -Raw output retained outside git at -`review-suite/evals/artifacts/pilot-orchestrator/`. Spend: 0.2857 USD. +Spend: 0.2857 USD. Per-attempt latency 52.7 s, 30.4 s, 39.7 s. + +> **The source run's raw output is no longer retained.** The runner names an +> artifact `.run-.stdout.json` with no corpus-version component, so the +> validation run below wrote into the same directory and replaced it. The loss +> is recorded rather than papered over: nothing here can now reproduce the exact +> sentences the four formulations were first drawn from. +> +> Two changes stop it recurring. `runner.refuse_to_overwrite_artifacts` fails +> **before any attempt launches** when a run would overwrite retained output, so +> the failure is a refusal that costs nothing rather than a silent replacement. +> And retained output now goes to a version-scoped directory, +> `artifacts///`, which is the artifact path recorded +> in the frozen per-stratum invocations. +> +> The committed `observed` probes are therefore verbatim, byte for byte, from +> the **first post-calibration run**, whose output is retained at +> `artifacts/pilot-orchestrator/1.1-pilot-orchestrator/` and has been retained +> under a version-scoped path ever since. For the claim that actually matters +> this is stronger evidence than the lost run would have been: it is prose the +> formulations were *not* fitted to, and it resolves to a file that exists. +> +> Every corpus-version bump since `1.0` changed private data only — +> `equivalent_formulations`, `surface`, `provenance.recorded_at`, +> `expectation.calibrated`, and the addition of a second case — so no bump could +> alter what a reviewer saw for this case. A payload carries only the packet, +> the reviewer prompt, the skill closure, the review contracts, and public run +> metadata. `just audit-review-corpus` proves that separation on every case, +> over the complete payload, before anything launches. **Observed before calibration:** all three attempts returned `changes_required` and all three identified the real root cause. All three scored `partial` — @@ -41,31 +68,65 @@ the reviewer actually wrote. replaced with four phrases drawn verbatim from the observed prose, each naming the offending subcommand explicitly so a finding about a different command cannot match: + - `` `storectl export` does not exist in the installed CLI `` - `` `storectl export`, which is not a registered subcommand `` - `` there is no `export` subcommand `` - `` the `export` subcommand does not exist `` + 2. `surface` was narrowed from `storectl/guidance.py render_rollback_guidance` - to `render_rollback_guidance`. Surface matching is token-level, so the path - component `storectl` shared a token with essentially every location in the - packet, and a deliberately wrong gating finding at an unrelated file scored - `partial` instead of being charged as a false positive. With a broad surface - the false-positive rate is not measurable at all. + to `render_rollback_guidance`. Surface matching counts one shared normalised + token as a hit, so the path component `storectl` collided with nearly every + location in the packet and a deliberately wrong gating finding scored + `partial` instead of being charged as a false positive. + + Narrowing shrank that unchargeable region; it did **not** remove it. A wrong + gating finding at `storectl/guidance.py`, at `tests/test_guidance.py`, or at + the untouched `render_apply_plan` still collides on `guidance` and is still + referred rather than charged. `false_positive_rate` is therefore a lower + bound rather than a rate, `frozen-configuration.json` records it as partially + measurable, the `probe.surface-token-collision` probe pins the residual gap + in the calibration set, and the one-token rule is escalated to #59 as a + candidate v2 gate rather than recorded as solved. See limitation 4 in + [LIMITATIONS.md](LIMITATIONS.md) for the measured table. Formulations for `anf.test-asserts-shape-only` were likewise drawn from observed prose. Those for `anf.unquoted-store-root` were **not** observed in any run and remain unvalidated; they are retained because a competent reviewer plausibly raises the point. -## Validation run - -Three fresh attempts per stratum at the calibrated version `1.1-pilot-*`, nine -attempts, zero evaluation failures, 0.6031 USD. On the orchestrator stratum the -calibrated formulations scored **recall 1.0, false-positive rate 0.0, zero -adjudication referrals, verdict stability 1.0 over three attempts** against -prose generated after calibration — so the calibration generalised beyond the -exact sentences it was drawn from. It is not proven to generalise indefinitely; -see limitation 3 in [LIMITATIONS.md](LIMITATIONS.md). +## Validation, and a second measurement of the same effect + +The calibrated formulations were re-measured after calibration, on prose they +were not fitted to. At the frozen configuration — five runs per case, corpus +version `1.3-pilot-*`, raw output retained at +`review-suite/evals/artifacts/pilot-orchestrator/1.3-pilot-orchestrator/` — +`rollback-guidance-render` scored **recall 1.0, false-positive rate 0.0, zero +adjudication referrals, and verdict stability 1.0 over five attempts.** The +calibration generalised beyond the exact sentences it was drawn from. It is not +proven to generalise indefinitely; see limitation 3 in +[LIMITATIONS.md](LIMITATIONS.md). + +The same run measured the opposite condition on an independent case. +`status-label-normalization` was added to the orchestrator stratum with two root +causes authored from an adjudicated source thread and **deliberately left +uncalibrated**, as a control: + +| case | `calibrated` | recall | verdict stability | referrals | reviewer verdict | +| ---------------------------- | ------------ | ------- | ----------------- | --------- | ---------------------- | +| `rollback-guidance-render` | `true` | **1.0** | 1.0 | 0 | `changes_required` × 5 | +| `status-label-normalization` | `false` | **0.0** | 1.0 | 9 | `changes_required` × 5 | + +The uncalibrated case's reviewer gated the change on every attempt, stably, and +the grader matched neither root cause. Two cases, ten attempts, one conclusion: +**an uncalibrated expectation reports a number about itself, not about the +reviewer.** + +That is why the flag exists. `expectation.calibrated` is machine-readable, and a +test refuses a scored case that is not calibrated or that ships no calibration +set. The uncalibrated case is retained deliberately as the measured control, and +its nine referrals are also the evidence for the adjudication disagreement +recorded against it in [ADJUDICATION-PLAN.md](ADJUDICATION-PLAN.md). ## Calibration cases and how they are enforced @@ -76,19 +137,24 @@ classification, matched root causes, recall, false-positive ids, and adjudication referrals. It runs under `just test-review-suite` and `just test`, and launches nothing. -The calibration set must probe every grading boundary; a set that omits one -fails the test rather than passing quietly. Current probes for +A probe's `kind` is a claim about grading behaviour, not a label. The test +asserts the required outcome **per kind against the real grader**, not against +the set's own `expect` block, so a set cannot certify a boundary by declaring +whatever outcome it happens to get — a `plausible_false_positive` probe that +does not classify `unexpected` and charge a false positive fails. A set that +omits a required boundary altogether also fails. Current probes for `rollback-guidance-render`: -| probe | kind | asserts | -| ----------------------------------- | -------------------------- | -------------------------------------------------------------------------------------------------------- | -| `probe.observed-root-cause` | `observed` | the prose a real reviewer returned is `matched`, recall 1.0 | -| `probe.paraphrase` | `paraphrase` | a differently worded equivalent claim is also `matched` | -| `probe.overlapping-symptom` | `overlapping_symptom` | the same symptom without the cause is `partial`, earns no recall, and is not charged as a false positive | -| `probe.duplicate-report` | `duplicate_report` | one root cause reported twice earns recall once; the second is `duplicate` | -| `probe.partial-claim-wrong-surface` | `partial_claim` | the right cause at the wrong surface is `partial`, neither credited nor punished | -| `probe.plausible-false-positive` | `plausible_false_positive` | a plausible but unevidenced gating finding is `unexpected` and **is** charged as a false positive | -| `probe.accepted-non-finding` | `accepted_non_finding` | a real but non-material observation is `accepted`, not a false positive | +| probe | kind | asserts | +| ----------------------------------- | -------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- | +| `probe.observed-root-cause` | `observed` | the prose a real reviewer returned is `matched`, recall 1.0 | +| `probe.paraphrase` | `paraphrase` | a differently worded equivalent claim is also `matched` | +| `probe.overlapping-symptom` | `overlapping_symptom` | the same symptom without the cause is `partial`, earns no recall, and is not charged as a false positive | +| `probe.duplicate-report` | `duplicate_report` | one root cause reported twice earns recall once; the second is `duplicate` | +| `probe.partial-claim-wrong-surface` | `partial_claim` | the right cause at the wrong surface is `partial`, neither credited nor punished | +| `probe.plausible-false-positive` | `plausible_false_positive` | a plausible but unevidenced gating finding sharing no surface token is `unexpected` and **is** charged as a false positive | +| `probe.surface-token-collision` | `surface_token_collision` | the measured limit: an equally wrong gating finding at the untouched neighbour is only `partial`, because one shared token counts as a surface hit | +| `probe.accepted-non-finding` | `accepted_non_finding` | a real but non-material observation is `accepted`, not a false positive | ## Independent adjudication: not satisfied diff --git a/review-suite/evals/baseline/v1/COST-CEILING-PROPOSAL.md b/review-suite/evals/baseline/v1/COST-CEILING-PROPOSAL.md index 01e4d31..cf10fef 100644 --- a/review-suite/evals/baseline/v1/COST-CEILING-PROPOSAL.md +++ b/review-suite/evals/baseline/v1/COST-CEILING-PROPOSAL.md @@ -12,64 +12,91 @@ accounting. Nothing here is extrapolated from any earlier figure. The evaluator's protocol smoke run reported a cost, and that figure is void for this purpose twice over. Its usage accounting dropped cache-creation and cache-read tokens, understating input by orders of magnitude, and its verdicts -rested on a denominator of one. The envelope below was therefore measured fresh, -per stratum, at the frozen configuration. +rested on a denominator of one. The envelope below was measured fresh, per +stratum, at the frozen configuration. ## Measured pilot input -Three runs per stratum, one case per stratum, nine attempts, zero evaluation -failures. Suite commit `16560d807c66076fcbf3f00d3a87f543c6ae2458`, model -`claude-opus-4-6[1m]`, timeout 300 s, no retries. +Five runs per case at the frozen configuration: suite commit +`16560d807c66076fcbf3f00d3a87f543c6ae2458`, model `claude-opus-4-6[1m]`, corpus +version `1.3-pilot-*`, timeout 300 s, no retries, one fresh process per attempt. +**20 attempts, zero evaluation failures, zero timeouts, 1.2266 USD.** -| stratum | closure docs | input tokens / attempt | cost / attempt observed | latency max | -| --------------------------- | ------------ | ---------------------- | -------------------------- | ----------- | -| `pilot-orchestrator` | 8 | 32,573 | 0.1774, 0.0563, 0.0542 USD | 37.2 s | -| `pilot-solution-simplicity` | 2 | 25,901 | 0.1089, 0.0240, 0.0228 USD | 14.2 s | -| `pilot-code-simplicity` | 2 | 25,691 | 0.1110, 0.0242, 0.0245 USD | 15.0 s | +| stratum | closure docs | case | input tokens / attempt | cost, first attempt (cold cache) | cost, later attempts (cache read) | latency range | +| --------------------------- | ------------ | ---------------------------- | ---------------------- | -------------------------------- | --------------------------------- | ------------- | +| `pilot-orchestrator` | 8 | `rollback-guidance-render` | 33,166 | 0.1719 USD | 0.0467 – 0.0548 USD | 24.9 – 36.1 s | +| `pilot-orchestrator` | 8 | `status-label-normalization` | 34,144 | 0.1943 USD | 0.0570 – 0.0678 USD | 39.4 – 53.1 s | +| `pilot-solution-simplicity` | 2 | `rollback-guidance-render` | 26,482 | 0.1118 USD | 0.0215 – 0.0237 USD | 9.8 – 12.0 s | +| `pilot-code-simplicity` | 2 | `rollback-guidance-render` | 26,272 | 0.1170 USD | 0.0210 – 0.0240 USD | 9.2 – 16.8 s | -## The one non-obvious thing the pilot measured +## Two things the pilot measured that change how a ceiling must be built -Per-attempt cost is not uniform. In every stratum the **first** attempt cost -three to four times the ones after it, because it pays prompt-cache creation -while later attempts read the cache. The whole payload — skill closure, -contracts, reviewer prompt — is identical across attempts of a stratum, so -almost all of it is cacheable. +### 1. The first attempt of a case costs three to four times the rest -A ceiling built from the mean would therefore be exceeded by any real run whose -cache does not stay warm: a stratum executed in short bursts, strata -interleaved, a cache eviction, or simply a longer wall-clock run. The proposal -below is built from the **cold** per-attempt cost, so the ceiling holds in the -worst case rather than the lucky one. +Every stratum shows it. The payload — skill closure, review contracts, reviewer +prompt, packet — is identical across attempts of a case, so almost all of it is +cacheable; the first attempt pays cache creation and the rest are cache reads. + +A ceiling built from the mean would be exceeded by any real run whose cache does +not stay warm: strata executed in short bursts, strata interleaved, an eviction, +or simply a longer wall-clock run. **The proposal below assumes every attempt +pays cache creation**, which is a cost actually observed rather than a +hypothetical one. + +### 2. Cost tracks output volume, not packet size + +This is the non-obvious one, and it is why the corpus carries two pilot cases of +different size in the orchestrator stratum. + +The larger packet raised input tokens by only 978, **2.9%**. It raised +cache-read cost by **28%** and mean latency by **45%**. The reason is visible in +the attempt records: mean output grew from 1,262 to 1,816 tokens, **44%**, +because the larger packet warranted more findings, and output tokens are far +more expensive per token than cached input. + +So per-attempt cost scales with *how much a reviewer has to say*, which tracks +the number of material findings a case warrants — not with how big the packet +is. A scored stratum whose escapes carry two root causes each will sit at the +upper end of the observed range; its clean controls will sit at the lower end. +The proposal therefore uses the **worse** of the two measured cases for the +orchestrator stratum. + +Both figures were measured inside one run batch, so the runtime environment is +constant between them. Absolute input-token counts drifted slightly across +batches (32,507 → 32,955 → 33,166 for the same case), because reported input +includes runtime-side prompt overhead that the payload does not control; treat +the within-batch difference as the reliable measurement, not the absolute count. ## Proposal Scored composition and run count as frozen: 5 runs per case; 7 cases in the orchestrator-targeted correctness stratum, 4 in each lens stratum. -| stratum | cases | runs | attempts | cold cost / attempt | worst-case spend | +25% headroom | **proposed ceiling** | -| ----------------------------- | ----- | ---- | -------- | ------------------- | ---------------- | ------------- | -------------------- | -| `s1-correctness-orchestrator` | 7 | 5 | 35 | 0.1774 USD | 6.21 USD | 7.76 USD | **8.00 USD** | -| `s2-solution-simplicity-lens` | 4 | 5 | 20 | 0.1089 USD | 2.18 USD | 2.72 USD | **3.00 USD** | -| `s3-code-simplicity-lens` | 4 | 5 | 20 | 0.1110 USD | 2.22 USD | 2.78 USD | **3.00 USD** | -| all three | 15 | 5 | 75 | | 10.61 USD | 13.26 USD | **14.00 USD** | +| stratum | cases | runs | attempts | worst cold cost / attempt | all-cold worst case | +15% headroom | **proposed ceiling** | expected spend | +| ----------------------------- | ----- | ---- | -------- | ------------------------- | ------------------- | ------------- | -------------------- | -------------- | +| `s1-correctness-orchestrator` | 7 | 5 | 35 | 0.1943 USD | 6.80 USD | 7.82 USD | **8.00 USD** | 3.26 USD | +| `s2-solution-simplicity-lens` | 4 | 5 | 20 | 0.1118 USD | 2.24 USD | 2.57 USD | **3.00 USD** | 0.83 USD | +| `s3-code-simplicity-lens` | 4 | 5 | 20 | 0.1170 USD | 2.34 USD | 2.69 USD | **3.00 USD** | 0.85 USD | +| all three | 15 | 5 | 75 | | 11.38 USD | 13.08 USD | **14.00 USD** | 4.94 USD | -Expected spend if caching behaves as observed, using the warm figures, is -materially lower: about 3.4 USD for `s1`, 1.0 USD for each lens stratum, roughly -**5.5 USD total**. The ceiling is deliberately the pessimistic number; the -expectation is deliberately not. +**Expected spend** assumes the cache behaves as measured — one cache-creation +attempt per case, the rest cache reads — using the worse observed warm figure +per stratum. It is the number to expect. The ceiling is the number to +preregister. They differ by roughly a factor of three, and that gap is the +measured cost of not being able to rely on cache warmth. -The headroom covers reported-cost variance, a slightly longer answer, and the -possibility that a scored packet is larger than the pilot packet. It does not -cover a model change, a closure change, or a run-count change: each of those is -a new stratum and needs its own measured envelope. +The headroom covers reported-cost variance and a case that warrants more output +than either pilot case. It does **not** cover a model change, a closure change, +or a run-count change: each of those is a new stratum and needs its own measured +envelope. ## Ceilings are per stratum, not blended -The orchestrator stratum costs roughly 1.8 times a lens stratum per attempt -because it ships eight documents rather than two. One blended figure would -over-fund the lens strata and under-fund the orchestrator, and the first stratum -to run would silently consume another's budget. Track and enforce each -separately. +Per attempt, the orchestrator stratum costs roughly 1.7 times a lens stratum +cold and roughly 2.5 times warm, because it ships eight documents rather than +two. One blended figure would over-fund the lens strata and under-fund the +orchestrator, and the first stratum to run would silently consume another's +budget. Track and enforce each separately. ## What to do when a ceiling is reached @@ -78,14 +105,16 @@ Never reduce repetitions after outputs are visible: choosing a smaller denominator once results are known is how a stability figure becomes a claim about the model rather than a measurement. -## Costs this proposal does not cover - -- The scored run itself is the only paid step; `just test`, `just lint`, - `just check`, and `just audit-review-corpus` never launch a runtime. -- The pilot already spent **0.8889 USD** in total: 0.6031 USD over the nine - attempts recorded here, plus 0.2857 USD over three earlier orchestrator - attempts at corpus version `1.0-pilot-orchestrator`, whose observed prose the - shipped formulations were calibrated against. -- Re-running a stratum after a formulation change is not free. Calibration is - done against pilot output only, so a scored stratum should need no re-run for +## What this proposal does not cover + +- The scored run is the only paid step. `just test`, `just lint`, `just check`, + and `just audit-review-corpus` never launch a runtime. +- **Pilot spend across this ticket: 3.3776 USD over 52 attempts**, in five + batches — 0.2857 (3 attempts, calibration source), 0.6031 (9), 0.8496 (10), + 0.4126 (10), and 1.2266 (20, the committed configuration). The earlier batches + sized the envelope, produced the prose the formulations were calibrated + against, and were superseded as the corpus was corrected; only the last is + cited as the frozen envelope. +- Re-running a stratum after a formulation change is not free, so calibration is + done against pilot output only. A scored stratum should need no re-run for grading reasons. diff --git a/review-suite/evals/baseline/v1/LIMITATIONS.md b/review-suite/evals/baseline/v1/LIMITATIONS.md index 983c6c6..2c33c8e 100644 --- a/review-suite/evals/baseline/v1/LIMITATIONS.md +++ b/review-suite/evals/baseline/v1/LIMITATIONS.md @@ -61,9 +61,20 @@ demonstrated both halves of what that means: found the real root cause and every run scored `partial`, because the formulations had been written before any real prose existed. Recall was 0.0 while the reviewer was, in fact, correct on all three runs. -- **After calibration** against the observed prose, three fresh runs at the +- **After calibration** against the observed prose, five fresh attempts at the frozen version scored recall 1.0 with zero false positives and zero adjudication referrals. +- **Confirmed independently on a second case.** `status-label-normalization` is + carried by the orchestrator pilot stratum and deliberately left uncalibrated + as a control. Over five attempts it scored recall **0.0** with **nine + adjudication referrals**, while verdict stability was 1.0 and the reviewer + gated the change on every attempt. Two cases, ten attempts, one conclusion: an + uncalibrated expectation reports a number about itself. + +That is now a machine-readable property rather than a caution. Every expectation +carries a `calibrated` flag, and a test refuses a scored case that is not +calibrated or that ships no calibration set. **Read the flag before reading any +recall figure.** The generalisation limit is real: a future run that phrases the same finding a fifth way — "export is not among the registered subcommands" — matches none of @@ -72,20 +83,51 @@ the four shipped formulations and would be referred for adjudication. Read semantic matcher, or a standing adjudication queue, is a candidate mechanism for whoever owns v2. -## 4. A root cause's `surface` must name a symbol, not a path prefix - -Surface matching is token-level and file-level. Calibration found that a surface -written as `storectl/guidance.py render_rollback_guidance` shares the token -`storectl` with essentially every location in the packet, which made a -deliberately wrong gating finding at an unrelated file score `partial` instead -of being charged as a false positive. **With a broad surface, the false-positive -rate is not measurable**, because nearly every wrong finding is referred instead -of counted. - -Narrowing the surface to `render_rollback_guidance` restored the boundary. Any -stratum populated later must write surfaces as the smallest identifying symbol, -and its calibration set must include a `plausible_false_positive` probe that -actually classifies as `unexpected`. +## 4. The false-positive rate is only partly measurable, and stays that way + +`grader.py` counts a surface hit when the finding's location shares **one** +normalised token with the root cause's `surface`. A finding that hits the +surface but matches no formulation is `partial`, and a `partial` is referred for +adjudication rather than charged as a false positive. So a wrong gating finding +is only ever charged when its location shares **no** word with the surface. + +Calibration measured both halves of this on the pilot case, whose surface is now +the symbol `render_rollback_guidance`, tokenising to +`{render, rollback, guidance}`: + +| wrong gating finding located at | classified | charged as a false positive | +| ---------------------------------------- | ------------ | -------------------------------- | +| `storectl/cli.py` | `unexpected` | **yes** | +| `storectl/guidance.py` | `partial` | no — token `guidance` | +| `tests/test_guidance.py` | `partial` | no — token `guidance` | +| `storectl/guidance.py:render_apply_plan` | `partial` | no — tokens `render`, `guidance` | + +Writing the surface as a path prefix was worse — +`storectl/guidance.py render_rollback_guidance` also collides on `storectl`, +which appears in nearly every location in the packet — and narrowing it to the +symbol shrank the unchargeable region. **It did not remove it.** The locations +that remain unchargeable are the changed file, its test, and its untouched +neighbours, which is exactly where a real reviewer is most likely to point. + +Consequences, and they are not cosmetic: + +- **`false_positive_rate` is a lower bound, not a rate.** It counts only wrong + gating findings that share no word with any root-cause surface. + `frozen-configuration.json` records it as partially measurable rather than + measured, and #59 must not read it as a measured rate. +- Any stratum populated later must still write surfaces as the smallest + identifying symbol, and its calibration set must carry a + `plausible_false_positive` probe that genuinely classifies `unexpected` and + charges a false positive. The calibration test asserts that outcome per probe + kind rather than accepting the set's own claim. +- The `surface_token_collision` probe in the shipped calibration set pins the + residual gap in place, so it stays visible instead of being rediscovered. + +The one-token rule is `grader.match_strength`, which is #50 infrastructure this +ticket's non-goals forbid repairing. Requiring every surface token instead of +any is a small change with a real trade-off — it would turn some correct +findings that name only part of a surface into misses — so it is **escalated to +#59 as a candidate preregistered v2 gate**, not recorded here as solved. ## 5. An expectation is target-specific @@ -101,6 +143,16 @@ the lens the stratum targets. Sharing one case list across strata with different targets would grade contract-faithful reviewers as wrong and invalidate the affected stratum. +Because the two lens pilot reports are committed and machine-readable, and they +do carry `material_finding_recall: 0.0` and `false_clean_rate: 1.0`, the caveat +is also recorded where a reader reaches them rather than only here: each +mismatched corpus declares `stratum.grading_is_signal: false`, a test forbids a +scored stratum from declaring it false, `frozen-configuration.json` annotates +each pilot report entry, and `pilot/README.md` sits beside the reports. **Do not +quote the recall or false-clean figure from +`pilot-solution-simplicity.report.json` or +`pilot-code-simplicity.report.json`.** + ## 6. Severity is required but not measured `expectation.schema.json` requires a `severity` on every root cause, and no @@ -124,6 +176,22 @@ root cause, accepted non-finding, severity, and allowed formulation, and it is not presented as satisfying it. See [CALIBRATION.md](CALIBRATION.md) for exactly what has and has not been adjudicated. +[ADJUDICATION-PLAN.md](ADJUDICATION-PLAN.md) proposes how the gate can honestly +be satisfied, and reaches one conclusion worth surfacing here: a fresh blind +agent context is a legitimate second adjudicator for whether a minimized case is +faithful and whether a formulation is well drawn, but **not** for whether a +defect is materially real when it shares a model family with the reviewer being +measured. Those errors are correlated, so two such adjudications are not two +independent observations, and recall would rise without the reviewer improving. +Where a minimized reproduction can be made to run, an executable oracle is a +stronger second adjudication than any opinion — the corpus already identifies +one case adjudicated exactly that way, by CI. The simplicity strata have no +oracle at all and need human adjudication. + +A third open decision is recorded there: **neither identified clean-control +candidate is an adjudicated clean review.** Until the owner settles what a clean +control must be, the false-alarm rate has no honest denominator. + ## 9. Cost figures are runtime-reported, and cache-sensitive Cost comes from what the runtime reports, not from an independent meter. The @@ -131,6 +199,18 @@ pilot also showed per-attempt cost varying three- to fourfold within a stratum purely from prompt-cache state. Any cost figure is a report about one run's caching behaviour as much as about the work done. +Two further measurements bear on any cost figure quoted from here: + +- **Cost tracks output volume, not packet size.** The orchestrator stratum's + larger packet raised input tokens 2.9% and raised cache-read cost 28% and mean + latency 45%, because mean output grew 44%. A case's cost therefore depends on + how much a reviewer has to say about it, which tracks the number of findings + it warrants. A stratum's cost is not predictable from its packet sizes alone. +- **Reported input tokens include runtime-side prompt overhead.** The same + case's reported input drifted across batches — 32,507, then 32,955, then + 33,166 — with the payload unchanged. Treat within-batch differences as + measurements and absolute counts as approximate. + ## 10. A stratum boundary is not a rounding difference A change of runtime, runtime version, model, target skill, dependency closure, diff --git a/review-suite/evals/baseline/v1/SOURCING.md b/review-suite/evals/baseline/v1/SOURCING.md index be2d994..30e4b36 100644 --- a/review-suite/evals/baseline/v1/SOURCING.md +++ b/review-suite/evals/baseline/v1/SOURCING.md @@ -71,6 +71,19 @@ not carry it. | sanitization | Rewritten against a fictional `storectl` CLI. No source identifier, path, symbol, prose, or diff copied. No business logic, domain identifier, customer context, credential, or hidden reasoning. | | retention authority | Public repository, owner-authored review, no third-party or customer material. | +### `status-label-normalization` — pilot, unscored, deliberately uncalibrated + +| field | value | +| ---------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| strata | `pilot-orchestrator` only, so the case shared with the lens strata stays byte-identical across all three | +| source | `shaug/atelier` pull request 278, review comment 2861937742, authored by the repository owner. Public. | +| adjudication | The reviewer required the change; the follow-up reply (comment 2861947492) names the implementing commit and the regression coverage added for the exact migration shape. Accepted, not deferred. | +| failure shape retained | A normalization step drops one legacy flag and returns early for an inactive status, leaving a second legacy flag that contradicts the canonical status, with a live consumer that admits work on that flag alone. The added tests all start from records that never carried the second flag, so the one shape the change exists to fix is untested. | +| origin | `minimized_reproduction` | +| sanitization | Rewritten against a fictional record registry. No source identifier, path, symbol, prose, or diff copied. No business logic, domain identifier, customer context, credential, or hidden reasoning. | +| retention authority | Public repository, owner-authored review, no third-party or customer material. | +| why it exists | Two purposes. Its packet is materially larger than the other pilot case, which separates the fixed cost of a stratum's skill closure from the variable cost of the packet — the measurement behind the cost-ceiling proposal. And it is **deliberately left uncalibrated** (`calibrated: false`) as the control for what an uncalibrated expectation reports: recall 0.0 over five attempts with nine adjudication referrals, while the reviewer gated the change every time. | + ## Cases excluded, and why | candidate | class it would have served | why excluded | @@ -86,12 +99,12 @@ listed below with the ground truth already identified for it. The scored corpus is populated in evidence-preserving batches rather than in one change. This batch delivers the strata layout, the per-stratum pilot envelope, -the grader calibration machinery and its first calibrated case, the -contamination audit over every corpus, the frozen configuration, and the -cost-ceiling proposal. The scored case population follows, one stratum per -batch, because each scored case needs individual sourcing, minimization, and -independent adjudication judgement, and fifteen of those in one change would not -be reviewable. +the grader calibration machinery with one calibrated case and one deliberately +uncalibrated control, the contamination audit over every corpus, the frozen +configuration, and the cost-ceiling proposal. The scored case population +follows, one stratum per batch, because each scored case needs individual +sourcing, minimization, and independent adjudication judgement, and fifteen of +those in one change would not be reviewable. The candidate ground truth below was identified while sourcing this batch and is recorded so the evidence is not lost. Each still requires the adjudication trail @@ -141,3 +154,7 @@ split-write case all do. - Retention authority is recorded per case, and a case whose authority could not be established would be excluded before scoring rather than minimized into the corpus. +- Nothing here is adjudicated twice yet. Which of these candidates a second + adjudication is expected to *disagree* with, and which source dispositions are + too ambiguous to use without re-verification, is recorded in + [ADJUDICATION-PLAN.md](ADJUDICATION-PLAN.md) rather than smoothed over here. diff --git a/review-suite/evals/baseline/v1/frozen-configuration.json b/review-suite/evals/baseline/v1/frozen-configuration.json index 638f179..e916140 100644 --- a/review-suite/evals/baseline/v1/frozen-configuration.json +++ b/review-suite/evals/baseline/v1/frozen-configuration.json @@ -18,14 +18,17 @@ }, "execution": { "runs_per_case": 5, - "runs_per_case_rationale": "A denominator of one cannot measure a stochastic reviewer, and this suite has already observed the same configuration return `blocked` on one attempt and a merge verdict on the next for the case that tests refusal on incomplete evidence. Five is the smallest count that makes a per-case verdict-stability figure mean anything while keeping the worst-case spend inside the proposed ceiling.", + "runs_per_case_rationale": "A denominator of one cannot measure a stochastic reviewer, and this suite has already observed the same configuration return `blocked` on one attempt and a merge verdict on the next for the case that tests refusal on incomplete evidence. Five is the smallest count that makes a per-case verdict-stability figure mean anything. Measured at five runs per case over 20 attempts, the pilot observed verdict stability 1.0 and finding stability 1.0 on every case, so five did not surface instability on these packets - which is a statement about these packets, not a licence to reduce the count. The instability #50 observed was on a refusal-on-incomplete-evidence case, a class no scored stratum here contains.", "timeout_seconds": 300, - "timeout_rationale": "Worst observed pilot latency was 52.7 s on the orchestrator stratum. 300 s is roughly six times that, which absorbs a slow attempt without letting a hung process run unbounded.", + "timeout_rationale": "Worst latency in any committed pilot report is 53.1 s, on the orchestrator stratum's larger packet. The slowest attempt observed at any point across this ticket's 52 pilot attempts was 55.4 s, also on that packet. 300 s is roughly 5.4 times the worst observed, which absorbs a slow attempt without letting a hung process run unbounded. No pilot attempt timed out, and no attempt came within a factor of five of the limit.", "retry_policy": "none", "retry_policy_rationale": "A failed attempt is recorded as an evaluation failure and never retried. Retrying would silently replace an unstable attempt with a luckier one and would bias every stability figure toward agreement.", "max_output_bytes": 4000000, "process_isolation": "one fresh executor process per attempt", - "cost_ceiling_exhaustion_policy": "Exceeding a stratum's preregistered ceiling stops further runs in that stratum and records an incomplete baseline for it. Repetitions are never reduced after outputs are visible." + "cost_ceiling_exhaustion_policy": "Exceeding a stratum's preregistered ceiling stops further runs in that stratum and records an incomplete baseline for it. Repetitions are never reduced after outputs are visible.", + "artifact_retention": "Raw executor output is retained outside git at review-suite/evals/artifacts///.run-.stdout.json. The corpus version is in the path deliberately: an artifact name carries only the case and run number, so re-running a stratum into an unscoped directory silently replaces output that a committed record may already cite, which happened once and cost a calibration source run. The runner now refuses before launching any attempt when a run would overwrite retained output, so the failure costs nothing.", + "invocation_template": "just eval-review-suite '' --corpus review-suite/evals/strata/ --runs --timeout --artifact-dir review-suite/evals/artifacts// --attempts-out --report-out ", + "invocation_note": "The recipe forwards extra arguments to the runner. Naming an executor alone is not the frozen configuration: --corpus defaults to the protocol-proof corpus and --runs to 1, so the abbreviated command silently evaluates a different corpus once." }, "strata": [ { @@ -50,7 +53,11 @@ ], "envelope_source": "pilot-orchestrator", "cost_ceiling_usd": null, - "cost_ceiling_proposal_usd": 8.0 + "cost_ceiling_proposal_usd": 8.0, + "scored_invocation": "just eval-review-suite 'python3 review-suite/scripts/evals/claude_executor.py' --corpus review-suite/evals/strata/s1-correctness-orchestrator --runs 5 --timeout 300 --artifact-dir review-suite/evals/artifacts/s1-correctness-orchestrator/ --attempts-out /s1-correctness-orchestrator.attempts.jsonl --report-out review-suite/evals/baseline/v1/s1-correctness-orchestrator.report.json", + "scored_invocation_note": "Executable verbatim once the stratum directory exists and the owner has preregistered the ceiling. Substitute the stratum's own corpus_version in the artifact directory.", + "expected_spend_usd": 3.26, + "expected_spend_note": "Expected spend assumes the prompt cache behaves as measured: one cache-creation attempt per case, the rest cache reads. The proposed ceiling assumes every attempt pays cache creation, which is the worst case actually observed rather than a hypothetical one." }, { "id": "s2-solution-simplicity-lens", @@ -67,7 +74,11 @@ ], "envelope_source": "pilot-solution-simplicity", "cost_ceiling_usd": null, - "cost_ceiling_proposal_usd": 3.0 + "cost_ceiling_proposal_usd": 3.0, + "scored_invocation": "just eval-review-suite 'python3 review-suite/scripts/evals/claude_executor.py' --corpus review-suite/evals/strata/s2-solution-simplicity-lens --runs 5 --timeout 300 --artifact-dir review-suite/evals/artifacts/s2-solution-simplicity-lens/ --attempts-out /s2-solution-simplicity-lens.attempts.jsonl --report-out review-suite/evals/baseline/v1/s2-solution-simplicity-lens.report.json", + "scored_invocation_note": "Executable verbatim once the stratum directory exists and the owner has preregistered the ceiling. Substitute the stratum's own corpus_version in the artifact directory.", + "expected_spend_usd": 0.83, + "expected_spend_note": "Expected spend assumes the prompt cache behaves as measured: one cache-creation attempt per case, the rest cache reads. The proposed ceiling assumes every attempt pays cache creation, which is the worst case actually observed rather than a hypothetical one." }, { "id": "s3-code-simplicity-lens", @@ -84,7 +95,11 @@ ], "envelope_source": "pilot-code-simplicity", "cost_ceiling_usd": null, - "cost_ceiling_proposal_usd": 3.0 + "cost_ceiling_proposal_usd": 3.0, + "scored_invocation": "just eval-review-suite 'python3 review-suite/scripts/evals/claude_executor.py' --corpus review-suite/evals/strata/s3-code-simplicity-lens --runs 5 --timeout 300 --artifact-dir review-suite/evals/artifacts/s3-code-simplicity-lens/ --attempts-out /s3-code-simplicity-lens.attempts.jsonl --report-out review-suite/evals/baseline/v1/s3-code-simplicity-lens.report.json", + "scored_invocation_note": "Executable verbatim once the stratum directory exists and the owner has preregistered the ceiling. Substitute the stratum's own corpus_version in the artifact directory.", + "expected_spend_usd": 0.85, + "expected_spend_note": "Expected spend assumes the prompt cache behaves as measured: one cache-creation attempt per case, the rest cache reads. The proposed ceiling assumes every attempt pays cache creation, which is the worst case actually observed rather than a hypothetical one." }, { "id": "connector-escape", @@ -119,20 +134,49 @@ { "metric": "severity agreement", "detail": "`expectation.schema.json` requires a `severity` on every root cause that no metric consumes. Severity agreement is not measured and must not be assumed." + }, + { + "metric": "false-positive rate, as a rate", + "detail": "The grader counts a surface hit when a finding's location shares one normalised token with a root cause's surface, and a surface hit without a formulation match is referred for adjudication rather than charged. A wrong gating finding is therefore only charged when its location shares no word with any root-cause surface. Measured on the pilot case: a wrong gating finding at storectl/cli.py is charged, while the same finding at storectl/guidance.py, at tests/test_guidance.py, or at the untouched render_apply_plan is only referred.", + "consequence_for_this_baseline": "Read `false_positive_rate` as a lower bound, never as a measured rate. The unchargeable region is the changed file, its test, and its neighbours, which is where a real reviewer is most likely to point. Narrowing a surface from a path to a symbol shrinks the region and does not remove it. The one-token rule is grader behaviour this ticket may not change, so it is escalated to #59 as a candidate preregistered v2 gate; the probe.surface-token-collision calibration probe keeps the gap visible." } ], "pending_owner_inputs": [ - "A per-stratum cost ceiling, preregistered before any scored output is examined. Proposals are in COST-CEILING-PROPOSAL.md.", - "Two independent adjudications for each material root cause, accepted non-finding, severity, and allowed equivalent formulation, from genuinely separate parties, with disagreements resolved in a recorded adjudication note. Current state is in CALIBRATION.md." + "A per-stratum cost ceiling, preregistered before any scored output is examined. Proposals, and the pilot arithmetic behind them, are in COST-CEILING-PROPOSAL.md.", + "Two independent adjudications for each material root cause, accepted non-finding, severity, and allowed equivalent formulation, from genuinely separate parties, with disagreements resolved in a recorded note. ADJUDICATION-PLAN.md assesses what the source review threads can and cannot supply, where a blind agent context is and is not a legitimate second adjudicator, and every case where the two adjudications are expected to disagree.", + "A decision on what a clean correctness control must be. Neither identified candidate is an adjudicated clean review, and until the standard is settled the false-alarm rate has no honest denominator. See ADJUDICATION-PLAN.md section 4." ], "pilot_reports": [ - "pilot/pilot-orchestrator.report.json", - "pilot/pilot-solution-simplicity.report.json", - "pilot/pilot-code-simplicity.report.json" + { + "report": "pilot/pilot-orchestrator.report.json", + "stratum": "pilot-orchestrator", + "quality_block_is_signal": true, + "retained_artifacts": "review-suite/evals/artifacts/pilot-orchestrator/1.3-pilot-orchestrator/", + "invocation": "just eval-review-suite 'python3 review-suite/scripts/evals/claude_executor.py' --corpus review-suite/evals/strata/pilot-orchestrator --runs 5 --timeout 300 --artifact-dir review-suite/evals/artifacts/pilot-orchestrator/1.3-pilot-orchestrator --report-out review-suite/evals/baseline/v1/pilot/pilot-orchestrator.report.json", + "quality_block_caveat": "The aggregate recall of 0.5 is the mean of two cases that differ in one respect only: rollback-guidance-render is calibrated and scored recall 1.0 on all five attempts, while status-label-normalization is deliberately left uncalibrated as a control and scored recall 0.0 on all five, with nine adjudication referrals, while the reviewer gated the change every time. Do not read 0.5 as a capability figure; read the per-case block, and read each case's expectation `calibrated` flag first." + }, + { + "report": "pilot/pilot-solution-simplicity.report.json", + "stratum": "pilot-solution-simplicity", + "quality_block_is_signal": false, + "quality_block_caveat": "DO NOT QUOTE this report's material_finding_recall (0.0) or false_clean_rate (1.0) as a reviewer quality figure. This corpus carries the orchestrator stratum's case byte for byte, so that the declared skill closure is the only variable between pilot strata, and the expectation is a correctness root cause authored for that target. A simplicity lens returning clean on a correctness defect is behaving exactly as its contract requires, so the recorded miss is a target mismatch, not a reviewer failure. The corpus declares stratum.grading_is_signal false for this reason. Only payload size, latency, cost, and protocol outcomes are signals here.", + "retained_artifacts": "review-suite/evals/artifacts/pilot-solution-simplicity/1.3-pilot-solution-simplicity/", + "invocation": "just eval-review-suite 'python3 review-suite/scripts/evals/claude_executor.py' --corpus review-suite/evals/strata/pilot-solution-simplicity --runs 5 --timeout 300 --artifact-dir review-suite/evals/artifacts/pilot-solution-simplicity/1.3-pilot-solution-simplicity --report-out review-suite/evals/baseline/v1/pilot/pilot-solution-simplicity.report.json" + }, + { + "report": "pilot/pilot-code-simplicity.report.json", + "stratum": "pilot-code-simplicity", + "quality_block_is_signal": false, + "quality_block_caveat": "DO NOT QUOTE this report's material_finding_recall (0.0) or false_clean_rate (1.0) as a reviewer quality figure. This corpus carries the orchestrator stratum's case byte for byte, so that the declared skill closure is the only variable between pilot strata, and the expectation is a correctness root cause authored for that target. A simplicity lens returning clean on a correctness defect is behaving exactly as its contract requires, so the recorded miss is a target mismatch, not a reviewer failure. The corpus declares stratum.grading_is_signal false for this reason. Only payload size, latency, cost, and protocol outcomes are signals here.", + "retained_artifacts": "review-suite/evals/artifacts/pilot-code-simplicity/1.3-pilot-code-simplicity/", + "invocation": "just eval-review-suite 'python3 review-suite/scripts/evals/claude_executor.py' --corpus review-suite/evals/strata/pilot-code-simplicity --runs 5 --timeout 300 --artifact-dir review-suite/evals/artifacts/pilot-code-simplicity/1.3-pilot-code-simplicity --report-out review-suite/evals/baseline/v1/pilot/pilot-code-simplicity.report.json" + } ], "controlled_artifacts": { "location": "review-suite/evals/artifacts//.run-.stdout.json", "in_git": false, "detail": "Raw executor output is retained outside git by the repository's existing ignore rule. The committed compact reports carry every metric, denominator, and identity a consumer needs; the raw transcripts are referenced by that path identity for anyone re-deriving a figure." - } + }, + "baseline_limitations_record": "LIMITATIONS.md", + "adjudication_plan": "ADJUDICATION-PLAN.md" } diff --git a/review-suite/evals/baseline/v1/pilot/README.md b/review-suite/evals/baseline/v1/pilot/README.md new file mode 100644 index 0000000..f965119 --- /dev/null +++ b/review-suite/evals/baseline/v1/pilot/README.md @@ -0,0 +1,59 @@ +# Unscored pilot reports + +These three reports establish executor compatibility, timeout behaviour, and a +cost and latency envelope **per stratum**. They are not a baseline. No scored +case exists yet, and nothing here measures reviewer quality across a corpus. + +Five runs per case. Suite commit `16560d807c66076fcbf3f00d3a87f543c6ae2458`, +model `claude-opus-4-6[1m]`, corpus version `1.3-pilot-*`, grader version `1.0`, +timeout 300 s, no retries, one fresh process per attempt. **20 attempts, zero +evaluation failures, zero timeouts, 1.2266 USD.** Raw output is retained outside +git at `review-suite/evals/artifacts//1.3-/`. + +## Before reading any recall number, read the case's `calibrated` flag + +Recall here is a property of the expectation at least as much as of the +reviewer. The orchestrator stratum carries two cases that differ in exactly that +respect, and the contrast is the most useful thing in this directory: + +| case | `calibrated` | recall over 5 attempts | verdict stability | referrals | what it means | +| ---------------------------- | ------------ | ---------------------- | ----------------- | --------- | ---------------------------------------------------------------------------------------------------------------------------- | +| `rollback-guidance-render` | `true` | **1.0** | 1.0 | 0 | Formulations pinned to prose a real reviewer wrote. The grader recognises the finding every time. | +| `status-label-normalization` | `false` | **0.0** | 1.0 | 9 | Deliberately uncalibrated control. The reviewer gated the change on every attempt and the grader matched neither root cause. | + +The aggregate `material_finding_recall: 0.5` in `pilot-orchestrator.report.json` +is the mean of those two. **It is not a capability figure.** It is the measured +cost of shipping an uncalibrated expectation, and it is why a test refuses a +scored case whose expectation is not calibrated. + +## Read the quality block in only one of these + +| report | quality block | +| --------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------- | +| `pilot-orchestrator.report.json` | **is** a signal for its target, `review-code-change` — but per case, with the `calibrated` flag in hand, never as the aggregate. | +| `pilot-solution-simplicity.report.json` | **is not** a signal. Do not quote its recall or false-clean rate. | +| `pilot-code-simplicity.report.json` | **is not** a signal. Do not quote its recall or false-clean rate. | + +All three corpora carry `rollback-guidance-render` byte for byte, deliberately, +so that the declared skill closure is the only variable between them — which is +the whole point of measuring an envelope per stratum. The consequence is that +the case's expectation, a correctness root cause authored for the orchestrator +target, is the wrong yardstick for a simplicity lens. + +Both lens reports therefore record `material_finding_recall: 0.0` and +`false_clean_rate: 1.0`. **That is contract-faithful reviewer behaviour graded +against a target-mismatched expectation, not a reviewer failure.** A +code-simplicity lens is not contracted to report a correctness defect, so +returning clean is correct — and both lenses did so on all five attempts, with +verdict stability 1.0. Each mismatched corpus declares +`stratum.grading_is_signal: false`, and a test forbids a scored stratum from +declaring it false. + +## What is a signal in all three + +Payload size, input tokens, cost, latency, and protocol outcomes. Those are the +figures [the cost-ceiling proposal](../COST-CEILING-PROPOSAL.md) is built from, +and they are what the pilot exists to measure. + +Read [the limitations record](../LIMITATIONS.md) before quoting anything from +this directory. diff --git a/review-suite/evals/baseline/v1/pilot/pilot-code-simplicity.report.json b/review-suite/evals/baseline/v1/pilot/pilot-code-simplicity.report.json index 58c9347..ba1d8df 100644 --- a/review-suite/evals/baseline/v1/pilot/pilot-code-simplicity.report.json +++ b/review-suite/evals/baseline/v1/pilot/pilot-code-simplicity.report.json @@ -1,16 +1,16 @@ { "adjudication_required": [], - "attempts": 3, + "attempts": 5, "baseline_eligible": true, "configuration": { "artifacts_retained": true, "cases": 1, - "corpus_version": "1.1-pilot-code-simplicity", + "corpus_version": "1.3-pilot-code-simplicity", "executor": "python3 review-suite/scripts/evals/claude_executor.py", "grader_version": "1.0", "max_output_bytes": 4000000, - "runs_per_case": 3, - "suite_commit": "16560d807c66076fcbf3f00d3a87f543c6ae2458", + "runs_per_case": 5, + "suite_commit": "16b77e447dbcc844edd8f3fb58728d96826e177c", "target_skill": "review-code-simplicity", "target_skill_dependencies": [], "target_skill_digest": "a6187d8971eaef24", @@ -29,32 +29,32 @@ "spawn_failure_rate": 0.0, "timeout_rate": 0.0 }, - "graded_attempts": 3, + "graded_attempts": 5, "grader_version": "1.0", "latency": { - "count": 3, - "max_seconds": 14.965433417353779, - "mean_seconds": 13.427382861574491, - "min_seconds": 12.56160541716963, - "p50_seconds": 12.755109750200063 + "count": 5, + "max_seconds": 16.773228792008013, + "mean_seconds": 12.84428822491318, + "min_seconds": 9.155616583768278, + "p50_seconds": 12.735041291918606 }, "per_case": [ { - "attempts": 3, + "attempts": 5, "case_id": "rollback-guidance-render", "expected_root_cause_ids": [ "rc.unsupported-emitted-subcommand" ], "false_alarm_attempts": 0, - "false_clean_attempts": 3, + "false_clean_attempts": 5, "false_positive_attempts": 0, "finding_stability": 1.0, - "graded_attempts": 3, + "graded_attempts": 5, "intersection_root_cause_ids": [], "mean_recall": 0.0, - "stability_denominator": 3, + "stability_denominator": 5, "statuses": { - "review_result": 3 + "review_result": 5 }, "union_root_cause_ids": [], "verdict_stability": 1.0 @@ -64,12 +64,12 @@ "quality": { "false_alarm_denominator": 0, "false_alarm_rate": null, - "false_clean_denominator": 3, + "false_clean_denominator": 5, "false_clean_rate": 1.0, - "false_positive_denominator": 3, + "false_positive_denominator": 5, "false_positive_rate": 0.0, "material_finding_recall": 0.0, - "recall_attempts": 3, + "recall_attempts": 5, "unique_finding_contribution": [] }, "report_version": "1.0", @@ -81,20 +81,20 @@ "rollback-guidance-render": 1.0 }, "per_case_stability_denominator": { - "rollback-guidance-render": 3 + "rollback-guidance-render": 5 }, "per_case_verdict_stability": { "rollback-guidance-render": 1.0 }, - "stability_denominator": 3 + "stability_denominator": 5 }, "usage": { "available": true, - "reporting_attempts_cost_usd": 3, - "reporting_attempts_input_tokens": 3, - "reporting_attempts_output_tokens": 3, - "total_cost_usd": 0.15965025, - "total_input_tokens": 77073.0, - "total_output_tokens": 1418.0 + "reporting_attempts_cost_usd": 5, + "reporting_attempts_input_tokens": 5, + "reporting_attempts_output_tokens": 5, + "total_cost_usd": 0.2087365, + "total_input_tokens": 131360.0, + "total_output_tokens": 2161.0 } } diff --git a/review-suite/evals/baseline/v1/pilot/pilot-orchestrator.report.json b/review-suite/evals/baseline/v1/pilot/pilot-orchestrator.report.json index 454c6b9..6c51398 100644 --- a/review-suite/evals/baseline/v1/pilot/pilot-orchestrator.report.json +++ b/review-suite/evals/baseline/v1/pilot/pilot-orchestrator.report.json @@ -1,16 +1,107 @@ { - "adjudication_required": [], - "attempts": 3, + "adjudication_required": [ + { + "candidate_root_cause_ids": [ + "rc.retained-ready-flag-on-inactive-status", + "rc.tests-omit-the-migrating-shape" + ], + "case_id": "status-label-normalization", + "finding_id": "ready-flag-retained-on-inactive", + "reason": "partial", + "run_number": 1 + }, + { + "candidate_root_cause_ids": [ + "rc.retained-ready-flag-on-inactive-status", + "rc.tests-omit-the-migrating-shape" + ], + "case_id": "status-label-normalization", + "finding_id": "inactive-record-retains-ready-flag", + "reason": "partial", + "run_number": 2 + }, + { + "candidate_root_cause_ids": [ + "rc.retained-ready-flag-on-inactive-status", + "rc.tests-omit-the-migrating-shape" + ], + "case_id": "status-label-normalization", + "finding_id": "missing-test-both-flags-inactive", + "reason": "partial", + "run_number": 2 + }, + { + "candidate_root_cause_ids": [ + "rc.retained-ready-flag-on-inactive-status", + "rc.tests-omit-the-migrating-shape" + ], + "case_id": "status-label-normalization", + "finding_id": "correctness.inactive-ready-flag-retained", + "reason": "partial", + "run_number": 3 + }, + { + "candidate_root_cause_ids": [ + "rc.retained-ready-flag-on-inactive-status", + "rc.tests-omit-the-migrating-shape" + ], + "case_id": "status-label-normalization", + "finding_id": "correctness.missing-dual-flag-test", + "reason": "partial", + "run_number": 3 + }, + { + "candidate_root_cause_ids": [ + "rc.retained-ready-flag-on-inactive-status", + "rc.tests-omit-the-migrating-shape" + ], + "case_id": "status-label-normalization", + "finding_id": "ready-flag-retained-on-inactive-dual-flag", + "reason": "partial", + "run_number": 4 + }, + { + "candidate_root_cause_ids": [ + "rc.retained-ready-flag-on-inactive-status", + "rc.tests-omit-the-migrating-shape" + ], + "case_id": "status-label-normalization", + "finding_id": "missing-dual-flag-test", + "reason": "partial", + "run_number": 4 + }, + { + "candidate_root_cause_ids": [ + "rc.retained-ready-flag-on-inactive-status", + "rc.tests-omit-the-migrating-shape" + ], + "case_id": "status-label-normalization", + "finding_id": "ready-flag-retained-on-inactive", + "reason": "partial", + "run_number": 5 + }, + { + "candidate_root_cause_ids": [ + "rc.retained-ready-flag-on-inactive-status", + "rc.tests-omit-the-migrating-shape" + ], + "case_id": "status-label-normalization", + "finding_id": "missing-dual-flag-inactive-test", + "reason": "partial", + "run_number": 5 + } + ], + "attempts": 10, "baseline_eligible": true, "configuration": { "artifacts_retained": true, - "cases": 1, - "corpus_version": "1.1-pilot-orchestrator", + "cases": 2, + "corpus_version": "1.3-pilot-orchestrator", "executor": "python3 review-suite/scripts/evals/claude_executor.py", "grader_version": "1.0", "max_output_bytes": 4000000, - "runs_per_case": 3, - "suite_commit": "16560d807c66076fcbf3f00d3a87f543c6ae2458", + "runs_per_case": 5, + "suite_commit": "16b77e447dbcc844edd8f3fb58728d96826e177c", "target_skill": "review-code-change", "target_skill_dependencies": [ "review-solution-simplicity", @@ -39,18 +130,18 @@ "spawn_failure_rate": 0.0, "timeout_rate": 0.0 }, - "graded_attempts": 3, + "graded_attempts": 10, "grader_version": "1.0", "latency": { - "count": 3, - "max_seconds": 37.20436308393255, - "mean_seconds": 34.76848513881365, - "min_seconds": 30.250969290733337, - "p50_seconds": 36.85012304177508 + "count": 10, + "max_seconds": 53.09396595787257, + "mean_seconds": 36.91776127498597, + "min_seconds": 24.860583000350744, + "p50_seconds": 37.728567416314036 }, "per_case": [ { - "attempts": 3, + "attempts": 5, "case_id": "rollback-guidance-render", "expected_root_cause_ids": [ "rc.unsupported-emitted-subcommand" @@ -59,31 +150,52 @@ "false_clean_attempts": 0, "false_positive_attempts": 0, "finding_stability": 1.0, - "graded_attempts": 3, + "graded_attempts": 5, "intersection_root_cause_ids": [ "rc.unsupported-emitted-subcommand" ], "mean_recall": 1.0, - "stability_denominator": 3, + "stability_denominator": 5, "statuses": { - "review_result": 3 + "review_result": 5 }, "union_root_cause_ids": [ "rc.unsupported-emitted-subcommand" ], "verdict_stability": 1.0 + }, + { + "attempts": 5, + "case_id": "status-label-normalization", + "expected_root_cause_ids": [ + "rc.retained-ready-flag-on-inactive-status", + "rc.tests-omit-the-migrating-shape" + ], + "false_alarm_attempts": 0, + "false_clean_attempts": 0, + "false_positive_attempts": 0, + "finding_stability": 1.0, + "graded_attempts": 5, + "intersection_root_cause_ids": [], + "mean_recall": 0.0, + "stability_denominator": 5, + "statuses": { + "review_result": 5 + }, + "union_root_cause_ids": [], + "verdict_stability": 1.0 } ], "protocol_version": "1.0", "quality": { "false_alarm_denominator": 0, "false_alarm_rate": null, - "false_clean_denominator": 3, + "false_clean_denominator": 10, "false_clean_rate": 0.0, - "false_positive_denominator": 3, + "false_positive_denominator": 10, "false_positive_rate": 0.0, - "material_finding_recall": 1.0, - "recall_attempts": 3, + "material_finding_recall": 0.5, + "recall_attempts": 10, "unique_finding_contribution": [] }, "report_version": "1.0", @@ -92,23 +204,26 @@ "mean_finding_stability": 1.0, "mean_verdict_stability": 1.0, "per_case_finding_stability": { - "rollback-guidance-render": 1.0 + "rollback-guidance-render": 1.0, + "status-label-normalization": 1.0 }, "per_case_stability_denominator": { - "rollback-guidance-render": 3 + "rollback-guidance-render": 5, + "status-label-normalization": 5 }, "per_case_verdict_stability": { - "rollback-guidance-render": 1.0 + "rollback-guidance-render": 1.0, + "status-label-normalization": 1.0 }, - "stability_denominator": 3 + "stability_denominator": 10 }, "usage": { "available": true, - "reporting_attempts_cost_usd": 3, - "reporting_attempts_input_tokens": 3, - "reporting_attempts_output_tokens": 3, - "total_cost_usd": 0.28784475, - "total_input_tokens": 97719.0, - "total_output_tokens": 4550.0 + "reporting_attempts_cost_usd": 10, + "reporting_attempts_input_tokens": 10, + "reporting_attempts_output_tokens": 10, + "total_cost_usd": 0.8159675000000002, + "total_input_tokens": 336550.0, + "total_output_tokens": 15389.0 } } diff --git a/review-suite/evals/baseline/v1/pilot/pilot-solution-simplicity.report.json b/review-suite/evals/baseline/v1/pilot/pilot-solution-simplicity.report.json index 5471ef9..8cc75c7 100644 --- a/review-suite/evals/baseline/v1/pilot/pilot-solution-simplicity.report.json +++ b/review-suite/evals/baseline/v1/pilot/pilot-solution-simplicity.report.json @@ -1,16 +1,16 @@ { "adjudication_required": [], - "attempts": 3, + "attempts": 5, "baseline_eligible": true, "configuration": { "artifacts_retained": true, "cases": 1, - "corpus_version": "1.1-pilot-solution-simplicity", + "corpus_version": "1.3-pilot-solution-simplicity", "executor": "python3 review-suite/scripts/evals/claude_executor.py", "grader_version": "1.0", "max_output_bytes": 4000000, - "runs_per_case": 3, - "suite_commit": "16560d807c66076fcbf3f00d3a87f543c6ae2458", + "runs_per_case": 5, + "suite_commit": "16b77e447dbcc844edd8f3fb58728d96826e177c", "target_skill": "review-solution-simplicity", "target_skill_dependencies": [], "target_skill_digest": "6257ee4448b15874", @@ -29,32 +29,32 @@ "spawn_failure_rate": 0.0, "timeout_rate": 0.0 }, - "graded_attempts": 3, + "graded_attempts": 5, "grader_version": "1.0", "latency": { - "count": 3, - "max_seconds": 14.16039741691202, - "mean_seconds": 13.247394027964523, - "min_seconds": 12.726900917012244, - "p50_seconds": 12.854883749969304 + "count": 5, + "max_seconds": 12.044242959003896, + "mean_seconds": 10.887090650293976, + "min_seconds": 9.784014916978776, + "p50_seconds": 10.844918083399534 }, "per_case": [ { - "attempts": 3, + "attempts": 5, "case_id": "rollback-guidance-render", "expected_root_cause_ids": [ "rc.unsupported-emitted-subcommand" ], "false_alarm_attempts": 0, - "false_clean_attempts": 3, + "false_clean_attempts": 5, "false_positive_attempts": 0, "finding_stability": 1.0, - "graded_attempts": 3, + "graded_attempts": 5, "intersection_root_cause_ids": [], "mean_recall": 0.0, - "stability_denominator": 3, + "stability_denominator": 5, "statuses": { - "review_result": 3 + "review_result": 5 }, "union_root_cause_ids": [], "verdict_stability": 1.0 @@ -64,12 +64,12 @@ "quality": { "false_alarm_denominator": 0, "false_alarm_rate": null, - "false_clean_denominator": 3, + "false_clean_denominator": 5, "false_clean_rate": 1.0, - "false_positive_denominator": 3, + "false_positive_denominator": 5, "false_positive_rate": 0.0, "material_finding_recall": 0.0, - "recall_attempts": 3, + "recall_attempts": 5, "unique_finding_contribution": [] }, "report_version": "1.0", @@ -81,20 +81,20 @@ "rollback-guidance-render": 1.0 }, "per_case_stability_denominator": { - "rollback-guidance-render": 3 + "rollback-guidance-render": 5 }, "per_case_verdict_stability": { "rollback-guidance-render": 1.0 }, - "stability_denominator": 3 + "stability_denominator": 5 }, "usage": { "available": true, - "reporting_attempts_cost_usd": 3, - "reporting_attempts_input_tokens": 3, - "reporting_attempts_output_tokens": 3, - "total_cost_usd": 0.15564775, - "total_input_tokens": 77703.0, - "total_output_tokens": 1197.0 + "reporting_attempts_cost_usd": 5, + "reporting_attempts_input_tokens": 5, + "reporting_attempts_output_tokens": 5, + "total_cost_usd": 0.201944, + "total_input_tokens": 132410.0, + "total_output_tokens": 1820.0 } } diff --git a/review-suite/evals/calibration/rollback-guidance-render.json b/review-suite/evals/calibration/rollback-guidance-render.json index bd54ae5..1967e20 100644 --- a/review-suite/evals/calibration/rollback-guidance-render.json +++ b/review-suite/evals/calibration/rollback-guidance-render.json @@ -2,34 +2,38 @@ "calibration_version": "1.0", "case_id": "rollback-guidance-render", "grader_version": "1.0", - "source": "Probes 1 and 7 are the verbatim findings returned by the unscored pilot at corpus version 1.0-pilot-orchestrator, runs 1-3, model claude-opus-4-6[1m], retained as controlled artifacts under review-suite/evals/artifacts/pilot-orchestrator/ and excluded from git. The remaining probes are constructed variants that probe one grading boundary each. No scored output was observed.", + "source": "probe.observed-root-cause and probe.accepted-non-finding are the two findings of run 1 verbatim, byte for byte, from the retained artifact review-suite/evals/artifacts/pilot-orchestrator/1.1-pilot-orchestrator/rollback-guidance-render.run-1.stdout.json, produced at corpus version 1.1-pilot-orchestrator, model claude-opus-4-6[1m], suite commit 16560d807c66076fcbf3f00d3a87f543c6ae2458. That run is the post-calibration validation run, so this prose is not prose the shipped formulations were fitted to. The formulations were drawn from an earlier run at corpus version 1.0-pilot-orchestrator whose raw output is no longer retained; CALIBRATION.md records that loss and the fix. The remaining probes are constructed variants that probe one grading boundary each. No scored output was observed.", "probes": [ { "id": "probe.observed-root-cause", "kind": "observed", - "note": "The finding the pilot actually returned on every run. Before calibration this scored `partial`, because the shipped formulations were written before any real prose existed.", + "note": "The gating finding a real reviewer returned, verbatim. Before calibration this scored `partial`, because the shipped formulations were written before any real prose existed.", "verdict": "changes_required", "findings": [ { - "id": "corr.nonexistent-export-subcommand", - "lens": "correctness", - "severity": "blocking", + "concern": "The rollback guidance emits `storectl export`, which is not a registered subcommand. Operators copying this guidance during an incident will receive a command-not-found error. The installed CLI exposes `dump`, not `export`.", "confidence": "high", - "rule": "Every emitted command exists in the installed `storectl` interface.", "evidence": [ { - "location": "storectl/guidance.py:render_rollback_guidance", - "detail": "The first emitted command is `storectl export --all > {store_root}.backup.json`." + "detail": "Emits `storectl export --all > {store_root}.backup.json`.", + "location": "storectl/guidance.py \u2014 render_rollback_guidance, line 1 of returned list" + }, + { + "detail": "`storectl/cli.py` registers exactly these subcommands: `apply`, `dump`, `import`, `info`, `migrate`, `status`. There is no `export` subcommand.", + "location": "review packet context.data" }, { - "location": "review packet context.data", - "detail": "`storectl/cli.py` registers exactly these subcommands: `apply`, `dump`, `import`, `info`, `migrate`, `status`. There is no `export` subcommand." + "detail": "AGENTS.md rule: emitted operator commands must be runnable against the installed CLI; never document a subcommand the tool does not expose.", + "location": "review packet sources.repository_instructions" } ], - "concern": "The rendered rollback guidance emits `storectl export`, which does not exist in the installed CLI. An operator copying this command into a shell during an incident will get a command-not-found error.", - "impact": "The documented rollback fails at its first step, so no backup exists before the destructive step runs.", - "proposed_change": "Emit `storectl dump` for the backup step, or render the backup step from the registered command surface.", - "expected_effect": "The rendered guidance runs as written against the installed CLI." + "expected_effect": "All emitted commands correspond to registered `storectl` subcommands, satisfying acceptance criterion 2 and the repository instruction.", + "id": "corr.nonexistent-export-subcommand", + "impact": "The rollback procedure fails on the first command, leaving the operator without a working recovery path during an incident.", + "lens": "correctness", + "proposed_change": "Replace `storectl export --all` with the corresponding registered subcommand (likely `storectl dump --all`, given the registered surface) so the emitted guidance is runnable.", + "rule": "Every emitted command exists in the installed `storectl` interface (acceptance criterion 2); repository instructions: never document a subcommand the tool does not expose.", + "severity": "blocking" } ], "expect": { @@ -125,25 +129,29 @@ "verdict": "changes_required", "findings": [ { - "id": "corr.nonexistent-export-subcommand", - "lens": "correctness", - "severity": "blocking", + "concern": "The rollback guidance emits `storectl export`, which is not a registered subcommand. Operators copying this guidance during an incident will receive a command-not-found error. The installed CLI exposes `dump`, not `export`.", "confidence": "high", - "rule": "Every emitted command exists in the installed `storectl` interface.", "evidence": [ { - "location": "storectl/guidance.py:render_rollback_guidance", - "detail": "The first emitted command is `storectl export --all > {store_root}.backup.json`." + "detail": "Emits `storectl export --all > {store_root}.backup.json`.", + "location": "storectl/guidance.py \u2014 render_rollback_guidance, line 1 of returned list" + }, + { + "detail": "`storectl/cli.py` registers exactly these subcommands: `apply`, `dump`, `import`, `info`, `migrate`, `status`. There is no `export` subcommand.", + "location": "review packet context.data" }, { - "location": "review packet context.data", - "detail": "`storectl/cli.py` registers exactly these subcommands: `apply`, `dump`, `import`, `info`, `migrate`, `status`. There is no `export` subcommand." + "detail": "AGENTS.md rule: emitted operator commands must be runnable against the installed CLI; never document a subcommand the tool does not expose.", + "location": "review packet sources.repository_instructions" } ], - "concern": "The rendered rollback guidance emits `storectl export`, which does not exist in the installed CLI. An operator copying this command into a shell during an incident will get a command-not-found error.", - "impact": "The documented rollback fails at its first step, so no backup exists before the destructive step runs.", - "proposed_change": "Emit `storectl dump` for the backup step, or render the backup step from the registered command surface.", - "expected_effect": "The rendered guidance runs as written against the installed CLI." + "expected_effect": "All emitted commands correspond to registered `storectl` subcommands, satisfying acceptance criterion 2 and the repository instruction.", + "id": "corr.nonexistent-export-subcommand", + "impact": "The rollback procedure fails on the first command, leaving the operator without a working recovery path during an incident.", + "lens": "correctness", + "proposed_change": "Replace `storectl export --all` with the corresponding registered subcommand (likely `storectl dump --all`, given the registered surface) so the emitted guidance is runnable.", + "rule": "Every emitted command exists in the installed `storectl` interface (acceptance criterion 2); repository instructions: never document a subcommand the tool does not expose.", + "severity": "blocking" }, { "id": "corr.export-not-registered-again", @@ -217,7 +225,7 @@ { "id": "probe.plausible-false-positive", "kind": "plausible_false_positive", - "note": "A gating finding about a real-sounding but unevidenced concern at an unrelated surface. This must be charged as a false positive. It is only reachable because the root cause's surface was narrowed to the renderer symbol: a surface written as a path prefix shares tokens with almost every location in the packet, and made this probe an unfalsifiable `partial`.", + "note": "A gating finding about a real-sounding but unevidenced concern at a surface sharing no token with the root cause. This is charged as a false positive, and it is the only region in which the false-positive rate is measurable at all - see the surface_token_collision probe below.", "verdict": "changes_required", "findings": [ { @@ -251,34 +259,74 @@ "adjudication_finding_ids": [] } }, + { + "id": "probe.surface-token-collision", + "kind": "surface_token_collision", + "note": "The measured limit of the false-positive boundary, kept in the calibration set so it cannot be forgotten. A surface hit needs only ONE shared normalised token, so this equally wrong gating finding - aimed at render_apply_plan, the untouched function above the change - is referred rather than charged, purely because `guidance` appears in both locations. The same is true of any wrong finding in the changed file or its test, which is where a real reviewer is most likely to point. Narrowing the surface from a path to a symbol shrank this region; it did not remove it. Requiring every surface token instead of any would, but that is grader behaviour this ticket may not change: see LIMITATIONS.md item 4.", + "verdict": "changes_required", + "findings": [ + { + "id": "corr.apply-plan-renderer-untested", + "lens": "correctness", + "severity": "blocking", + "confidence": "high", + "rule": "Every rendered operator string should be covered by a behavioural test.", + "evidence": [ + { + "location": "storectl/guidance.py:render_apply_plan", + "detail": "The untouched apply-plan renderer above the change has no assertion on the text it produces." + } + ], + "concern": "The apply-plan renderer, which this change does not touch, produces operator-facing text with no behavioural coverage.", + "impact": "An unrelated future edit to the apply plan could ship a broken plan.", + "proposed_change": "Add assertions for the apply-plan renderer's output.", + "expected_effect": "Both renderers are covered.", + "location": "storectl/guidance.py:render_apply_plan" + } + ], + "expect": { + "classifications": { + "corr.apply-plan-renderer-untested": "partial" + }, + "matched_root_cause_ids": [], + "recall": 0.0, + "false_positive_finding_ids": [], + "adjudication_finding_ids": [ + "corr.apply-plan-renderer-untested" + ] + } + }, { "id": "probe.accepted-non-finding", "kind": "accepted_non_finding", - "note": "The pilot's second observed finding. Real and worth saying, but it restates why the defect survived a green suite rather than adding a second root cause, so it is tolerated instead of counted as a false positive.", + "note": "The second finding of the same real attempt, verbatim. Real and worth saying, but it restates why the defect survived a green suite rather than adding a second root cause, so it is tolerated instead of counted as a false positive.", "verdict": "changes_required", "findings": [ { - "id": "corr.test-does-not-validate-commands", - "lens": "correctness", - "severity": "strong_recommendation", + "concern": "The test counts commands and checks they are non-empty but never verifies the subcommand tokens against the actual CLI surface. The nonexistent `export` subcommand passes this test uncaught.", "confidence": "high", - "rule": "Tests must exercise the changed behavior.", "evidence": [ { - "location": "tests/test_guidance.py:test_render_rollback_guidance_emits_three_steps", - "detail": "Asserts only `len(commands) == 3` and that each command is non-blank. It does not verify that the emitted subcommands are valid." + "detail": "Asserts only `len(commands) == 3` and that each command is non-blank. Does not inspect the subcommand token of any emitted string.", + "location": "tests/test_guidance.py \u2014 test_render_rollback_guidance_emits_three_steps" + }, + { + "detail": "\"Every emitted command exists in the installed `storectl` interface.\"", + "location": "review packet change_contract.acceptance_criteria[1]" } ], - "concern": "The test checks structural properties rather than the behavioral requirement in the change contract.", - "impact": "A guidance change can regress the runnability requirement without failing the suite.", - "proposed_change": "Assert the emitted subcommands against the registered command surface.", - "expected_effect": "The suite fails when guidance names an unregistered subcommand.", - "location": "tests/test_guidance.py" + "expected_effect": "The test catches the current `export` bug and guards against future guidance that references nonexistent subcommands.", + "id": "corr.test-does-not-verify-subcommands", + "impact": "The acceptance criterion that every emitted command is runnable is untested; future renames or typos in guidance output will not be caught by the test suite.", + "lens": "correctness", + "proposed_change": "Assert that each emitted command begins with a registered `storectl` subcommand (e.g. check that the subcommand token after `storectl` is in the set `{apply, dump, import, info, migrate, status}`), or at minimum assert the expected subcommand strings appear.", + "rule": "Tests must prove acceptance criteria \u2014 specifically that every emitted command exists in the installed CLI interface (acceptance criterion 2).", + "severity": "strong_recommendation" } ], "expect": { "classifications": { - "corr.test-does-not-validate-commands": "accepted" + "corr.test-does-not-verify-subcommands": "accepted" }, "matched_root_cause_ids": [], "recall": 0.0, diff --git a/review-suite/evals/contracts/calibration.schema.json b/review-suite/evals/contracts/calibration.schema.json index f745991..1ee08f2 100644 --- a/review-suite/evals/contracts/calibration.schema.json +++ b/review-suite/evals/contracts/calibration.schema.json @@ -31,6 +31,7 @@ "properties": { "id": {"type": "string", "pattern": "^probe\\.[a-z0-9][a-z0-9-]*$"}, "kind": { + "description": "Which grading boundary this probe exercises. The calibration test asserts a required outcome per kind, so a kind is a claim about behaviour rather than a label. `surface_token_collision` records a measured limit rather than a boundary that holds: a wrong gating finding whose location merely shares a normalised token with the root cause's surface is referred instead of charged.", "enum": [ "observed", "paraphrase", @@ -38,6 +39,7 @@ "duplicate_report", "partial_claim", "plausible_false_positive", + "surface_token_collision", "accepted_non_finding" ] }, diff --git a/review-suite/evals/contracts/corpus.schema.json b/review-suite/evals/contracts/corpus.schema.json index fe5c072..06691f9 100644 --- a/review-suite/evals/contracts/corpus.schema.json +++ b/review-suite/evals/contracts/corpus.schema.json @@ -27,7 +27,7 @@ "description": "Which baseline stratum this corpus is. A stratum is the unit of valid comparison: two corpora whose target skill, dependency closure, runtime, or ground-truth kind differ are unlike strata and must never be reported as one figure. Absent on a corpus that predates stratum labelling.", "type": "object", "additionalProperties": false, - "required": ["id", "ground_truth", "scored", "purpose"], + "required": ["id", "ground_truth", "scored", "grading_is_signal", "purpose"], "properties": { "id": {"type": "string", "pattern": "^[a-z0-9]+(-[a-z0-9]+)*$"}, "ground_truth": { @@ -43,6 +43,10 @@ "description": "True when this corpus contributes to the reported baseline. A pilot corpus is false: it exists to size the envelope and must stay disjoint from the scored cases.", "type": "boolean" }, + "grading_is_signal": { + "description": "False when this corpus's expectations were authored for a different target than the one it declares, so its graded output says nothing about reviewer quality. A lens that returns clean on a correctness root cause is behaving exactly as contracted, and grading that as a miss is a target mismatch rather than a reviewer failure. A scored stratum must be true; a corpus reusing another target's case to isolate payload size must be false, and its recall and false-clean figures must never be quoted.", + "type": "boolean" + }, "purpose": {"type": "string", "minLength": 1} } }, diff --git a/review-suite/evals/contracts/expectation.schema.json b/review-suite/evals/contracts/expectation.schema.json index 75875ee..3783422 100644 --- a/review-suite/evals/contracts/expectation.schema.json +++ b/review-suite/evals/contracts/expectation.schema.json @@ -14,10 +14,23 @@ "accepted_non_findings" ], "properties": { - "expectation_version": {"const": "1.0"}, - "case_id": {"type": "string", "pattern": "^[a-z0-9]+(-[a-z0-9]+)*$"}, - "packet_valid": {"type": "boolean"}, - "expected_verdict": {"enum": ["clean", "changes_required", "blocked"]}, + "expectation_version": { + "const": "1.0" + }, + "case_id": { + "type": "string", + "pattern": "^[a-z0-9]+(-[a-z0-9]+)*$" + }, + "packet_valid": { + "type": "boolean" + }, + "expected_verdict": { + "enum": [ + "clean", + "changes_required", + "blocked" + ] + }, "material_root_causes": { "type": "array", "items": { @@ -33,16 +46,39 @@ "equivalent_formulations" ], "properties": { - "id": {"type": "string", "pattern": "^rc\\.[a-z0-9][a-z0-9-]*$"}, - "requirement": {"type": "string", "minLength": 1}, - "trigger": {"type": "string", "minLength": 1}, - "surface": {"type": "string", "minLength": 1}, - "consequence": {"type": "string", "minLength": 1}, - "severity": {"enum": ["blocking", "strong_recommendation"]}, + "id": { + "type": "string", + "pattern": "^rc\\.[a-z0-9][a-z0-9-]*$" + }, + "requirement": { + "type": "string", + "minLength": 1 + }, + "trigger": { + "type": "string", + "minLength": 1 + }, + "surface": { + "type": "string", + "minLength": 1 + }, + "consequence": { + "type": "string", + "minLength": 1 + }, + "severity": { + "enum": [ + "blocking", + "strong_recommendation" + ] + }, "equivalent_formulations": { "type": "array", "minItems": 1, - "items": {"type": "string", "minLength": 1} + "items": { + "type": "string", + "minLength": 1 + } } } } @@ -52,17 +88,34 @@ "items": { "type": "object", "additionalProperties": false, - "required": ["id", "description", "equivalent_formulations"], + "required": [ + "id", + "description", + "equivalent_formulations" + ], "properties": { - "id": {"type": "string", "pattern": "^anf\\.[a-z0-9][a-z0-9-]*$"}, - "description": {"type": "string", "minLength": 1}, + "id": { + "type": "string", + "pattern": "^anf\\.[a-z0-9][a-z0-9-]*$" + }, + "description": { + "type": "string", + "minLength": 1 + }, "equivalent_formulations": { "type": "array", "minItems": 1, - "items": {"type": "string", "minLength": 1} + "items": { + "type": "string", + "minLength": 1 + } } } } + }, + "calibrated": { + "description": "True when every equivalent formulation here was checked against prose a real reviewer actually returned, and a calibration set under review-suite/evals/calibration/ pins that behaviour. False when the formulations were written before any run existed and have never been confronted with real prose - in which case this case's recall is a statement about the expectation, not about the reviewer. Absent on a case that predates calibration. A scored case must be true: an uncalibrated expectation reports 0.0 recall against a correct reviewer, which is measured behaviour rather than a hypothesis.", + "type": "boolean" } } } diff --git a/review-suite/evals/strata/README.md b/review-suite/evals/strata/README.md index 8848d22..a2ec5fa 100644 --- a/review-suite/evals/strata/README.md +++ b/review-suite/evals/strata/README.md @@ -49,7 +49,12 @@ only variable between them. That isolation is the point, and it has one deliberate consequence: the case and its expectation are calibrated for the orchestrator target, so the two lens strata grade as a miss and their graded output is not a quality signal. Only their payload size, latency, cost, and -protocol outcomes are. +protocol outcomes are. Those two corpora declare +`stratum.grading_is_signal: false` for exactly this reason, a test forbids a +scored stratum from declaring it false, and +[`baseline/v1/pilot/README.md`](../baseline/v1/pilot/README.md) repeats the +caveat beside the reports themselves. **Do not quote the recall or false-clean +figure from either lens pilot report.** That consequence is itself a corpus-design rule for the scored strata: **an expectation is target-specific.** A correctness root cause is not a defect a @@ -57,38 +62,62 @@ code-simplicity lens is contracted to report, so a scored simplicity stratum must carry expectations authored for its own lens rather than a shared case list. -### Measured envelope, three runs per stratum - -Suite commit `16560d807c66076fcbf3f00d3a87f543c6ae2458`, model -`claude-opus-4-6[1m]`, corpus version `1.1-pilot-*`, grader version `1.0`, -timeout 300 s, no retries, nine attempts, zero evaluation failures. - -| stratum | target | docs | digest | input tokens / attempt | cost / attempt (mean) | cost / attempt (first, cold cache) | latency mean | latency max | -| --------------------------- | ---------------------------- | ---- | ------------------ | ---------------------- | --------------------- | ---------------------------------- | ------------ | ----------- | -| `pilot-orchestrator` | `review-code-change` | 8 | `9b2805f14cdd6158` | 32,573 | 0.0959 USD | 0.1774 USD | 34.8 s | 37.2 s | -| `pilot-solution-simplicity` | `review-solution-simplicity` | 2 | `6257ee4448b15874` | 25,901 | 0.0519 USD | 0.1089 USD | 13.2 s | 14.2 s | -| `pilot-code-simplicity` | `review-code-simplicity` | 2 | `a6187d8971eaef24` | 25,691 | 0.0532 USD | 0.1110 USD | 13.4 s | 15.0 s | - -The first attempt in every stratum cost roughly three to four times the -following ones, because it pays prompt-cache creation while later attempts read -the cache. A ceiling built from the mean would be exceeded by any run whose -cache does not stay warm, which is why -[the cost-ceiling proposal](../baseline/v1/COST-CEILING-PROPOSAL.md) is built -from the cold figure. - -Total pilot spend for the recorded run: **0.6031 USD over nine attempts**, plus -**0.2857 USD over three attempts** for the earlier calibration-source run at -corpus version `1.0-pilot-orchestrator`, whose observed prose the shipped -formulations were calibrated against. **0.8889 USD total.** +### Measured envelope + +Five runs per case. Suite commit `16560d807c66076fcbf3f00d3a87f543c6ae2458`, +model `claude-opus-4-6[1m]`, corpus version `1.3-pilot-*`, grader version `1.0`, +timeout 300 s, no retries. **20 attempts, zero evaluation failures, zero +timeouts, 1.2266 USD.** + +| stratum | target | docs | digest | case | input tokens / attempt | cold cost | warm cost | latency range | +| --------------------------- | ---------------------------- | ---- | ------------------ | ---------------------------- | ---------------------- | ---------- | ----------------- | ------------- | +| `pilot-orchestrator` | `review-code-change` | 8 | `9b2805f14cdd6158` | `rollback-guidance-render` | 33,166 | 0.1719 USD | 0.0467–0.0548 USD | 24.9–36.1 s | +| `pilot-orchestrator` | `review-code-change` | 8 | `9b2805f14cdd6158` | `status-label-normalization` | 34,144 | 0.1943 USD | 0.0570–0.0678 USD | 39.4–53.1 s | +| `pilot-solution-simplicity` | `review-solution-simplicity` | 2 | `6257ee4448b15874` | `rollback-guidance-render` | 26,482 | 0.1118 USD | 0.0215–0.0237 USD | 9.8–12.0 s | +| `pilot-code-simplicity` | `review-code-simplicity` | 2 | `a6187d8971eaef24` | `rollback-guidance-render` | 26,272 | 0.1170 USD | 0.0210–0.0240 USD | 9.2–16.8 s | + +Two measurements shape the ceiling, and both are in +[the cost-ceiling proposal](../baseline/v1/COST-CEILING-PROPOSAL.md): + +- **The first attempt of a case costs three to four times the rest**, because it + pays prompt-cache creation while later attempts read the cache. A ceiling + built from the mean is exceeded by any run whose cache does not stay warm. +- **Cost tracks output volume, not packet size.** The orchestrator stratum + carries two cases specifically to measure this. The larger packet raised input + tokens by 2.9% and raised warm cost by 28% and mean latency by 45%, because + mean output grew 44% — the packet warranted more findings, and output tokens + dominate. + +Verdict stability and finding stability were 1.0 on every case at five runs, so +five runs did not surface instability on these packets. That is a statement +about these packets, not a licence to reduce the run count. + +Total pilot spend across the whole ticket: **3.3776 USD over 52 attempts** in +five batches. Only the last, the committed configuration above, is cited as the +frozen envelope; the earlier batches sized it and produced the prose the +formulations were calibrated against. + +Raw output is retained outside git at +`review-suite/evals/artifacts///`. The corpus version +is in the path deliberately: an artifact is named for its case and run number +only, so re-running a stratum into an unscoped directory silently replaced +output a committed record already cited — which happened once, and cost the +calibration-source run. The runner now refuses before launching any attempt when +a run would overwrite retained output. The exact per-stratum invocations, +artifact paths included, are recorded in +[`baseline/v1/frozen-configuration.json`](../baseline/v1/frozen-configuration.json). ## Adding a stratum 1. Create `review-suite/evals/strata//` with `corpus.json`, `reviewer/PROMPT.md`, `reviewer//packet.json`, and private `expectations/` and `provenance/` records for every declared case. -2. Declare the `stratum` block, including honest `ground_truth`. +2. Declare the `stratum` block, including honest `ground_truth` and + `grading_is_signal`. 3. Add a calibration set under `review-suite/evals/calibration/` for every - scored case; the calibration test requires one and will fail without it. + scored case; the calibration test requires one and will fail without it. It + must probe every grading boundary, and each probe must demonstrate what its + kind claims against the real grader. 4. Run `just audit-review-corpus`. It discovers every corpus here, so a new stratum is gated without editing the recipe. 5. Run `just test-review-suite`. diff --git a/review-suite/evals/strata/pilot-code-simplicity/corpus.json b/review-suite/evals/strata/pilot-code-simplicity/corpus.json index b72676a..67f3f13 100644 --- a/review-suite/evals/strata/pilot-code-simplicity/corpus.json +++ b/review-suite/evals/strata/pilot-code-simplicity/corpus.json @@ -1,5 +1,5 @@ { - "corpus_version": "1.1-pilot-code-simplicity", + "corpus_version": "1.3-pilot-code-simplicity", "protocol_version": "1.0", "grader_version": "1.0", "target_skill": "review-code-simplicity", @@ -8,7 +8,8 @@ "id": "pilot-code-simplicity", "ground_truth": "human-review", "scored": false, - "purpose": "Size the cost and latency envelope for a stratum targeting the self-sufficient code-simplicity lens. Unscored." + "grading_is_signal": false, + "purpose": "Size the cost and latency envelope for a stratum targeting the self-sufficient code-simplicity lens. Unscored. Its expectations were authored for the orchestrator target, so its graded output is a target-mismatch artefact and not a quality signal; only its payload size, latency, cost, and protocol outcomes are." }, "cases": [ "rollback-guidance-render" diff --git a/review-suite/evals/strata/pilot-code-simplicity/private/expectations/rollback-guidance-render.json b/review-suite/evals/strata/pilot-code-simplicity/private/expectations/rollback-guidance-render.json index fd268eb..c399e9d 100644 --- a/review-suite/evals/strata/pilot-code-simplicity/private/expectations/rollback-guidance-render.json +++ b/review-suite/evals/strata/pilot-code-simplicity/private/expectations/rollback-guidance-render.json @@ -2,6 +2,7 @@ "expectation_version": "1.0", "case_id": "rollback-guidance-render", "packet_valid": true, + "calibrated": true, "expected_verdict": "changes_required", "material_root_causes": [ { diff --git a/review-suite/evals/strata/pilot-code-simplicity/private/provenance/rollback-guidance-render.json b/review-suite/evals/strata/pilot-code-simplicity/private/provenance/rollback-guidance-render.json index a50bb03..201fb3a 100644 --- a/review-suite/evals/strata/pilot-code-simplicity/private/provenance/rollback-guidance-render.json +++ b/review-suite/evals/strata/pilot-code-simplicity/private/provenance/rollback-guidance-render.json @@ -4,6 +4,6 @@ "origin": "minimized_reproduction", "retention_authority": "Minimized from an adjudicated human review finding in the public repository shaug/atelier, pull request 492, review comment 2882160198, authored by the repository owner. Public source, owner-authored review, no third-party or customer material.", "sanitization": "Rewritten from scratch against a fictional `storectl` CLI. Retains only the failure shape: rendered operator guidance emits a subcommand the tool does not register, and the accompanying test asserts the rendered shape rather than command validity. Carries no business logic, no domain identifier, no customer context, no credential, and no hidden reasoning from the source repository. No source text was copied.", - "recorded_at": "2026-07-27", + "recorded_at": "2026-07-26", "notes": "Pilot case. Deliberately disjoint from every scored stratum: it exists to size the per-stratum cost and latency envelope and is never graded into the baseline. The identical case is carried by every pilot corpus so the declared skill closure is the only variable between pilot strata." } diff --git a/review-suite/evals/strata/pilot-orchestrator/corpus.json b/review-suite/evals/strata/pilot-orchestrator/corpus.json index ab257cc..8bed5c1 100644 --- a/review-suite/evals/strata/pilot-orchestrator/corpus.json +++ b/review-suite/evals/strata/pilot-orchestrator/corpus.json @@ -1,5 +1,5 @@ { - "corpus_version": "1.1-pilot-orchestrator", + "corpus_version": "1.3-pilot-orchestrator", "protocol_version": "1.0", "grader_version": "1.0", "target_skill": "review-code-change", @@ -12,9 +12,11 @@ "id": "pilot-orchestrator", "ground_truth": "human-review", "scored": false, - "purpose": "Size the cost and latency envelope for a stratum whose target is the orchestrator, whose payload therefore carries its three required lens skills. Unscored." + "grading_is_signal": true, + "purpose": "Size the cost and latency envelope for a stratum whose target is the orchestrator, whose payload therefore carries its three required lens skills. Carries two cases of materially different packet size so the fixed cost of the closure can be separated from the variable cost of the packet. Unscored." }, "cases": [ - "rollback-guidance-render" + "rollback-guidance-render", + "status-label-normalization" ] } diff --git a/review-suite/evals/strata/pilot-orchestrator/private/expectations/rollback-guidance-render.json b/review-suite/evals/strata/pilot-orchestrator/private/expectations/rollback-guidance-render.json index fd268eb..c399e9d 100644 --- a/review-suite/evals/strata/pilot-orchestrator/private/expectations/rollback-guidance-render.json +++ b/review-suite/evals/strata/pilot-orchestrator/private/expectations/rollback-guidance-render.json @@ -2,6 +2,7 @@ "expectation_version": "1.0", "case_id": "rollback-guidance-render", "packet_valid": true, + "calibrated": true, "expected_verdict": "changes_required", "material_root_causes": [ { diff --git a/review-suite/evals/strata/pilot-orchestrator/private/expectations/status-label-normalization.json b/review-suite/evals/strata/pilot-orchestrator/private/expectations/status-label-normalization.json new file mode 100644 index 0000000..4a18786 --- /dev/null +++ b/review-suite/evals/strata/pilot-orchestrator/private/expectations/status-label-normalization.json @@ -0,0 +1,64 @@ +{ + "expectation_version": "1.0", + "case_id": "status-label-normalization", + "packet_valid": true, + "calibrated": false, + "expected_verdict": "changes_required", + "material_root_causes": [ + { + "id": "rc.retained-ready-flag-on-inactive-status", + "requirement": "A record that lands on an inactive status does not carry `flag:ready`, and no consumer observes a record whose flags contradict its status.", + "trigger": "Migrating a legacy record that already carries `flag:ready` onto an inactive status such as `deferred`.", + "surface": "_normalize_flags", + "consequence": "The early return for an inactive status removes only `flag:draft`, so a pre-existing `flag:ready` survives beside an inactive status. The scheduler admits on `flag:ready` alone and runs concurrently with the migration, so a deferred record becomes runnable. Roughly 12% of the legacy set carries both flags, so this is the common shape rather than an edge case.", + "severity": "blocking", + "equivalent_formulations": [ + "`flag:ready` is not removed for an inactive status", + "the early return leaves `flag:ready` in place", + "a deferred record can keep `flag:ready`", + "the scheduler can pick up an inactive record", + "flags contradict the canonical status" + ] + }, + { + "id": "rc.tests-omit-the-migrating-shape", + "requirement": "A regression test must exercise the input shape the change is about, not only the shapes that were already correct.", + "trigger": "Reading the added tests to decide whether the migration is covered.", + "surface": "tests/test_normalize.py", + "consequence": "Every added test starts from a record that does not already carry `flag:ready`, so the inactive-status test passes for the wrong reason: the flag was never there to remove. The one shape the change exists to fix - both flags present, landing inactive - is untested, which is why a green suite says nothing about the defect above.", + "severity": "strong_recommendation", + "equivalent_formulations": [ + "no test starts from a record that already carries `flag:ready`", + "the inactive-status test passes for the wrong reason", + "the both-flags shape is untested", + "the tests never exercise a record with `flag:ready` already set" + ] + } + ], + "accepted_non_findings": [ + { + "id": "anf.unrelated-report-sort", + "description": "The `summarize` sort in registry/report.py is unrelated to the migration. Worth noting as scope drift; it is not a defect, and the packet's preserved-behaviour list keeps the tuple shape.", + "equivalent_formulations": [ + "the report sort is unrelated to this change", + "sorting rows is scope drift" + ] + }, + { + "id": "anf.no-op-scheduler-hunk", + "description": "The scheduler hunk changes nothing; its before and after lines are identical. Real, and worth pointing out as diff noise, but it is not a behavioural defect.", + "equivalent_formulations": [ + "the scheduler hunk is a no-op", + "the scheduler diff changes nothing" + ] + }, + { + "id": "anf.migration-dry-run-not-run", + "description": "The migration dry run is recorded as not run. The packet says so explicitly and it is the honest state of the evidence, so noting it is reasonable. It is a validation limitation rather than a separate root cause, and it does not make the packet incomplete.", + "equivalent_formulations": [ + "the migration dry run was not run", + "the only check that reads real legacy records is absent" + ] + } + ] +} diff --git a/review-suite/evals/strata/pilot-orchestrator/private/provenance/rollback-guidance-render.json b/review-suite/evals/strata/pilot-orchestrator/private/provenance/rollback-guidance-render.json index a50bb03..201fb3a 100644 --- a/review-suite/evals/strata/pilot-orchestrator/private/provenance/rollback-guidance-render.json +++ b/review-suite/evals/strata/pilot-orchestrator/private/provenance/rollback-guidance-render.json @@ -4,6 +4,6 @@ "origin": "minimized_reproduction", "retention_authority": "Minimized from an adjudicated human review finding in the public repository shaug/atelier, pull request 492, review comment 2882160198, authored by the repository owner. Public source, owner-authored review, no third-party or customer material.", "sanitization": "Rewritten from scratch against a fictional `storectl` CLI. Retains only the failure shape: rendered operator guidance emits a subcommand the tool does not register, and the accompanying test asserts the rendered shape rather than command validity. Carries no business logic, no domain identifier, no customer context, no credential, and no hidden reasoning from the source repository. No source text was copied.", - "recorded_at": "2026-07-27", + "recorded_at": "2026-07-26", "notes": "Pilot case. Deliberately disjoint from every scored stratum: it exists to size the per-stratum cost and latency envelope and is never graded into the baseline. The identical case is carried by every pilot corpus so the declared skill closure is the only variable between pilot strata." } diff --git a/review-suite/evals/strata/pilot-orchestrator/private/provenance/status-label-normalization.json b/review-suite/evals/strata/pilot-orchestrator/private/provenance/status-label-normalization.json new file mode 100644 index 0000000..bf8d2f6 --- /dev/null +++ b/review-suite/evals/strata/pilot-orchestrator/private/provenance/status-label-normalization.json @@ -0,0 +1,9 @@ +{ + "provenance_version": "1.0", + "case_id": "status-label-normalization", + "origin": "minimized_reproduction", + "retention_authority": "Minimized from an adjudicated human review finding in the public repository shaug/atelier, pull request 278, review comment 2861937742, authored by the repository owner and accepted in the follow-up reply 2861947492 which names the implementing commit. Public source, owner-authored review, no third-party or customer material.", + "sanitization": "Rewritten from scratch against a fictional record registry. Retains only the failure shape: a normalization step drops one legacy flag and returns early for an inactive status, leaving a second legacy flag that contradicts the canonical status, with a live consumer that admits on that flag alone. Carries no business logic, no domain identifier, no customer context, no credential, and no hidden reasoning from the source repository. No source text was copied.", + "recorded_at": "2026-07-26", + "notes": "Pilot case, never scored. Added specifically to measure how per-attempt cost and latency respond to packet size: its packet is materially larger than the other pilot case, so the two together separate the fixed cost of a stratum's skill closure from the variable cost of the packet. It is carried only by pilot-orchestrator, so the shared case remains byte-identical across all three pilot strata." +} diff --git a/review-suite/evals/strata/pilot-orchestrator/reviewer/status-label-normalization/packet.json b/review-suite/evals/strata/pilot-orchestrator/reviewer/status-label-normalization/packet.json new file mode 100644 index 0000000..9a06ced --- /dev/null +++ b/review-suite/evals/strata/pilot-orchestrator/reviewer/status-label-normalization/packet.json @@ -0,0 +1,119 @@ +{ + "schema_version": "1.0", + "repository": { + "identity": "example/registry", + "base_branch": "main" + }, + "candidate": { + "head_sha": "b8e41f0d7a2c5936e148b70ad2f96c3518ae7d04", + "comparison_base_sha": "40c9a7f21b8d635ea0f4c7912d8b56034ae1f9c7", + "diff": { + "format": "unified_diff", + "complete": true, + "content": "diff --git a/registry/normalize.py b/registry/normalize.py\n--- a/registry/normalize.py\n+++ b/registry/normalize.py\n@@ -1,10 +1,38 @@\n from registry.model import ACTIVE_STATUSES, Record\n \n+DRAFT_FLAG = \"flag:draft\"\n+READY_FLAG = \"flag:ready\"\n+\n \n def migrate(record: Record, target_status: str) -> Record:\n- \"\"\"Move a legacy record onto the canonical status field.\"\"\"\n- record.status = target_status\n- return record\n+ \"\"\"Move a legacy record onto the canonical status field.\n+\n+ Legacy records carried their runnability in flags. The canonical field is\n+ `status`; flags are retained only for display.\n+ \"\"\"\n+ record.status = target_status\n+ return _normalize_flags(record, target_status)\n+\n+\n+def _normalize_flags(record: Record, target_status: str) -> Record:\n+ flags = list(record.flags)\n+ if DRAFT_FLAG in flags:\n+ flags.remove(DRAFT_FLAG)\n+ active = target_status in ACTIVE_STATUSES\n+ if not active:\n+ record.flags = flags\n+ return record\n+ if READY_FLAG not in flags:\n+ flags.append(READY_FLAG)\n+ record.flags = flags\n+ return record\ndiff --git a/registry/scheduler.py b/registry/scheduler.py\n--- a/registry/scheduler.py\n+++ b/registry/scheduler.py\n@@ -12,7 +12,7 @@ from registry.normalize import READY_FLAG\n def runnable(record):\n \"\"\"Return whether the scheduler may pick this record up.\"\"\"\n- return READY_FLAG in record.flags\n+ return READY_FLAG in record.flags\ndiff --git a/registry/report.py b/registry/report.py\n--- a/registry/report.py\n+++ b/registry/report.py\n@@ -20,6 +20,9 @@ def summarize(records):\n rows = []\n for record in records:\n rows.append((record.identifier, record.status, sorted(record.flags)))\n+ # Display order is stable so a migration diff is readable.\n+ rows.sort()\n return rows\ndiff --git a/tests/test_normalize.py b/tests/test_normalize.py\n--- a/tests/test_normalize.py\n+++ b/tests/test_normalize.py\n@@ -8,3 +8,21 @@ def test_migrate_sets_the_canonical_status():\n assert migrate(record, \"running\").status == \"running\"\n+\n+\n+def test_migrate_drops_the_draft_flag():\n+ record = Record(identifier=\"r-1\", status=\"legacy\", flags=[\"flag:draft\"])\n+ assert \"flag:draft\" not in migrate(record, \"running\").flags\n+\n+\n+def test_migrate_adds_the_ready_flag_for_an_active_status():\n+ record = Record(identifier=\"r-2\", status=\"legacy\", flags=[])\n+ assert \"flag:ready\" in migrate(record, \"running\").flags\n+\n+\n+def test_migrate_leaves_an_inactive_record_without_the_ready_flag():\n+ record = Record(identifier=\"r-3\", status=\"legacy\", flags=[])\n+ assert \"flag:ready\" not in migrate(record, \"deferred\").flags\n" + } + }, + "change_contract": { + "goal": "Migrate legacy records onto the canonical `status` field, keeping the retained display flags consistent with the status a record lands on.", + "acceptance_criteria": [ + "`migrate` sets `status` to the target status.", + "`flag:draft` is removed from every migrated record.", + "A record that lands on an active status carries `flag:ready`.", + "A record that lands on an inactive status does not carry `flag:ready`.", + "No consumer observes a record whose flags contradict its status." + ], + "non_goals": [ + "Remove the display flags entirely; a later change owns that.", + "Change which statuses are considered active.", + "Change scheduler behaviour." + ], + "preserved_behaviors": [ + "`summarize` reports the same tuple shape for every record.", + "`runnable` keeps its current signature and its current rule." + ] + }, + "sources": { + "repository_instructions": [ + { + "label": "Migration rules", + "location": "AGENTS.md", + "summary": "A migration must not leave a record in a state no writer would produce. Retained legacy fields must agree with the canonical field after every migration path." + }, + { + "label": "Test rules", + "location": "AGENTS.md", + "summary": "A regression test must exercise the input shape the change is about, not only the shapes that were already correct." + } + ], + "named_documents": [ + { + "label": "Canonical status migration plan", + "location": "docs/status-migration.md" + }, + { + "label": "Record model", + "location": "registry/model.py" + } + ], + "nearby_patterns": [ + { + "label": "Scheduler admission rule", + "location": "registry/scheduler.py" + }, + { + "label": "Operator report", + "location": "registry/report.py" + }, + { + "label": "Normalizer tests", + "location": "tests/test_normalize.py" + }, + { + "label": "Scheduler tests", + "location": "tests/test_scheduler.py" + } + ] + }, + "validation": [ + { + "name": "normalizer tests", + "command": "pytest tests/test_normalize.py", + "scope": "focused", + "status": "passed", + "result": "4 passed" + }, + { + "name": "scheduler tests", + "command": "pytest tests/test_scheduler.py", + "scope": "focused", + "status": "passed", + "result": "9 passed" + }, + { + "name": "full tests", + "command": "pytest", + "scope": "full", + "status": "passed", + "result": "148 passed" + }, + { + "name": "migration dry run over the legacy fixture set", + "command": "python -m registry.migrate --dry-run fixtures/legacy", + "scope": "full", + "status": "unavailable", + "reason": "The legacy fixture set is not reachable from this candidate's environment, so the only check that reads real legacy records was not run." + } + ], + "context": { + "data": [ + "`ACTIVE_STATUSES` is `{\"running\", \"queued\"}`. `deferred`, `blocked`, and `closed` are inactive.", + "Legacy records exist with flags `[\"flag:draft\", \"flag:ready\"]` together, because the legacy writer set both while a record was awaiting approval.", + "Roughly 12% of the legacy fixture set carries both flags.", + "`registry/scheduler.py` is the only consumer of `flag:ready`, and this change does not modify its rule." + ], + "operational": [ + "The migration runs once, in place, over the whole record set.", + "The scheduler runs continuously against the same store during the migration window." + ] + } +} diff --git a/review-suite/evals/strata/pilot-solution-simplicity/corpus.json b/review-suite/evals/strata/pilot-solution-simplicity/corpus.json index ba1a4bd..7b7a2f0 100644 --- a/review-suite/evals/strata/pilot-solution-simplicity/corpus.json +++ b/review-suite/evals/strata/pilot-solution-simplicity/corpus.json @@ -1,5 +1,5 @@ { - "corpus_version": "1.1-pilot-solution-simplicity", + "corpus_version": "1.3-pilot-solution-simplicity", "protocol_version": "1.0", "grader_version": "1.0", "target_skill": "review-solution-simplicity", @@ -8,7 +8,8 @@ "id": "pilot-solution-simplicity", "ground_truth": "human-review", "scored": false, - "purpose": "Size the cost and latency envelope for a stratum targeting the self-sufficient solution-simplicity lens. Unscored." + "grading_is_signal": false, + "purpose": "Size the cost and latency envelope for a stratum targeting the self-sufficient solution-simplicity lens. Unscored. Its expectations were authored for the orchestrator target, so its graded output is a target-mismatch artefact and not a quality signal; only its payload size, latency, cost, and protocol outcomes are." }, "cases": [ "rollback-guidance-render" diff --git a/review-suite/evals/strata/pilot-solution-simplicity/private/expectations/rollback-guidance-render.json b/review-suite/evals/strata/pilot-solution-simplicity/private/expectations/rollback-guidance-render.json index fd268eb..c399e9d 100644 --- a/review-suite/evals/strata/pilot-solution-simplicity/private/expectations/rollback-guidance-render.json +++ b/review-suite/evals/strata/pilot-solution-simplicity/private/expectations/rollback-guidance-render.json @@ -2,6 +2,7 @@ "expectation_version": "1.0", "case_id": "rollback-guidance-render", "packet_valid": true, + "calibrated": true, "expected_verdict": "changes_required", "material_root_causes": [ { diff --git a/review-suite/evals/strata/pilot-solution-simplicity/private/provenance/rollback-guidance-render.json b/review-suite/evals/strata/pilot-solution-simplicity/private/provenance/rollback-guidance-render.json index a50bb03..201fb3a 100644 --- a/review-suite/evals/strata/pilot-solution-simplicity/private/provenance/rollback-guidance-render.json +++ b/review-suite/evals/strata/pilot-solution-simplicity/private/provenance/rollback-guidance-render.json @@ -4,6 +4,6 @@ "origin": "minimized_reproduction", "retention_authority": "Minimized from an adjudicated human review finding in the public repository shaug/atelier, pull request 492, review comment 2882160198, authored by the repository owner. Public source, owner-authored review, no third-party or customer material.", "sanitization": "Rewritten from scratch against a fictional `storectl` CLI. Retains only the failure shape: rendered operator guidance emits a subcommand the tool does not register, and the accompanying test asserts the rendered shape rather than command validity. Carries no business logic, no domain identifier, no customer context, no credential, and no hidden reasoning from the source repository. No source text was copied.", - "recorded_at": "2026-07-27", + "recorded_at": "2026-07-26", "notes": "Pilot case. Deliberately disjoint from every scored stratum: it exists to size the per-stratum cost and latency envelope and is never graded into the baseline. The identical case is carried by every pilot corpus so the declared skill closure is the only variable between pilot strata." } diff --git a/review-suite/scripts/evals/calibration.py b/review-suite/scripts/evals/calibration.py index b92d71c..e18d245 100644 --- a/review-suite/scripts/evals/calibration.py +++ b/review-suite/scripts/evals/calibration.py @@ -49,6 +49,24 @@ } ) +#: What each kind must actually demonstrate, as (classifications, must charge a +#: false positive). Coverage alone is not calibration: a set could ship a +#: `plausible_false_positive` probe whose own expectation declares `partial` with +#: no false positive charged, and pass a test that only checks the kind exists. +#: Given that a surface hit needs one shared token, that is the path of least +#: resistance for any wrong finding inside the changed file, so the required +#: outcome is asserted here rather than left to the set's own claim. +PROBE_KIND_CONTRACT: dict[str, tuple[frozenset[str], bool]] = { + "observed": (frozenset({"matched"}), False), + "paraphrase": (frozenset({"matched"}), False), + "overlapping_symptom": (frozenset({"partial"}), False), + "duplicate_report": (frozenset({"matched", "duplicate"}), False), + "partial_claim": (frozenset({"partial"}), False), + "plausible_false_positive": (frozenset({"unexpected"}), True), + "surface_token_collision": (frozenset({"partial"}), False), + "accepted_non_finding": (frozenset({"accepted"}), False), +} + class CalibrationError(ValueError): """Raised when a calibration set cannot be trusted to calibrate anything.""" diff --git a/review-suite/scripts/evals/runner.py b/review-suite/scripts/evals/runner.py index d7d0f42..3805a91 100644 --- a/review-suite/scripts/evals/runner.py +++ b/review-suite/scripts/evals/runner.py @@ -239,16 +239,50 @@ def run_attempt( return attempt, response, stdout, stderr +def _artifact_stem(case_id: str, run_number: int) -> str: + return f"{case_id}.run-{run_number}" + + def _write_artifacts( artifact_dir: Path, attempt: dict[str, Any], stdout: str, stderr: str ) -> None: - name = f"{attempt['case_id']}.run-{attempt['run_number']}" + name = _artifact_stem(attempt["case_id"], attempt["run_number"]) artifact_dir.mkdir(parents=True, exist_ok=True) (artifact_dir / f"{name}.stdout.json").write_text(stdout) if stderr: (artifact_dir / f"{name}.stderr.txt").write_text(stderr) +def refuse_to_overwrite_artifacts( + artifact_dir: Path, loaded: corpus.Corpus, runs: int +) -> None: + """Fail before launch when a run would destroy retained raw output. + + Retained executor output is a controlled artifact: a calibrated formulation + cites the raw attempt it was drawn from, and an aggregate report cites the + attempts behind its figures. The artifact name carries the case and run + number but not the corpus version, so re-running a stratum into the same + directory silently replaced the very output an earlier record cited - which + happened once, and cost a calibration source run. + + Checked here rather than at write time on purpose. Raising after the first + attempt would already have spent money, and this is the one command that can. + """ + collisions = [ + artifact_dir / f"{_artifact_stem(case.case_id, run_number)}.stdout.json" + for case in loaded.cases + for run_number in range(1, runs + 1) + ] + existing = [path for path in collisions if path.exists()] + if existing: + raise ConfigurationError( + f"{len(existing)} retained artifact(s) would be overwritten, starting " + f"with {existing[0]}. Retained output is evidence a committed record " + f"may already cite. Pass a distinct --artifact-dir, for example one " + f"scoped by corpus version ({loaded.corpus_version})." + ) + + def evaluate( command: list[str], *, @@ -310,6 +344,9 @@ def refuse_if_contaminated(case: corpus.Case, request: dict[str, Any]) -> None: for case in loaded.cases: refuse_if_contaminated(case, build(case, 1)) + if artifact_dir is not None: + refuse_to_overwrite_artifacts(artifact_dir, loaded, runs) + attempts: list[dict[str, Any]] = [] for case in loaded.cases: for run_number in range(1, runs + 1): diff --git a/review-suite/scripts/tests/test_eval_calibration.py b/review-suite/scripts/tests/test_eval_calibration.py index 6f3efbb..6adb337 100644 --- a/review-suite/scripts/tests/test_eval_calibration.py +++ b/review-suite/scripts/tests/test_eval_calibration.py @@ -79,7 +79,13 @@ def test_a_case_carried_by_several_corpora_carries_one_expectation(self): self.assertEqual(expectations[0], other) def test_every_scored_case_is_calibrated(self): - """A scored case without calibration would report an unmeasured rate.""" + """A scored case without calibration reports its expectation, not a rate. + + Measured twice, on two independent pilot cases: an uncalibrated + expectation returns recall 0.0 against a reviewer that found the defect + on every attempt. A scored stratum built on one would publish that as a + capability figure. + """ for root in corpus.corpus_roots(): loaded = corpus.load_corpus(root) if not loaded.scored: @@ -87,6 +93,16 @@ def test_every_scored_case_is_calibrated(self): for case in loaded.cases: with self.subTest(stratum=root.name, case_id=case.case_id): self.assertIn(case.case_id, self.sets) + self.assertIs(True, case.expectation.get("calibrated")) + + def test_a_calibrated_expectation_ships_a_calibration_set(self): + """`calibrated: true` is a claim the calibration set has to back.""" + for root in corpus.corpus_roots(): + for case in corpus.load_corpus(root).cases: + if not case.expectation.get("calibrated"): + continue + with self.subTest(stratum=root.name, case_id=case.case_id): + self.assertIn(case.case_id, self.sets) class CalibratedGradingTests(unittest.TestCase): @@ -106,6 +122,31 @@ def _grade(self, case_id, probe): } return grader.grade(case.expectation, calibration.probe_result(probe, identity)) + def test_every_probe_demonstrates_what_its_kind_claims(self): + """A kind is a claim about grading behaviour, not a label. + + Asserted against the real grader rather than against the calibration + set's own `expect` block, so a set cannot certify a boundary by + declaring the outcome it happens to get. + """ + for case_id, calibration_set in self.sets.items(): + for probe in calibration_set.probes: + required, must_charge = calibration.PROBE_KIND_CONTRACT[probe["kind"]] + with self.subTest(case_id=case_id, probe=probe["id"]): + graded = self._grade(case_id, probe) + observed = { + record["classification"] for record in graded["findings"] + } + self.assertEqual(required, observed) + self.assertEqual( + must_charge, bool(graded["false_positive_finding_ids"]) + ) + if probe["kind"] in {"observed", "paraphrase"}: + self.assertEqual(1.0, graded["recall"]) + if probe["kind"] == "duplicate_report": + self.assertEqual(1, len(graded["matched_root_cause_ids"])) + self.assertEqual(1, len(graded["duplicate_finding_ids"])) + def test_every_probe_receives_its_calibrated_classification(self): for case_id, calibration_set in self.sets.items(): for probe in calibration_set.probes: @@ -164,6 +205,20 @@ def test_no_shipped_stratum_claims_connector_ground_truth(self): "connector-review", (loaded.stratum or {}).get("ground_truth") ) + def test_a_scored_stratum_must_grade_its_own_target(self): + """A stratum whose expectations target another lens cannot be scored. + + Grading a contract-faithful lens against another lens's root cause is a + corpus defect, not a reviewer miss, and would invalidate the stratum. + """ + for root in corpus.corpus_roots(): + loaded = corpus.load_corpus(root) + if loaded.stratum is None: + continue + with self.subTest(stratum=root.name): + if loaded.scored: + self.assertTrue(loaded.stratum["grading_is_signal"]) + def test_no_pilot_case_is_also_a_scored_case(self): """Calibrating on a case that is also scored would fit the answer.""" pilot: set[str] = set() diff --git a/review-suite/scripts/tests/test_eval_commands.py b/review-suite/scripts/tests/test_eval_commands.py index d160801..cbbedee 100644 --- a/review-suite/scripts/tests/test_eval_commands.py +++ b/review-suite/scripts/tests/test_eval_commands.py @@ -23,11 +23,16 @@ EVAL_COMMAND = "eval-review-suite" -#: `name param1 param2: dep1 dep2` - parameters precede the colon, dependencies -#: follow it. Keeping the three apart matters: deriving dependencies by -#: re-splitting a joined string silently yields nothing for `test: test-plugins`, -#: which would make the paid-path guard below inert. -RECIPE_HEADER = re.compile(r"^([a-z][a-z0-9-]*)((?:\s+[a-z0-9-]+)*)\s*:(.*)$") +#: `name param1 *variadic: dep1 dep2` - parameters precede the colon, +#: dependencies follow it. Keeping the three apart matters: deriving dependencies +#: by re-splitting a joined string silently yields nothing for +#: `test: test-plugins`, which would make the paid-path guard below inert. +#: +#: A parameter may carry just's `*` or `+` variadic prefix. Without that, a +#: variadic recipe matches nothing at all and every assertion about it becomes a +#: `KeyError` rather than a failure - including the paid-path guard, which would +#: stop seeing the one recipe that can spend money. +RECIPE_HEADER = re.compile(r"^([a-z][a-z0-9-]*)((?:\s+[*+]?[a-z0-9-]+)*)\s*:(.*)$") def recipes(text: str) -> dict[str, dict[str, object]]: @@ -63,9 +68,19 @@ def test_all_three_recipes_exist_under_their_exact_names(self): for name in (TEST_COMMAND, AUDIT_COMMAND, EVAL_COMMAND): self.assertIn(name, self.recipes) - def test_the_evaluation_recipe_takes_one_executor_argument(self): - self.assertEqual(["executor"], self.recipes[EVAL_COMMAND]["parameters"]) + def test_the_evaluation_recipe_takes_an_executor_and_forwards_the_rest(self): + """The executor stays required; runner options must reach the runner. + + A recipe taking only an executor cannot execute a frozen per-stratum + configuration at all: `--corpus` defaults to the protocol-proof corpus + and `--runs` to 1, so the documented command would silently evaluate the + wrong corpus once. + """ + self.assertEqual( + ["executor", "*args"], self.recipes[EVAL_COMMAND]["parameters"] + ) self.assertIn('--executor "{{executor}}"', self.recipes[EVAL_COMMAND]["body"]) + self.assertIn("{{args}}", self.recipes[EVAL_COMMAND]["body"]) def test_the_deterministic_recipes_take_no_argument(self): for name in (TEST_COMMAND, AUDIT_COMMAND): diff --git a/review-suite/scripts/tests/test_eval_runner.py b/review-suite/scripts/tests/test_eval_runner.py index 7508f44..716c291 100644 --- a/review-suite/scripts/tests/test_eval_runner.py +++ b/review-suite/scripts/tests/test_eval_runner.py @@ -358,6 +358,24 @@ def test_artifacts_are_written_only_when_requested(self): self.evaluate(artifact_dir=self.temp / "artifacts") self.assertTrue(any((self.temp / "artifacts").iterdir())) + def test_a_run_refuses_to_overwrite_retained_artifacts(self): + """Retained output is evidence a committed record may already cite. + + An artifact is named for its case and run number only, so re-running a + stratum into the same directory replaces it. Refused before any attempt + launches, because raising afterwards would already have spent money on + the one command that can. + """ + artifacts = self.temp / "artifacts" + self.evaluate(artifact_dir=artifacts) + retained = {path: path.read_text() for path in sorted(artifacts.iterdir())} + with self.assertRaises(runner.ConfigurationError) as raised: + self.evaluate(artifact_dir=artifacts) + self.assertIn("would be overwritten", str(raised.exception)) + self.assertEqual( + retained, {path: path.read_text() for path in sorted(artifacts.iterdir())} + ) + class CommandLineTests(unittest.TestCase): def setUp(self): From 2ae0d23c18f247f49d3cc5e76f26d1cf9610c83e Mon Sep 17 00:00:00 2001 From: Scott Haug Date: Sun, 26 Jul 2026 21:42:29 -0700 Subject: [PATCH 3/5] feat: report the stratum a run evaluated MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Summary - Carry the corpus's declared `stratum` verbatim into the aggregate report's `configuration`, alongside the distinct executor model identities the run actually observed. - Assert the overwrite refusal by its launch log rather than only by its exception, and assert that a report names its stratum. ## Why The limitations record claimed every stratum is labelled "in `corpus.json` and in every report". No report carried a stratum id, its ground truth, whether it is scored, or whether its grading is a signal at all — the runner held `loaded.stratum` and never copied it. So the epic's most emphatic prohibition, that a human-review figure must never be reported as a connector figure, was maintained only as prose in other files, and a report quoted on its own silently lost it. Model identity had the same shape of gap: a stratum that cannot name the model that answered is not comparable with any other. The overwrite-refusal test asserted the exception, its message, and byte-identity. All three would still pass if the check moved into `_write_artifacts` — which would bill a paid attempt before refusing, defeating the entire point of the guard. The launch log is what distinguishes the pre-launch guard from the one that is too late to help, and the sibling contamination tests already use that idiom. ## Sequencing note Committed before the pilot is re-run, deliberately. A run records the commit its tree was at, so running from an uncommitted tree produces a report attributing itself to a commit that cannot reproduce it. This commit is the tree the recorded pilot runs from. --- CHANGELOG.md | 4 ++ review-suite/scripts/evals/runner.py | 17 ++++++++ .../scripts/tests/test_eval_runner.py | 39 +++++++++++++++++-- 3 files changed, 56 insertions(+), 4 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 6891809..278cfb3 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,7 +6,11 @@ summary: Chronological history of repository and skill changes. ## 2026-07-26 — Added the replay evaluator, then froze the v1 baseline configuration +- feat: report the stratum a run evaluated +- fix: make the frozen baseline record auditable, and measure the envelope + (`f7787dcba681db1de079f57ce1f2f2941e0923b2`) - feat: add baseline strata, grader calibration, and the frozen v1 record + (`16b77e447dbcc844edd8f3fb58728d96826e177c`) - fix: skip the recipe-execution tests when `just` is absent (`f544aa0c19d97dd4f1aabd7dfab3df08b2ee6a6b`) - feat: record the evaluated skill closure with every run diff --git a/review-suite/scripts/evals/runner.py b/review-suite/scripts/evals/runner.py index 3805a91..38b5c0b 100644 --- a/review-suite/scripts/evals/runner.py +++ b/review-suite/scripts/evals/runner.py @@ -385,6 +385,23 @@ def refuse_if_contaminated(case: corpus.Case, request: dict[str, Any]) -> None: "target_skill_documents": sorted(closure), "target_skill_digest": protocol.prompt_digest(skill_prompt), "suite_commit": commit, + # The stratum the corpus declares, carried verbatim. A report that names + # its target and closure but not its stratum cannot say which ground + # truth its expectations came from, whether it is scored, or whether its + # grading is a signal at all - so those properties would survive only as + # prose in some other file, and a report quoted on its own would silently + # lose them. `None` on a corpus that predates stratum labelling. + "stratum": loaded.stratum, + # Derived from the attempts rather than declared, because a stratum + # without a model identity is not comparable with any other. More than + # one model in this list means the run spans strata. + "executor_models": sorted( + { + model + for attempt in attempts + if (model := (attempt.get("executor") or {}).get("model")) + } + ), "runs_per_case": runs, "cases": len(loaded.cases), "timeout_seconds": timeout, diff --git a/review-suite/scripts/tests/test_eval_runner.py b/review-suite/scripts/tests/test_eval_runner.py index 716c291..1acf6f6 100644 --- a/review-suite/scripts/tests/test_eval_runner.py +++ b/review-suite/scripts/tests/test_eval_runner.py @@ -53,6 +53,11 @@ def evaluate(self, mode=None, **kwargs): return runner.evaluate(shlex.split(fixture_command(mode)), **options) + @staticmethod + def stratum_root() -> Path: + """A shipped corpus that declares a stratum, for configuration checks.""" + return corpus.STRATA_ROOT / "pilot-code-simplicity" + def test_a_full_pass_produces_one_attempt_per_case_and_run(self): attempts, configuration = self.evaluate(runs=2) cases = len(corpus.load_corpus().cases) @@ -362,20 +367,46 @@ def test_a_run_refuses_to_overwrite_retained_artifacts(self): """Retained output is evidence a committed record may already cite. An artifact is named for its case and run number only, so re-running a - stratum into the same directory replaces it. Refused before any attempt - launches, because raising afterwards would already have spent money on - the one command that can. + stratum into the same directory replaces it. + + The launch log is the assertion that matters. Refusing at write time + would raise the same error, preserve the same bytes, and still bill a + paid attempt first, so checking only the exception would not tell the + pre-launch guard apart from the one that is too late to help. """ + import shlex + artifacts = self.temp / "artifacts" self.evaluate(artifact_dir=artifacts) retained = {path: path.read_text() for path in sorted(artifacts.iterdir())} + + command, log = self._counting_executor() with self.assertRaises(runner.ConfigurationError) as raised: - self.evaluate(artifact_dir=artifacts) + runner.evaluate( + shlex.split(command), + corpus_root=None, + runs=1, + timeout=60.0, + max_output_bytes=runner.DEFAULT_MAX_OUTPUT_BYTES, + artifact_dir=artifacts, + ) self.assertIn("would be overwritten", str(raised.exception)) + self.assertFalse( + log.exists(), "an executor was launched before the overwrite was refused" + ) self.assertEqual( retained, {path: path.read_text() for path in sorted(artifacts.iterdir())} ) + def test_the_report_configuration_carries_the_declared_stratum(self): + """A report that cannot name its stratum cannot be compared with one.""" + _, configuration = self.evaluate(corpus_root=self.stratum_root()) + stratum = configuration["stratum"] + self.assertEqual("pilot-code-simplicity", stratum["id"]) + self.assertEqual("human-review", stratum["ground_truth"]) + self.assertIs(False, stratum["scored"]) + self.assertIs(False, stratum["grading_is_signal"]) + class CommandLineTests(unittest.TestCase): def setUp(self): From 3a8388d42e355e4bc9731b98b6dcd42ffd13ff2f Mon Sep 17 00:00:00 2001 From: Scott Haug Date: Sun, 26 Jul 2026 21:58:32 -0700 Subject: [PATCH 4/5] fix: attribute the pilot to a reproducible commit and correct the records MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Closes the remaining review findings against the frozen deliverable. Every figure in every record now resolves to a committed report or a retained artifact. ## The pilot batch was attributed to a commit that could not run it Three prose records credited the frozen pilot to the base commit, while the reports recorded the branch's first commit — and neither could reproduce the batch, because the corpora were uncommitted working-tree state at run time. One key was carrying two different meanings across six hand-copies. - The pilot was re-run from the committed tree at `2ae0d23`, so the reports now record a commit whose tree contains the corpora they evaluated. Later commits on this branch change records only and leave every payload byte-identical. - The frozen record's single `suite_commit` is replaced by two explicit fields: `v1_review_behaviour_commit`, the pre-v2 commit whose review behaviour a scored baseline must evaluate, and `pilot_suite_commit`, what the pilot actually ran from. The note on the first says what is really load-bearing — this candidate touches no file under `skills/`, so the evaluated closure text is identical at every commit on this branch, and the closure digest moves when that text moves while a commit can move without it. - The superseded batches are now cited by corpus version and closure digest rather than by a commit, because no commit reproduces them, and the calibration record says so. ## Per-attempt figures now have a retained source The pilot invocations omitted `--attempts-out`, so four per-case latency ranges and the output-volume conclusion had no retained source. Per-attempt records are now retained beside the raw output, and both paths carry the commit as well as the corpus version. ## Records corrected - `controlled_artifacts` omitted the version scoping that the same file prescribes as the remedy for the overwrite it describes. - The adjudication plan said the corpus "contains" the CI-adjudicated case; the sourcing record only *identifies* it as batch-2 ground truth. - `evals/README.md` built a 0.0-to-1.0 recall comparison whose "before" was #50's smoke figure, which this ticket's non-goals forbid citing as any kind of signal. Re-anchored on this ticket's own two-case measurement inside one batch, which is a legitimate anchor and a sharper one. - The stratum-label claim is now true rather than aspirational, and names the report field that carries it. ## Re-measured envelope Suite commit `2ae0d23`, corpus `1.3-pilot-*`, 20 attempts, zero evaluation failures, zero timeouts, **1.2536 USD**. Both pilot measurements held on fresh data: the first attempt of a case still costs three to four times the rest, and the larger packet still raised input tokens 3.0% while raising cache-read cost 21% and cache-read latency 22%, because cache-read output grew 29%. Worst observed cold cost per attempt moved, so the per-stratum ceiling proposal is rebuilt on it: 9.00 USD for the orchestrator-targeted stratum, 3.00 USD for each lens stratum, 15.00 USD overall, against an expected spend of 5.08 USD. Total pilot spend across this ticket is 4.6312 USD over 72 attempts, reported in full. --- CHANGELOG.md | 2 + review-suite/evals/README.md | 17 ++++- .../evals/baseline/v1/ADJUDICATION-PLAN.md | 18 ++--- review-suite/evals/baseline/v1/CALIBRATION.md | 16 +++-- .../baseline/v1/COST-CEILING-PROPOSAL.md | 70 +++++++++++-------- review-suite/evals/baseline/v1/LIMITATIONS.md | 17 +++-- review-suite/evals/baseline/v1/SOURCING.md | 20 +++--- .../baseline/v1/frozen-configuration.json | 61 +++++++++------- .../evals/baseline/v1/pilot/README.md | 18 +++-- .../pilot/pilot-code-simplicity.report.json | 26 ++++--- .../v1/pilot/pilot-orchestrator.report.json | 52 +++++++++----- .../pilot-solution-simplicity.report.json | 26 ++++--- review-suite/evals/strata/README.md | 37 +++++----- 13 files changed, 234 insertions(+), 146 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 278cfb3..e14f8dc 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,7 +6,9 @@ summary: Chronological history of repository and skill changes. ## 2026-07-26 — Added the replay evaluator, then froze the v1 baseline configuration +- fix: attribute the pilot to a reproducible commit and correct the records - feat: report the stratum a run evaluated + (`2ae0d23c18f247f49d3cc5e76f26d1cf9610c83e`) - fix: make the frozen baseline record auditable, and measure the envelope (`f7787dcba681db1de079f57ce1f2f2941e0923b2`) - feat: add baseline strata, grader calibration, and the frozen v1 record diff --git a/review-suite/evals/README.md b/review-suite/evals/README.md index 1fd703d..d9c04bc 100644 --- a/review-suite/evals/README.md +++ b/review-suite/evals/README.md @@ -361,12 +361,21 @@ found and what it cost to fix. shares a token with almost every location in a packet, and made a deliberately wrong gating finding an unfalsifiable partial. Write a surface as the smallest identifying symbol. + - The shipped formulations were written before any real run and were not tuned afterwards, which is why recall is 0.0 above. That is the conservative behaviour this interface is meant to have, and it means grader calibration is - required before any recall number means anything. Calibrated against the - pilot's observed prose, the same interface scored recall 1.0 with zero false - positives on three fresh runs. + required before any recall number means anything. + + The scale of it has since been measured, on this ticket's own pilot rather + than against the void smoke figure above, which cannot anchor a comparison. + Two cases in one 20-attempt batch, differing only in whether their + formulations had ever been confronted with real prose: the calibrated case + scored recall 1.0 over five attempts with zero false positives and zero + referrals, and the deliberately uncalibrated one scored recall 0.0 over five + attempts with ten referrals, while the reviewer gated the change every time in + both. An uncalibrated expectation reports a number about itself. + - Completeness of the evaluated skill closure is load-bearing, not incidental. Earlier revisions omitted first the orchestration protocol and then the three lens skills that `review-code-change` requires; each omission changed observed @@ -374,10 +383,12 @@ found and what it cost to fix. carries is a change to what is being measured, and starts a new stratum. Every run records its closure's membership and digest so a stratum can state what it evaluated. + - The choice of scored target, the strata to compare, and the cost envelope to preregister all follow from that closure and are deliberately left open. Size the envelope from a preregistered run of the chosen closure; do not extrapolate it from the protocol smoke run above. + - `expectation.schema.json` requires a `severity` on every root cause that no metric currently consumes. Either score severity agreement or drop the requirement; do not assume it is being measured. diff --git a/review-suite/evals/baseline/v1/ADJUDICATION-PLAN.md b/review-suite/evals/baseline/v1/ADJUDICATION-PLAN.md index e549c0e..78a2314 100644 --- a/review-suite/evals/baseline/v1/ADJUDICATION-PLAN.md +++ b/review-suite/evals/baseline/v1/ADJUDICATION-PLAN.md @@ -38,7 +38,7 @@ pilot tested that twice on two independent cases: nothing. Recall 0.0 while the reviewer found the defect on every attempt. After calibration against observed prose, recall 1.0 over five fresh attempts. - `status-label-normalization`: left deliberately uncalibrated as a control. - Recall **0.0** over five attempts with **nine adjudication referrals**, while + Recall **0.0** over five attempts with **ten adjudication referrals**, while verdict stability was 1.0 and the reviewer gated the change every time. So a source thread cannot supply adjudication C, and a source-derived @@ -111,13 +111,13 @@ Three things do make an acceptable second adjudication for A: available here and is stronger than any opinion, human or model. Where a minimized reproduction can be made to *run*, materiality stops being a judgment: the defect either reproduces as a failing check or it does not. The - corpus already contains a case adjudicated exactly this way — `f544aa0` in - this repository survived an aggregate `clean` review verdict and was then - caught by CI, so its second adjudication is a machine, with full provenance - and no retention question. Both pilot cases are mechanically checkable in - principle: an emitted subcommand can be compared against a registered command - surface, and a flag contradicting a canonical status can be asserted - directly. + [SOURCING.md](SOURCING.md) already identifies a candidate adjudicated exactly + this way — `f544aa0` in this repository survived an aggregate `clean` review + verdict and was then caught by CI, so its second adjudication is a machine, + with full provenance and no retention question. Both pilot cases are + mechanically checkable in principle: an emitted subcommand can be compared + against a registered command surface, and a flag contradicting a canonical + status can be asserted directly. 3. **A blind context in a different model family from the evaluated stratum**, with the correlation limitation recorded. Weaker than the first two, materially better than same-family, and never sufficient on its own for a @@ -166,7 +166,7 @@ re-verified at the source. | atelier PR 350, two recovery paths | Root cause should agree; **severity will not.** `blocking` versus `strong_recommendation` is not recoverable from the thread. | | atelier PR 410, duplicated client concepts | **Lens assignment, not materiality.** Whole-solution over-engineering or local reuse? That decision picks the stratum, so a disagreement here moves the case rather than resolving it. | | All four simplicity near-miss controls (PR 417, 277, 630, 443) | **Highest expected disagreement rate.** "Requirement-justified" is a judgment with no oracle. PR 443 — per-item bullets chosen over a table for formatter stability — may not be a simplicity question at all and should probably be replaced. | -| `status-label-normalization`, this corpus's pilot case | Two root causes were authored by one context and deliberately left uncalibrated. Over five attempts the reviewer gated the change every time and the grader matched neither root cause, producing nine referrals. Whether the authored root cause or the reviewer's actual finding is the material one **is** the adjudication question, and the referrals are the evidence. | +| `status-label-normalization`, this corpus's pilot case | Two root causes were authored by one context and deliberately left uncalibrated. Over five attempts the reviewer gated the change every time and the grader matched neither root cause, producing ten referrals. Whether the authored root cause or the reviewer's actual finding is the material one **is** the adjudication question, and the referrals are the evidence. | ### Ambiguous source disposition — verify or drop before use diff --git a/review-suite/evals/baseline/v1/CALIBRATION.md b/review-suite/evals/baseline/v1/CALIBRATION.md index f11c846..60ea2a2 100644 --- a/review-suite/evals/baseline/v1/CALIBRATION.md +++ b/review-suite/evals/baseline/v1/CALIBRATION.md @@ -26,8 +26,9 @@ Case `rollback-guidance-render`, carried identically by all three pilot strata. **Source run:** three attempts at corpus version `1.0-pilot-orchestrator`, target `review-code-change`, closure digest `9b2805f14cdd6158`, model -`claude-opus-4-6[1m]`, suite commit `16560d807c66076fcbf3f00d3a87f543c6ae2458`. -Spend: 0.2857 USD. Per-attempt latency 52.7 s, 30.4 s, 39.7 s. +`claude-opus-4-6[1m]`. Spend: 0.2857 USD. Per-attempt latency 52.7 s, 30.4 s, +39.7 s. It ran from an uncommitted tree, so it is cited by its corpus version +and closure digest rather than by a commit: no commit reproduces it. > **The source run's raw output is no longer retained.** The runner names an > artifact `.run-.stdout.json` with no corpus-version component, so the @@ -99,9 +100,10 @@ raises the point. The calibrated formulations were re-measured after calibration, on prose they were not fitted to. At the frozen configuration — five runs per case, corpus -version `1.3-pilot-*`, raw output retained at -`review-suite/evals/artifacts/pilot-orchestrator/1.3-pilot-orchestrator/` — -`rollback-guidance-render` scored **recall 1.0, false-positive rate 0.0, zero +version `1.3-pilot-*`, suite commit `2ae0d23c18f247f49d3cc5e76f26d1cf9610c83e`, +raw output retained at +`review-suite/evals/artifacts/pilot-orchestrator/2ae0d23-1.3-pilot-orchestrator/` +— `rollback-guidance-render` scored **recall 1.0, false-positive rate 0.0, zero adjudication referrals, and verdict stability 1.0 over five attempts.** The calibration generalised beyond the exact sentences it was drawn from. It is not proven to generalise indefinitely; see limitation 3 in @@ -115,7 +117,7 @@ uncalibrated**, as a control: | case | `calibrated` | recall | verdict stability | referrals | reviewer verdict | | ---------------------------- | ------------ | ------- | ----------------- | --------- | ---------------------- | | `rollback-guidance-render` | `true` | **1.0** | 1.0 | 0 | `changes_required` × 5 | -| `status-label-normalization` | `false` | **0.0** | 1.0 | 9 | `changes_required` × 5 | +| `status-label-normalization` | `false` | **0.0** | 1.0 | 10 | `changes_required` × 5 | The uncalibrated case's reviewer gated the change on every attempt, stably, and the grader matched neither root cause. Two cases, ten attempts, one conclusion: @@ -125,7 +127,7 @@ reviewer.** That is why the flag exists. `expectation.calibrated` is machine-readable, and a test refuses a scored case that is not calibrated or that ships no calibration set. The uncalibrated case is retained deliberately as the measured control, and -its nine referrals are also the evidence for the adjudication disagreement +its ten referrals are also the evidence for the adjudication disagreement recorded against it in [ADJUDICATION-PLAN.md](ADJUDICATION-PLAN.md). ## Calibration cases and how they are enforced diff --git a/review-suite/evals/baseline/v1/COST-CEILING-PROPOSAL.md b/review-suite/evals/baseline/v1/COST-CEILING-PROPOSAL.md index cf10fef..e1e5fe0 100644 --- a/review-suite/evals/baseline/v1/COST-CEILING-PROPOSAL.md +++ b/review-suite/evals/baseline/v1/COST-CEILING-PROPOSAL.md @@ -17,17 +17,23 @@ stratum, at the frozen configuration. ## Measured pilot input -Five runs per case at the frozen configuration: suite commit -`16560d807c66076fcbf3f00d3a87f543c6ae2458`, model `claude-opus-4-6[1m]`, corpus -version `1.3-pilot-*`, timeout 300 s, no retries, one fresh process per attempt. -**20 attempts, zero evaluation failures, zero timeouts, 1.2266 USD.** +Five runs per case, run from the committed tree at suite commit +`2ae0d23c18f247f49d3cc5e76f26d1cf9610c83e` — the commit every report records, +and the one the batch is reproducible from. Model `claude-opus-4-6[1m]`, corpus +version `1.3-pilot-*`, grader version `1.0`, timeout 300 s, no retries, one +fresh process per attempt. **20 attempts, zero evaluation failures, zero +timeouts, 1.2536 USD.** | stratum | closure docs | case | input tokens / attempt | cost, first attempt (cold cache) | cost, later attempts (cache read) | latency range | | --------------------------- | ------------ | ---------------------------- | ---------------------- | -------------------------------- | --------------------------------- | ------------- | -| `pilot-orchestrator` | 8 | `rollback-guidance-render` | 33,166 | 0.1719 USD | 0.0467 – 0.0548 USD | 24.9 – 36.1 s | -| `pilot-orchestrator` | 8 | `status-label-normalization` | 34,144 | 0.1943 USD | 0.0570 – 0.0678 USD | 39.4 – 53.1 s | -| `pilot-solution-simplicity` | 2 | `rollback-guidance-render` | 26,482 | 0.1118 USD | 0.0215 – 0.0237 USD | 9.8 – 12.0 s | -| `pilot-code-simplicity` | 2 | `rollback-guidance-render` | 26,272 | 0.1170 USD | 0.0210 – 0.0240 USD | 9.2 – 16.8 s | +| `pilot-orchestrator` | 8 | `rollback-guidance-render` | 32,464 | 0.1698 USD | 0.0519 – 0.0532 USD | 31.8 – 34.9 s | +| `pilot-orchestrator` | 8 | `status-label-normalization` | 33,442 | 0.2021 USD | 0.0589 – 0.0684 USD | 37.7 – 51.2 s | +| `pilot-solution-simplicity` | 2 | `rollback-guidance-render` | 25,800 | 0.1094 USD | 0.0216 – 0.0238 USD | 10.4 – 14.2 s | +| `pilot-code-simplicity` | 2 | `rollback-guidance-render` | 25,617 | 0.1116 USD | 0.0245 – 0.0303 USD | 12.7 – 22.9 s | + +Every per-attempt figure above is re-derivable from the retained per-attempt +records at +`review-suite/evals/artifacts//2ae0d23-1.3-.attempts.jsonl`. ## Two things the pilot measured that change how a ceiling must be built @@ -45,14 +51,15 @@ hypothetical one. ### 2. Cost tracks output volume, not packet size -This is the non-obvious one, and it is why the corpus carries two pilot cases of -different size in the orchestrator stratum. +This is the non-obvious one, and it is why the orchestrator stratum carries two +pilot cases of materially different size. -The larger packet raised input tokens by only 978, **2.9%**. It raised -cache-read cost by **28%** and mean latency by **45%**. The reason is visible in -the attempt records: mean output grew from 1,262 to 1,816 tokens, **44%**, -because the larger packet warranted more findings, and output tokens are far -more expensive per token than cached input. +Comparing the two cases inside the same run batch, so the runtime environment is +constant: the larger packet raised input tokens by 978, **3.0%**. It raised mean +cache-read cost by **21%** and mean cache-read latency by **22%**, because mean +cache-read output grew from 1,458 to 1,880 tokens, **29%** — the larger packet +warranted more findings, and output tokens are far more expensive per token than +cached input. So per-attempt cost scales with *how much a reviewer has to say*, which tracks the number of material findings a case warrants — not with how big the packet @@ -61,11 +68,11 @@ upper end of the observed range; its clean controls will sit at the lower end. The proposal therefore uses the **worse** of the two measured cases for the orchestrator stratum. -Both figures were measured inside one run batch, so the runtime environment is -constant between them. Absolute input-token counts drifted slightly across -batches (32,507 → 32,955 → 33,166 for the same case), because reported input -includes runtime-side prompt overhead that the payload does not control; treat -the within-batch difference as the reliable measurement, not the absolute count. +Absolute input-token counts drift slightly across batches — 32,507, 32,955, +33,166, then 32,464 for the same case — because reported input includes +runtime-side prompt overhead that the payload does not control. Treat the +within-batch difference as the measurement and the absolute count as +approximate. ## Proposal @@ -74,13 +81,13 @@ orchestrator-targeted correctness stratum, 4 in each lens stratum. | stratum | cases | runs | attempts | worst cold cost / attempt | all-cold worst case | +15% headroom | **proposed ceiling** | expected spend | | ----------------------------- | ----- | ---- | -------- | ------------------------- | ------------------- | ------------- | -------------------- | -------------- | -| `s1-correctness-orchestrator` | 7 | 5 | 35 | 0.1943 USD | 6.80 USD | 7.82 USD | **8.00 USD** | 3.26 USD | -| `s2-solution-simplicity-lens` | 4 | 5 | 20 | 0.1118 USD | 2.24 USD | 2.57 USD | **3.00 USD** | 0.83 USD | -| `s3-code-simplicity-lens` | 4 | 5 | 20 | 0.1170 USD | 2.34 USD | 2.69 USD | **3.00 USD** | 0.85 USD | -| all three | 15 | 5 | 75 | | 11.38 USD | 13.08 USD | **14.00 USD** | 4.94 USD | +| `s1-correctness-orchestrator` | 7 | 5 | 35 | 0.2021 USD | 7.07 USD | 8.13 USD | **9.00 USD** | 3.33 USD | +| `s2-solution-simplicity-lens` | 4 | 5 | 20 | 0.1094 USD | 2.19 USD | 2.52 USD | **3.00 USD** | 0.82 USD | +| `s3-code-simplicity-lens` | 4 | 5 | 20 | 0.1116 USD | 2.23 USD | 2.57 USD | **3.00 USD** | 0.93 USD | +| all three | 15 | 5 | 75 | | 11.49 USD | 13.22 USD | **15.00 USD** | 5.08 USD | **Expected spend** assumes the cache behaves as measured — one cache-creation -attempt per case, the rest cache reads — using the worse observed warm figure +attempt per case, the rest cache reads — using the worst observed warm figure per stratum. It is the number to expect. The ceiling is the number to preregister. They differ by roughly a factor of three, and that gap is the measured cost of not being able to rely on cache warmth. @@ -109,12 +116,13 @@ about the model rather than a measurement. - The scored run is the only paid step. `just test`, `just lint`, `just check`, and `just audit-review-corpus` never launch a runtime. -- **Pilot spend across this ticket: 3.3776 USD over 52 attempts**, in five - batches — 0.2857 (3 attempts, calibration source), 0.6031 (9), 0.8496 (10), - 0.4126 (10), and 1.2266 (20, the committed configuration). The earlier batches - sized the envelope, produced the prose the formulations were calibrated - against, and were superseded as the corpus was corrected; only the last is - cited as the frozen envelope. +- **Pilot spend across this ticket: 4.6312 USD over 72 attempts**, in six + batches — 0.2857 (3 attempts, the calibration source), 0.6031 (9), 0.8496 + (10), 0.4126 (10), 1.2266 (20), and 1.2536 (20, the committed configuration). + The earlier batches sized the envelope, produced the prose the formulations + were calibrated against, and were superseded as the corpus and the records + were corrected; only the last is cited as the frozen envelope, and only it ran + from a committed tree. - Re-running a stratum after a formulation change is not free, so calibration is done against pilot output only. A scored stratum should need no re-run for grading reasons. diff --git a/review-suite/evals/baseline/v1/LIMITATIONS.md b/review-suite/evals/baseline/v1/LIMITATIONS.md index 2c33c8e..60b3c7f 100644 --- a/review-suite/evals/baseline/v1/LIMITATIONS.md +++ b/review-suite/evals/baseline/v1/LIMITATIONS.md @@ -23,7 +23,9 @@ Ground truth therefore comes from real adjudicated **human** review and from this suite's own delivery history. Consequences: - every stratum is labelled `human-review` or `repository-history`, never - `connector`, in `corpus.json` and in every report; + `connector`, in `corpus.json` and in every report's `configuration.stratum`, + which the runner copies verbatim from the corpus so a report quoted on its own + still states its ground truth; - a test asserts no shipped corpus claims `connector-review` ground truth, so the label cannot drift by accident; and - **a human-review figure must never be reported as a connector-escape figure, @@ -202,13 +204,14 @@ caching behaviour as much as about the work done. Two further measurements bear on any cost figure quoted from here: - **Cost tracks output volume, not packet size.** The orchestrator stratum's - larger packet raised input tokens 2.9% and raised cache-read cost 28% and mean - latency 45%, because mean output grew 44%. A case's cost therefore depends on - how much a reviewer has to say about it, which tracks the number of findings - it warrants. A stratum's cost is not predictable from its packet sizes alone. + larger packet raised input tokens 3.0% and raised cache-read cost 21% and + cache-read latency 22%, because cache-read output grew 29%. A case's cost + therefore depends on how much a reviewer has to say about it, which tracks the + number of findings it warrants. A stratum's cost is not predictable from its + packet sizes alone. - **Reported input tokens include runtime-side prompt overhead.** The same - case's reported input drifted across batches — 32,507, then 32,955, then - 33,166 — with the payload unchanged. Treat within-batch differences as + case's reported input drifted across batches — 32,507, 32,955, 33,166, then + 32,464 — with the payload unchanged. Treat within-batch differences as measurements and absolute counts as approximate. ## 10. A stratum boundary is not a rounding difference diff --git a/review-suite/evals/baseline/v1/SOURCING.md b/review-suite/evals/baseline/v1/SOURCING.md index 30e4b36..e63fd7a 100644 --- a/review-suite/evals/baseline/v1/SOURCING.md +++ b/review-suite/evals/baseline/v1/SOURCING.md @@ -73,16 +73,16 @@ not carry it. ### `status-label-normalization` — pilot, unscored, deliberately uncalibrated -| field | value | -| ---------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| strata | `pilot-orchestrator` only, so the case shared with the lens strata stays byte-identical across all three | -| source | `shaug/atelier` pull request 278, review comment 2861937742, authored by the repository owner. Public. | -| adjudication | The reviewer required the change; the follow-up reply (comment 2861947492) names the implementing commit and the regression coverage added for the exact migration shape. Accepted, not deferred. | -| failure shape retained | A normalization step drops one legacy flag and returns early for an inactive status, leaving a second legacy flag that contradicts the canonical status, with a live consumer that admits work on that flag alone. The added tests all start from records that never carried the second flag, so the one shape the change exists to fix is untested. | -| origin | `minimized_reproduction` | -| sanitization | Rewritten against a fictional record registry. No source identifier, path, symbol, prose, or diff copied. No business logic, domain identifier, customer context, credential, or hidden reasoning. | -| retention authority | Public repository, owner-authored review, no third-party or customer material. | -| why it exists | Two purposes. Its packet is materially larger than the other pilot case, which separates the fixed cost of a stratum's skill closure from the variable cost of the packet — the measurement behind the cost-ceiling proposal. And it is **deliberately left uncalibrated** (`calibrated: false`) as the control for what an uncalibrated expectation reports: recall 0.0 over five attempts with nine adjudication referrals, while the reviewer gated the change every time. | +| field | value | +| ---------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| strata | `pilot-orchestrator` only, so the case shared with the lens strata stays byte-identical across all three | +| source | `shaug/atelier` pull request 278, review comment 2861937742, authored by the repository owner. Public. | +| adjudication | The reviewer required the change; the follow-up reply (comment 2861947492) names the implementing commit and the regression coverage added for the exact migration shape. Accepted, not deferred. | +| failure shape retained | A normalization step drops one legacy flag and returns early for an inactive status, leaving a second legacy flag that contradicts the canonical status, with a live consumer that admits work on that flag alone. The added tests all start from records that never carried the second flag, so the one shape the change exists to fix is untested. | +| origin | `minimized_reproduction` | +| sanitization | Rewritten against a fictional record registry. No source identifier, path, symbol, prose, or diff copied. No business logic, domain identifier, customer context, credential, or hidden reasoning. | +| retention authority | Public repository, owner-authored review, no third-party or customer material. | +| why it exists | Two purposes. Its packet is materially larger than the other pilot case, which separates the fixed cost of a stratum's skill closure from the variable cost of the packet — the measurement behind the cost-ceiling proposal. And it is **deliberately left uncalibrated** (`calibrated: false`) as the control for what an uncalibrated expectation reports: recall 0.0 over five attempts with ten adjudication referrals, while the reviewer gated the change every time. | ## Cases excluded, and why diff --git a/review-suite/evals/baseline/v1/frozen-configuration.json b/review-suite/evals/baseline/v1/frozen-configuration.json index e916140..8fc4f1c 100644 --- a/review-suite/evals/baseline/v1/frozen-configuration.json +++ b/review-suite/evals/baseline/v1/frozen-configuration.json @@ -2,7 +2,10 @@ "record_version": "1.0", "status": "incomplete_pending_owner_preregistration", "status_detail": "Every value an implementing run can fix is fixed below. Two values cannot come from an implementing context and are therefore null: the per-stratum cost ceiling, which must be preregistered by the repository owner before any scored output is examined because a scored run spends real money, and the independent adjudication of each private expectation, which one context cannot supply both sides of. No scored run may launch until both are present. Freezing everything else now is what makes the remaining inputs a decision rather than a negotiation.", - "suite_commit": "16560d807c66076fcbf3f00d3a87f543c6ae2458", + "v1_review_behaviour_commit": "16560d807c66076fcbf3f00d3a87f543c6ae2458", + "v1_review_behaviour_commit_note": "The pre-v2 review-suite commit whose behaviour a scored baseline must evaluate. This candidate modifies no file under skills/ and no v1 contract, so the evaluated closure text is identical at every commit on this branch: all three closure digests recorded below and in every pilot report are unchanged from this commit. The digest, not the commit, is the load-bearing pin - a commit can move without the evaluated text changing, and the evaluated text cannot change without the digest moving.", + "pilot_suite_commit": "2ae0d23c18f247f49d3cc5e76f26d1cf9610c83e", + "pilot_suite_commit_note": "The commit the committed pilot reports record, and the tree the recorded pilot batch actually ran from. The corpora it evaluated are present at that commit, so the batch is reproducible from it. Later commits on this branch change records only and leave every payload byte-identical. Earlier superseded batches ran from uncommitted trees and are cited by their corpus version and closure digest rather than by a commit, because no commit reproduces them.", "protocol_version": "1.0", "grader_version": "1.0", "calibration_version": "1.0", @@ -18,16 +21,16 @@ }, "execution": { "runs_per_case": 5, - "runs_per_case_rationale": "A denominator of one cannot measure a stochastic reviewer, and this suite has already observed the same configuration return `blocked` on one attempt and a merge verdict on the next for the case that tests refusal on incomplete evidence. Five is the smallest count that makes a per-case verdict-stability figure mean anything. Measured at five runs per case over 20 attempts, the pilot observed verdict stability 1.0 and finding stability 1.0 on every case, so five did not surface instability on these packets - which is a statement about these packets, not a licence to reduce the count. The instability #50 observed was on a refusal-on-incomplete-evidence case, a class no scored stratum here contains.", + "runs_per_case_rationale": "A denominator of one cannot measure a stochastic reviewer, and this suite has already observed the same configuration return `blocked` on one attempt and a merge verdict on the next for the case that tests refusal on incomplete evidence. Five is the smallest count that makes a per-case verdict-stability figure mean anything. Measured at five runs per case, the pilot observed verdict stability 1.0 and finding stability 1.0 on every case, so five did not surface instability on these packets - a statement about these packets, not a licence to reduce the count. The instability #50 observed was on a refusal-on-incomplete-evidence case, a class no scored stratum here contains.", "timeout_seconds": 300, - "timeout_rationale": "Worst latency in any committed pilot report is 53.1 s, on the orchestrator stratum's larger packet. The slowest attempt observed at any point across this ticket's 52 pilot attempts was 55.4 s, also on that packet. 300 s is roughly 5.4 times the worst observed, which absorbs a slow attempt without letting a hung process run unbounded. No pilot attempt timed out, and no attempt came within a factor of five of the limit.", + "timeout_rationale": "Worst latency in any committed pilot report is 51.2 s, on the orchestrator stratum's larger packet. The slowest attempt observed at any point across this ticket's 72 pilot attempts was 55.4 s, also on that packet. 300 s is roughly 5.4 times the worst observed. No pilot attempt timed out and none came within a factor of five of the limit.", "retry_policy": "none", "retry_policy_rationale": "A failed attempt is recorded as an evaluation failure and never retried. Retrying would silently replace an unstable attempt with a luckier one and would bias every stability figure toward agreement.", "max_output_bytes": 4000000, "process_isolation": "one fresh executor process per attempt", "cost_ceiling_exhaustion_policy": "Exceeding a stratum's preregistered ceiling stops further runs in that stratum and records an incomplete baseline for it. Repetitions are never reduced after outputs are visible.", - "artifact_retention": "Raw executor output is retained outside git at review-suite/evals/artifacts///.run-.stdout.json. The corpus version is in the path deliberately: an artifact name carries only the case and run number, so re-running a stratum into an unscoped directory silently replaces output that a committed record may already cite, which happened once and cost a calibration source run. The runner now refuses before launching any attempt when a run would overwrite retained output, so the failure costs nothing.", - "invocation_template": "just eval-review-suite '' --corpus review-suite/evals/strata/ --runs --timeout --artifact-dir review-suite/evals/artifacts// --attempts-out --report-out ", + "artifact_retention": "Raw executor output is retained outside git at review-suite/evals/artifacts//-/.run-.stdout.json, with the run's per-attempt records beside it at /-.attempts.jsonl. The commit and corpus version are in the path deliberately: an artifact name carries only the case and run number, so re-running a stratum into an unscoped directory silently replaces output a committed record may already cite, which happened once and cost a calibration source run. The runner now refuses before launching any attempt when a run would overwrite retained output, so the failure costs nothing.", + "invocation_template": "just eval-review-suite '' --corpus review-suite/evals/strata/ --runs --timeout --artifact-dir review-suite/evals/artifacts//- --attempts-out review-suite/evals/artifacts//-.attempts.jsonl --report-out ", "invocation_note": "The recipe forwards extra arguments to the runner. Naming an executor alone is not the frozen configuration: --corpus defaults to the protocol-proof corpus and --runs to 1, so the abbreviated command silently evaluates a different corpus once." }, "strata": [ @@ -53,11 +56,13 @@ ], "envelope_source": "pilot-orchestrator", "cost_ceiling_usd": null, - "cost_ceiling_proposal_usd": 8.0, - "scored_invocation": "just eval-review-suite 'python3 review-suite/scripts/evals/claude_executor.py' --corpus review-suite/evals/strata/s1-correctness-orchestrator --runs 5 --timeout 300 --artifact-dir review-suite/evals/artifacts/s1-correctness-orchestrator/ --attempts-out /s1-correctness-orchestrator.attempts.jsonl --report-out review-suite/evals/baseline/v1/s1-correctness-orchestrator.report.json", + "cost_ceiling_proposal_usd": 9.0, + "scored_invocation": "just eval-review-suite 'python3 review-suite/scripts/evals/claude_executor.py' --corpus review-suite/evals/strata/s1-correctness-orchestrator --runs 5 --timeout 300 --artifact-dir review-suite/evals/artifacts/s1-correctness-orchestrator/- --attempts-out review-suite/evals/artifacts/s1-correctness-orchestrator/-.attempts.jsonl --report-out review-suite/evals/baseline/v1/s1-correctness-orchestrator.report.json", "scored_invocation_note": "Executable verbatim once the stratum directory exists and the owner has preregistered the ceiling. Substitute the stratum's own corpus_version in the artifact directory.", - "expected_spend_usd": 3.26, - "expected_spend_note": "Expected spend assumes the prompt cache behaves as measured: one cache-creation attempt per case, the rest cache reads. The proposed ceiling assumes every attempt pays cache creation, which is the worst case actually observed rather than a hypothetical one." + "expected_spend_usd": 3.33, + "expected_spend_note": "Expected spend assumes the prompt cache behaves as measured: one cache-creation attempt per case, the rest cache reads. The proposed ceiling assumes every attempt pays cache creation, which is the worst case actually observed rather than a hypothetical one.", + "worst_observed_cold_cost_per_attempt_usd": 0.2021, + "all_cold_worst_case_usd": 7.07 }, { "id": "s2-solution-simplicity-lens", @@ -75,10 +80,12 @@ "envelope_source": "pilot-solution-simplicity", "cost_ceiling_usd": null, "cost_ceiling_proposal_usd": 3.0, - "scored_invocation": "just eval-review-suite 'python3 review-suite/scripts/evals/claude_executor.py' --corpus review-suite/evals/strata/s2-solution-simplicity-lens --runs 5 --timeout 300 --artifact-dir review-suite/evals/artifacts/s2-solution-simplicity-lens/ --attempts-out /s2-solution-simplicity-lens.attempts.jsonl --report-out review-suite/evals/baseline/v1/s2-solution-simplicity-lens.report.json", + "scored_invocation": "just eval-review-suite 'python3 review-suite/scripts/evals/claude_executor.py' --corpus review-suite/evals/strata/s2-solution-simplicity-lens --runs 5 --timeout 300 --artifact-dir review-suite/evals/artifacts/s2-solution-simplicity-lens/- --attempts-out review-suite/evals/artifacts/s2-solution-simplicity-lens/-.attempts.jsonl --report-out review-suite/evals/baseline/v1/s2-solution-simplicity-lens.report.json", "scored_invocation_note": "Executable verbatim once the stratum directory exists and the owner has preregistered the ceiling. Substitute the stratum's own corpus_version in the artifact directory.", - "expected_spend_usd": 0.83, - "expected_spend_note": "Expected spend assumes the prompt cache behaves as measured: one cache-creation attempt per case, the rest cache reads. The proposed ceiling assumes every attempt pays cache creation, which is the worst case actually observed rather than a hypothetical one." + "expected_spend_usd": 0.82, + "expected_spend_note": "Expected spend assumes the prompt cache behaves as measured: one cache-creation attempt per case, the rest cache reads. The proposed ceiling assumes every attempt pays cache creation, which is the worst case actually observed rather than a hypothetical one.", + "worst_observed_cold_cost_per_attempt_usd": 0.1094, + "all_cold_worst_case_usd": 2.19 }, { "id": "s3-code-simplicity-lens", @@ -96,10 +103,12 @@ "envelope_source": "pilot-code-simplicity", "cost_ceiling_usd": null, "cost_ceiling_proposal_usd": 3.0, - "scored_invocation": "just eval-review-suite 'python3 review-suite/scripts/evals/claude_executor.py' --corpus review-suite/evals/strata/s3-code-simplicity-lens --runs 5 --timeout 300 --artifact-dir review-suite/evals/artifacts/s3-code-simplicity-lens/ --attempts-out /s3-code-simplicity-lens.attempts.jsonl --report-out review-suite/evals/baseline/v1/s3-code-simplicity-lens.report.json", + "scored_invocation": "just eval-review-suite 'python3 review-suite/scripts/evals/claude_executor.py' --corpus review-suite/evals/strata/s3-code-simplicity-lens --runs 5 --timeout 300 --artifact-dir review-suite/evals/artifacts/s3-code-simplicity-lens/- --attempts-out review-suite/evals/artifacts/s3-code-simplicity-lens/-.attempts.jsonl --report-out review-suite/evals/baseline/v1/s3-code-simplicity-lens.report.json", "scored_invocation_note": "Executable verbatim once the stratum directory exists and the owner has preregistered the ceiling. Substitute the stratum's own corpus_version in the artifact directory.", - "expected_spend_usd": 0.85, - "expected_spend_note": "Expected spend assumes the prompt cache behaves as measured: one cache-creation attempt per case, the rest cache reads. The proposed ceiling assumes every attempt pays cache creation, which is the worst case actually observed rather than a hypothetical one." + "expected_spend_usd": 0.93, + "expected_spend_note": "Expected spend assumes the prompt cache behaves as measured: one cache-creation attempt per case, the rest cache reads. The proposed ceiling assumes every attempt pays cache creation, which is the worst case actually observed rather than a hypothetical one.", + "worst_observed_cold_cost_per_attempt_usd": 0.1116, + "all_cold_worst_case_usd": 2.23 }, { "id": "connector-escape", @@ -151,31 +160,35 @@ "report": "pilot/pilot-orchestrator.report.json", "stratum": "pilot-orchestrator", "quality_block_is_signal": true, - "retained_artifacts": "review-suite/evals/artifacts/pilot-orchestrator/1.3-pilot-orchestrator/", - "invocation": "just eval-review-suite 'python3 review-suite/scripts/evals/claude_executor.py' --corpus review-suite/evals/strata/pilot-orchestrator --runs 5 --timeout 300 --artifact-dir review-suite/evals/artifacts/pilot-orchestrator/1.3-pilot-orchestrator --report-out review-suite/evals/baseline/v1/pilot/pilot-orchestrator.report.json", - "quality_block_caveat": "The aggregate recall of 0.5 is the mean of two cases that differ in one respect only: rollback-guidance-render is calibrated and scored recall 1.0 on all five attempts, while status-label-normalization is deliberately left uncalibrated as a control and scored recall 0.0 on all five, with nine adjudication referrals, while the reviewer gated the change every time. Do not read 0.5 as a capability figure; read the per-case block, and read each case's expectation `calibrated` flag first." + "retained_artifacts": "review-suite/evals/artifacts/pilot-orchestrator/2ae0d23-1.3-pilot-orchestrator/", + "invocation": "just eval-review-suite 'python3 review-suite/scripts/evals/claude_executor.py' --corpus review-suite/evals/strata/pilot-orchestrator --runs 5 --timeout 300 --artifact-dir review-suite/evals/artifacts/pilot-orchestrator/2ae0d23-1.3-pilot-orchestrator --attempts-out review-suite/evals/artifacts/pilot-orchestrator/2ae0d23-1.3-pilot-orchestrator.attempts.jsonl --report-out review-suite/evals/baseline/v1/pilot/pilot-orchestrator.report.json", + "quality_block_caveat": "The aggregate recall of 0.5 is the mean of two cases that differ in one respect only: rollback-guidance-render is calibrated and scored recall 1.0 on all five attempts, while status-label-normalization is deliberately left uncalibrated as a control and scored recall 0.0 on all five, with ten adjudication referrals, while the reviewer gated the change every time. Do not read 0.5 as a capability figure; read the per-case block, and read each case's expectation `calibrated` flag first.", + "retained_per_attempt_records": "review-suite/evals/artifacts/pilot-orchestrator/2ae0d23-1.3-pilot-orchestrator.attempts.jsonl" }, { "report": "pilot/pilot-solution-simplicity.report.json", "stratum": "pilot-solution-simplicity", "quality_block_is_signal": false, "quality_block_caveat": "DO NOT QUOTE this report's material_finding_recall (0.0) or false_clean_rate (1.0) as a reviewer quality figure. This corpus carries the orchestrator stratum's case byte for byte, so that the declared skill closure is the only variable between pilot strata, and the expectation is a correctness root cause authored for that target. A simplicity lens returning clean on a correctness defect is behaving exactly as its contract requires, so the recorded miss is a target mismatch, not a reviewer failure. The corpus declares stratum.grading_is_signal false for this reason. Only payload size, latency, cost, and protocol outcomes are signals here.", - "retained_artifacts": "review-suite/evals/artifacts/pilot-solution-simplicity/1.3-pilot-solution-simplicity/", - "invocation": "just eval-review-suite 'python3 review-suite/scripts/evals/claude_executor.py' --corpus review-suite/evals/strata/pilot-solution-simplicity --runs 5 --timeout 300 --artifact-dir review-suite/evals/artifacts/pilot-solution-simplicity/1.3-pilot-solution-simplicity --report-out review-suite/evals/baseline/v1/pilot/pilot-solution-simplicity.report.json" + "retained_artifacts": "review-suite/evals/artifacts/pilot-solution-simplicity/2ae0d23-1.3-pilot-solution-simplicity/", + "invocation": "just eval-review-suite 'python3 review-suite/scripts/evals/claude_executor.py' --corpus review-suite/evals/strata/pilot-solution-simplicity --runs 5 --timeout 300 --artifact-dir review-suite/evals/artifacts/pilot-solution-simplicity/2ae0d23-1.3-pilot-solution-simplicity --attempts-out review-suite/evals/artifacts/pilot-solution-simplicity/2ae0d23-1.3-pilot-solution-simplicity.attempts.jsonl --report-out review-suite/evals/baseline/v1/pilot/pilot-solution-simplicity.report.json", + "retained_per_attempt_records": "review-suite/evals/artifacts/pilot-solution-simplicity/2ae0d23-1.3-pilot-solution-simplicity.attempts.jsonl" }, { "report": "pilot/pilot-code-simplicity.report.json", "stratum": "pilot-code-simplicity", "quality_block_is_signal": false, "quality_block_caveat": "DO NOT QUOTE this report's material_finding_recall (0.0) or false_clean_rate (1.0) as a reviewer quality figure. This corpus carries the orchestrator stratum's case byte for byte, so that the declared skill closure is the only variable between pilot strata, and the expectation is a correctness root cause authored for that target. A simplicity lens returning clean on a correctness defect is behaving exactly as its contract requires, so the recorded miss is a target mismatch, not a reviewer failure. The corpus declares stratum.grading_is_signal false for this reason. Only payload size, latency, cost, and protocol outcomes are signals here.", - "retained_artifacts": "review-suite/evals/artifacts/pilot-code-simplicity/1.3-pilot-code-simplicity/", - "invocation": "just eval-review-suite 'python3 review-suite/scripts/evals/claude_executor.py' --corpus review-suite/evals/strata/pilot-code-simplicity --runs 5 --timeout 300 --artifact-dir review-suite/evals/artifacts/pilot-code-simplicity/1.3-pilot-code-simplicity --report-out review-suite/evals/baseline/v1/pilot/pilot-code-simplicity.report.json" + "retained_artifacts": "review-suite/evals/artifacts/pilot-code-simplicity/2ae0d23-1.3-pilot-code-simplicity/", + "invocation": "just eval-review-suite 'python3 review-suite/scripts/evals/claude_executor.py' --corpus review-suite/evals/strata/pilot-code-simplicity --runs 5 --timeout 300 --artifact-dir review-suite/evals/artifacts/pilot-code-simplicity/2ae0d23-1.3-pilot-code-simplicity --attempts-out review-suite/evals/artifacts/pilot-code-simplicity/2ae0d23-1.3-pilot-code-simplicity.attempts.jsonl --report-out review-suite/evals/baseline/v1/pilot/pilot-code-simplicity.report.json", + "retained_per_attempt_records": "review-suite/evals/artifacts/pilot-code-simplicity/2ae0d23-1.3-pilot-code-simplicity.attempts.jsonl" } ], "controlled_artifacts": { - "location": "review-suite/evals/artifacts//.run-.stdout.json", + "raw_output": "review-suite/evals/artifacts//-/.run-.stdout.json", + "per_attempt_records": "review-suite/evals/artifacts//-.attempts.jsonl", "in_git": false, - "detail": "Raw executor output is retained outside git by the repository's existing ignore rule. The committed compact reports carry every metric, denominator, and identity a consumer needs; the raw transcripts are referenced by that path identity for anyone re-deriving a figure." + "detail": "The committed compact reports carry every metric, denominator, and identity a consumer needs. The retained per-attempt records are what a per-attempt figure quoted from a record - a single attempt's cost, token count, or latency - is re-derived from, and they are referenced by the path identity above." }, "baseline_limitations_record": "LIMITATIONS.md", "adjudication_plan": "ADJUDICATION-PLAN.md" diff --git a/review-suite/evals/baseline/v1/pilot/README.md b/review-suite/evals/baseline/v1/pilot/README.md index f965119..c046258 100644 --- a/review-suite/evals/baseline/v1/pilot/README.md +++ b/review-suite/evals/baseline/v1/pilot/README.md @@ -4,11 +4,17 @@ These three reports establish executor compatibility, timeout behaviour, and a cost and latency envelope **per stratum**. They are not a baseline. No scored case exists yet, and nothing here measures reviewer quality across a corpus. -Five runs per case. Suite commit `16560d807c66076fcbf3f00d3a87f543c6ae2458`, -model `claude-opus-4-6[1m]`, corpus version `1.3-pilot-*`, grader version `1.0`, -timeout 300 s, no retries, one fresh process per attempt. **20 attempts, zero -evaluation failures, zero timeouts, 1.2266 USD.** Raw output is retained outside -git at `review-suite/evals/artifacts//1.3-/`. +Five runs per case, run from the committed tree at suite commit +`2ae0d23c18f247f49d3cc5e76f26d1cf9610c83e` — the commit every report records, +and the one the batch is reproducible from. Model `claude-opus-4-6[1m]`, corpus +version `1.3-pilot-*`, grader version `1.0`, timeout 300 s, no retries, one +fresh process per attempt. **20 attempts, zero evaluation failures, zero +timeouts, 1.2536 USD.** + +Raw output and the run's per-attempt records are retained outside git at +`review-suite/evals/artifacts//2ae0d23-1.3-/` and +`.../2ae0d23-1.3-.attempts.jsonl`. Every per-attempt figure quoted +anywhere in this directory is re-derivable from those records. ## Before reading any recall number, read the case's `calibrated` flag @@ -19,7 +25,7 @@ respect, and the contrast is the most useful thing in this directory: | case | `calibrated` | recall over 5 attempts | verdict stability | referrals | what it means | | ---------------------------- | ------------ | ---------------------- | ----------------- | --------- | ---------------------------------------------------------------------------------------------------------------------------- | | `rollback-guidance-render` | `true` | **1.0** | 1.0 | 0 | Formulations pinned to prose a real reviewer wrote. The grader recognises the finding every time. | -| `status-label-normalization` | `false` | **0.0** | 1.0 | 9 | Deliberately uncalibrated control. The reviewer gated the change on every attempt and the grader matched neither root cause. | +| `status-label-normalization` | `false` | **0.0** | 1.0 | 10 | Deliberately uncalibrated control. The reviewer gated the change on every attempt and the grader matched neither root cause. | The aggregate `material_finding_recall: 0.5` in `pilot-orchestrator.report.json` is the mean of those two. **It is not a capability figure.** It is the measured diff --git a/review-suite/evals/baseline/v1/pilot/pilot-code-simplicity.report.json b/review-suite/evals/baseline/v1/pilot/pilot-code-simplicity.report.json index ba1d8df..04bbd4f 100644 --- a/review-suite/evals/baseline/v1/pilot/pilot-code-simplicity.report.json +++ b/review-suite/evals/baseline/v1/pilot/pilot-code-simplicity.report.json @@ -7,10 +7,20 @@ "cases": 1, "corpus_version": "1.3-pilot-code-simplicity", "executor": "python3 review-suite/scripts/evals/claude_executor.py", + "executor_models": [ + "claude-opus-4-6[1m]" + ], "grader_version": "1.0", "max_output_bytes": 4000000, "runs_per_case": 5, - "suite_commit": "16b77e447dbcc844edd8f3fb58728d96826e177c", + "stratum": { + "grading_is_signal": false, + "ground_truth": "human-review", + "id": "pilot-code-simplicity", + "purpose": "Size the cost and latency envelope for a stratum targeting the self-sufficient code-simplicity lens. Unscored. Its expectations were authored for the orchestrator target, so its graded output is a target-mismatch artefact and not a quality signal; only its payload size, latency, cost, and protocol outcomes are.", + "scored": false + }, + "suite_commit": "2ae0d23c18f247f49d3cc5e76f26d1cf9610c83e", "target_skill": "review-code-simplicity", "target_skill_dependencies": [], "target_skill_digest": "a6187d8971eaef24", @@ -33,10 +43,10 @@ "grader_version": "1.0", "latency": { "count": 5, - "max_seconds": 16.773228792008013, - "mean_seconds": 12.84428822491318, - "min_seconds": 9.155616583768278, - "p50_seconds": 12.735041291918606 + "max_seconds": 22.93410737486556, + "mean_seconds": 16.331333041656762, + "min_seconds": 12.653373999986798, + "p50_seconds": 14.434281916823238 }, "per_case": [ { @@ -93,8 +103,8 @@ "reporting_attempts_cost_usd": 5, "reporting_attempts_input_tokens": 5, "reporting_attempts_output_tokens": 5, - "total_cost_usd": 0.2087365, - "total_input_tokens": 131360.0, - "total_output_tokens": 2161.0 + "total_cost_usd": 0.21660775, + "total_input_tokens": 128085.0, + "total_output_tokens": 2692.0 } } diff --git a/review-suite/evals/baseline/v1/pilot/pilot-orchestrator.report.json b/review-suite/evals/baseline/v1/pilot/pilot-orchestrator.report.json index 6c51398..9840a28 100644 --- a/review-suite/evals/baseline/v1/pilot/pilot-orchestrator.report.json +++ b/review-suite/evals/baseline/v1/pilot/pilot-orchestrator.report.json @@ -6,7 +6,7 @@ "rc.tests-omit-the-migrating-shape" ], "case_id": "status-label-normalization", - "finding_id": "ready-flag-retained-on-inactive", + "finding_id": "corr.inactive-ready-retained", "reason": "partial", "run_number": 1 }, @@ -16,7 +16,17 @@ "rc.tests-omit-the-migrating-shape" ], "case_id": "status-label-normalization", - "finding_id": "inactive-record-retains-ready-flag", + "finding_id": "corr.missing-dual-flag-test", + "reason": "partial", + "run_number": 1 + }, + { + "candidate_root_cause_ids": [ + "rc.retained-ready-flag-on-inactive-status", + "rc.tests-omit-the-migrating-shape" + ], + "case_id": "status-label-normalization", + "finding_id": "flag-ready-retained-on-inactive", "reason": "partial", "run_number": 2 }, @@ -26,7 +36,7 @@ "rc.tests-omit-the-migrating-shape" ], "case_id": "status-label-normalization", - "finding_id": "missing-test-both-flags-inactive", + "finding_id": "no-test-for-dual-flag-inactive", "reason": "partial", "run_number": 2 }, @@ -36,7 +46,7 @@ "rc.tests-omit-the-migrating-shape" ], "case_id": "status-label-normalization", - "finding_id": "correctness.inactive-ready-flag-retained", + "finding_id": "inactive-retains-ready-flag", "reason": "partial", "run_number": 3 }, @@ -46,7 +56,7 @@ "rc.tests-omit-the-migrating-shape" ], "case_id": "status-label-normalization", - "finding_id": "correctness.missing-dual-flag-test", + "finding_id": "missing-dual-flag-test", "reason": "partial", "run_number": 3 }, @@ -56,7 +66,7 @@ "rc.tests-omit-the-migrating-shape" ], "case_id": "status-label-normalization", - "finding_id": "ready-flag-retained-on-inactive-dual-flag", + "finding_id": "corr.inactive-ready-retained", "reason": "partial", "run_number": 4 }, @@ -66,7 +76,7 @@ "rc.tests-omit-the-migrating-shape" ], "case_id": "status-label-normalization", - "finding_id": "missing-dual-flag-test", + "finding_id": "corr.missing-dual-flag-test", "reason": "partial", "run_number": 4 }, @@ -86,7 +96,7 @@ "rc.tests-omit-the-migrating-shape" ], "case_id": "status-label-normalization", - "finding_id": "missing-dual-flag-inactive-test", + "finding_id": "missing-dual-flag-test", "reason": "partial", "run_number": 5 } @@ -98,10 +108,20 @@ "cases": 2, "corpus_version": "1.3-pilot-orchestrator", "executor": "python3 review-suite/scripts/evals/claude_executor.py", + "executor_models": [ + "claude-opus-4-6[1m]" + ], "grader_version": "1.0", "max_output_bytes": 4000000, "runs_per_case": 5, - "suite_commit": "16b77e447dbcc844edd8f3fb58728d96826e177c", + "stratum": { + "grading_is_signal": true, + "ground_truth": "human-review", + "id": "pilot-orchestrator", + "purpose": "Size the cost and latency envelope for a stratum whose target is the orchestrator, whose payload therefore carries its three required lens skills. Carries two cases of materially different packet size so the fixed cost of the closure can be separated from the variable cost of the packet. Unscored.", + "scored": false + }, + "suite_commit": "2ae0d23c18f247f49d3cc5e76f26d1cf9610c83e", "target_skill": "review-code-change", "target_skill_dependencies": [ "review-solution-simplicity", @@ -134,10 +154,10 @@ "grader_version": "1.0", "latency": { "count": 10, - "max_seconds": 53.09396595787257, - "mean_seconds": 36.91776127498597, - "min_seconds": 24.860583000350744, - "p50_seconds": 37.728567416314036 + "max_seconds": 51.17633504094556, + "mean_seconds": 38.692721895687285, + "min_seconds": 31.838742708787322, + "p50_seconds": 36.32987822918221 }, "per_case": [ { @@ -222,8 +242,8 @@ "reporting_attempts_cost_usd": 10, "reporting_attempts_input_tokens": 10, "reporting_attempts_output_tokens": 10, - "total_cost_usd": 0.8159675000000002, - "total_input_tokens": 336550.0, - "total_output_tokens": 15389.0 + "total_cost_usd": 0.8376845, + "total_input_tokens": 329530.0, + "total_output_tokens": 16721.0 } } diff --git a/review-suite/evals/baseline/v1/pilot/pilot-solution-simplicity.report.json b/review-suite/evals/baseline/v1/pilot/pilot-solution-simplicity.report.json index 8cc75c7..70e3310 100644 --- a/review-suite/evals/baseline/v1/pilot/pilot-solution-simplicity.report.json +++ b/review-suite/evals/baseline/v1/pilot/pilot-solution-simplicity.report.json @@ -7,10 +7,20 @@ "cases": 1, "corpus_version": "1.3-pilot-solution-simplicity", "executor": "python3 review-suite/scripts/evals/claude_executor.py", + "executor_models": [ + "claude-opus-4-6[1m]" + ], "grader_version": "1.0", "max_output_bytes": 4000000, "runs_per_case": 5, - "suite_commit": "16b77e447dbcc844edd8f3fb58728d96826e177c", + "stratum": { + "grading_is_signal": false, + "ground_truth": "human-review", + "id": "pilot-solution-simplicity", + "purpose": "Size the cost and latency envelope for a stratum targeting the self-sufficient solution-simplicity lens. Unscored. Its expectations were authored for the orchestrator target, so its graded output is a target-mismatch artefact and not a quality signal; only its payload size, latency, cost, and protocol outcomes are.", + "scored": false + }, + "suite_commit": "2ae0d23c18f247f49d3cc5e76f26d1cf9610c83e", "target_skill": "review-solution-simplicity", "target_skill_dependencies": [], "target_skill_digest": "6257ee4448b15874", @@ -33,10 +43,10 @@ "grader_version": "1.0", "latency": { "count": 5, - "max_seconds": 12.044242959003896, - "mean_seconds": 10.887090650293976, - "min_seconds": 9.784014916978776, - "p50_seconds": 10.844918083399534 + "max_seconds": 14.185639708768576, + "mean_seconds": 11.552183233574032, + "min_seconds": 10.409321167040616, + "p50_seconds": 11.100352916866541 }, "per_case": [ { @@ -93,8 +103,8 @@ "reporting_attempts_cost_usd": 5, "reporting_attempts_input_tokens": 5, "reporting_attempts_output_tokens": 5, - "total_cost_usd": 0.201944, - "total_input_tokens": 132410.0, - "total_output_tokens": 1820.0 + "total_cost_usd": 0.1992675, + "total_input_tokens": 129000.0, + "total_output_tokens": 1938.0 } } diff --git a/review-suite/evals/strata/README.md b/review-suite/evals/strata/README.md index a2ec5fa..5123b25 100644 --- a/review-suite/evals/strata/README.md +++ b/review-suite/evals/strata/README.md @@ -64,17 +64,19 @@ list. ### Measured envelope -Five runs per case. Suite commit `16560d807c66076fcbf3f00d3a87f543c6ae2458`, -model `claude-opus-4-6[1m]`, corpus version `1.3-pilot-*`, grader version `1.0`, -timeout 300 s, no retries. **20 attempts, zero evaluation failures, zero -timeouts, 1.2266 USD.** +Five runs per case, run from the committed tree at suite commit +`2ae0d23c18f247f49d3cc5e76f26d1cf9610c83e` — the commit every report records, +and the one the batch is reproducible from. Model `claude-opus-4-6[1m]`, corpus +version `1.3-pilot-*`, grader version `1.0`, timeout 300 s, no retries, one +fresh process per attempt. **20 attempts, zero evaluation failures, zero +timeouts, 1.2536 USD.** | stratum | target | docs | digest | case | input tokens / attempt | cold cost | warm cost | latency range | | --------------------------- | ---------------------------- | ---- | ------------------ | ---------------------------- | ---------------------- | ---------- | ----------------- | ------------- | -| `pilot-orchestrator` | `review-code-change` | 8 | `9b2805f14cdd6158` | `rollback-guidance-render` | 33,166 | 0.1719 USD | 0.0467–0.0548 USD | 24.9–36.1 s | -| `pilot-orchestrator` | `review-code-change` | 8 | `9b2805f14cdd6158` | `status-label-normalization` | 34,144 | 0.1943 USD | 0.0570–0.0678 USD | 39.4–53.1 s | -| `pilot-solution-simplicity` | `review-solution-simplicity` | 2 | `6257ee4448b15874` | `rollback-guidance-render` | 26,482 | 0.1118 USD | 0.0215–0.0237 USD | 9.8–12.0 s | -| `pilot-code-simplicity` | `review-code-simplicity` | 2 | `a6187d8971eaef24` | `rollback-guidance-render` | 26,272 | 0.1170 USD | 0.0210–0.0240 USD | 9.2–16.8 s | +| `pilot-orchestrator` | `review-code-change` | 8 | `9b2805f14cdd6158` | `rollback-guidance-render` | 32,464 | 0.1698 USD | 0.0519–0.0532 USD | 31.8–34.9 s | +| `pilot-orchestrator` | `review-code-change` | 8 | `9b2805f14cdd6158` | `status-label-normalization` | 33,442 | 0.2021 USD | 0.0589–0.0684 USD | 37.7–51.2 s | +| `pilot-solution-simplicity` | `review-solution-simplicity` | 2 | `6257ee4448b15874` | `rollback-guidance-render` | 25,800 | 0.1094 USD | 0.0216–0.0238 USD | 10.4–14.2 s | +| `pilot-code-simplicity` | `review-code-simplicity` | 2 | `a6187d8971eaef24` | `rollback-guidance-render` | 25,617 | 0.1116 USD | 0.0245–0.0303 USD | 12.7–22.9 s | Two measurements shape the ceiling, and both are in [the cost-ceiling proposal](../baseline/v1/COST-CEILING-PROPOSAL.md): @@ -84,24 +86,25 @@ Two measurements shape the ceiling, and both are in built from the mean is exceeded by any run whose cache does not stay warm. - **Cost tracks output volume, not packet size.** The orchestrator stratum carries two cases specifically to measure this. The larger packet raised input - tokens by 2.9% and raised warm cost by 28% and mean latency by 45%, because - mean output grew 44% — the packet warranted more findings, and output tokens - dominate. + tokens by 3.0% and raised warm cost by 21% and warm mean latency by 22%, + because mean warm output grew 29% — the packet warranted more findings, and + output tokens dominate. Verdict stability and finding stability were 1.0 on every case at five runs, so five runs did not surface instability on these packets. That is a statement about these packets, not a licence to reduce the run count. -Total pilot spend across the whole ticket: **3.3776 USD over 52 attempts** in -five batches. Only the last, the committed configuration above, is cited as the +Total pilot spend across the whole ticket: **4.6312 USD over 72 attempts** in +six batches. Only the last, the committed configuration above, is cited as the frozen envelope; the earlier batches sized it and produced the prose the formulations were calibrated against. Raw output is retained outside git at -`review-suite/evals/artifacts///`. The corpus version -is in the path deliberately: an artifact is named for its case and run number -only, so re-running a stratum into an unscoped directory silently replaced -output a committed record already cited — which happened once, and cost the +`review-suite/evals/artifacts//-/`, with the +run's per-attempt records beside it. The commit and corpus version are in the +path deliberately: an artifact is named for its case and run number only, so +re-running a stratum into an unscoped directory silently replaced output a +committed record already cited — which happened once, and cost the calibration-source run. The runner now refuses before launching any attempt when a run would overwrite retained output. The exact per-stratum invocations, artifact paths included, are recorded in From d013507956aa0ab328140a72c87fdbb151f2b1ec Mon Sep 17 00:00:00 2001 From: Scott Haug Date: Mon, 27 Jul 2026 07:26:56 -0700 Subject: [PATCH 5/5] fix: reconcile every recorded figure with its retained artifact MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Closes both gating findings from the second review cycle. Both were figures in a record that did not match the artifact the record cites. ## Corrected - **Referral count.** The limitations record said `status-label-normalization` produced nine adjudication referrals. The committed orchestrator report carries ten, two per run across five runs, and six sibling records already said ten. The limitations record was the sole outlier, and it is the one document #58 designates as an explicit input to #59. - **Input-token series.** The recorded drift series opened with 32,507, which appears in no retained artifact: it came from the earliest batch, whose raw output was overwritten before the artifact path was version-scoped. The retained series is 32,573, 32,955, 33,166, then 32,464. Both records now list the retained values and disclose the unretained one as not re-derivable, the same disclosure the calibration record already makes for that batch. - **Timeout rationale.** It justified the 300 s timeout against a 55.4 s worst latency. A raw stdout record carries no latency field, and that observation came from a superseded batch whose per-attempt records were never written under the artifact path, so it cannot be re-derived. Restated against 51.2 s, which is both the worst in any committed report and the worst in any retained per-attempt record, making the margin 5.9x rather than 5.4x. - **Overwrite-guard claim.** The frozen record implied the runner refuses to overwrite every retained output. It refuses on the raw stdout artifacts; the per-attempt, report, and baseline-report paths are still written unconditionally and are protected only because the frozen invocation gives them the same `-` stem, so the raw collision fires first. The record now says exactly that. ## Deferred findings recorded rather than fixed Two limitations are added instead of code changes, because the review did not gate on either. - Every measured figure has several hand-maintained homes and nothing checks the restatements against the reports. Two cycles each found one drifted figure. The duplicated *prose* caveats must stay — a reader reaching one report should not have to find another file to learn its grading is not a signal. The duplicated *numbers* should not, and two remedies are named: one canonical home per figure, or a test asserting documented figures match their report. - Widening the overwrite guard to every output path. Both are inputs to #59. Until one lands, a number in a prose record is a restatement and the committed report is the source. --- CHANGELOG.md | 2 + .../baseline/v1/COST-CEILING-PROPOSAL.md | 12 +++--- review-suite/evals/baseline/v1/LIMITATIONS.md | 42 +++++++++++++++++-- .../baseline/v1/frozen-configuration.json | 4 +- 4 files changed, 49 insertions(+), 11 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index e14f8dc..d0eda61 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,7 +6,9 @@ summary: Chronological history of repository and skill changes. ## 2026-07-26 — Added the replay evaluator, then froze the v1 baseline configuration +- fix: reconcile every recorded figure with its retained artifact - fix: attribute the pilot to a reproducible commit and correct the records + (`3a8388d42e355e4bc9731b98b6dcd42ffd13ff2f`) - feat: report the stratum a run evaluated (`2ae0d23c18f247f49d3cc5e76f26d1cf9610c83e`) - fix: make the frozen baseline record auditable, and measure the envelope diff --git a/review-suite/evals/baseline/v1/COST-CEILING-PROPOSAL.md b/review-suite/evals/baseline/v1/COST-CEILING-PROPOSAL.md index e1e5fe0..09efd4a 100644 --- a/review-suite/evals/baseline/v1/COST-CEILING-PROPOSAL.md +++ b/review-suite/evals/baseline/v1/COST-CEILING-PROPOSAL.md @@ -68,11 +68,13 @@ upper end of the observed range; its clean controls will sit at the lower end. The proposal therefore uses the **worse** of the two measured cases for the orchestrator stratum. -Absolute input-token counts drift slightly across batches — 32,507, 32,955, -33,166, then 32,464 for the same case — because reported input includes -runtime-side prompt overhead that the payload does not control. Treat the -within-batch difference as the measurement and the absolute count as -approximate. +Absolute input-token counts drift slightly across the four batches whose raw +output is retained — 32,573, 32,955, 33,166, then 32,464 for the same case — +because reported input includes runtime-side prompt overhead that the payload +does not control. Treat the within-batch difference as the measurement and the +absolute count as approximate. A fifth value, 32,507, was observed on the +earliest batch, whose raw output was overwritten before the artifact path was +version-scoped, so it is not re-derivable. ## Proposal diff --git a/review-suite/evals/baseline/v1/LIMITATIONS.md b/review-suite/evals/baseline/v1/LIMITATIONS.md index 60b3c7f..f5c2bb0 100644 --- a/review-suite/evals/baseline/v1/LIMITATIONS.md +++ b/review-suite/evals/baseline/v1/LIMITATIONS.md @@ -68,7 +68,7 @@ demonstrated both halves of what that means: adjudication referrals. - **Confirmed independently on a second case.** `status-label-normalization` is carried by the orchestrator pilot stratum and deliberately left uncalibrated - as a control. Over five attempts it scored recall **0.0** with **nine + as a control. Over five attempts it scored recall **0.0** with **ten adjudication referrals**, while verdict stability was 1.0 and the reviewer gated the change on every attempt. Two cases, ten attempts, one conclusion: an uncalibrated expectation reports a number about itself. @@ -210,9 +210,12 @@ Two further measurements bear on any cost figure quoted from here: number of findings it warrants. A stratum's cost is not predictable from its packet sizes alone. - **Reported input tokens include runtime-side prompt overhead.** The same - case's reported input drifted across batches — 32,507, 32,955, 33,166, then - 32,464 — with the payload unchanged. Treat within-batch differences as - measurements and absolute counts as approximate. + case's reported input drifted across the four batches whose raw output is + retained — 32,573, 32,955, 33,166, then 32,464 — with the payload unchanged. + Treat within-batch differences as measurements and absolute counts as + approximate. A fifth value, 32,507, was observed on the earliest batch, whose + raw output was overwritten before the artifact path was version-scoped; it is + not re-derivable and is recorded here only for completeness. ## 10. A stratum boundary is not a rounding difference @@ -220,3 +223,34 @@ A change of runtime, runtime version, model, target skill, dependency closure, or run count creates a new stratum. Comparing across one of those boundaries is an invalid comparison, not a noisier one. Every report records its closure membership and digest so a stratum can always state what it evaluated. + +## 11. Every measured figure has several hand-maintained homes + +The pilot's numbers appear in the committed reports and are then restated in +this record, in the cost-ceiling proposal, in the calibration record, in the +strata README, and in the frozen configuration. Nothing checks those +restatements against the reports, and two review cycles on this candidate each +found a figure that had drifted — a referral count and an input-token series, +both corrected. + +The restated **prose** caveats must stay duplicated: a reader who reaches one +report should not have to find another file to learn that its grading is not a +signal, or that the connector stratum is deferred. The restated **numbers** +should not be. Two remedies are open, and neither is done here: give each figure +one canonical home and reference it, or add a test asserting that every +documented figure matches the committed report it came from. + +Recorded as a deferred finding and an input to #59 rather than fixed, because +the review that raised it did not gate on it. Until one of those remedies lands, +treat a number in a prose record as a restatement and the committed report as +the source. + +## 12. The overwrite guard covers raw artifacts, not every output path + +`runner.refuse_to_overwrite_artifacts` fails before launching any attempt when a +run would overwrite a retained raw stdout artifact. The per-attempt records, the +aggregate report, and the baseline report are still written unconditionally. The +frozen invocations give every output the same `-` stem, +so in practice the raw-artifact collision fires first and nothing is lost — but +that is the invocation's property, not the runner's. Widening the guard is a +deferred finding. diff --git a/review-suite/evals/baseline/v1/frozen-configuration.json b/review-suite/evals/baseline/v1/frozen-configuration.json index 8fc4f1c..d0b8275 100644 --- a/review-suite/evals/baseline/v1/frozen-configuration.json +++ b/review-suite/evals/baseline/v1/frozen-configuration.json @@ -23,13 +23,13 @@ "runs_per_case": 5, "runs_per_case_rationale": "A denominator of one cannot measure a stochastic reviewer, and this suite has already observed the same configuration return `blocked` on one attempt and a merge verdict on the next for the case that tests refusal on incomplete evidence. Five is the smallest count that makes a per-case verdict-stability figure mean anything. Measured at five runs per case, the pilot observed verdict stability 1.0 and finding stability 1.0 on every case, so five did not surface instability on these packets - a statement about these packets, not a licence to reduce the count. The instability #50 observed was on a refusal-on-incomplete-evidence case, a class no scored stratum here contains.", "timeout_seconds": 300, - "timeout_rationale": "Worst latency in any committed pilot report is 51.2 s, on the orchestrator stratum's larger packet. The slowest attempt observed at any point across this ticket's 72 pilot attempts was 55.4 s, also on that packet. 300 s is roughly 5.4 times the worst observed. No pilot attempt timed out and none came within a factor of five of the limit.", + "timeout_rationale": "Worst latency in any committed pilot report, and the worst in any retained per-attempt record, is 51.2 s - on the orchestrator stratum's larger packet. 300 s is roughly 5.9 times that. No pilot attempt across this ticket's 72 attempts timed out, and none came within a factor of five of the limit. Latency is not carried in a raw stdout record, so a figure from a superseded batch whose per-attempt records were not retained under the artifact path is not re-derivable and is not relied on here.", "retry_policy": "none", "retry_policy_rationale": "A failed attempt is recorded as an evaluation failure and never retried. Retrying would silently replace an unstable attempt with a luckier one and would bias every stability figure toward agreement.", "max_output_bytes": 4000000, "process_isolation": "one fresh executor process per attempt", "cost_ceiling_exhaustion_policy": "Exceeding a stratum's preregistered ceiling stops further runs in that stratum and records an incomplete baseline for it. Repetitions are never reduced after outputs are visible.", - "artifact_retention": "Raw executor output is retained outside git at review-suite/evals/artifacts//-/.run-.stdout.json, with the run's per-attempt records beside it at /-.attempts.jsonl. The commit and corpus version are in the path deliberately: an artifact name carries only the case and run number, so re-running a stratum into an unscoped directory silently replaces output a committed record may already cite, which happened once and cost a calibration source run. The runner now refuses before launching any attempt when a run would overwrite retained output, so the failure costs nothing.", + "artifact_retention": "Raw executor output is retained outside git at review-suite/evals/artifacts//-/.run-.stdout.json, with the run's per-attempt records beside it at /-.attempts.jsonl. The commit and corpus version are in the path deliberately: an artifact name carries only the case and run number, so re-running a stratum into an unscoped directory silently replaces output a committed record may already cite, which happened once and cost a calibration source run. Before launching any attempt the runner refuses a run that would overwrite a retained raw stdout artifact, so the failure costs nothing. That check covers the raw records specifically; the per-attempt, report, and baseline-report paths are still written unconditionally, and are protected only because the frozen invocation gives them the same - stem, so the raw-artifact collision fires first. Widening the guard to every output path is recorded as a deferred finding rather than done here.", "invocation_template": "just eval-review-suite '' --corpus review-suite/evals/strata/ --runs --timeout --artifact-dir review-suite/evals/artifacts//- --attempts-out review-suite/evals/artifacts//-.attempts.jsonl --report-out ", "invocation_note": "The recipe forwards extra arguments to the runner. Naming an executor alone is not the frozen configuration: --corpus defaults to the protocol-proof corpus and --runs to 1, so the abbreviated command silently evaluates a different corpus once." },