Curate the scored corpus and capture the frozen v1 review baseline

[shaug]

Parent epic: #49

Blocked by: #50

Goal

Use the evaluator produced by #50 to curate and freeze a representative result-blind corpus, calibrate its private grading expectations, and capture a reproducible behavioral baseline for the unmodified v1 review suite.

This ticket ends with corpus and baseline artifacts. #59 owns baseline interpretation, v2 mechanism decisions, preregistered v2 gates, and downstream issue revisions.

Inputs from #50

Require these verified outcomes before starting:

• provider-neutral fresh-process executor protocol;
• automated contamination checks;
• deterministic protocol/grading-interface tests;
• at least one documented real-runtime executor;
• versioned corpus/expectation/provenance format;
• machine-readable per-run and aggregate reporting;
• payload assembly that carries a target skill's declared dependency closure and records which closure was sent; and
• usage accounting that includes cache-creation and cache-read tokens, plus recorded runtime/model identity per real attempt.

If any input is absent or incompatible with result-blind replay, return blocked rather than repairing evaluator infrastructure inside this ticket.

#50's real-runtime smoke evaluation proves the protocol only. It is not a capability, quality, or cost signal and must not be used as a prior, a sanity check, or a cost estimate for this ticket's pilot or baseline.

Its verdicts are demonstrably not repeatable. Across #50's smoke attempts the case testing refusal on incomplete evidence returned blocked in one run and a merge verdict in another. Every stability figure it reports rests on a denominator of 1. Treat that instability as a reason to set a run count per case sufficient to measure it, not as a finding about reviewer quality.

Declare a target per stratum

Each scored stratum must declare the target skill it measures, and the corpus must record that target alongside the dependency closure the payload sends for it.

This is load-bearing rather than bookkeeping. A skill whose contract requires sibling skills cannot be evaluated from its own directory alone: review-code-change requires review-solution-simplicity, review-correctness, and review-code-simplicity to be readable and returns an aggregate blocked result naming any missing one. A stratum targeting the orchestrator therefore sends a materially larger payload than a stratum targeting a single self-sufficient lens, and an expectation set that does not account for this grades a compliant reviewer as wrong.

Choosing which target each scored stratum measures is this ticket's decision. #50 owns only the payload mechanism and the self-consistency of its own synthetic cases. The closure is declared in the corpus rather than in code, so a target can be swapped without a code change.

Measured closure size to size the envelope against

From #50's protocol-proof corpus, for target_skill: review-code-change with the three lenses declared as its closure:

• 8 documents — each lens SKILL.md plus its rubric, alongside review-code-change/SKILL.md and references/orchestration-protocol.md;
• 41,588 characters of skill text, digest 9b2805f14cdd6158; and
• 191,422 input tokens across 6 attempts.

These are payload-size measurements, not baseline results. Use them to size a per-stratum cost ceiling before the pilot; do not treat the token figure as a cost estimate, since it predates a fixed-accounting run at this ticket's own run count.

Corpus composition

Freeze a versioned corpus with provenance, retention authority, reviewer-visible artifacts, private grading evidence, and stable root-cause identifiers. It must include at least:

Source of ground truth, and the stratum label it forces

This repository has no pull-request review history: across all of its PRs there are zero reviews, zero review threads, and zero comments. There is no in-repo connector material to curate, and none may be invented — a corpus of fabricated escapes would reproduce exactly the defect this epic exists to correct.

Source real, already-adjudicated review findings from these repositories instead:

• shaug/atelier — public, ~899 owner-authored review comments. This is the source in use.
• this suite's own delivery history — real adjudicated findings from Build the result-blind review replay evaluator and corpus contract #50's review cycles, including one defect that survived an aggregate clean verdict and was caught by CI.
• shaug/eldritchdark — private, ~361 reviews. Assessed and excluded; not read. The public sources covered every required case class with clean provenance, so reading a private repository bought nothing and carried disclosure risk into a public repo. Do not read it unless a required class provably cannot be sourced publicly, and record that finding first.

Both carry human review by the repository owner, not connector output. Label the resulting stratum human-review, never connector, in the corpus metadata and in every report. A private repository that does contain connector review history was considered and deliberately excluded on third-party authority and disclosure grounds; do not seek it out, name it, or derive from it.

The connector stratum is therefore deferred, not satisfied. Record its absence as an explicit, prominent limitation in the baseline limitations record so #59 interprets the baseline knowing that connector-escape recall was never measured. Do not present a human-review figure as a connector-escape figure, and do not blend the two.

Sanitize on the way in. This repository is public, and any private source would make sanitization irreversible if it failed: retain only minimized reproductions that preserve the failure shape and carry no business logic, domain identifiers, customer context, credentials, or hidden reasoning. When a case cannot be minimized without losing the behavior it demonstrates, drop it and record the exclusion rather than weakening the sanitization rule.

Correctness and verification

Read every "connector" slot below as satisfied by an equivalent adjudicated human-review finding from the designated sources, subject to the labeling and sanitization rules above.

• three accepted material escapes, including:
  • one multi-file contract or untouched-consumer propagation failure;
  • one concurrency, retry, idempotency, transaction, or data-integrity failure; and
  • one validation gap where passing tests did not exercise the changed risk;
• two clean correctness controls; and
• two adjudicated rejected, deferred, speculative, or polish-only findings as negative controls.

This suite's own delivery history is admissible source material and should be used where it fits. #50 produced real adjudicated findings across seven review cycles on real candidates, including one defect that survived an aggregate clean verdict and was then caught by CI — a genuine escape past a clean review, in this repository, with complete provenance and no retention question.

Existing simplicity lenses

• two demonstrated whole-solution over-engineering cases and two requirement-justified near-miss controls for review-solution-simplicity;
• two demonstrated local code-complexity/reuse cases and two behavior-clarifying or non-material near-miss controls for review-code-simplicity.

Existing synthetic fixtures may satisfy a slot only when they exercise the same reviewer behavior through the real runtime. At least three scored cases must require reasoning across multiple files or an untouched downstream surface.

Do not retain private code, credentials, customer data, hidden reasoning, or connector-only metadata. Replace any case whose retention authority cannot be established with a minimized synthetic reproduction before freezing the corpus.

Progress: all four strata populated; owner adjudication is the only remaining gate

Batch 1 is merged (PR #62, merge commit 4788c60). It delivered the strata layout, the per-stratum pilot envelope, grader calibration, the frozen-configuration record, and the supporting documents under review-suite/evals/baseline/v1/ — SOURCING.md, CALIBRATION.md, COST-CEILING-PROPOSAL.md, ADJUDICATION-PLAN.md, LIMITATIONS.md.

Batch 2 is merged (PR #66, merge commit c4e01d3). It populated s1-correctness-orchestrator with 7 cases (3 accepted escapes, 2 clean controls under the settled standard, 2 negative controls), each adjudicated by an executable oracle under review-suite/scripts/evals/oracles/ — a runnable module independent of the reviewer's model, not an opinion. The stratum ships scored: false, every expectation calibrated: false, and no case has been run through any runtime, for the reason recorded immediately below.

Batches 3–4 are merged as one delivery (PR #73, merge commit 05bea2e). It populated s2-solution-simplicity-lens and s3-code-simplicity-lens, 4 cases each. Corpus minima are now met — all four strata are populated. Neither of these two strata has an executable oracle for its subject ("over-engineered", "unnecessary duplication"), so their independent adjudication needs the owner directly, not a second implementing context. Every case's source citation below was independently verified to resolve to a real PR, review comment, or commit — not just quoted from provenance prose.

Owner adjudication — 8 of 8 CONFIRMED, complete

The owner adjudicated all 8 cases directly, independent of the recommendations below — case 7 from genuine recollection of the underlying PR, not from trusting the framing offered. Full reasoning is in the audit-trail comment. Standing standard applied throughout: over-engineering is implementing solutions to unspecified/unjustifiable requirements or protecting against improbable edge cases; a simpler design is not automatically correct if it compromises the data model or creates a one-way door. This standard is a required input to #59, not scoped to these 8 cases alone.

s2-solution-simplicity-lens:

1. reconciliation-outcome-type — CONFIRMED clean. atelier PR #417 comment, self-accepted, implemented in 668f2c7. Enums over booleans is the general case, not the exception: typed, self-descriptive, expandable, and boolean fields becoming boolean parameters is its own anti-pattern.
2. record-status-transition-guard — CONFIRMED clean. atelier PR #277 comment, accepted in this reply, implemented in 79703c5. Protecting against an error case within the scope of the requirements is good engineering, not over-engineering.
3. registry-client-layering — CONFIRMED material (rc.three-client-concepts-duplicate-binding). atelier PR #410 comment, accepted in f631cb0. Reframed the standard rather than accepting it as given: the defect is not "reaching through an abstraction barrier" per se, it is failing to question and re-justify the barrier itself; a barrier that holds up under scrutiny should be maintained, not locally worked around. Cross-reference with case 7 resolved: case 7's local duplication is the symptom of this case's design defect, not an incidental double-count — see the note below both cases.
4. setup-service-path-gateway — CONFIRMED material (rc.gateway-abstracts-a-pure-function-with-one-implementation). atelier PR #160 comment, accepted in fcbf469. The mirror case of revert: remove modular code review contract #3: an abstraction that provides no obvious value and only obfuscates the implementation is the failure mode, not abstraction itself — good engineering is designing good abstractions, not more of them.

s3-code-simplicity-lens:
5. compat-accessor-boundary-duplication — CONFIRMED clean. atelier PR #630 comment, responding to an earlier comment on the same thread that raised a typeless-seam concern this change resolves. Preferring strong types, edge validation into strong types, and domain-modeling types over a loose seam is the standard, not a stylistic preference.
6. env-inventory-bullet-format — CONFIRMED clean. atelier PR #443 comment. Markdown must stay readable in raw form — that is the actual point of the format over a full markup language, and the same reason prose is wrapped and tables are formatted rather than left to a renderer.
7. metrics-label-formatting-duplication — CONFIRMED material (rc.label-expression-copied-into-two-new-functions). Same atelier PR #410 comment as case 3, read at the local-implementation level. Confirmed from direct recollection of this PR and the difficulty of getting the implementing agent to understand the point of the abstraction barrier at the time — independent of the framing offered here.
8. watcher-check-policy-duplication — CONFIRMED material (rc.local-policy-duplicated-across-call-sites). This repository's own commit 9351619.

Case 3 / case 7 relationship — resolved. Case 7's local duplication is not an incidental double-count of case 3's design defect; it is that defect's downstream symptom. A design that doesn't fit the requirement's actual shape forces the local implementation to twist itself into a suspect shape to accommodate it. Preferred fix order: solve the whole-design defect (case 3) first; the local symptom (case 7) is worth showing separately precisely because it demonstrates the shared root cause, not despite that.

Baseline-interpretation caveat, carried to #59: because case 7 is causally downstream of case 3, recall on the two is likely correlated rather than independent — a reviewer that catches the design flaw is more likely to also catch its local symptom, and one that misses it may only surface the narrower local finding or neither. Reporting them as two independent recall samples could overstate the precision of any aggregate recall figure derived from this pair.

Adjudication is now complete for all 8 cases. This will be folded into the corpus JSON (adjudication.second per case) as part of the frozen scored-baseline delivery — not a separate implementation batch.

Grading method — SETTLED by the owner: score three-way

Batch 2 surfaced a real conflict: calibrating the grader against a case requires observing that case's actual reviewer prose, but a scored case must never have been observed before scoring — otherwise the baseline measures a case the corpus author already looked at, the same contamination the payload-blindness tests exist to prevent, entering through the grader instead. Batch 1 demonstrated the stakes concretely: the same real defect measured recall 0.0 uncalibrated and 1.0 calibrated on identical reviewer behavior.

Resolution: score every case three-way — matched, missed, or referred for adjudication — without calibrating on scored-case prose first. A grader miss must surface as a labeled referred outcome, not silently become a false reviewer-miss. This reuses the adjudication_required output already built rather than sacrificing corpus size (rejected: splitting each class into calibrate/score halves) or gambling on untuned formulations (rejected as the sole method for v1: relying on calibration transferring from the pilot, unproven at corpus scale and biased downward if it fails).

Once enough scored runs exist, check post hoc whether transfer from pilot calibration would have matched the referred bucket — that evidence should inform #59's v2 grading design, not gate this baseline.

Corpus population is complete; no further implementation batch is expected. Both owner inputs that gated the scored baseline are now satisfied:

• Per-stratum cost ceiling — APPROVED by the owner at 15.00 USD total for the three declared strata, allocated 9.00 / 3.00 / 3.00 as proposed below. Treat 15.00 USD as the preregistered hard ceiling: exceeding it stops further runs and records incomplete baseline evidence rather than silently reducing repetitions. Measured proposal from the pilot: 9.00 USD for s1-correctness-orchestrator, 3.00 for s2-solution-simplicity-lens, 3.00 for s3-code-simplicity-lens, 15.00 for all three. Ceilings assume every attempt is cold, since a case's first attempt costs 3–4× the rest under prompt caching. See COST-CEILING-PROPOSAL.md.

• Clean-control standard — SETTLED by the owner. A clean control must be an adjudicated-rejected finding: a case where a finding was actually raised and the owner dispositioned it as not material. The recorded rejection is the evidence of cleanliness, and a reviewer that re-raises the rejected finding is charged a false alarm.

  Do not treat "no review comments" as evidence of cleanliness. Absence of comment is ambiguous between reviewed-and-clean and nobody-looked, and a control whose cleanliness rests on it can charge a false alarm against a reviewer that correctly found a real, unnoticed defect. That would punish correct behavior — the same defect as grading a contract-faithful reviewer wrong, arriving from the opposite direction.

  Residual gap, record it as a limitation: this standard measures re-raising an adjudicated-immaterial finding. It does not measure whether a reviewer invents a novel finding on a wholly clean diff. False-alarm rate under this standard is therefore a lower bound on invention, and must be reported as such rather than as a general false-alarm rate.

• Independent adjudications — COMPLETE, 8 of 8 CONFIRMED, see the "Owner adjudication" section above. s1-correctness-orchestrator's 7 cases are settled by executable oracle (review-suite/scripts/evals/oracles/), not owner adjudication, per batch 2. ADJUDICATION-PLAN.md records the method that produced these recommendations.

Grader calibration

Before scoring v1:

• obtain two independent adjudications for each material root cause, accepted non-finding, severity, and allowed equivalent formulation;
• resolve disagreements in a recorded adjudication note;
• test the grader against paraphrases, overlapping symptoms, duplicate reports, partially correct claims, and plausible false positives;
• keep reviewer-visible inputs physically and structurally separate from private expectations; and
• reject a corpus version if an automated contamination test can recover expected outcomes from the executor payload.

The target reviewer must never see case labels that imply outcome, expected findings, prior review conclusions, connector dispositions, or grader data.

Frozen baseline protocol

Delivery stops at the freeze boundary

Steps 1 and 2 below are in scope for an implementing run. Steps 3–6 are not, and must not be started without two things the implementer cannot supply itself:

• an explicit per-stratum cost ceiling preregistered by the repository owner, since a scored run spends real money and the ceiling must be fixed before any scored output is examined; and
• the two independent adjudications per material root cause required below. A single implementing context generating both sides of an adjudication does not satisfy this and must not pretend to.

An implementing run therefore delivers: the curated and sanitized corpus, provenance and retention records, the grader calibration cases, contamination audits, the unscored pilot, and a concrete per-stratum cost-ceiling proposal grounded in real pilot numbers. It then stops and reports, leaving the ceiling and the adjudications to the owner. Reaching that boundary in good order is a successful outcome, not a failure.

1. Run an unscored pilot on separate calibration cases to establish executor compatibility, timeout behavior, and a cost/latency envelope. Establish the envelope per stratum: an orchestrator-targeted stratum's per-attempt payload and cost differ materially from a single-lens stratum's, so one blended figure is not a usable ceiling. Derive it from this ticket's own pilot against the fixed usage accounting, never from a figure predating that fix.
2. Before examining scored outputs, freeze:
   • exact suite commit;
   • corpus and grader versions;
   • runtime/model/version;
   • executor version;
   • target skill and dependency closure per stratum;
   • run count per case;
   • timeout and retry policy;
   • cost ceiling per stratum; and
   • metrics to report.
3. Run the unmodified v1 suite in fresh isolated processes.
4. Preserve raw structured results outside git when sensitive or large; commit the immutable configuration and compact aggregate report.
5. Report per-case and aggregate recall, false-clean rate, false-positive rate, verdict stability, unique discovery contribution, malformed/blocked/runtime-failure rate, latency, and available cost.
6. Record confidence/uncertainty appropriate to the observed run count; do not present point estimates as exact model capability.

A runtime/model change creates a new baseline stratum. Never silently compare unlike strata. A change of target skill or dependency closure likewise creates a new stratum: strata whose payloads differ in composition are not comparable, and reporting them as one figure is an invalid comparison.

Failure behavior

• Missing retention authority excludes a case before scoring.
• Grader disagreement that cannot be resolved excludes or marks the case unscorable; it is not a reviewer miss.
• A scored case is never calibrated on its own prose before scoring. Score every scored case matched, missed, or referred for adjudication; a grader miss that stems from unmet formulations is a referral, not a silent reviewer-miss.
• Contamination invalidates the affected run and corpus version.
• Runtime, timeout, malformed-output, and protocol failures remain separate from valid clean/changes-required/blocked reviews.
• Exceeding the preregistered cost ceiling stops further runs and records incomplete baseline evidence; it does not silently reduce repetitions after outputs are visible.
• A reviewer that correctly refuses a merge verdict under its own contract — including an aggregate blocked result naming a skill absent from its payload — is a correct outcome for that payload, never a reviewer miss. An expectation set that scores such a refusal as wrong is a corpus defect and invalidates the affected stratum.
• Baseline limitations are explicit inputs to Use the v1 baseline to finalize and preregister the review v2 graph #59.

Expected change surface

• Versioned scored corpus and private expectation layout under the canonical review-suite/ evaluation area from Build the result-blind review replay evaluator and corpus contract #50.
• Sanitized/minimized case artifacts and provenance/retention records.
• Grader calibration cases and adjudication records.
• Frozen v1 configuration, per-attempt controlled artifacts, compact report, and limitations record.
• README and CHANGELOG.

Do not modify review prompts, rubrics, packet/result semantics, orchestration, caller workflows, downstream ticket contracts, or native graph edges.

Validation and delivery boundary

• Run just audit-review-corpus before every pilot or scored launch.
• Run just eval-review-suite '<frozen executor command>' with the preregistered baseline configuration; preserve the exact command/configuration in the committed manifest.
• Run focused grader/calibration tests through just test-review-suite.
• Run just format, just lint, and just test before publication.
• Deliver one coherent reviewable candidate containing minimized corpus artifacts, calibration/adjudication records, immutable configuration, and the compact baseline report. Keep large or sensitive raw transcripts outside git, referenced by controlled artifact identity.
• If the approved corpus cannot fit the repository's reviewability guardrails even after minimizing raw artifacts, stop and split corpus population into evidence-preserving batches before publication; do not silently omit required case classes.

Acceptance criteria

• The corpus satisfies every lens, positive-case, negative-control, and multi-file minimum above.
• Every retained case has provenance and retention authority; restricted material is absent.
• Every case is sourced from real adjudicated review history, none invented; the stratum is labeled human-review, and the deferred connector stratum is recorded as a prominent baseline limitation.
• No retained artifact derived from a private source carries business logic, domain identifiers, customer context, credentials, or hidden reasoning.
• A per-stratum cost-ceiling proposal grounded in real pilot numbers is delivered; no scored run was launched without an owner-preregistered ceiling.
• Independent adjudications come from genuinely separate parties; no single context supplied both sides.
• Independent adjudication and disagreement resolution are recorded for private expectations.
• Automated contamination tests prove executor payloads cannot expose expected outcomes or disposition labels.
• Grader calibration distinguishes equivalent root causes from overlapping symptoms, partial claims, duplicates, and plausible false positives.
• No scored case is calibrated on its own prose before scoring; every scored result reports matched, missed, or referred-for-adjudication rather than collapsing a grader miss into a reviewer miss.
• Pilot and scored cases are separate.
• Exact v1 suite, corpus, grader, runtime/model, executor, per-stratum target skill and dependency closure, run count, timeout/retry, per-stratum cost ceiling, and metrics are frozen before scored outputs are examined.
• Every scored stratum declares its target skill and the dependency closure its payload sends.
• The cost/latency envelope is established per stratum from this ticket's own pilot against fixed usage accounting, not from any pre-fix figure.
• No expectation grades a contract-faithful refusal — including an aggregate blocked naming a skill absent from the payload — as a reviewer miss.
• A real-runtime v1 baseline is captured against the exact pre-v2 suite commit. Gated: requires the owner-preregistered per-stratum cost ceiling and the independent adjudications; an implementing run stops at the freeze boundary and does not satisfy this criterion alone.
• Per-case and aggregate quality, stability, failure, latency, available cost, and uncertainty are reported.
• Runtime/model drift creates an explicit new stratum rather than an invalid comparison, as does a change of target skill or dependency closure.
• Raw sensitive/large artifacts remain controlled while the committed report is sufficient for Use the v1 baseline to finalize and preregister the review v2 graph #59 to interpret.
• Review behavior and live downstream ticket definitions remain unchanged.
• just format, just lint, and just test pass.

Non-goals

• Change review prompts, lens rubrics, verdict semantics, orchestration, or caller workflows.
• Interpret the baseline or choose v2 mechanisms; that is Use the v1 baseline to finalize and preregister the review v2 graph #59.
• Revise Make clean verdicts require passing validation and current-head lens evidence #51–Run the preregistered review v2 ablation and integration closeout #57 or mutate their native dependency graph; that is Use the v1 baseline to finalize and preregister the review v2 graph #59.
• Treat connector findings as ground truth without adjudication.
• Tune grading expectations after observing scored outputs.
• Require paid model evaluation in ordinary deterministic repository checks.
• Claim cross-model improvement by comparing different runtime strata.
• Repair evaluator infrastructure from Build the result-blind review replay evaluator and corpus contract #50, including payload assembly or usage accounting; return blocked instead.
• Cite Build the result-blind review replay evaluator and corpus contract #50's protocol smoke evaluation as a capability, quality, or cost signal.
• Invent, synthesize, or infer an "escape" that no real review actually missed.
• Derive material from the excluded private connector-bearing repository, or name it.
• Launch a scored paid baseline without an owner-preregistered per-stratum cost ceiling.
• Self-supply both sides of an independent adjudication.
• Report a human-review figure as a connector-escape figure, or blend the two strata.