feat: fold owner adjudications and run the frozen v1 baseline

CHANGELOG.md

@@ -4,13 +4,25 @@ summary: Chronological history of repository and skill changes.
 
 # Changelog
 
-## 2026-07-27 — Bound epic delegation, hardened command execution, populated the solution-simplicity and code-simplicity strata, enforced acceptance-gated closeout, populated the correctness stratum, and recovered carved suffixes
+## 2026-07-27 — Bound epic delegation, hardened command execution, populated the solution-simplicity and code-simplicity strata, enforced acceptance-gated closeout, populated the correctness stratum, recovered carved suffixes, folded owner adjudications, and ran the frozen v1 baseline
 
+- docs: record the small-sample caveat the frozen protocol's step 6 requires
+- fix: enforce owner_disposition exactly when owner_confirmed
+  (`e0027dd24be391706a8269d84a9766abb95ca95b`)
+- feat: run the frozen v1 baseline and record real scored results
+  (`28fb2e57474fbf776beff50f3fc3f0f5cedfcd6a`)
+- docs: freeze the v1 configuration for scoring, before any scored output
+  (`07066d22a64bb218938a60d905e52745ca717c1a`)
+- feat: fold owner adjudications into the corpus and mark all strata scored
+  (`bf99b86b3844bea2bd248bd0828283158bee85dd`)
+- fix: score a partial or ambiguous match as referred, not a silent
+  reviewer-miss (`732e975391d0ea1b92d6d1ec312bdf4fb44d5948`)
 - feat: bind epic delegation to trusted ticket skill
+  (`569b11ec60977c19c66092690ffdada0dbac1eb4`)
 - fix: execute carve commands from explicit argv
   (`7da1a75ad585bddec6be1cc4743e77a1744c4e98`)
 - fix: correct a stale reference, a stale validation entry, and an inverted case
-  (`c7a80c0fb51cc39bbb16b174a09b4cabf7d164b3`)
+  (`c7a80c0e05ea76c0a7626c02dbf0b1605da37739`)
 - fix: make the last two before-state and sanitization defects actually resolved
   (`41de65daadc5d53bfbb299cb4ecd6d040ac47ab9`)
 - fix: sanitize the repository-history case and correct the changelog order

review-suite/evals/baseline/v1/LIMITATIONS.md

@@ -770,3 +770,156 @@ across two strata found a defect at every prior stopping point; that is the
 strongest evidence yet in this record that curation discipline alone does not
 converge, and that a mechanical diff-to-expectation consistency check remains
 the more durable fix, still undone.
+
+## 32. Recall on `registry-client-layering` and `metrics-label-formatting-duplication` is not independent
+
+Both cases trace to the same source: `atelier` PR #410 review comment
+`2870262209`, read at two levels. `registry-client-layering` (`s2`) reads it as
+a whole-solution over-engineering finding - three overlapping client concepts
+that should converge on one already-bound client.
+`metrics-label-formatting-duplication` (`s3`) reads the same comment at the
+local-implementation level - the repeated argument-threading pattern the same
+reviewer separately flagged as a local reuse defect.
+
+The owner confirmed this is deliberate dual-level coverage of one causal chain,
+not an incidental double-count, and explained why in the owner's own words:
+"sometimes a local implementation smells because the whole design of the
+solution doesn't match the shape of the requirements, and the implementation
+must twist itself into suspect shapes to accommodate. prefer solving the
+whole-design first before addressing local implementation, but showing how a
+local implementation is suspect because the design itself doesn't fit can help
+emphasize the problem with both." Case 7's local duplication is case 3's design
+defect made visible at the implementation layer, not an independent finding that
+happens to share a source.
+
+**Consequence for any aggregate figure built from this pair: recall on cases 3
+and 7 is likely correlated, not independent.** A reviewer that catches the
+whole-solution design flaw is more likely to also catch its local symptom,
+because the two are causally linked rather than statistically unrelated; a
+reviewer that misses the design flaw may still surface only the narrower local
+finding, or neither. Treating five or seven scored root causes as seven
+independent Bernoulli trials when two of them are this tightly coupled
+overstates the precision of the resulting recall figure. This caveat is a
+required input to #59's interpretation of the baseline, not a defect in this
+delivery: both cases are real, both are correctly adjudicated, and neither
+should be dropped. What must not happen is reporting an aggregate recall figure
+without disclosing that this pair is not an independent sample.
+
+## 33. Owner adjudication is complete; both scoring gates are satisfied
+
+All 8 `owner_required` cases across `s2-solution-simplicity-lens` and
+`s3-code-simplicity-lens` are now `owner_confirmed`, each with a concrete
+recorded disposition (`provenance.adjudication.owner_disposition`) rather than
+only a recommendation. Every one of the 8 confirmed the recommendation already
+on record; none was overturned. Combined with `s1-correctness-orchestrator`'s 7
+oracle-settled cases, independent adjudication is complete for every populated
+stratum, and the per-stratum cost ceiling (9.00 / 3.00 / 3.00 USD, 15.00 total)
+is preregistered by the owner. Both inputs the frozen baseline protocol gated on
+are satisfied, and all three strata now declare `scored: true`.
+
+The full record of each owner disposition, including the standing
+over-engineering standard applied across all 8 cases, is in the tracker rather
+than duplicated here: see
+[the audit-trail comment](https://github.com/shaug/agent-scripts/issues/58#issuecomment-5099314609)
+and
+[its follow-up](https://github.com/shaug/agent-scripts/issues/58#issuecomment-5099339357).
+Each case's provenance carries a concise summary and links back to these two
+comments rather than re-deriving the reasoning.
+
+## 34. A miss from an unrecognized formulation was silently counted as a reviewer miss until this delivery
+
+The three-way grading method the owner settled after batch 2 - score every case
+matched, missed, or referred for adjudication, never calibrating a scored case
+on its own prose first - was recorded as policy but not implemented in the
+grader until this delivery. `grader.grade()` computed `missed_root_cause_ids` as
+"every expected root cause not matched," which counted a root cause with a
+`partial` or `ambiguous` candidate finding - exactly the case the referral
+bucket exists to catch - as a flat miss. Confirmed directly: the exact scenario
+the settled method describes (a finding at the right surface, using words the
+shipped formulations do not recognise) returned
+`missed_root_cause_ids: ["rc.deadline"]` and `recall: 0.0`, not a referral.
+
+Fixed: `grade()` now computes `referred_root_cause_ids` from every partial or
+ambiguous candidate not otherwise matched, and excludes those from
+`missed_root_cause_ids`. A root cause is now a scored miss only when nothing
+found pointed at it at all. `recall` is unchanged in formula (matched /
+expected) and remains a lower bound; the fix is entirely in what counts as a
+miss versus a referral. `report.py` surfaces the new bucket at both levels:
+`per_case[...].ever_referred_root_cause_ids` and the aggregate
+`quality.referred_rate` / `quality.referred_denominator`, so a reader never has
+to infer a referral from an unexplained gap between matched and expected root
+causes.
+
+This was found while implementing the frozen baseline, not by a separate audit
+pass: reading the settled method's own acceptance criterion ("a grader miss that
+stems from unmet formulations is a referral, not a silent reviewer-miss")
+against the code that was supposed to implement it. Recorded as a limitation
+rather than only a fix, because it means every scored-adjacent recall figure
+this suite has reported before this commit - including any figure derived by
+hand from `adjudication_required` output during earlier batches - undercounted
+recall by however many root causes were referred rather than missed. No such
+figure was published as a baseline result prior to this delivery, so nothing
+downstream needs correction; the record exists so #59 knows the method was
+policy-only, not code-enforced, until this exact commit.
+
+## 35. The first `s1-correctness-orchestrator` scored launch was killed mid-run by an operator-side shell timeout, and its partial spend is real
+
+The first attempt to run the scored `s1-correctness-orchestrator` suite (7
+cases, 5 runs each, 35 attempts) was terminated by an external shell-level
+timeout after 21 of 35 attempts completed (5 cases entirely or partially run),
+before the runner reached the point of writing `--attempts-out` or
+`--report-out`. No report or per-attempt record was ever produced from this run,
+so no committed figure is affected. The raw per-attempt `stdout.json` files this
+partial run did write were inspected only for their `usage.cost_usd` field, to
+account for spend, never for review content used to calibrate any case, and were
+then deleted; the stratum was relaunched in full, unmodified, from the same
+frozen commit (`2644c12`) and the same invocation template, this time run to
+completion. The committed `s1-correctness-orchestrator.report.json` reflects
+that one complete, coherent 35-attempt run.
+
+The killed partial run still spent real money: 1.54512325 USD across its 21
+completed attempts, on top of the 1.80750125 USD the complete run cost. Neither
+figure alone is the full cost of scoring this stratum; true total spend on
+`s1-correctness-orchestrator` across both the discarded and the complete run is
+3.35262450 USD, still comfortably under its 9.00 USD ceiling, and true total
+spend across all three strata's scoring activity this batch, including the
+discarded run, is 5.2213545 USD against the 15.00 USD total ceiling. This is
+recorded because the protocol requires reporting actual cost faithfully, and a
+reader comparing only the three committed reports' `usage.total_cost_usd`
+figures (3.67623125 USD) would understate real spend by the discarded run's
+cost. This was an operator-side execution accident, not a defect in the suite,
+the grader, or the corpus, and required no code change.
+
+## 36. Every scored figure is a small-sample point estimate, not a measured capability
+
+The frozen protocol's step 6 requires recording confidence and uncertainty
+appropriate to the observed run count, and forbids presenting a point estimate
+as exact model capability. This is stated here explicitly because the three
+committed reports do not carry a confidence interval field, and a reader who
+only opens `quality` could otherwise read `material_finding_recall` or any other
+rate as more precise than the sample size supports.
+
+Every per-case figure in `s1-correctness-orchestrator.report.json`,
+`s2-solution-simplicity-lens.report.json`, and
+`s3-code-simplicity-lens.report.json` rests on exactly 5 attempts per case -
+`per_case[...].attempts` is 5 everywhere across all 15 cases. A per-case
+`mean_recall` can therefore only take the values 0.0, 0.2, 0.4, 0.6, 0.8, or
+1.0; a single attempt landing differently moves it by a fifth. Aggregate
+denominators are larger but still small: `recall_attempts` is 15 for
+`s1-correctness-orchestrator` and 10 for each of `s2-solution-simplicity-lens`
+and `s3-code-simplicity-lens`; `false_positive_denominator` is 35, 20, and 20
+respectively. None of these support a precise probability. A recall of 0.0 or
+1.0 in this baseline means exactly what the 5 (or 10, or 15, or 35) observed
+attempts did, not that the reviewer succeeds or fails on this case class with
+certainty.
+
+Read every reported rate as **this sample's outcome**, not as the true long-run
+rate for the target skill on this case class. This applies in both directions: a
+0.0 recall case (several appear in this baseline, all correctly bucketed to
+`referred` rather than `missed` per item 34's fix) does not prove the reviewer
+can never find that root cause, and a 1.0 verdict-stability figure does not
+prove the reviewer is deterministic on that packet beyond the 5 attempts
+actually run. Widening any of these estimates - larger `runs_per_case`, a
+held-out confirmation sample, or an explicit interval calculation - is a
+candidate decision for whoever owns #59's interpretation of this baseline, not
+something this delivery resolves.