feat: fold owner adjudications and run the frozen v1 baseline - #77
Merged
Conversation
…iewer-miss Implements the three-way grading method the owner settled on #58 after batch 2: score every case matched, missed, or referred for adjudication, never calibrating a scored case's own prose first. The method was recorded as policy in the ticket body but the grader never implemented it. ## The defect `grader.grade()` computed `missed_root_cause_ids` as every expected root cause not fully `matched`, which counted a root cause with a `partial` or `ambiguous` candidate finding as a flat miss — exactly the scenario the referral bucket exists to catch: a finding at the right surface, using words the shipped formulations do not recognise. Confirmed directly before fixing: that exact scenario returned `missed_root_cause_ids: ["rc.deadline"]` and `recall: 0.0`, not a referral, collapsing "the formulation didn't recognise real prose" into "the reviewer missed this." ## Fix `grade()` now computes `referred_root_cause_ids` from every partial or ambiguous candidate not otherwise matched, and excludes those from `missed_root_cause_ids`. A root cause is a scored miss only when nothing found pointed at it at all. `recall`'s formula is unchanged (matched / expected) and remains a lower bound; the fix is entirely in what counts as a miss versus a referral. `report.py` surfaces the new bucket at both levels: per-case `ever_referred_root_cause_ids`, and the aggregate `quality.referred_rate` / `quality.referred_denominator`, so a reader never has to infer a referral from an unexplained gap between matched and expected root causes. ## Tests Two new grader tests pin the exact partial and ambiguous scenarios against `missed_root_cause_ids` and `referred_root_cause_ids`, plus a genuine-miss control proving referral does not swallow a case where nothing pointed at the root cause at all. A new report test proves the aggregate and per-case surfacing. The calibration test module's requirement that a scored case be `calibrated: true` is inverted to the opposite, now-correct invariant: a scored case must never be calibrated on its own prose, per the settled method — the prior test enforced exactly the contamination the method exists to prevent. The oracle test module's `owner_required` assertions are widened to also accept `owner_confirmed`, the new state a case moves to once the owner adjudicates it directly. ## Recorded Limitation 34: this was policy-only, not code-enforced, until this commit. No scored baseline figure was published before this delivery, so nothing downstream needs correction; the record exists so #59 knows the method's enforcement history.
…ored Both gates the frozen baseline protocol required are satisfied: the owner preregistered the $15 total cost ceiling (9.00/3.00/3.00 per stratum) in batch 1, and the owner has now directly adjudicated all 8 `owner_required` cases across `s2-solution-simplicity-lens` and `s3-code-simplicity-lens`. ## Adjudication folded into the corpus Every one of the 8 cases' `provenance.adjudication.second` moves from `owner_required` to `owner_confirmed`. Each case gains an `owner_disposition` object: `outcome` (`confirmed_material` or `confirmed_clean`), a concise 2-4 sentence `reasoning` in the owner's own terms, and `source` linking to the tracker comments carrying the complete reasoning rather than duplicating it into JSON. All 8 confirmed the recommendation already on record; none was overturned. The schema (`provenance.schema.json`) gains the `owner_confirmed` enum value and the `owner_disposition` object shape. ## Case 3 / case 7: deliberate dual-level coverage, not a double-count `registry-client-layering` (s2) and `metrics-label-formatting-duplication` (s3) share one source: `atelier` PR #410 review comment `2870262209`. The owner resolved this explicitly: case 7's local duplication is case 3's design defect made visible at the implementation layer, not an independent finding that happens to share a source. Recorded as limitation 32, including the methodological consequence for any aggregate figure: recall on this pair is likely correlated rather than independent, and must carry that caveat. ## All three strata now `scored: true` `s1-correctness-orchestrator` (oracle-settled, from batch 2), `s2-solution-simplicity-lens`, and `s3-code-simplicity-lens` (owner-confirmed, this commit) all flip from `scored: false` to `scored: true`. Every expectation across all three stays `calibrated: false` permanently — that is the correct, intended state under the three-way method, not a gap. ## Recorded Limitation 33 records that both scoring gates are satisfied and points to the two tracker comments carrying the complete adjudication reasoning rather than re-deriving it here.
Both owner-gated inputs the frozen baseline protocol required are now satisfied: the $15 total cost ceiling (9.00/3.00/3.00 per stratum, preregistered in batch 1) and complete independent adjudication across all three declared strata (7 oracle-settled in s1, 4 owner-confirmed in each of s2 and s3, no overturns, previous commit). Every configuration field the protocol requires frozen before scoring - suite commit, corpus and grader versions, runtime/model/version, executor version, per-stratum target skill and dependency closure, run count, timeout/retry policy, per-stratum cost ceiling, and metrics to report - was already fixed in batch 1 and is unchanged by this commit. This transition records that both preconditions are now met and clears the `blocked_on`/`pending_owner_inputs` fields that described the prior state. This is the exact commit the scored suite runs from. Its own SHA is what each per-attempt record and aggregate report's `configuration.suite_commit` will name, per the same pattern batch 1's pilot established: commit the frozen decisions, then run from that committed tree, so the run is reproducible from a real commit rather than an uncommitted working tree. No scored output has been examined as of this commit.
## Summary - Run the unmodified v1 suite (5 runs/case, real `claude` runtime) for all three declared strata from commit 2644c12, and commit each compact report: s1-correctness-orchestrator ($1.80750125), s2-solution-simplicity-lens ($0.9657435), s3-code-simplicity-lens ($0.9029865). Total committed spend $3.67623125 of the $15.00 ceiling; every stratum finished well under its own preregistered ceiling (9.00/3.00/3.00). - Record the actual per-stratum state, spend, and a `scored_reports` block in `frozen-configuration.json`, including the case-3/case-7 correlated-recall caveat pointer between s2 and s3. - Add LIMITATIONS.md item 35: a first s1 launch was killed mid-run by an external shell timeout after 21 of 35 attempts, before any report was written. That partial run's raw artifacts were discarded (never cited by any report) and s1 was relaunched in full from the same commit and invocation. Its real spend, 1.54512325 USD, is disclosed alongside the complete run's cost so total real spend (5.2213545 USD) is reported faithfully rather than understated by the three committed reports alone. ## Why - #58's frozen baseline protocol (steps 3-6) is now authorized: both owner-gated inputs (cost ceiling, independent adjudication) were satisfied as of the prior commit. This delivers the ticket's own terminal deliverable: corpus and baseline artifacts.
- Backfill the SHA for the previous batch's final commit and add entries for the three-way scoring fix, the owner-adjudication fold, the configuration freeze, and this batch's baseline run. - Keep the changelog's newest-first, SHA-backfilled convention current before this branch is proposed for review.
## Summary - The `provenance.schema.json` description says `owner_disposition` is present only when `adjudication.second` is `owner_confirmed`, but the repository's schema-subset validator has no conditional keyword to enforce it, and no field marked it required. Add `corpus._provenance_semantics`, a cross-field check parallel to the existing `_expectation_semantics`, wired into `load_case` so it runs on every corpus load - `just audit-review-corpus`, the test suite, and the runner all get it for free. - Add two `MutatedCorpusTests` cases proving the check actually rejects both directions: `owner_confirmed` with no disposition, and a disposition under any other `second` value. ## Why - Found during the one authorized `review-code-change` cycle on this batch's final delivery (correctness lens): the current 8 `owner_confirmed` cases are all internally consistent, but nothing would have caught a future case shipping the same silent gap this ticket's adjudication record exists to prevent.
…uires ## Summary - Add LIMITATIONS.md item 36: every reported recall, false-clean, false-alarm, false-positive, and stability figure in the three committed scored reports rests on 5 attempts per case (10-35 per aggregate denominator, depending on stratum and metric). State plainly that these are this-sample outcomes, not exact model-capability numbers, with the concrete denominators cited from the reports themselves. ## Why - #58's frozen baseline protocol has six steps; step 6 explicitly requires recording confidence/uncertainty appropriate to the observed run count and forbids presenting a point estimate as exact capability. Nothing in the delivered reports or prior limitations record carried this for the actual scored figures - only the abstract policy line in frozen-configuration.json's `metrics_to_report` list. Acceptance criterion 17 ("uncertainty... are reported") requires this to be stated, not left implicit in the reports' denominators.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #58
Summary
This is #58's final delivery: it folds the owner's completed adjudications
into the corpus and runs the frozen v1 baseline protocol's remaining steps
(3-6). Both owner-gated inputs (the $15 cost ceiling and independent
adjudication for all four strata) were satisfied before this branch started,
per #58's own tracker comments.
owner_requiredcases (4 ins2-solution-simplicity-lens, 4 ins3-code-simplicity-lens) toowner_confirmed, each with a concreteowner_disposition(outcome, concise reasoning, and a link to the twotracker comments carrying the full reasoning). All 8 confirmed the
recommended disposition; none was overturned. Set
stratum.scored: trueforall three declared strata now that every case in each is adjudicated.
method the owner settled after batch 2: a
partial/ambiguous-matched rootcause was silently counted as a flat miss instead of
referred for adjudication.grader.pyandreport.pynow surfacereferred_root_cause_ids/ever_referred_root_cause_ids/referred_rateas their own bucket.clauderuntime, 5 runs/case) for all three strata from the frozen commit, and
commit the three resulting compact reports. Total spend across the three
committed reports: $3.67623125 of the $15.00 ceiling ($1.80750125/$9.00
s1, $0.9657435/$3.00 s2, $0.9029865/$3.00 s3) - every stratum finished well
under its own ceiling.
LIMITATIONS.md(item 32):
registry-client-layering(s2) andmetrics-label-formatting-duplication(s3) share source PR #410's commentby deliberate owner design (dual-level coverage of one design defect, not a
double-count), so their recall must not be treated as two independent
samples.
frozen-configuration.jsonwith real per-stratum state, spend, anda
scored_reportsblock; disclose inLIMITATIONS.md(item 35) that afirst
s1launch was killed mid-run by an external shell timeout after21/35 attempts (before any report was written) and was cleanly relaunched -
its real cost ($1.54512325) is disclosed alongside the complete run's cost.
explicitly (
LIMITATIONS.mditem 36): every reported figure rests on 5attempts per case (10-35 per aggregate denominator), and is this sample's
outcome, not an exact model-capability number.
corpus._provenance_semantics, found during this branch's oneauthorized
review-code-changecycle: the schema documentsowner_dispositionas present only whensecondisowner_confirmed, butthe repository's schema-subset validator has no conditional keyword to
enforce it. The new check runs on every corpus load, with two tests proving
it rejects both directions.
Test plan
just formatjust lint(includesskills-ref validate)just audit-review-corpus(7 corpuses)just test(213 review-suite tests, plus all skill test suites)review-code-changecycle: solution-simplicity clean; correctnessreturned one
strong_recommendationfinding (owner_dispositionenforcement gap), fixed exactly that finding.