diff --git a/CHANGELOG.md b/CHANGELOG.md index c6669aa..7941231 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,13 +4,25 @@ summary: Chronological history of repository and skill changes. # Changelog -## 2026-07-27 — Bound epic delegation, hardened command execution, populated the solution-simplicity and code-simplicity strata, enforced acceptance-gated closeout, populated the correctness stratum, and recovered carved suffixes +## 2026-07-27 — Bound epic delegation, hardened command execution, populated the solution-simplicity and code-simplicity strata, enforced acceptance-gated closeout, populated the correctness stratum, recovered carved suffixes, folded owner adjudications, and ran the frozen v1 baseline +- docs: record the small-sample caveat the frozen protocol's step 6 requires +- fix: enforce owner_disposition exactly when owner_confirmed + (`e0027dd24be391706a8269d84a9766abb95ca95b`) +- feat: run the frozen v1 baseline and record real scored results + (`28fb2e57474fbf776beff50f3fc3f0f5cedfcd6a`) +- docs: freeze the v1 configuration for scoring, before any scored output + (`07066d22a64bb218938a60d905e52745ca717c1a`) +- feat: fold owner adjudications into the corpus and mark all strata scored + (`bf99b86b3844bea2bd248bd0828283158bee85dd`) +- fix: score a partial or ambiguous match as referred, not a silent + reviewer-miss (`732e975391d0ea1b92d6d1ec312bdf4fb44d5948`) - feat: bind epic delegation to trusted ticket skill + (`569b11ec60977c19c66092690ffdada0dbac1eb4`) - fix: execute carve commands from explicit argv (`7da1a75ad585bddec6be1cc4743e77a1744c4e98`) - fix: correct a stale reference, a stale validation entry, and an inverted case - (`c7a80c0fb51cc39bbb16b174a09b4cabf7d164b3`) + (`c7a80c0e05ea76c0a7626c02dbf0b1605da37739`) - fix: make the last two before-state and sanitization defects actually resolved (`41de65daadc5d53bfbb299cb4ecd6d040ac47ab9`) - fix: sanitize the repository-history case and correct the changelog order diff --git a/review-suite/evals/baseline/v1/LIMITATIONS.md b/review-suite/evals/baseline/v1/LIMITATIONS.md index 1641424..36ef572 100644 --- a/review-suite/evals/baseline/v1/LIMITATIONS.md +++ b/review-suite/evals/baseline/v1/LIMITATIONS.md @@ -770,3 +770,156 @@ across two strata found a defect at every prior stopping point; that is the strongest evidence yet in this record that curation discipline alone does not converge, and that a mechanical diff-to-expectation consistency check remains the more durable fix, still undone. + +## 32. Recall on `registry-client-layering` and `metrics-label-formatting-duplication` is not independent + +Both cases trace to the same source: `atelier` PR #410 review comment +`2870262209`, read at two levels. `registry-client-layering` (`s2`) reads it as +a whole-solution over-engineering finding - three overlapping client concepts +that should converge on one already-bound client. +`metrics-label-formatting-duplication` (`s3`) reads the same comment at the +local-implementation level - the repeated argument-threading pattern the same +reviewer separately flagged as a local reuse defect. + +The owner confirmed this is deliberate dual-level coverage of one causal chain, +not an incidental double-count, and explained why in the owner's own words: +"sometimes a local implementation smells because the whole design of the +solution doesn't match the shape of the requirements, and the implementation +must twist itself into suspect shapes to accommodate. prefer solving the +whole-design first before addressing local implementation, but showing how a +local implementation is suspect because the design itself doesn't fit can help +emphasize the problem with both." Case 7's local duplication is case 3's design +defect made visible at the implementation layer, not an independent finding that +happens to share a source. + +**Consequence for any aggregate figure built from this pair: recall on cases 3 +and 7 is likely correlated, not independent.** A reviewer that catches the +whole-solution design flaw is more likely to also catch its local symptom, +because the two are causally linked rather than statistically unrelated; a +reviewer that misses the design flaw may still surface only the narrower local +finding, or neither. Treating five or seven scored root causes as seven +independent Bernoulli trials when two of them are this tightly coupled +overstates the precision of the resulting recall figure. This caveat is a +required input to #59's interpretation of the baseline, not a defect in this +delivery: both cases are real, both are correctly adjudicated, and neither +should be dropped. What must not happen is reporting an aggregate recall figure +without disclosing that this pair is not an independent sample. + +## 33. Owner adjudication is complete; both scoring gates are satisfied + +All 8 `owner_required` cases across `s2-solution-simplicity-lens` and +`s3-code-simplicity-lens` are now `owner_confirmed`, each with a concrete +recorded disposition (`provenance.adjudication.owner_disposition`) rather than +only a recommendation. Every one of the 8 confirmed the recommendation already +on record; none was overturned. Combined with `s1-correctness-orchestrator`'s 7 +oracle-settled cases, independent adjudication is complete for every populated +stratum, and the per-stratum cost ceiling (9.00 / 3.00 / 3.00 USD, 15.00 total) +is preregistered by the owner. Both inputs the frozen baseline protocol gated on +are satisfied, and all three strata now declare `scored: true`. + +The full record of each owner disposition, including the standing +over-engineering standard applied across all 8 cases, is in the tracker rather +than duplicated here: see +[the audit-trail comment](https://github.com/shaug/agent-scripts/issues/58#issuecomment-5099314609) +and +[its follow-up](https://github.com/shaug/agent-scripts/issues/58#issuecomment-5099339357). +Each case's provenance carries a concise summary and links back to these two +comments rather than re-deriving the reasoning. + +## 34. A miss from an unrecognized formulation was silently counted as a reviewer miss until this delivery + +The three-way grading method the owner settled after batch 2 - score every case +matched, missed, or referred for adjudication, never calibrating a scored case +on its own prose first - was recorded as policy but not implemented in the +grader until this delivery. `grader.grade()` computed `missed_root_cause_ids` as +"every expected root cause not matched," which counted a root cause with a +`partial` or `ambiguous` candidate finding - exactly the case the referral +bucket exists to catch - as a flat miss. Confirmed directly: the exact scenario +the settled method describes (a finding at the right surface, using words the +shipped formulations do not recognise) returned +`missed_root_cause_ids: ["rc.deadline"]` and `recall: 0.0`, not a referral. + +Fixed: `grade()` now computes `referred_root_cause_ids` from every partial or +ambiguous candidate not otherwise matched, and excludes those from +`missed_root_cause_ids`. A root cause is now a scored miss only when nothing +found pointed at it at all. `recall` is unchanged in formula (matched / +expected) and remains a lower bound; the fix is entirely in what counts as a +miss versus a referral. `report.py` surfaces the new bucket at both levels: +`per_case[...].ever_referred_root_cause_ids` and the aggregate +`quality.referred_rate` / `quality.referred_denominator`, so a reader never has +to infer a referral from an unexplained gap between matched and expected root +causes. + +This was found while implementing the frozen baseline, not by a separate audit +pass: reading the settled method's own acceptance criterion ("a grader miss that +stems from unmet formulations is a referral, not a silent reviewer-miss") +against the code that was supposed to implement it. Recorded as a limitation +rather than only a fix, because it means every scored-adjacent recall figure +this suite has reported before this commit - including any figure derived by +hand from `adjudication_required` output during earlier batches - undercounted +recall by however many root causes were referred rather than missed. No such +figure was published as a baseline result prior to this delivery, so nothing +downstream needs correction; the record exists so #59 knows the method was +policy-only, not code-enforced, until this exact commit. + +## 35. The first `s1-correctness-orchestrator` scored launch was killed mid-run by an operator-side shell timeout, and its partial spend is real + +The first attempt to run the scored `s1-correctness-orchestrator` suite (7 +cases, 5 runs each, 35 attempts) was terminated by an external shell-level +timeout after 21 of 35 attempts completed (5 cases entirely or partially run), +before the runner reached the point of writing `--attempts-out` or +`--report-out`. No report or per-attempt record was ever produced from this run, +so no committed figure is affected. The raw per-attempt `stdout.json` files this +partial run did write were inspected only for their `usage.cost_usd` field, to +account for spend, never for review content used to calibrate any case, and were +then deleted; the stratum was relaunched in full, unmodified, from the same +frozen commit (`2644c12`) and the same invocation template, this time run to +completion. The committed `s1-correctness-orchestrator.report.json` reflects +that one complete, coherent 35-attempt run. + +The killed partial run still spent real money: 1.54512325 USD across its 21 +completed attempts, on top of the 1.80750125 USD the complete run cost. Neither +figure alone is the full cost of scoring this stratum; true total spend on +`s1-correctness-orchestrator` across both the discarded and the complete run is +3.35262450 USD, still comfortably under its 9.00 USD ceiling, and true total +spend across all three strata's scoring activity this batch, including the +discarded run, is 5.2213545 USD against the 15.00 USD total ceiling. This is +recorded because the protocol requires reporting actual cost faithfully, and a +reader comparing only the three committed reports' `usage.total_cost_usd` +figures (3.67623125 USD) would understate real spend by the discarded run's +cost. This was an operator-side execution accident, not a defect in the suite, +the grader, or the corpus, and required no code change. + +## 36. Every scored figure is a small-sample point estimate, not a measured capability + +The frozen protocol's step 6 requires recording confidence and uncertainty +appropriate to the observed run count, and forbids presenting a point estimate +as exact model capability. This is stated here explicitly because the three +committed reports do not carry a confidence interval field, and a reader who +only opens `quality` could otherwise read `material_finding_recall` or any other +rate as more precise than the sample size supports. + +Every per-case figure in `s1-correctness-orchestrator.report.json`, +`s2-solution-simplicity-lens.report.json`, and +`s3-code-simplicity-lens.report.json` rests on exactly 5 attempts per case - +`per_case[...].attempts` is 5 everywhere across all 15 cases. A per-case +`mean_recall` can therefore only take the values 0.0, 0.2, 0.4, 0.6, 0.8, or +1.0; a single attempt landing differently moves it by a fifth. Aggregate +denominators are larger but still small: `recall_attempts` is 15 for +`s1-correctness-orchestrator` and 10 for each of `s2-solution-simplicity-lens` +and `s3-code-simplicity-lens`; `false_positive_denominator` is 35, 20, and 20 +respectively. None of these support a precise probability. A recall of 0.0 or +1.0 in this baseline means exactly what the 5 (or 10, or 15, or 35) observed +attempts did, not that the reviewer succeeds or fails on this case class with +certainty. + +Read every reported rate as **this sample's outcome**, not as the true long-run +rate for the target skill on this case class. This applies in both directions: a +0.0 recall case (several appear in this baseline, all correctly bucketed to +`referred` rather than `missed` per item 34's fix) does not prove the reviewer +can never find that root cause, and a 1.0 verdict-stability figure does not +prove the reviewer is deterministic on that packet beyond the 5 attempts +actually run. Widening any of these estimates - larger `runs_per_case`, a +held-out confirmation sample, or an explicit interval calculation - is a +candidate decision for whoever owns #59's interpretation of this baseline, not +something this delivery resolves. diff --git a/review-suite/evals/baseline/v1/frozen-configuration.json b/review-suite/evals/baseline/v1/frozen-configuration.json index 109ba74..cd67023 100644 --- a/review-suite/evals/baseline/v1/frozen-configuration.json +++ b/review-suite/evals/baseline/v1/frozen-configuration.json @@ -1,7 +1,7 @@ { "record_version": "1.0", - "status": "corpus_minima_met_adjudication_partial", - "status_detail": "All three declared strata are populated: s1-correctness-orchestrator (7 cases, every one oracle-settled), s2-solution-simplicity-lens (4 cases), and s3-code-simplicity-lens (4 cases). Neither s2 nor s3 has an executable oracle for its subject; every case in both records adjudication.second: owner_required with a recommended disposition and residual risk. The per-stratum cost ceiling is preregistered at 9.00 / 3.00 / 3.00 USD, 15.00 total. The clean-control standard is settled: an adjudicated-rejected finding. The grading method is settled: score matched, missed, or referred for adjudication, never calibrating a scored case on its own prose first. No case in any populated stratum has been run through any runtime. Corpus minima being met is a necessary condition for a scored baseline, not a sufficient one: the owner's direct adjudication of s2 and s3 (8 cases) and the frozen-baseline freeze itself remain outstanding.", + "status": "v1_scored", + "status_detail": "Both owner-gated inputs were satisfied before this run launched: the per-stratum cost ceiling (9.00 / 3.00 / 3.00 USD, 15.00 total) was preregistered in batch 1, and the owner directly adjudicated all 8 owner_required cases across s2-solution-simplicity-lens and s3-code-simplicity-lens, confirming every recommended disposition with no overturns. Combined with s1-correctness-orchestrator's 7 oracle-settled cases, independent adjudication was complete for every populated stratum before any scored output was examined. The unmodified v1 suite has now been run from this exact commit (2644c12) for all three declared strata, in fresh isolated processes, via the real runtime, with the three-way scoring method applied throughout. Actual total spend was 3.67623125 USD against the 15.00 USD ceiling (1.80750125 / 9.00 on s1, 0.9657435 / 3.00 on s2, 0.9029865 / 3.00 on s3) - every stratum finished well under its own ceiling and no repetition count was reduced after output became visible. See `scored_reports` below for the committed compact report for each stratum and its retained raw artifact path.", "v1_review_behaviour_commit": "16560d807c66076fcbf3f00d3a87f543c6ae2458", "v1_review_behaviour_commit_note": "The pre-v2 review-suite commit whose behaviour a scored baseline must evaluate. This candidate modifies no file under skills/ and no v1 contract, so the evaluated closure text is identical at every commit on this branch: all three closure digests recorded below and in every pilot report are unchanged from this commit. The digest, not the commit, is the load-bearing pin - a commit can move without the evaluated text changing, and the evaluated text cannot change without the digest moving.", "pilot_suite_commit": "2ae0d23c18f247f49d3cc5e76f26d1cf9610c83e", @@ -36,7 +36,7 @@ "strata": [ { "id": "s1-correctness-orchestrator", - "state": "populated_not_scored", + "state": "scored", "target_skill": "review-code-change", "target_skill_dependencies": [ "review-solution-simplicity", @@ -75,12 +75,17 @@ ], "corpus_version": "0.1-s1-populated", "adjudication": "Every case adjudicated twice: the recorded source disposition, and an executable oracle that runs the stated requirement. No case needs the owner.", - "blocked_on": "Calibration. Every expectation is `calibrated: false` and no case has been run through any runtime, deliberately. See limitation 14: a scored stratum cannot be both calibrated and result-blind, and the owner must pick a resolution before `scored` flips to true.", - "cost_ceiling_status": "preregistered_by_owner" + "blocked_on": null, + "cost_ceiling_status": "preregistered_by_owner", + "scoring_blocked_on": null, + "actual_spend_usd": 1.80750125, + "actual_attempts": 35, + "actual_graded_attempts": 35, + "actual_spend_note": "First launch of this stratum was killed by an external shell timeout after 21 of 35 attempts (5 cases fully or partially run), before any report or attempts-out file was written. That partial run cost 1.54512325 USD across the completed attempts; its raw artifacts were discarded (no report ever cited them) and the stratum was relaunched in full from the same frozen commit and invocation. The 1.80750125 USD recorded here is the cost of the complete, coherent 35-attempt run whose report is committed; the 1.54512325 USD spent on the discarded partial run is additional real spend against the 15.00 USD total ceiling, bringing true total spend across all scoring activity on this stratum to 3.35262450 USD - still well under the 9.00 USD per-stratum ceiling." }, { "id": "s2-solution-simplicity-lens", - "state": "populated_not_scored", + "state": "scored", "target_skill": "review-solution-simplicity", "target_skill_dependencies": [], "closure_documents": 2, @@ -110,11 +115,15 @@ "setup-service-path-gateway" ], "corpus_version": "0.1-s2-populated", - "blocked_on": "Two independent adjudications for every case (source disposition as the first, the owner as the second - no oracle exists) and the three-way grading method now settled in LIMITATIONS.md item 14: no case may be calibrated on its own prose before scoring." + "blocked_on": null, + "scoring_blocked_on": null, + "actual_spend_usd": 0.9657435, + "actual_attempts": 20, + "actual_graded_attempts": 20 }, { "id": "s3-code-simplicity-lens", - "state": "populated_not_scored", + "state": "scored", "target_skill": "review-code-simplicity", "target_skill_dependencies": [], "closure_documents": 2, @@ -144,7 +153,11 @@ "watcher-check-policy-duplication" ], "corpus_version": "0.1-s3-populated", - "blocked_on": "Two independent adjudications for every case (source disposition as the first, the owner as the second - no oracle exists)." + "blocked_on": null, + "scoring_blocked_on": null, + "actual_spend_usd": 0.9029865, + "actual_attempts": 20, + "actual_graded_attempts": 20 }, { "id": "connector-escape", @@ -186,9 +199,46 @@ "consequence_for_this_baseline": "Read `false_positive_rate` as a lower bound, never as a measured rate. The unchargeable region is the changed file, its test, and its neighbours, which is where a real reviewer is most likely to point. Narrowing a surface from a path to a symbol shrinks the region and does not remove it. The one-token rule is grader behaviour this ticket may not change, so it is escalated to #59 as a candidate preregistered v2 gate; the probe.surface-token-collision calibration probe keeps the gap visible." } ], - "pending_owner_inputs": [ - "Second adjudications for every case in s2-solution-simplicity-lens (4) and s3-code-simplicity-lens (4). No oracle exists for either subject; recommended dispositions are recorded per case." + "pending_owner_inputs": [], + "scored_reports": [ + { + "report": "s1-correctness-orchestrator.report.json", + "stratum": "s1-correctness-orchestrator", + "quality_block_is_signal": true, + "actual_spend_usd": 1.80750125, + "cost_ceiling_usd": 9.0, + "retained_artifacts": "review-suite/evals/artifacts/s1-correctness-orchestrator/2644c12-0.1-s1-populated/", + "retained_per_attempt_records": "review-suite/evals/artifacts/s1-correctness-orchestrator/2644c12-0.1-s1-populated.attempts.jsonl", + "invocation": "just eval-review-suite 'python3 review-suite/scripts/evals/claude_executor.py' --corpus review-suite/evals/strata/s1-correctness-orchestrator --runs 5 --timeout 300 --artifact-dir review-suite/evals/artifacts/s1-correctness-orchestrator/2644c12-0.1-s1-populated --attempts-out review-suite/evals/artifacts/s1-correctness-orchestrator/2644c12-0.1-s1-populated.attempts.jsonl --report-out review-suite/evals/baseline/v1/s1-correctness-orchestrator.report.json" + }, + { + "report": "s2-solution-simplicity-lens.report.json", + "stratum": "s2-solution-simplicity-lens", + "quality_block_is_signal": true, + "actual_spend_usd": 0.9657435, + "cost_ceiling_usd": 3.0, + "retained_artifacts": "review-suite/evals/artifacts/s2-solution-simplicity-lens/2644c12-0.1-s2-populated/", + "retained_per_attempt_records": "review-suite/evals/artifacts/s2-solution-simplicity-lens/2644c12-0.1-s2-populated.attempts.jsonl", + "invocation": "just eval-review-suite 'python3 review-suite/scripts/evals/claude_executor.py' --corpus review-suite/evals/strata/s2-solution-simplicity-lens --runs 5 --timeout 300 --artifact-dir review-suite/evals/artifacts/s2-solution-simplicity-lens/2644c12-0.1-s2-populated --attempts-out review-suite/evals/artifacts/s2-solution-simplicity-lens/2644c12-0.1-s2-populated.attempts.jsonl --report-out review-suite/evals/baseline/v1/s2-solution-simplicity-lens.report.json", + "correlated_recall_caveat": "Case 3, registry-client-layering, shares its source PR #410 comment with s3's case 7, metrics-label-formatting-duplication - deliberate dual-level coverage of one design defect, not an incidental double-count (owner resolution recorded in LIMITATIONS.md item 32 and in this case's own provenance.owner_disposition). Recall on the pair is not an independent sample: a reviewer that catches the design flaw in case 3 is more likely to catch its local symptom in s3's case 7, so the two cases' recall figures must not be combined or presented as two independent Bernoulli trials." + }, + { + "report": "s3-code-simplicity-lens.report.json", + "stratum": "s3-code-simplicity-lens", + "quality_block_is_signal": true, + "actual_spend_usd": 0.9029865, + "cost_ceiling_usd": 3.0, + "retained_artifacts": "review-suite/evals/artifacts/s3-code-simplicity-lens/2644c12-0.1-s3-populated/", + "retained_per_attempt_records": "review-suite/evals/artifacts/s3-code-simplicity-lens/2644c12-0.1-s3-populated.attempts.jsonl", + "invocation": "just eval-review-suite 'python3 review-suite/scripts/evals/claude_executor.py' --corpus review-suite/evals/strata/s3-code-simplicity-lens --runs 5 --timeout 300 --artifact-dir review-suite/evals/artifacts/s3-code-simplicity-lens/2644c12-0.1-s3-populated --attempts-out review-suite/evals/artifacts/s3-code-simplicity-lens/2644c12-0.1-s3-populated.attempts.jsonl --report-out review-suite/evals/baseline/v1/s3-code-simplicity-lens.report.json", + "correlated_recall_caveat": "Case 7, metrics-label-formatting-duplication, is the causal local symptom of s2's case 3, registry-client-layering - the same PR #410 comment resolved at two levels by owner design, not a double-count. See LIMITATIONS.md item 32 for the full caveat; the two cases' recall figures must not be combined or treated as independent." + } ], + "scored_totals": { + "total_actual_spend_usd": 3.67623125, + "total_cost_ceiling_usd": 15.0, + "note": "Sum of the three committed scored reports' usage.total_cost_usd. This does not include 1.54512325 USD spent on a first s1 launch that was killed by an external shell timeout before any report was produced; that partial run's raw artifacts were discarded and s1 was relaunched in full from the same commit and invocation, so the committed s1 report reflects one complete, coherent 35-attempt run. True total real spend across all scoring activity this batch, including the discarded partial run, is 5.2213545 USD - still well under the 15.00 USD ceiling." + }, "pilot_reports": [ { "report": "pilot/pilot-orchestrator.report.json", @@ -229,6 +279,8 @@ "resolved_owner_inputs": [ "Per-stratum cost ceiling: preregistered at 9.00 / 3.00 / 3.00 USD, 15.00 total.", "Clean-control standard: an adjudicated-rejected finding, with false-alarm rate reported as a lower bound on invention.", - "Grading method: score every case matched, missed, or referred for adjudication; never calibrate a scored case on its own prose first." - ] + "Grading method: score every case matched, missed, or referred for adjudication; never calibrate a scored case on its own prose first.", + "Independent adjudication: complete for all four strata (7 oracle-settled in s1, 4 owner-confirmed in each of s2 and s3, no overturns)." + ], + "frozen_before_scoring_at": "2026-07-27" } diff --git a/review-suite/evals/baseline/v1/s1-correctness-orchestrator.report.json b/review-suite/evals/baseline/v1/s1-correctness-orchestrator.report.json new file mode 100644 index 0000000..bcbcf03 --- /dev/null +++ b/review-suite/evals/baseline/v1/s1-correctness-orchestrator.report.json @@ -0,0 +1,307 @@ +{ + "adjudication_required": [ + { + "candidate_root_cause_ids": [ + "rc.probe-checks-status-not-absence" + ], + "case_id": "optional-tool-probe", + "finding_id": "missing-filenotfounderror-guard", + "reason": "partial", + "run_number": 2 + }, + { + "candidate_root_cause_ids": [ + "rc.probe-checks-status-not-absence" + ], + "case_id": "optional-tool-probe", + "finding_id": "probe-unhandled-filenotfounderror", + "reason": "partial", + "run_number": 3 + }, + { + "candidate_root_cause_ids": [ + "rc.probe-checks-status-not-absence" + ], + "case_id": "optional-tool-probe", + "finding_id": "corr.probe-filenotfounderror", + "reason": "partial", + "run_number": 4 + }, + { + "candidate_root_cause_ids": [ + "rc.probe-checks-status-not-absence" + ], + "case_id": "optional-tool-probe", + "finding_id": "absent-tool-crashes", + "reason": "partial", + "run_number": 5 + } + ], + "attempts": 35, + "baseline_eligible": true, + "configuration": { + "artifacts_retained": true, + "cases": 7, + "corpus_version": "0.1-s1-populated", + "executor": "python3 review-suite/scripts/evals/claude_executor.py", + "executor_models": [ + "claude-opus-4-6[1m]" + ], + "grader_version": "1.0", + "max_output_bytes": 4000000, + "runs_per_case": 5, + "stratum": { + "grading_is_signal": true, + "ground_truth": "human-review", + "id": "s1-correctness-orchestrator", + "purpose": "Correctness and verification cases for the orchestrator target, whose payload therefore carries its three required lens skills. Every case carries the recorded source disposition as its first adjudication and an executable oracle - a runnable module independent of the reviewer's model - as its second. SCORED: both owner-gated inputs are satisfied (the $15 total cost ceiling and complete independent adjudication across all three strata), and every expectation stays `calibrated: false` permanently, per the owner-settled three-way grading method - a scored case is never calibrated on its own prose, and a grader miss surfaces as `referred_root_cause_ids` rather than a silent reviewer-miss.", + "scored": true + }, + "suite_commit": "2644c129046f00a0191270aa555f0f3560796b1a", + "target_skill": "review-code-change", + "target_skill_dependencies": [ + "review-solution-simplicity", + "review-correctness", + "review-code-simplicity" + ], + "target_skill_digest": "9b2805f14cdd6158", + "target_skill_documents": [ + "review-code-change/SKILL.md", + "review-code-change/references/orchestration-protocol.md", + "review-code-simplicity/SKILL.md", + "review-code-simplicity/references/code-simplicity-rubric.md", + "review-correctness/SKILL.md", + "review-correctness/references/correctness-rubric.md", + "review-solution-simplicity/SKILL.md", + "review-solution-simplicity/references/solution-simplicity-rubric.md" + ], + "timeout_seconds": 300.0 + }, + "failures": { + "blocked_rate": 0.0, + "malformed_output_rate": 0.0, + "output_too_large_rate": 0.0, + "protocol_mismatch_rate": 0.0, + "runtime_failure_rate": 0.0, + "spawn_failure_rate": 0.0, + "timeout_rate": 0.0 + }, + "graded_attempts": 35, + "grader_version": "1.0", + "latency": { + "count": 35, + "max_seconds": 57.21804962493479, + "mean_seconds": 28.033873178557094, + "min_seconds": 7.84957620780915, + "p50_seconds": 23.597682165913284 + }, + "per_case": [ + { + "attempts": 5, + "case_id": "dependency-hint-parser-coverage", + "ever_referred_root_cause_ids": [], + "expected_root_cause_ids": [], + "false_alarm_attempts": 0, + "false_clean_attempts": 0, + "false_positive_attempts": 0, + "finding_stability": 1.0, + "graded_attempts": 5, + "intersection_root_cause_ids": [], + "mean_recall": null, + "stability_denominator": 5, + "statuses": { + "review_result": 5 + }, + "union_root_cause_ids": [], + "verdict_stability": 1.0 + }, + { + "attempts": 5, + "case_id": "dependency-strictness-propagation", + "ever_referred_root_cause_ids": [], + "expected_root_cause_ids": [ + "rc.sibling-call-site-keeps-permissive-mode" + ], + "false_alarm_attempts": 0, + "false_clean_attempts": 0, + "false_positive_attempts": 5, + "finding_stability": 1.0, + "graded_attempts": 5, + "intersection_root_cause_ids": [], + "mean_recall": 0.0, + "stability_denominator": 5, + "statuses": { + "review_result": 5 + }, + "union_root_cause_ids": [], + "verdict_stability": 1.0 + }, + { + "attempts": 5, + "case_id": "optional-tool-probe", + "ever_referred_root_cause_ids": [ + "rc.probe-checks-status-not-absence" + ], + "expected_root_cause_ids": [ + "rc.probe-checks-status-not-absence" + ], + "false_alarm_attempts": 0, + "false_clean_attempts": 0, + "false_positive_attempts": 1, + "finding_stability": 0.8, + "graded_attempts": 5, + "intersection_root_cause_ids": [], + "mean_recall": 0.2, + "stability_denominator": 5, + "statuses": { + "review_result": 5 + }, + "union_root_cause_ids": [ + "rc.probe-checks-status-not-absence" + ], + "verdict_stability": 1.0 + }, + { + "attempts": 5, + "case_id": "post-bootstrap-module-load", + "ever_referred_root_cause_ids": [], + "expected_root_cause_ids": [], + "false_alarm_attempts": 0, + "false_clean_attempts": 0, + "false_positive_attempts": 0, + "finding_stability": 1.0, + "graded_attempts": 5, + "intersection_root_cause_ids": [], + "mean_recall": null, + "stability_denominator": 5, + "statuses": { + "review_result": 5 + }, + "union_root_cause_ids": [], + "verdict_stability": 1.0 + }, + { + "attempts": 5, + "case_id": "process-isolation-assertion", + "ever_referred_root_cause_ids": [], + "expected_root_cause_ids": [], + "false_alarm_attempts": 2, + "false_clean_attempts": 0, + "false_positive_attempts": 2, + "finding_stability": 1.0, + "graded_attempts": 5, + "intersection_root_cause_ids": [], + "mean_recall": null, + "stability_denominator": 5, + "statuses": { + "review_result": 5 + }, + "union_root_cause_ids": [], + "verdict_stability": 0.6 + }, + { + "attempts": 5, + "case_id": "session-continuation-summary", + "ever_referred_root_cause_ids": [], + "expected_root_cause_ids": [], + "false_alarm_attempts": 0, + "false_clean_attempts": 0, + "false_positive_attempts": 0, + "finding_stability": 1.0, + "graded_attempts": 5, + "intersection_root_cause_ids": [], + "mean_recall": null, + "stability_denominator": 5, + "statuses": { + "review_result": 5 + }, + "union_root_cause_ids": [], + "verdict_stability": 1.0 + }, + { + "attempts": 5, + "case_id": "stale-claim-release-guard", + "ever_referred_root_cause_ids": [], + "expected_root_cause_ids": [ + "rc.guard-skipped-when-snapshot-owner-absent" + ], + "false_alarm_attempts": 0, + "false_clean_attempts": 0, + "false_positive_attempts": 5, + "finding_stability": 1.0, + "graded_attempts": 5, + "intersection_root_cause_ids": [], + "mean_recall": 0.0, + "stability_denominator": 5, + "statuses": { + "review_result": 5 + }, + "union_root_cause_ids": [], + "verdict_stability": 1.0 + } + ], + "protocol_version": "1.0", + "quality": { + "false_alarm_denominator": 20, + "false_alarm_rate": 0.1, + "false_clean_denominator": 15, + "false_clean_rate": 0.0, + "false_positive_denominator": 35, + "false_positive_rate": 0.37142857142857144, + "material_finding_recall": 0.06666666666666667, + "recall_attempts": 15, + "referred_denominator": 35, + "referred_rate": 0.11428571428571428, + "unique_finding_contribution": [ + { + "case_id": "optional-tool-probe", + "only_some_attempts": [ + "rc.probe-checks-status-not-absence" + ] + } + ] + }, + "report_version": "1.0", + "simulation": false, + "stability": { + "mean_finding_stability": 0.9714285714285714, + "mean_verdict_stability": 0.9428571428571428, + "per_case_finding_stability": { + "dependency-hint-parser-coverage": 1.0, + "dependency-strictness-propagation": 1.0, + "optional-tool-probe": 0.8, + "post-bootstrap-module-load": 1.0, + "process-isolation-assertion": 1.0, + "session-continuation-summary": 1.0, + "stale-claim-release-guard": 1.0 + }, + "per_case_stability_denominator": { + "dependency-hint-parser-coverage": 5, + "dependency-strictness-propagation": 5, + "optional-tool-probe": 5, + "post-bootstrap-module-load": 5, + "process-isolation-assertion": 5, + "session-continuation-summary": 5, + "stale-claim-release-guard": 5 + }, + "per_case_verdict_stability": { + "dependency-hint-parser-coverage": 1.0, + "dependency-strictness-propagation": 1.0, + "optional-tool-probe": 1.0, + "post-bootstrap-module-load": 1.0, + "process-isolation-assertion": 0.6, + "session-continuation-summary": 1.0, + "stale-claim-release-guard": 1.0 + }, + "stability_denominator": 35 + }, + "usage": { + "available": true, + "reporting_attempts_cost_usd": 35, + "reporting_attempts_input_tokens": 35, + "reporting_attempts_output_tokens": 35, + "total_cost_usd": 1.80750125, + "total_input_tokens": 1145500.0, + "total_output_tokens": 39204.0 + } +} diff --git a/review-suite/evals/baseline/v1/s2-solution-simplicity-lens.report.json b/review-suite/evals/baseline/v1/s2-solution-simplicity-lens.report.json new file mode 100644 index 0000000..a04ecdb --- /dev/null +++ b/review-suite/evals/baseline/v1/s2-solution-simplicity-lens.report.json @@ -0,0 +1,240 @@ +{ + "adjudication_required": [ + { + "candidate_root_cause_ids": [ + "rc.three-client-concepts-duplicate-binding" + ], + "case_id": "registry-client-layering", + "finding_id": "sol.duplicate-threading", + "reason": "partial", + "run_number": 1 + }, + { + "candidate_root_cause_ids": [ + "rc.three-client-concepts-duplicate-binding" + ], + "case_id": "registry-client-layering", + "finding_id": "sol.duplicated-threading", + "reason": "partial", + "run_number": 2 + }, + { + "candidate_root_cause_ids": [ + "rc.three-client-concepts-duplicate-binding" + ], + "case_id": "registry-client-layering", + "finding_id": "sol.duplicate-threading", + "reason": "partial", + "run_number": 3 + }, + { + "candidate_root_cause_ids": [ + "rc.three-client-concepts-duplicate-binding" + ], + "case_id": "registry-client-layering", + "finding_id": "sol.queue-ops-duplicates-threading", + "reason": "partial", + "run_number": 4 + }, + { + "candidate_root_cause_ids": [ + "rc.three-client-concepts-duplicate-binding" + ], + "case_id": "registry-client-layering", + "finding_id": "sol.duplicate-parameter-threading", + "reason": "partial", + "run_number": 5 + }, + { + "candidate_root_cause_ids": [ + "rc.gateway-abstracts-a-pure-function-with-one-implementation" + ], + "case_id": "setup-service-path-gateway", + "finding_id": "ss.unsupported-gateway-and-dep-bag", + "reason": "partial", + "run_number": 5 + } + ], + "attempts": 20, + "baseline_eligible": true, + "configuration": { + "artifacts_retained": true, + "cases": 4, + "corpus_version": "0.1-s2-populated", + "executor": "python3 review-suite/scripts/evals/claude_executor.py", + "executor_models": [ + "claude-opus-4-6[1m]" + ], + "grader_version": "1.0", + "max_output_bytes": 4000000, + "runs_per_case": 5, + "stratum": { + "grading_is_signal": true, + "ground_truth": "human-review", + "id": "s2-solution-simplicity-lens", + "purpose": "Whole-solution over-engineering and requirement-justified near-miss cases for the self-sufficient solution-simplicity lens. No executable oracle exists for this subject - 'over-engineered' and 'requirement-justified' are design judgements a runnable check cannot decide - so every case's second adjudication is `owner_confirmed`: the owner adjudicated all four directly, confirming every recommended disposition with no overturns. SCORED: both owner-gated inputs are satisfied, and every expectation stays `calibrated: false` permanently, per the owner-settled three-way grading method.", + "scored": true + }, + "suite_commit": "2644c129046f00a0191270aa555f0f3560796b1a", + "target_skill": "review-solution-simplicity", + "target_skill_dependencies": [], + "target_skill_digest": "6257ee4448b15874", + "target_skill_documents": [ + "review-solution-simplicity/SKILL.md", + "review-solution-simplicity/references/solution-simplicity-rubric.md" + ], + "timeout_seconds": 300.0 + }, + "failures": { + "blocked_rate": 0.0, + "malformed_output_rate": 0.0, + "output_too_large_rate": 0.0, + "protocol_mismatch_rate": 0.0, + "runtime_failure_rate": 0.0, + "spawn_failure_rate": 0.0, + "timeout_rate": 0.0 + }, + "graded_attempts": 20, + "grader_version": "1.0", + "latency": { + "count": 20, + "max_seconds": 32.41287258313969, + "mean_seconds": 19.12018592285458, + "min_seconds": 9.453158125281334, + "p50_seconds": 16.976669353898615 + }, + "per_case": [ + { + "attempts": 5, + "case_id": "reconciliation-outcome-type", + "ever_referred_root_cause_ids": [], + "expected_root_cause_ids": [], + "false_alarm_attempts": 0, + "false_clean_attempts": 0, + "false_positive_attempts": 0, + "finding_stability": 1.0, + "graded_attempts": 5, + "intersection_root_cause_ids": [], + "mean_recall": null, + "stability_denominator": 5, + "statuses": { + "review_result": 5 + }, + "union_root_cause_ids": [], + "verdict_stability": 1.0 + }, + { + "attempts": 5, + "case_id": "record-status-transition-guard", + "ever_referred_root_cause_ids": [], + "expected_root_cause_ids": [], + "false_alarm_attempts": 0, + "false_clean_attempts": 0, + "false_positive_attempts": 0, + "finding_stability": 1.0, + "graded_attempts": 5, + "intersection_root_cause_ids": [], + "mean_recall": null, + "stability_denominator": 5, + "statuses": { + "review_result": 5 + }, + "union_root_cause_ids": [], + "verdict_stability": 1.0 + }, + { + "attempts": 5, + "case_id": "registry-client-layering", + "ever_referred_root_cause_ids": [ + "rc.three-client-concepts-duplicate-binding" + ], + "expected_root_cause_ids": [ + "rc.three-client-concepts-duplicate-binding" + ], + "false_alarm_attempts": 0, + "false_clean_attempts": 0, + "false_positive_attempts": 0, + "finding_stability": 1.0, + "graded_attempts": 5, + "intersection_root_cause_ids": [], + "mean_recall": 0.0, + "stability_denominator": 5, + "statuses": { + "review_result": 5 + }, + "union_root_cause_ids": [], + "verdict_stability": 1.0 + }, + { + "attempts": 5, + "case_id": "setup-service-path-gateway", + "ever_referred_root_cause_ids": [ + "rc.gateway-abstracts-a-pure-function-with-one-implementation" + ], + "expected_root_cause_ids": [ + "rc.gateway-abstracts-a-pure-function-with-one-implementation" + ], + "false_alarm_attempts": 0, + "false_clean_attempts": 0, + "false_positive_attempts": 4, + "finding_stability": 1.0, + "graded_attempts": 5, + "intersection_root_cause_ids": [], + "mean_recall": 0.0, + "stability_denominator": 5, + "statuses": { + "review_result": 5 + }, + "union_root_cause_ids": [], + "verdict_stability": 1.0 + } + ], + "protocol_version": "1.0", + "quality": { + "false_alarm_denominator": 10, + "false_alarm_rate": 0.0, + "false_clean_denominator": 10, + "false_clean_rate": 0.0, + "false_positive_denominator": 20, + "false_positive_rate": 0.2, + "material_finding_recall": 0.0, + "recall_attempts": 10, + "referred_denominator": 20, + "referred_rate": 0.3, + "unique_finding_contribution": [] + }, + "report_version": "1.0", + "simulation": false, + "stability": { + "mean_finding_stability": 1.0, + "mean_verdict_stability": 1.0, + "per_case_finding_stability": { + "reconciliation-outcome-type": 1.0, + "record-status-transition-guard": 1.0, + "registry-client-layering": 1.0, + "setup-service-path-gateway": 1.0 + }, + "per_case_stability_denominator": { + "reconciliation-outcome-type": 5, + "record-status-transition-guard": 5, + "registry-client-layering": 5, + "setup-service-path-gateway": 5 + }, + "per_case_verdict_stability": { + "reconciliation-outcome-type": 1.0, + "record-status-transition-guard": 1.0, + "registry-client-layering": 1.0, + "setup-service-path-gateway": 1.0 + }, + "stability_denominator": 20 + }, + "usage": { + "available": true, + "reporting_attempts_cost_usd": 20, + "reporting_attempts_input_tokens": 20, + "reporting_attempts_output_tokens": 20, + "total_cost_usd": 0.9657435, + "total_input_tokens": 523090.0, + "total_output_tokens": 14031.0 + } +} diff --git a/review-suite/evals/baseline/v1/s3-code-simplicity-lens.report.json b/review-suite/evals/baseline/v1/s3-code-simplicity-lens.report.json new file mode 100644 index 0000000..f362251 --- /dev/null +++ b/review-suite/evals/baseline/v1/s3-code-simplicity-lens.report.json @@ -0,0 +1,294 @@ +{ + "adjudication_required": [ + { + "candidate_root_cause_ids": [ + "rc.label-expression-copied-into-two-new-functions" + ], + "case_id": "metrics-label-formatting-duplication", + "finding_id": "cs.inline-label-duplicates-format-sample-point", + "reason": "partial", + "run_number": 1 + }, + { + "candidate_root_cause_ids": [ + "rc.label-expression-copied-into-two-new-functions" + ], + "case_id": "metrics-label-formatting-duplication", + "finding_id": "cs.inline-label-duplicates-format-sample-point", + "reason": "partial", + "run_number": 2 + }, + { + "candidate_root_cause_ids": [ + "rc.label-expression-copied-into-two-new-functions" + ], + "case_id": "metrics-label-formatting-duplication", + "finding_id": "cs.inline-label-duplicates-format-sample-point", + "reason": "partial", + "run_number": 3 + }, + { + "candidate_root_cause_ids": [ + "rc.label-expression-copied-into-two-new-functions" + ], + "case_id": "metrics-label-formatting-duplication", + "finding_id": "cs.inline-label-duplication", + "reason": "partial", + "run_number": 4 + }, + { + "candidate_root_cause_ids": [ + "rc.label-expression-copied-into-two-new-functions" + ], + "case_id": "metrics-label-formatting-duplication", + "finding_id": "cs.duplicated-label-format", + "reason": "partial", + "run_number": 5 + }, + { + "candidate_root_cause_ids": [ + "rc.local-policy-duplicated-across-call-sites" + ], + "case_id": "watcher-check-policy-duplication", + "finding_id": "cs.inline-gating-policy-duplication", + "reason": "partial", + "run_number": 1 + }, + { + "candidate_root_cause_ids": [ + "rc.local-policy-duplicated-across-call-sites" + ], + "case_id": "watcher-check-policy-duplication", + "finding_id": "cs.inline-gating-policy", + "reason": "partial", + "run_number": 2 + }, + { + "candidate_root_cause_ids": [ + "rc.local-policy-duplicated-across-call-sites" + ], + "case_id": "watcher-check-policy-duplication", + "finding_id": "cs.unused-monkeypatch", + "reason": "partial", + "run_number": 2 + }, + { + "candidate_root_cause_ids": [ + "rc.local-policy-duplicated-across-call-sites" + ], + "case_id": "watcher-check-policy-duplication", + "finding_id": "cs.inline-gating-policy", + "reason": "partial", + "run_number": 3 + }, + { + "candidate_root_cause_ids": [ + "rc.local-policy-duplicated-across-call-sites" + ], + "case_id": "watcher-check-policy-duplication", + "finding_id": "cs.unused-monkeypatch", + "reason": "partial", + "run_number": 3 + }, + { + "candidate_root_cause_ids": [ + "rc.local-policy-duplicated-across-call-sites" + ], + "case_id": "watcher-check-policy-duplication", + "finding_id": "cs.inline-gating-policy", + "reason": "partial", + "run_number": 4 + }, + { + "candidate_root_cause_ids": [ + "rc.local-policy-duplicated-across-call-sites" + ], + "case_id": "watcher-check-policy-duplication", + "finding_id": "cs.inline-gating-policy", + "reason": "partial", + "run_number": 5 + } + ], + "attempts": 20, + "baseline_eligible": true, + "configuration": { + "artifacts_retained": true, + "cases": 4, + "corpus_version": "0.1-s3-populated", + "executor": "python3 review-suite/scripts/evals/claude_executor.py", + "executor_models": [ + "claude-opus-4-6[1m]" + ], + "grader_version": "1.0", + "max_output_bytes": 4000000, + "runs_per_case": 5, + "stratum": { + "grading_is_signal": true, + "ground_truth": "human-review", + "id": "s3-code-simplicity-lens", + "purpose": "Local code-complexity, reuse, and DRY cases for the self-sufficient code-simplicity lens. No executable oracle exists for this subject - 'this is a duplication defect' and 'this apparent duplication is justified' are design judgements a runnable check cannot decide - so every case's second adjudication is `owner_confirmed`: the owner adjudicated all four directly, confirming every recommended disposition with no overturns. One pair, `metrics-label-formatting-duplication` here and `registry-client-layering` in s2, shares one source PR read at two levels; the owner confirmed this is deliberate dual-level coverage of one causal chain, not a double-count, and LIMITATIONS.md carries the resulting correlated-recall caveat. SCORED: both owner-gated inputs are satisfied, and every expectation stays `calibrated: false` permanently.", + "scored": true + }, + "suite_commit": "2644c129046f00a0191270aa555f0f3560796b1a", + "target_skill": "review-code-simplicity", + "target_skill_dependencies": [], + "target_skill_digest": "a6187d8971eaef24", + "target_skill_documents": [ + "review-code-simplicity/SKILL.md", + "review-code-simplicity/references/code-simplicity-rubric.md" + ], + "timeout_seconds": 300.0 + }, + "failures": { + "blocked_rate": 0.0, + "malformed_output_rate": 0.0, + "output_too_large_rate": 0.0, + "protocol_mismatch_rate": 0.0, + "runtime_failure_rate": 0.0, + "spawn_failure_rate": 0.0, + "timeout_rate": 0.0 + }, + "graded_attempts": 20, + "grader_version": "1.0", + "latency": { + "count": 20, + "max_seconds": 26.260644875001162, + "mean_seconds": 16.369523791619578, + "min_seconds": 9.683177208062261, + "p50_seconds": 15.369316895958036 + }, + "per_case": [ + { + "attempts": 5, + "case_id": "compat-accessor-boundary-duplication", + "ever_referred_root_cause_ids": [], + "expected_root_cause_ids": [], + "false_alarm_attempts": 0, + "false_clean_attempts": 0, + "false_positive_attempts": 0, + "finding_stability": 1.0, + "graded_attempts": 5, + "intersection_root_cause_ids": [], + "mean_recall": null, + "stability_denominator": 5, + "statuses": { + "review_result": 5 + }, + "union_root_cause_ids": [], + "verdict_stability": 1.0 + }, + { + "attempts": 5, + "case_id": "env-inventory-bullet-format", + "ever_referred_root_cause_ids": [], + "expected_root_cause_ids": [], + "false_alarm_attempts": 0, + "false_clean_attempts": 0, + "false_positive_attempts": 0, + "finding_stability": 1.0, + "graded_attempts": 5, + "intersection_root_cause_ids": [], + "mean_recall": null, + "stability_denominator": 5, + "statuses": { + "review_result": 5 + }, + "union_root_cause_ids": [], + "verdict_stability": 1.0 + }, + { + "attempts": 5, + "case_id": "metrics-label-formatting-duplication", + "ever_referred_root_cause_ids": [ + "rc.label-expression-copied-into-two-new-functions" + ], + "expected_root_cause_ids": [ + "rc.label-expression-copied-into-two-new-functions" + ], + "false_alarm_attempts": 0, + "false_clean_attempts": 0, + "false_positive_attempts": 0, + "finding_stability": 1.0, + "graded_attempts": 5, + "intersection_root_cause_ids": [], + "mean_recall": 0.0, + "stability_denominator": 5, + "statuses": { + "review_result": 5 + }, + "union_root_cause_ids": [], + "verdict_stability": 1.0 + }, + { + "attempts": 5, + "case_id": "watcher-check-policy-duplication", + "ever_referred_root_cause_ids": [ + "rc.local-policy-duplicated-across-call-sites" + ], + "expected_root_cause_ids": [ + "rc.local-policy-duplicated-across-call-sites" + ], + "false_alarm_attempts": 0, + "false_clean_attempts": 0, + "false_positive_attempts": 0, + "finding_stability": 1.0, + "graded_attempts": 5, + "intersection_root_cause_ids": [], + "mean_recall": 0.0, + "stability_denominator": 5, + "statuses": { + "review_result": 5 + }, + "union_root_cause_ids": [], + "verdict_stability": 1.0 + } + ], + "protocol_version": "1.0", + "quality": { + "false_alarm_denominator": 10, + "false_alarm_rate": 0.0, + "false_clean_denominator": 10, + "false_clean_rate": 0.0, + "false_positive_denominator": 20, + "false_positive_rate": 0.0, + "material_finding_recall": 0.0, + "recall_attempts": 10, + "referred_denominator": 20, + "referred_rate": 0.5, + "unique_finding_contribution": [] + }, + "report_version": "1.0", + "simulation": false, + "stability": { + "mean_finding_stability": 1.0, + "mean_verdict_stability": 1.0, + "per_case_finding_stability": { + "compat-accessor-boundary-duplication": 1.0, + "env-inventory-bullet-format": 1.0, + "metrics-label-formatting-duplication": 1.0, + "watcher-check-policy-duplication": 1.0 + }, + "per_case_stability_denominator": { + "compat-accessor-boundary-duplication": 5, + "env-inventory-bullet-format": 5, + "metrics-label-formatting-duplication": 5, + "watcher-check-policy-duplication": 5 + }, + "per_case_verdict_stability": { + "compat-accessor-boundary-duplication": 1.0, + "env-inventory-bullet-format": 1.0, + "metrics-label-formatting-duplication": 1.0, + "watcher-check-policy-duplication": 1.0 + }, + "stability_denominator": 20 + }, + "usage": { + "available": true, + "reporting_attempts_cost_usd": 20, + "reporting_attempts_input_tokens": 20, + "reporting_attempts_output_tokens": 20, + "total_cost_usd": 0.9029865000000001, + "total_input_tokens": 515510.0, + "total_output_tokens": 12021.0 + } +} diff --git a/review-suite/evals/contracts/provenance.schema.json b/review-suite/evals/contracts/provenance.schema.json index b1b96fa..3d5db0f 100644 --- a/review-suite/evals/contracts/provenance.schema.json +++ b/review-suite/evals/contracts/provenance.schema.json @@ -60,16 +60,43 @@ "minLength": 1 }, "second": { - "description": "The second adjudication. `oracle` means a machine settled it: the stated requirement runs, fails against the candidate, and holds once the stated root cause is corrected. `owner_required` means no oracle exists and the owner must supply it - never a fresh agent context sharing a model family with the reviewer being measured, whose errors correlate with it. `none` means neither, which no case may ship with once it is scored.", + "description": "The second adjudication. `oracle` means a machine settled it: the stated requirement runs, fails against the candidate, and holds once the stated root cause is corrected. `owner_required` means no oracle exists and the owner must supply it - never a fresh agent context sharing a model family with the reviewer being measured, whose errors correlate with it. `owner_confirmed` means the owner has directly adjudicated the case; `owner_disposition` then records the outcome. `none` means neither, which no case may ship with once it is scored.", "enum": [ "oracle", "owner_required", + "owner_confirmed", "none" ] }, "notes": { "type": "string", "minLength": 1 + }, + "owner_disposition": { + "description": "Present only when `second` is `owner_confirmed`. A concise record of the owner's actual adjudication, distinct from the `notes` field's pre-adjudication recommendation: `outcome` is the owner's confirmed verdict on materiality, `reasoning` is a short summary in the owner's own terms rather than the recommender's framing, and `source` links to the full reasoning recorded in the tracker rather than duplicating it here.", + "type": "object", + "additionalProperties": false, + "required": [ + "outcome", + "reasoning", + "source" + ], + "properties": { + "outcome": { + "enum": [ + "confirmed_material", + "confirmed_clean" + ] + }, + "reasoning": { + "type": "string", + "minLength": 1 + }, + "source": { + "type": "string", + "minLength": 1 + } + } } } } diff --git a/review-suite/evals/strata/s1-correctness-orchestrator/corpus.json b/review-suite/evals/strata/s1-correctness-orchestrator/corpus.json index 01409cb..d0c3cb7 100644 --- a/review-suite/evals/strata/s1-correctness-orchestrator/corpus.json +++ b/review-suite/evals/strata/s1-correctness-orchestrator/corpus.json @@ -11,9 +11,9 @@ "stratum": { "id": "s1-correctness-orchestrator", "ground_truth": "human-review", - "scored": false, + "scored": true, "grading_is_signal": true, - "purpose": "Correctness and verification cases for the orchestrator target, whose payload therefore carries its three required lens skills. Populated and independently adjudicated: every case carries the recorded source disposition as its first adjudication and an executable oracle as its second. NOT yet scored, for one remaining reason: every expectation is uncalibrated, and no case has been run through any runtime, because observing a scored case's prose in order to calibrate it would fit the grader to the answer. Flip `scored` to true only once the owner preregisters how that conflict is resolved." + "purpose": "Correctness and verification cases for the orchestrator target, whose payload therefore carries its three required lens skills. Every case carries the recorded source disposition as its first adjudication and an executable oracle - a runnable module independent of the reviewer's model - as its second. SCORED: both owner-gated inputs are satisfied (the $15 total cost ceiling and complete independent adjudication across all three strata), and every expectation stays `calibrated: false` permanently, per the owner-settled three-way grading method - a scored case is never calibrated on its own prose, and a grader miss surfaces as `referred_root_cause_ids` rather than a silent reviewer-miss." }, "cases": [ "dependency-hint-parser-coverage", diff --git a/review-suite/evals/strata/s2-solution-simplicity-lens/corpus.json b/review-suite/evals/strata/s2-solution-simplicity-lens/corpus.json index 21e4203..6b92276 100644 --- a/review-suite/evals/strata/s2-solution-simplicity-lens/corpus.json +++ b/review-suite/evals/strata/s2-solution-simplicity-lens/corpus.json @@ -7,9 +7,9 @@ "stratum": { "id": "s2-solution-simplicity-lens", "ground_truth": "human-review", - "scored": false, + "scored": true, "grading_is_signal": true, - "purpose": "Whole-solution over-engineering and requirement-justified near-miss cases for the self-sufficient solution-simplicity lens. Populated and sourced from adjudicated human review, with the recorded disposition as the first adjudication. NO executable oracle exists for this subject: 'over-engineered' and 'requirement-justified' are design judgements a runnable check cannot decide. Every case therefore records adjudication.second: owner_required and a concrete recommended disposition for the owner to confirm or overturn. NOT scored: every expectation is uncalibrated and no case has been run through any runtime, per the owner-settled three-way grading method - a scored case is never calibrated on its own prose, and a grader miss must surface as referred rather than a silent reviewer-miss." + "purpose": "Whole-solution over-engineering and requirement-justified near-miss cases for the self-sufficient solution-simplicity lens. No executable oracle exists for this subject - 'over-engineered' and 'requirement-justified' are design judgements a runnable check cannot decide - so every case's second adjudication is `owner_confirmed`: the owner adjudicated all four directly, confirming every recommended disposition with no overturns. SCORED: both owner-gated inputs are satisfied, and every expectation stays `calibrated: false` permanently, per the owner-settled three-way grading method." }, "cases": [ "reconciliation-outcome-type", diff --git a/review-suite/evals/strata/s2-solution-simplicity-lens/private/provenance/reconciliation-outcome-type.json b/review-suite/evals/strata/s2-solution-simplicity-lens/private/provenance/reconciliation-outcome-type.json index 887b3de..49781a0 100644 --- a/review-suite/evals/strata/s2-solution-simplicity-lens/private/provenance/reconciliation-outcome-type.json +++ b/review-suite/evals/strata/s2-solution-simplicity-lens/private/provenance/reconciliation-outcome-type.json @@ -7,7 +7,12 @@ "recorded_at": "2026-07-27", "adjudication": { "first": "shaug/atelier PR 417 review comment 2870710594, which records its own acceptance: implemented in 668f2c7, adding the typed outcome exactly as the packet describes.", - "second": "owner_required", - "notes": "No executable oracle exists for whether machinery is 'requirement-justified' rather than over-engineered - that is the judgement itself. Recommended adjudication: CLEAN / NOT MATERIAL to gate on. The packet states the caller must distinguish an intentional block from a real failure and that a boolean cannot express that distinction, which is a concrete, checkable requirement rather than a preference for more types. The residual risk: a reviewer could reasonably propose a smaller fix (e.g. a second boolean flag, or a single sentinel value) instead of a four-member enum: whether that counts as 'this change over-shoots the requirement' rather than 'this looks like unnecessary machinery' is a distinction the owner should draw, since this case's accepted non-finding treats the whole enum as justified rather than picking apart its size." + "second": "owner_confirmed", + "notes": "No executable oracle exists for whether machinery is 'requirement-justified' rather than over-engineered - that is the judgement itself. Recommended adjudication: CLEAN / NOT MATERIAL to gate on. The packet states the caller must distinguish an intentional block from a real failure and that a boolean cannot express that distinction, which is a concrete, checkable requirement rather than a preference for more types. The residual risk: a reviewer could reasonably propose a smaller fix (e.g. a second boolean flag, or a single sentinel value) instead of a four-member enum: whether that counts as 'this change over-shoots the requirement' rather than 'this looks like unnecessary machinery' is a distinction the owner should draw, since this case's accepted non-finding treats the whole enum as justified rather than picking apart its size.", + "owner_disposition": { + "outcome": "confirmed_clean", + "reasoning": "Owner confirmed: over-engineering is rarely a boolean-vs-enum question. Booleans-as-state are often the real code smell; an enum is typed, self-descriptive, expandable, and boolean fields becoming boolean parameters is its own anti-pattern. Confirms the recommended CLEAN / NOT MATERIAL disposition.", + "source": "https://github.com/shaug/agent-scripts/issues/58#issuecomment-5099314609 (owner adjudication, 8 of 8); https://github.com/shaug/agent-scripts/issues/58#issuecomment-5099339357 (case 3/7 cross-reference resolution)" + } } } diff --git a/review-suite/evals/strata/s2-solution-simplicity-lens/private/provenance/record-status-transition-guard.json b/review-suite/evals/strata/s2-solution-simplicity-lens/private/provenance/record-status-transition-guard.json index 140151f..085f65f 100644 --- a/review-suite/evals/strata/s2-solution-simplicity-lens/private/provenance/record-status-transition-guard.json +++ b/review-suite/evals/strata/s2-solution-simplicity-lens/private/provenance/record-status-transition-guard.json @@ -7,7 +7,12 @@ "recorded_at": "2026-07-27", "adjudication": { "first": "shaug/atelier PR 277 review comment 2861848880, accepted in reply 2861868165, which names implementing commit 79703c5 and the regression tests added for the fail-closed path.", - "second": "owner_required", - "notes": "No executable oracle exists for whether added robustness machinery is over-engineered versus requirement-justified. Recommended adjudication: CLEAN / NOT MATERIAL to gate on. The packet states the created record is runnable by default and picked up within seconds, which makes a silent transition failure a real hazard rather than a hypothetical one, and the sibling script already carries the identical protection, which is evidence of an established pattern rather than one-off scaffolding. Residual risk for the owner to weigh: retry count and backoff are unspecified in the packet, so a reviewer could reasonably ask for that detail without the request being about over-engineering at all." + "second": "owner_confirmed", + "notes": "No executable oracle exists for whether added robustness machinery is over-engineered versus requirement-justified. Recommended adjudication: CLEAN / NOT MATERIAL to gate on. The packet states the created record is runnable by default and picked up within seconds, which makes a silent transition failure a real hazard rather than a hypothetical one, and the sibling script already carries the identical protection, which is evidence of an established pattern rather than one-off scaffolding. Residual risk for the owner to weigh: retry count and backoff are unspecified in the packet, so a reviewer could reasonably ask for that detail without the request being about over-engineering at all.", + "owner_disposition": { + "outcome": "confirmed_clean", + "reasoning": "Owner confirmed: protecting against or recovering from an error case that falls within the scope of the stated requirements is good engineering, not over-engineering. Confirms the recommended CLEAN / NOT MATERIAL disposition.", + "source": "https://github.com/shaug/agent-scripts/issues/58#issuecomment-5099314609 (owner adjudication, 8 of 8); https://github.com/shaug/agent-scripts/issues/58#issuecomment-5099339357 (case 3/7 cross-reference resolution)" + } } } diff --git a/review-suite/evals/strata/s2-solution-simplicity-lens/private/provenance/registry-client-layering.json b/review-suite/evals/strata/s2-solution-simplicity-lens/private/provenance/registry-client-layering.json index 2fd5ee1..3272975 100644 --- a/review-suite/evals/strata/s2-solution-simplicity-lens/private/provenance/registry-client-layering.json +++ b/review-suite/evals/strata/s2-solution-simplicity-lens/private/provenance/registry-client-layering.json @@ -7,7 +7,12 @@ "recorded_at": "2026-07-27", "adjudication": { "first": "shaug/atelier PR 410 review comment 2870262209, accepted in commit f631cb0: all three client classes were converged onto one bound BeadsClient instance.", - "second": "owner_required", - "notes": "No executable oracle exists for a solution-shape judgement like this. Recommended adjudication: MATERIAL. The packet states all three classes call the same transport and that one is already bound to the exact two values the other two re-thread, which is the layering defect stated directly rather than inferred. The only real counter-consideration: the diff is scoped to a queue-message client only, so a reviewer might argue the mutation-client duplication predates this change and is out of scope for it - the owner should weigh whether introducing a third instance of an existing anti-pattern is itself gating." + "second": "owner_confirmed", + "notes": "No executable oracle exists for a solution-shape judgement like this. Recommended adjudication: MATERIAL. The packet states all three classes call the same transport and that one is already bound to the exact two values the other two re-thread, which is the layering defect stated directly rather than inferred. The only real counter-consideration: the diff is scoped to a queue-message client only, so a reviewer might argue the mutation-client duplication predates this change and is out of scope for it - the owner should weigh whether introducing a third instance of an existing anti-pattern is itself gating.", + "owner_disposition": { + "outcome": "confirmed_material", + "reasoning": "Owner confirmed, and reframed the standard rather than accepting it as given: the defect is not reaching through an abstraction barrier per se, it is failing to question and re-justify the barrier itself. A barrier that holds up under scrutiny should be maintained, not locally worked around. See the case 3/case 7 note for the shared-source resolution: case 7's local duplication is this case's downstream symptom, confirmed as deliberate dual-level coverage rather than a double-count.", + "source": "https://github.com/shaug/agent-scripts/issues/58#issuecomment-5099314609 (owner adjudication, 8 of 8); https://github.com/shaug/agent-scripts/issues/58#issuecomment-5099339357 (case 3/7 cross-reference resolution)" + } } } diff --git a/review-suite/evals/strata/s2-solution-simplicity-lens/private/provenance/setup-service-path-gateway.json b/review-suite/evals/strata/s2-solution-simplicity-lens/private/provenance/setup-service-path-gateway.json index efc7c3f..190f2af 100644 --- a/review-suite/evals/strata/s2-solution-simplicity-lens/private/provenance/setup-service-path-gateway.json +++ b/review-suite/evals/strata/s2-solution-simplicity-lens/private/provenance/setup-service-path-gateway.json @@ -7,7 +7,12 @@ "recorded_at": "2026-07-27", "adjudication": { "first": "shaug/atelier PR 160 review comment 2848776499, accepted in commit fcbf469: the gateway indirection was removed entirely and the service now calls the concrete functions directly.", - "second": "owner_required", - "notes": "No executable oracle exists for over-engineering: whether an abstraction is 'unnecessary' given only one production implementation is a design judgement, not a property a runnable check can decide. Recommended adjudication: MATERIAL. The packet states one implementation exists, no second is planned, and a sibling service in the same package solves the identical testability need without an injected gateway - three converging facts a reviewer can point to without inventing a policy against dependency injection in general. The risk of over-adjudicating: a reviewer could reasonably value the injected-gateway pattern as forward consistency with a future second implementation the packet does not rule out; that is the strongest counter-argument for a NOT MATERIAL disposition and the owner should weigh it, not this record." + "second": "owner_confirmed", + "notes": "No executable oracle exists for over-engineering: whether an abstraction is 'unnecessary' given only one production implementation is a design judgement, not a property a runnable check can decide. Recommended adjudication: MATERIAL. The packet states one implementation exists, no second is planned, and a sibling service in the same package solves the identical testability need without an injected gateway - three converging facts a reviewer can point to without inventing a policy against dependency injection in general. The risk of over-adjudicating: a reviewer could reasonably value the injected-gateway pattern as forward consistency with a future second implementation the packet does not rule out; that is the strongest counter-argument for a NOT MATERIAL disposition and the owner should weigh it, not this record.", + "owner_disposition": { + "outcome": "confirmed_material", + "reasoning": "Owner confirmed: the mirror case of `registry-client-layering` - an abstraction that provides no obvious value and only obfuscates the implementation is the failure mode, not abstraction itself. Good engineering is designing good abstractions, not adding more of them.", + "source": "https://github.com/shaug/agent-scripts/issues/58#issuecomment-5099314609 (owner adjudication, 8 of 8); https://github.com/shaug/agent-scripts/issues/58#issuecomment-5099339357 (case 3/7 cross-reference resolution)" + } } } diff --git a/review-suite/evals/strata/s3-code-simplicity-lens/corpus.json b/review-suite/evals/strata/s3-code-simplicity-lens/corpus.json index 7f018c4..df6c02c 100644 --- a/review-suite/evals/strata/s3-code-simplicity-lens/corpus.json +++ b/review-suite/evals/strata/s3-code-simplicity-lens/corpus.json @@ -7,9 +7,9 @@ "stratum": { "id": "s3-code-simplicity-lens", "ground_truth": "human-review", - "scored": false, + "scored": true, "grading_is_signal": true, - "purpose": "Local code-complexity, reuse, and DRY cases for the self-sufficient code-simplicity lens. Populated and sourced from adjudicated human review and one adjudicated repository- history finding, with the recorded disposition as the first adjudication. NO executable oracle exists for this subject: 'this is a duplication defect' and 'this apparent duplication is justified' are design judgements a runnable check cannot decide. Every case therefore records adjudication.second: owner_required and a concrete recommended disposition for the owner to confirm or overturn. NOT scored: every expectation is uncalibrated and no case has been run through any runtime, per the owner-settled three-way grading method - a scored case is never calibrated on its own prose, and a grader miss must surface as referred rather than a silent reviewer-miss." + "purpose": "Local code-complexity, reuse, and DRY cases for the self-sufficient code-simplicity lens. No executable oracle exists for this subject - 'this is a duplication defect' and 'this apparent duplication is justified' are design judgements a runnable check cannot decide - so every case's second adjudication is `owner_confirmed`: the owner adjudicated all four directly, confirming every recommended disposition with no overturns. One pair, `metrics-label-formatting-duplication` here and `registry-client-layering` in s2, shares one source PR read at two levels; the owner confirmed this is deliberate dual-level coverage of one causal chain, not a double-count, and LIMITATIONS.md carries the resulting correlated-recall caveat. SCORED: both owner-gated inputs are satisfied, and every expectation stays `calibrated: false` permanently." }, "cases": [ "compat-accessor-boundary-duplication", diff --git a/review-suite/evals/strata/s3-code-simplicity-lens/private/provenance/compat-accessor-boundary-duplication.json b/review-suite/evals/strata/s3-code-simplicity-lens/private/provenance/compat-accessor-boundary-duplication.json index 0412bfc..3a87678 100644 --- a/review-suite/evals/strata/s3-code-simplicity-lens/private/provenance/compat-accessor-boundary-duplication.json +++ b/review-suite/evals/strata/s3-code-simplicity-lens/private/provenance/compat-accessor-boundary-duplication.json @@ -7,7 +7,12 @@ "recorded_at": "2026-07-27", "adjudication": { "first": "shaug/atelier PR 630 review comment 2906875752, the accepted response to the typeless-seam concern raised in 2906745973: the compatibility boundary was moved to the true downstream edge rather than removed.", - "second": "owner_required", - "notes": "No executable oracle exists for whether apparent duplication is justified by a migration boundary. Recommended adjudication: CLEAN / NOT MATERIAL to gate on. The packet states the two callers are on independently scheduled migration paths, which is a concrete reason rather than an assumption that duplication is always fine. Residual risk for the owner: a reviewer could reasonably ask whether the two schedules could still share one function with two call sites rather than two functions with one body each - a smaller version of the same question this case's own accepted non-finding already raises and tolerates." + "second": "owner_confirmed", + "notes": "No executable oracle exists for whether apparent duplication is justified by a migration boundary. Recommended adjudication: CLEAN / NOT MATERIAL to gate on. The packet states the two callers are on independently scheduled migration paths, which is a concrete reason rather than an assumption that duplication is always fine. Residual risk for the owner: a reviewer could reasonably ask whether the two schedules could still share one function with two call sites rather than two functions with one body each - a smaller version of the same question this case's own accepted non-finding already raises and tolerates.", + "owner_disposition": { + "outcome": "confirmed_clean", + "reasoning": "Owner confirmed: preferring strong types, edge validation into strong types, and domain-modeling types over a loose seam is the standard applied throughout, not a stylistic preference for this case alone.", + "source": "https://github.com/shaug/agent-scripts/issues/58#issuecomment-5099314609 (owner adjudication, 8 of 8); https://github.com/shaug/agent-scripts/issues/58#issuecomment-5099339357 (case 3/7 cross-reference resolution)" + } } } diff --git a/review-suite/evals/strata/s3-code-simplicity-lens/private/provenance/env-inventory-bullet-format.json b/review-suite/evals/strata/s3-code-simplicity-lens/private/provenance/env-inventory-bullet-format.json index 5406eb2..c34e582 100644 --- a/review-suite/evals/strata/s3-code-simplicity-lens/private/provenance/env-inventory-bullet-format.json +++ b/review-suite/evals/strata/s3-code-simplicity-lens/private/provenance/env-inventory-bullet-format.json @@ -7,7 +7,12 @@ "recorded_at": "2026-07-27", "adjudication": { "first": "shaug/atelier PR 443 review comment 2880556753, the accepted resolution: bullet sections per variable were chosen specifically to stay valid under the repository's Markdown wrap formatter and remain diff-friendly.", - "second": "owner_required", - "notes": "No executable oracle exists for whether apparent verbosity is justified by a formatting constraint. Recommended adjudication: CLEAN / NOT MATERIAL to gate on. The packet states a table previously produced malformed rows under the repository's own formatter, which is a concrete, checkable constraint rather than a stylistic preference for more text. Residual risk for the owner: a reviewer could reasonably propose a narrower fix (only wrapping the wide cell, or a two-column table) rather than the fully verbose bullet form, and this case's own accepted non-finding already flags the repeated owner field as one place that narrower fix could start." + "second": "owner_confirmed", + "notes": "No executable oracle exists for whether apparent verbosity is justified by a formatting constraint. Recommended adjudication: CLEAN / NOT MATERIAL to gate on. The packet states a table previously produced malformed rows under the repository's own formatter, which is a concrete, checkable constraint rather than a stylistic preference for more text. Residual risk for the owner: a reviewer could reasonably propose a narrower fix (only wrapping the wide cell, or a two-column table) rather than the fully verbose bullet form, and this case's own accepted non-finding already flags the repeated owner field as one place that narrower fix could start.", + "owner_disposition": { + "outcome": "confirmed_clean", + "reasoning": "Owner confirmed: Markdown must stay readable in its raw form - that is the actual reason for the simplified syntax over a full markup language, and the same reason prose is wrapped and tables are formatted rather than left to a renderer.", + "source": "https://github.com/shaug/agent-scripts/issues/58#issuecomment-5099314609 (owner adjudication, 8 of 8); https://github.com/shaug/agent-scripts/issues/58#issuecomment-5099339357 (case 3/7 cross-reference resolution)" + } } } diff --git a/review-suite/evals/strata/s3-code-simplicity-lens/private/provenance/metrics-label-formatting-duplication.json b/review-suite/evals/strata/s3-code-simplicity-lens/private/provenance/metrics-label-formatting-duplication.json index 4c95f99..757d17c 100644 --- a/review-suite/evals/strata/s3-code-simplicity-lens/private/provenance/metrics-label-formatting-duplication.json +++ b/review-suite/evals/strata/s3-code-simplicity-lens/private/provenance/metrics-label-formatting-duplication.json @@ -7,7 +7,12 @@ "recorded_at": "2026-07-27", "adjudication": { "first": "shaug/atelier PR 410 review comment 2870262209, accepted in commit f631cb0, read at the local-implementation level rather than the whole-solution level this same comment grounds in s2: the repeated argument-threading pattern the reviewer flagged is a local reuse defect independent of the architectural over-engineering question.", - "second": "owner_required", - "notes": "No executable oracle exists for a local-reuse judgement like this. Recommended adjudication: MATERIAL. The packet states the shared formatter already exists in the same module and that a fourth exporter is already planned, which makes the compounding cost concrete rather than hypothetical. Residual risk for the owner: this case and s2's `registry-client-layering` trace to the same source PR and comment, read at two different levels (whole-design versus local-implementation); the owner should confirm this reuse is the deliberate, recorded kind rather than an unintended duplicate sample of one real finding across two strata." + "second": "owner_confirmed", + "notes": "No executable oracle exists for a local-reuse judgement like this. Recommended adjudication: MATERIAL. The packet states the shared formatter already exists in the same module and that a fourth exporter is already planned, which makes the compounding cost concrete rather than hypothetical. Residual risk for the owner: this case and s2's `registry-client-layering` trace to the same source PR and comment, read at two different levels (whole-design versus local-implementation); the owner should confirm this reuse is the deliberate, recorded kind rather than an unintended duplicate sample of one real finding across two strata.", + "owner_disposition": { + "outcome": "confirmed_material", + "reasoning": "Owner confirmed from direct recollection of the real PR, independent of the recommendation offered - including the difficulty at the time of getting the implementing agent to understand the point of the abstraction barrier. Resolved as case 3's downstream symptom, not a double-count of the same finding: deliberate dual-level coverage of one causal chain. See LIMITATIONS.md for the correlated-recall caveat this creates for any aggregate figure over the pair.", + "source": "https://github.com/shaug/agent-scripts/issues/58#issuecomment-5099314609 (owner adjudication, 8 of 8); https://github.com/shaug/agent-scripts/issues/58#issuecomment-5099339357 (case 3/7 cross-reference resolution)" + } } } diff --git a/review-suite/evals/strata/s3-code-simplicity-lens/private/provenance/watcher-check-policy-duplication.json b/review-suite/evals/strata/s3-code-simplicity-lens/private/provenance/watcher-check-policy-duplication.json index d23954a..4c06acd 100644 --- a/review-suite/evals/strata/s3-code-simplicity-lens/private/provenance/watcher-check-policy-duplication.json +++ b/review-suite/evals/strata/s3-code-simplicity-lens/private/provenance/watcher-check-policy-duplication.json @@ -7,7 +7,12 @@ "recorded_at": "2026-07-27", "adjudication": { "first": "This repository's commit 93516194388116f4841fc191a8c78c191d0da5b1: the code-simplicity lens found and this commit resolved an inline duplicate of a shared policy, with a stated history of that duplication class causing real drift. This case models a new instance of the same duplication class the commit's own message warns against, rather than reproducing the commit's diff directly.", - "second": "owner_required", - "notes": "No executable oracle exists for a local-complexity judgement like this. Recommended adjudication: MATERIAL. The packet states two other call sites already use the shared predicate and that this exact duplication class had already caused real drift twice, which are concrete, checkable facts rather than a stylistic preference for less code. Residual risk for the owner: the severity is recorded as strong_recommendation rather than blocking, since the packet shows no currently observed drift, only a stated history of it; a reviewer could reasonably argue for blocking severity given that history, and the owner should decide which is right." + "second": "owner_confirmed", + "notes": "No executable oracle exists for a local-complexity judgement like this. Recommended adjudication: MATERIAL. The packet states two other call sites already use the shared predicate and that this exact duplication class had already caused real drift twice, which are concrete, checkable facts rather than a stylistic preference for less code. Residual risk for the owner: the severity is recorded as strong_recommendation rather than blocking, since the packet shows no currently observed drift, only a stated history of it; a reviewer could reasonably argue for blocking severity given that history, and the owner should decide which is right.", + "owner_disposition": { + "outcome": "confirmed_material", + "reasoning": "Owner confirmed material, consistent with the recommended disposition and with this repository's own commit history recording the drift this case demonstrates.", + "source": "https://github.com/shaug/agent-scripts/issues/58#issuecomment-5099314609 (owner adjudication, 8 of 8); https://github.com/shaug/agent-scripts/issues/58#issuecomment-5099339357 (case 3/7 cross-reference resolution)" + } } } diff --git a/review-suite/scripts/evals/corpus.py b/review-suite/scripts/evals/corpus.py index 791ce0e..a02f596 100644 --- a/review-suite/scripts/evals/corpus.py +++ b/review-suite/scripts/evals/corpus.py @@ -165,6 +165,35 @@ def _expectation_semantics(expectation: dict[str, Any]) -> list[str]: return errors +def _provenance_semantics(provenance: dict[str, Any]) -> list[str]: + """Cross-field adjudication rules JSON Schema cannot express clearly. + + `owner_disposition` is documented as present only when `second` is + `owner_confirmed` - a claim the schema's own subset validator has no way + to enforce, since it supports no conditional (`if`/`then`/`else`) keyword. + Checked here instead, so a case cannot silently claim owner adjudication + with no recorded disposition, or carry a disposition under an + `oracle`/`owner_required`/`none` second adjudication it does not belong + to. + """ + errors = [] + adjudication = provenance.get("adjudication") + if not adjudication: + return errors + second = adjudication.get("second") + has_disposition = "owner_disposition" in adjudication + if second == "owner_confirmed" and not has_disposition: + errors.append( + "adjudication.second is owner_confirmed but owner_disposition is missing" + ) + if second != "owner_confirmed" and has_disposition: + errors.append( + f"adjudication.owner_disposition is only valid when second is " + f"owner_confirmed, not {second!r}" + ) + return errors + + def load_case(root: Path, case_id: str) -> Case: """Load and validate one case's reviewer-visible and private parts.""" errors: list[str] = [] @@ -206,6 +235,7 @@ def load_case(root: Path, case_id: str) -> Case: errors.extend( f"expectation: {error}" for error in _expectation_semantics(expectation) ) + errors.extend(f"provenance: {error}" for error in _provenance_semantics(provenance)) packet_errors = protocol.VALIDATOR.validate_packet(packet) if expectation.get("packet_valid") and packet_errors: diff --git a/review-suite/scripts/evals/grader.py b/review-suite/scripts/evals/grader.py index a9e9323..a8143db 100644 --- a/review-suite/scripts/evals/grader.py +++ b/review-suite/scripts/evals/grader.py @@ -187,9 +187,26 @@ def grade(expectation: dict[str, Any] | None, result: dict[str, Any]) -> dict[st claimed.add(record["root_cause_id"]) records.append(record) + # Three-way scoring, settled by the owner on #58: a root cause with a + # partial or ambiguous candidate is referred for adjudication, not counted + # as a miss. Collapsing "unmatched because the formulation didn't + # recognise real prose" into "the reviewer missed this" is exactly the + # false reviewer-miss the referral bucket exists to prevent - a case is + # only a genuine miss when nothing pointed at it at all. + referred_ids = { + candidate_id + for record in records + if record["classification"] in {"partial", "ambiguous"} + for candidate_id in record["candidate_root_cause_ids"] + } - claimed + expected_ids = [rc["id"] for rc in root_causes] matched_ids = sorted(claimed) - missed_ids = [item for item in expected_ids if item not in claimed] + missed_ids = [ + item + for item in expected_ids + if item not in claimed and item not in referred_ids + ] expected_verdict = expectation["expected_verdict"] observed_verdict = result.get("verdict") gating = [ @@ -215,6 +232,14 @@ def grade(expectation: dict[str, Any] | None, result: dict[str, Any]) -> dict[st "expected_root_cause_ids": expected_ids, "matched_root_cause_ids": matched_ids, "missed_root_cause_ids": missed_ids, + "referred_root_cause_ids": sorted(referred_ids), + # Recall counts confirmed matches against the full expected set, never + # crediting a referral - a referred root cause is neither a match nor + # a scored miss, so it is absent from both the numerator and this + # denominator's alternative reading. It stays a lower bound: a case + # whose only unmatched root causes were all referred reports the same + # recall as one with genuine misses, and the referral bucket is what + # tells the two apart. "recall": (len(matched_ids) / len(expected_ids)) if expected_ids else None, "findings": records, "false_positive_finding_ids": [record["finding_id"] for record in gating], diff --git a/review-suite/scripts/evals/report.py b/review-suite/scripts/evals/report.py index 77de833..ea44727 100644 --- a/review-suite/scripts/evals/report.py +++ b/review-suite/scripts/evals/report.py @@ -93,10 +93,12 @@ def _case_summary(case_id: str, attempts: list[dict[str, Any]]) -> dict[str, Any union: set[str] = set() intersection: set[str] | None = None + referred_union: set[str] = set() for grade_record in grades: found = set(grade_record["matched_root_cause_ids"]) union |= found intersection = found if intersection is None else (intersection & found) + referred_union |= set(grade_record["referred_root_cause_ids"]) expected = grades[0]["expected_root_cause_ids"] if grades else [] return { @@ -107,6 +109,11 @@ def _case_summary(case_id: str, attempts: list[dict[str, Any]]) -> dict[str, Any "mean_recall": statistics.fmean(recalls) if recalls else None, "union_root_cause_ids": sorted(union), "intersection_root_cause_ids": sorted(intersection or set()), + # Three-way scoring: a root cause ever referred for adjudication on + # any attempt is neither a confirmed match nor a scored miss for this + # case. Reported separately so a reader never has to infer a referral + # from the absence of a match. + "ever_referred_root_cause_ids": sorted(referred_union), "verdict_stability": _modal_share([a["verdict"] for a in answered]), "finding_stability": _modal_share(matched_sets), "stability_denominator": len(answered), @@ -189,6 +196,16 @@ def aggregate( sum(1 for g in clean_expected if g["false_alarm"]), len(clean_expected) ), "false_alarm_denominator": len(clean_expected), + # Three-way scoring, settled on #58: a root cause referred for + # adjudication is neither a confirmed match nor a scored miss. + # Reported as its own rate over the same attempts recall is + # measured over, so a reader can see how much of the + # not-matched remainder is a genuine miss versus a referral + # awaiting the owner, rather than inferring it from a gap. + "referred_rate": _rate( + sum(1 for g in grades if g["referred_root_cause_ids"]), len(grades) + ), + "referred_denominator": len(grades), "unique_finding_contribution": unique_contribution, }, "stability": { diff --git a/review-suite/scripts/tests/test_eval_calibration.py b/review-suite/scripts/tests/test_eval_calibration.py index 6adb337..06cad99 100644 --- a/review-suite/scripts/tests/test_eval_calibration.py +++ b/review-suite/scripts/tests/test_eval_calibration.py @@ -1,4 +1,4 @@ -"""Calibration tests: every shipped expectation is graded as calibrated. +"""Calibration tests: pilot cases are calibrated, scored cases never are. These tests are the executable half of grader calibration. The calibration sets hold the probe reviews; this module replays each probe through the real grader @@ -7,6 +7,12 @@ prose, or loosens far enough to credit an overlapping symptom, fails here rather than quietly changing what a baseline means. +Calibration and scoring are mutually exclusive by the owner-settled method on +#58: a scored case is never calibrated on its own prose, and a grader miss on a +scored case is reported as `referred_root_cause_ids` (see `grader.py`) rather +than silently becoming recall damage. Calibration sets exist only for the +disjoint pilot corpus, whose cases are never scored. + Nothing in this module launches a runtime or spends money. """ @@ -78,13 +84,24 @@ def test_a_case_carried_by_several_corpora_carries_one_expectation(self): for other in expectations[1:]: self.assertEqual(expectations[0], other) - def test_every_scored_case_is_calibrated(self): - """A scored case without calibration reports its expectation, not a rate. - - Measured twice, on two independent pilot cases: an uncalibrated - expectation returns recall 0.0 against a reviewer that found the defect - on every attempt. A scored stratum built on one would publish that as a - capability figure. + def test_a_scored_case_is_never_calibrated(self): + """Settled by the owner on #58: never calibrate a scored case's prose. + + Batch 1 measured what an uncalibrated expectation reports - recall 0.0 + against a reviewer that found the defect on every attempt - and batch 2 + proposed calibrating scored cases against their own observed prose as + one fix. The owner rejected that: calibrating a case requires observing + its real reviewer prose, and a scored case must never have been + observed before scoring, or the baseline measures a case the corpus + author already looked at - the exact contamination the payload-blindness + tests exist to prevent, entering through the grader instead. + + The settled replacement is three-way scoring (matched / missed / + referred), which needs no calibration at all: a formulation miss + surfaces as `referred_root_cause_ids` rather than a silently wrong + recall figure. So `calibrated: true` on a scored case is not a missing + nice-to-have, it is itself the contamination this method exists to rule + out. """ for root in corpus.corpus_roots(): loaded = corpus.load_corpus(root) @@ -92,8 +109,29 @@ def test_every_scored_case_is_calibrated(self): continue for case in loaded.cases: with self.subTest(stratum=root.name, case_id=case.case_id): - self.assertIn(case.case_id, self.sets) - self.assertIs(True, case.expectation.get("calibrated")) + self.assertIs( + False, + case.expectation.get("calibrated"), + "a scored case must never be calibrated on its own " + "prose - see the three-way scoring method", + ) + + def test_a_scored_case_records_a_complete_second_adjudication(self): + """A scored stratum may not carry a case still marked as needing one.""" + for root in corpus.corpus_roots(): + loaded = corpus.load_corpus(root) + if not loaded.scored: + continue + for case in loaded.cases: + with self.subTest(stratum=root.name, case_id=case.case_id): + adjudication = case.provenance.get("adjudication") + self.assertIsNotNone(adjudication) + self.assertIn( + adjudication["second"], + {"oracle", "owner_confirmed"}, + "a scored case's second adjudication must be settled, " + "not still owner_required or none", + ) def test_a_calibrated_expectation_ships_a_calibration_set(self): """`calibrated: true` is a claim the calibration set has to back.""" diff --git a/review-suite/scripts/tests/test_eval_corpus.py b/review-suite/scripts/tests/test_eval_corpus.py index 5cb181b..360e9fe 100644 --- a/review-suite/scripts/tests/test_eval_corpus.py +++ b/review-suite/scripts/tests/test_eval_corpus.py @@ -225,6 +225,37 @@ def test_provenance_without_retention_authority_fails(self): path.write_text(json.dumps(document, indent=2)) self.assertAuditFails("retention_authority") + def test_owner_confirmed_without_a_disposition_fails(self): + """The schema subset validator has no conditional keyword, so a case + claiming owner_confirmed with no recorded disposition must still be + caught - by the cross-field check in `corpus._provenance_semantics`, + not by JSON Schema.""" + case_id = self._index()["cases"][0] + path = self.root / "private" / "provenance" / f"{case_id}.json" + document = json.loads(path.read_text()) + document["adjudication"] = { + "first": "recorded source", + "second": "owner_confirmed", + } + path.write_text(json.dumps(document, indent=2)) + self.assertAuditFails("owner_disposition is missing") + + def test_a_disposition_without_owner_confirmed_fails(self): + case_id = self._index()["cases"][0] + path = self.root / "private" / "provenance" / f"{case_id}.json" + document = json.loads(path.read_text()) + document["adjudication"] = { + "first": "recorded source", + "second": "oracle", + "owner_disposition": { + "outcome": "confirmed_material", + "reasoning": "irrelevant to an oracle-settled case", + "source": "https://example.invalid/irrelevant", + }, + } + path.write_text(json.dumps(document, indent=2)) + self.assertAuditFails("only valid when second is owner_confirmed") + def test_gating_expectation_without_a_root_cause_fails(self): case_id = next( item diff --git a/review-suite/scripts/tests/test_eval_grader.py b/review-suite/scripts/tests/test_eval_grader.py index accdcee..cc6bf16 100644 --- a/review-suite/scripts/tests/test_eval_grader.py +++ b/review-suite/scripts/tests/test_eval_grader.py @@ -171,6 +171,11 @@ def test_a_partial_match_is_referred_for_adjudication(self): ], grade["adjudication_required"], ) + # Three-way scoring: a partial match is referred, not a scored miss. + # Collapsing it into `missed_root_cause_ids` would be exactly the + # silent reviewer-miss the owner-settled grading method forbids. + self.assertEqual([], grade["missed_root_cause_ids"]) + self.assertEqual(["rc.deadline"], grade["referred_root_cause_ids"]) def test_a_finding_matching_two_root_causes_is_ambiguous(self): grade = grader.grade( @@ -191,6 +196,20 @@ def test_a_finding_matching_two_root_causes_is_ambiguous(self): ["rc.deadline", "rc.record"], grade["adjudication_required"][0]["candidate_root_cause_ids"], ) + # An ambiguous candidate is referred for both root causes, not a + # scored miss for either. + self.assertEqual([], grade["missed_root_cause_ids"]) + self.assertEqual(["rc.deadline", "rc.record"], grade["referred_root_cause_ids"]) + + def test_a_root_cause_with_no_candidate_finding_is_a_genuine_miss(self): + """Nothing pointed at it at all: this is the one case referral must not swallow.""" + grade = grader.grade( + expectation("changes_required", [ROOT_CAUSE]), + result("changes_required", []), + ) + self.assertEqual(["rc.deadline"], grade["missed_root_cause_ids"]) + self.assertEqual([], grade["referred_root_cause_ids"]) + self.assertEqual(0.0, grade["recall"]) def test_an_unexpected_gating_finding_is_a_false_positive(self): grade = grader.grade( diff --git a/review-suite/scripts/tests/test_eval_oracles.py b/review-suite/scripts/tests/test_eval_oracles.py index bd3dfef..bc2f227 100644 --- a/review-suite/scripts/tests/test_eval_oracles.py +++ b/review-suite/scripts/tests/test_eval_oracles.py @@ -163,14 +163,21 @@ def test_a_scored_stratum_records_a_second_adjudication_for_every_case(self): adjudication, "a scored case must record how it was adjudicated", ) - self.assertIn(adjudication["second"], {"oracle", "owner_required"}) + self.assertIn( + adjudication["second"], + {"oracle", "owner_confirmed"}, + "a scored case's second adjudication must be settled " + "(oracle or owner_confirmed), not still owner_required", + ) - def test_a_case_without_an_oracle_routes_to_the_owner(self): - """`owner_required` is the only honest alternative to an oracle. + def test_a_case_without_an_oracle_never_claims_oracle_adjudication(self): + """No oracle exists, so `oracle` is never an honest second adjudication. A fresh agent context is not a second adjudicator for materiality when it shares a model family with the reviewer being measured, so a case with no - oracle has to say it needs the owner rather than quietly claiming two. + oracle must say it needs the owner (`owner_required`) or that the owner + has settled it (`owner_confirmed`) - never claim a machine adjudication + it does not have. """ shipped = set(oracles.case_ids()) for root in corpus.corpus_roots(): @@ -179,7 +186,9 @@ def test_a_case_without_an_oracle_routes_to_the_owner(self): if adjudication is None or case.case_id in shipped: continue with self.subTest(stratum=root.name, case_id=case.case_id): - self.assertEqual("owner_required", adjudication["second"]) + self.assertIn( + adjudication["second"], {"owner_required", "owner_confirmed"} + ) if __name__ == "__main__": diff --git a/review-suite/scripts/tests/test_eval_report.py b/review-suite/scripts/tests/test_eval_report.py index 3f8f122..2375180 100644 --- a/review-suite/scripts/tests/test_eval_report.py +++ b/review-suite/scripts/tests/test_eval_report.py @@ -51,6 +51,7 @@ def grade( observed="changes_required", expected_ids=("rc.one",), matched_ids=("rc.one",), + referred_ids=(), false_positives=(), adjudication=(), ): @@ -63,7 +64,10 @@ def grade( "false_alarm": expected == "clean" and observed == "changes_required", "expected_root_cause_ids": list(expected_ids), "matched_root_cause_ids": list(matched_ids), - "missed_root_cause_ids": [i for i in expected_ids if i not in matched_ids], + "missed_root_cause_ids": [ + i for i in expected_ids if i not in matched_ids and i not in referred_ids + ], + "referred_root_cause_ids": list(referred_ids), "recall": (len(matched_ids) / len(expected_ids)) if expected_ids else None, "findings": [], "false_positive_finding_ids": list(false_positives), @@ -94,6 +98,33 @@ def test_every_required_metric_is_present(self): self.assertIn("total_cost_usd", aggregate["usage"]) self.assertEqual(report.REPORT_VERSION, aggregate["report_version"]) + def test_a_referred_root_cause_is_reported_separately_from_a_miss(self): + """Three-way scoring, settled on #58: referred is neither match nor miss. + + Reported at both levels - the per-case union and the aggregate rate - + so a reader never has to infer a referral from an unexplained gap + between `matched_root_cause_ids` and `expected_root_cause_ids`. + """ + aggregate = report.aggregate( + [ + attempt( + "subject-one", + 1, + grade=grade( + expected_ids=("rc.one", "rc.two"), + matched_ids=("rc.one",), + referred_ids=("rc.two",), + ), + ) + ], + configuration=CONFIGURATION, + ) + self.assertIn("referred_rate", aggregate["quality"]) + self.assertEqual(1.0, aggregate["quality"]["referred_rate"]) + self.assertEqual(1, aggregate["quality"]["referred_denominator"]) + per_case = aggregate["per_case"][0] + self.assertEqual(["rc.two"], per_case["ever_referred_root_cause_ids"]) + def test_recall_averages_only_graded_attempts(self): aggregate = report.aggregate( [