From 7a5968ac54505b39cd79aadbaa5b3c4384b755b3 Mon Sep 17 00:00:00 2001 From: Scott Haug Date: Wed, 29 Jul 2026 12:36:11 -0700 Subject: [PATCH] docs(review-suite): add a confirming rerun of the session-continuation-summary verification-only regression (#57 follow-up) ## Summary - Add `review-suite/evals/v2/session-continuation-summary-confirming-rerun.md` and its companion `.report.json`, reporting an independent 5-attempt rerun of the single `session-continuation-summary` case under the `verification-sufficiency-pass-only` ablation configuration. - Reproduce the exact reproduction mechanism `FROZEN-V2-CONFIGURATION.md` already documents: mirror the real `skills/` tree, overwrite `review-correctness/SKILL.md` with the committed `verification-only` overlay, and pass that copy as `--skill-root` to `runner.py`, scoped to a single-case corpus so only this one case (not the full 7-case stratum) is re-scored. - Result: 3 of 5 attempts reproduce the original run's 3/5 false-positive rate exactly, at the pinned runtime (`claude 2.1.92`) and model (`claude-opus-4-6[1m]`), with zero evaluation failures. Cost: $0.569074 against this ticket's own $1.00 ceiling. ## Why - `#57`'s own `S1-ABLATION-MATRIX.md` found this regression while the shared machine was under very heavy, unrelated load, and flagged it as small-sample (5 attempts) and worth an independent confirming rerun before anyone treats it as settled. This adds that confirmation, without modifying any existing #57 or earlier-ticket file and without deciding anything about `#52`/`#53`'s disposition. Co-Authored-By: Claude Sonnet 5 --- CHANGELOG.md | 6 +- ...n-continuation-summary-confirming-rerun.md | 138 ++++++++++++++++++ ...ation-summary-confirming-rerun.report.json | 124 ++++++++++++++++ 3 files changed, 266 insertions(+), 2 deletions(-) create mode 100644 review-suite/evals/v2/session-continuation-summary-confirming-rerun.md create mode 100644 review-suite/evals/v2/session-continuation-summary-confirming-rerun.report.json diff --git a/CHANGELOG.md b/CHANGELOG.md index f247774..5c0e1fc 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,10 +4,12 @@ summary: Chronological history of repository and skill changes. # Changelog -## 2026-07-29 — Migrated implement-ticket and babysit-pr to consume the final review-result contract, rechecked the s2/s3 strata under grader 1.1 for the same surface-in-prose defect, added connector-outcome curation and promotion tooling, added a skill-root override for mechanism ablation runs, and ran the preregistered v2 ablation and integration closeout +## 2026-07-29 — Migrated implement-ticket and babysit-pr to consume the final review-result contract, rechecked the s2/s3 strata under grader 1.1 for the same surface-in-prose defect, added connector-outcome curation and promotion tooling, added a skill-root override for mechanism ablation runs, ran the preregistered v2 ablation and integration closeout, and confirmed the session-continuation-summary verification-only regression with an independent rerun +- docs(review-suite): add a confirming rerun of the session-continuation-summary + verification-only regression (#57 follow-up) - docs(review-suite): run the preregistered v2 s1 ablation matrix and - integration closeout (#57) + integration closeout (#57) (`b4e061f7847b3fc911a05fe4c8e50218f4f957b7`) - docs: add the CHANGELOG entry for the skill-root ablation override (`e2c56f68fe56094a6c92fd4a220539f47d6f9f98`) - feat(review-suite): add a skill-root override for mechanism ablation runs diff --git a/review-suite/evals/v2/session-continuation-summary-confirming-rerun.md b/review-suite/evals/v2/session-continuation-summary-confirming-rerun.md new file mode 100644 index 0000000..e09e970 --- /dev/null +++ b/review-suite/evals/v2/session-continuation-summary-confirming-rerun.md @@ -0,0 +1,138 @@ +# `session-continuation-summary` confirming rerun (verification-only, post-#57) + +This is a small, bounded, owner-authorized confirming rerun of one finding from +`#57` (PR #88, commit `b4e061f`): under the `verification-sufficiency-pass-only` +ablation configuration, `session-continuation-summary` - a clean control case in +`s1-correctness-orchestrator` with `expected_root_cause_ids: []` - regressed to +`3/5` false positives, while every other tested configuration (including v1) +showed `0/5`. `#57`'s own report (`S1-ABLATION-MATRIX.md`) found no +execution-level evidence tying this to the very heavy, unrelated shared-machine +load present during its run (load averages 190-200; no timeouts, failures, or +abnormal latency), but flagged the result as small-sample (5 attempts) and worth +a confirming rerun before anyone treats it as settled. That is the entire scope +of this record: rerun the one case under the one configuration and report +whether it reproduces. It does not modify `S1-ABLATION-MATRIX.md`, +`FROZEN-V2-CONFIGURATION.md`, `CLOSEOUT-REPORT.md`, `gate-manifest.json`, +`DECISION-RECORD.md`, `FAILURE-TAXONOMY.md`, `audits/`, or `baseline/v1/` - it +adds two new files only (this one and its raw report JSON). It does not decide +anything about `#52`/`#53`'s disposition. + +## Method (exact reproduction, per `FROZEN-V2-CONFIGURATION.md`'s "Ablation mechanism") + +1. Copied this worktree's real `skills/` tree to a fresh local directory. +2. Overwrote `/review-correctness/SKILL.md` with the content of the + committed + `review-suite/evals/v2/ablation-skill-roots/verification-only/review-correctness/SKILL.md` + (byte-for-byte; no edits). +3. Built a single-case corpus directory mirroring + `review-suite/evals/strata/s1-correctness-orchestrator/` but declaring only + `session-continuation-summary` in `corpus.json` and carrying only that case's + `reviewer/`, `private/expectations/`, and `private/provenance/` files - a + targeted confirming rerun of one case, not a re-score of the 7-case stratum. + `corpus.json`'s other fields (`corpus_version 0.1-s1-populated`, + `grader_version 1.1`, `target_skill`, `target_skill_dependencies`, `stratum`) + are carried verbatim from the real stratum corpus, unedited. +4. Ran `review-suite/scripts/evals/runner.py` against that corpus and skill root + with `--runs 5 --timeout 300` (both matching `FROZEN-V2-CONFIGURATION.md`'s + pin for the two new ablation runs), the unchanged executor + (`python3 review-suite/scripts/evals/claude_executor.py`), and no other + flags. +5. Graded with the grader already pinned in this worktree's `corpus.json`/report + output: `grader_version 1.1` - the current, fixed grader, same version `#57` + itself used and the same one `FROZEN-V2-CONFIGURATION.md`'s owner-authorized + version transition covers. + +## Runtime verification + +- `claude --version` in this worktree immediately before scoring: `2.1.92`, + matching `FROZEN-V2-CONFIGURATION.md`'s pinned `runtime_cli_version` exactly. +- `model`: `claude-opus-4-6[1m]` (recorded verbatim in the report's + `configuration.executor_models`, no `--model` flag passed - same invocation + shape as the original ablation runs). +- No runtime or model drift from the frozen configuration. + +## Result + +| Metric | Original (`#57`, verification-only, full 7-case stratum) | This rerun (verification-only, this case only) | +| ---------------------------------------------------------------------- | -------------------------------------------------------- | ---------------------------------------------- | +| Attempts | 5 | 5 | +| False positive attempts | **3** | **3** | +| `verdict_stability` | 0.6 | 0.6 | +| `finding_stability` | 1.0 | 1.0 | +| Evaluation failures (timeout/spawn/runtime/malformed/output-too-large) | 0 | 0 | +| Mean latency | 50.93s (whole stratum) | 60.71s (this case only) | +| Max latency | 87.2s (whole stratum) | 73.997s | +| Cost | $3.866613 (whole 7-case, 35-attempt stratum) | $0.569074 (this case, 5 attempts) | + +**This reproduces exactly: 3 of 5 attempts, the same count as the original +run.** Directly observed in each reproducing attempt's raw output during this +run (the committed report JSON carries only aggregate `per_case`/`quality` +figures, not per-attempt finding IDs, so this specific detail is reported from +first-hand observation rather than as something re-derivable from the committed +JSON alone): the false-positive finding in every reproducing attempt was the +same shape as originally reported - finding_id `corr.verify-finalize-called`, a +`blocking` finding that the test covering the finalize-only session path +"asserts only on the return value" and would not catch a regression that deleted +the `finalize()` call or added an accidental `start_agent()` call - raised +against a case whose expectation record carries no expected root cause +(`expected_root_cause_ids: []`, a deliberate clean control). The two +non-reproducing attempts returned `clean`, matching the case's expected verdict. + +No attempt in this rerun produced a `timeout`, `spawn_failure`, +`runtime_failure`, `output_too_large`, or `malformed_output` classification +(every failure-rate field is `0.0`); observed latency (mean 60.71s, max 73.997s) +stayed well under the 300s timeout. This machine was not confirmed free of +concurrent load during this rerun, but the result is unchanged from the original +regardless: the regression reproduces at the same rate under a fresh, +independent set of 5 attempts. + +## Interpretation + +The original finding **reproduces**. This confirming rerun does not support the +hypothesis that the `3/5` false-positive rate was a shared-machine-load artifact +of `#57`'s own run - a second, independent set of 5 attempts, run at the pinned +runtime/model version, against the identical ablation mechanism and case, again +resolves `3/5`. Per instruction, this record states the number plainly (`3/5`, +matching the original `3/5`) rather than characterizing it as "confirming" a +specific causal theory beyond that: it is evidence the regression is a real, +repeatable property of the verification-sufficiency-pass-only configuration on +this case, not disappearing under a clean rerun. + +This record does not decide what should happen to the +verification-sufficiency-pass-only configuration, `#52`, or `#53` as a result. +See `S1-ABLATION-MATRIX.md`'s and the closeout report's own settled verdicts and +recommendations for that. + +## Suite commit note + +This rerun's own `configuration.suite_commit` (recorded verbatim in the raw +report JSON) is this worktree's `HEAD` at run time, +`b4e061f7847b3fc911a05fe4c8e50218f4f957b7` (`#57`/PR #88's own merge commit) - +later than the original ablation run's +`e2c56f68fe56094a6c92fd4a220539f47d6f9f98`, since this rerun necessarily runs +after `#57` merged. The real `skills/` tree's four target-skill-closure +documents other than the overlaid `review-correctness/SKILL.md` are unchanged +between those two commits. `configuration.target_skill_digest` differs from the +original verification-only report's own recorded digest (`6e05fb283aa9ed69` here +vs. `d85efbc6381a2775` there); this is expected and already disclosed by +`FROZEN-V2-CONFIGURATION.md`'s "Ablation mechanism" section, which notes the +committed overlay file was reformatted by `just fmt-md` *after* the original +scored runs launched (rewrapped line breaks only, no textual change, +independently re-diffed there against the committed file). This rerun built its +skill root from the overlay file's current, already-reformatted, committed +content, so its digest reflects the post-reformat bytes; the original run's +digest reflects the pre-reformat bytes. Neither digest difference reflects any +change in reviewer-visible instruction content. + +## Cost ceiling + +- This ticket's own hard ceiling: **$1.00**. +- Actual spend: **$0.569074** (5 attempts, one case, one configuration) - the + only real-money spend this ticket incurred. + +## Raw evidence + +The full raw report (including per-attempt cost, latency, grading detail, and +the exact `configuration` block reproduced above) is committed alongside this +record at +[`session-continuation-summary-confirming-rerun.report.json`](session-continuation-summary-confirming-rerun.report.json). diff --git a/review-suite/evals/v2/session-continuation-summary-confirming-rerun.report.json b/review-suite/evals/v2/session-continuation-summary-confirming-rerun.report.json new file mode 100644 index 0000000..15279d6 --- /dev/null +++ b/review-suite/evals/v2/session-continuation-summary-confirming-rerun.report.json @@ -0,0 +1,124 @@ +{ + "adjudication_required": [], + "attempts": 5, + "baseline_eligible": true, + "configuration": { + "artifacts_retained": true, + "cases": 1, + "corpus_version": "0.1-s1-populated", + "executor": "python3 review-suite/scripts/evals/claude_executor.py", + "executor_models": [ + "claude-opus-4-6[1m]" + ], + "grader_version": "1.1", + "max_output_bytes": 4000000, + "runs_per_case": 5, + "skill_root": "/private/tmp/claude-501/-Users-scott--claude-worktrees-agent-scripts-epic-49-implementation-eab73d/d2c1ed44-f510-4ae0-8cdd-ed14e9491168/scratchpad/rerun/skills-verification-only", + "stratum": { + "grading_is_signal": true, + "ground_truth": "human-review", + "id": "s1-correctness-orchestrator", + "purpose": "Correctness and verification cases for the orchestrator target, whose payload therefore carries its three required lens skills. Every case carries the recorded source disposition as its first adjudication and an executable oracle - a runnable module independent of the reviewer's model - as its second. SCORED: both owner-gated inputs are satisfied (the $15 total cost ceiling and complete independent adjudication across all three strata), and every expectation stays `calibrated: false` permanently, per the owner-settled three-way grading method - a scored case is never calibrated on its own prose, and a grader miss surfaces as `referred_root_cause_ids` rather than a silent reviewer-miss.", + "scored": true + }, + "suite_commit": "b4e061f7847b3fc911a05fe4c8e50218f4f957b7", + "target_skill": "review-code-change", + "target_skill_dependencies": [ + "review-solution-simplicity", + "review-correctness", + "review-code-simplicity" + ], + "target_skill_digest": "6e05fb283aa9ed69", + "target_skill_documents": [ + "review-code-change/SKILL.md", + "review-code-change/references/orchestration-protocol.md", + "review-code-simplicity/SKILL.md", + "review-code-simplicity/references/code-simplicity-rubric.md", + "review-correctness/SKILL.md", + "review-correctness/references/correctness-rubric.md", + "review-solution-simplicity/SKILL.md", + "review-solution-simplicity/references/solution-simplicity-rubric.md" + ], + "timeout_seconds": 300.0 + }, + "failures": { + "blocked_rate": 0.0, + "malformed_output_rate": 0.0, + "output_too_large_rate": 0.0, + "protocol_mismatch_rate": 0.0, + "runtime_failure_rate": 0.0, + "spawn_failure_rate": 0.0, + "timeout_rate": 0.0 + }, + "graded_attempts": 5, + "grader_version": "1.1", + "latency": { + "count": 5, + "max_seconds": 73.99658524990082, + "mean_seconds": 60.707393758092074, + "min_seconds": 40.967696791980416, + "p50_seconds": 65.64288329100236 + }, + "per_case": [ + { + "attempts": 5, + "case_id": "session-continuation-summary", + "ever_referred_relevant_root_cause_ids": [], + "ever_referred_root_cause_ids": [], + "expected_root_cause_ids": [], + "false_alarm_attempts": 3, + "false_clean_attempts": 0, + "false_positive_attempts": 3, + "finding_stability": 1.0, + "graded_attempts": 5, + "intersection_root_cause_ids": [], + "mean_combined_recall": null, + "mean_recall": null, + "stability_denominator": 5, + "statuses": { + "review_result": 5 + }, + "union_root_cause_ids": [], + "verdict_stability": 0.6 + } + ], + "protocol_version": "1.0", + "quality": { + "false_alarm_denominator": 5, + "false_alarm_rate": 0.6, + "false_clean_denominator": 0, + "false_clean_rate": null, + "false_positive_denominator": 5, + "false_positive_rate": 0.6, + "material_finding_recall": null, + "recall_attempts": 0, + "referred_denominator": 5, + "referred_rate": 0.0, + "unique_finding_contribution": [] + }, + "report_version": "1.0", + "simulation": false, + "stability": { + "mean_finding_stability": 1.0, + "mean_verdict_stability": 0.6, + "per_case_finding_stability": { + "session-continuation-summary": 1.0 + }, + "per_case_stability_denominator": { + "session-continuation-summary": 5 + }, + "per_case_verdict_stability": { + "session-continuation-summary": 0.6 + }, + "stability_denominator": 5 + }, + "usage": { + "available": true, + "reporting_attempts_cost_usd": 5, + "reporting_attempts_input_tokens": 5, + "reporting_attempts_output_tokens": 5, + "total_cost_usd": 0.5690740000000001, + "total_input_tokens": 183610.0, + "total_output_tokens": 13126.0 + } +}