Skip to content

docs(review-suite): confirm the session-continuation-summary verification-only regression #90

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
shaug merged 1 commit into from
Jul 29, 2026
Merged

docs(review-suite): confirm the session-continuation-summary verification-only regression #90

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,10 +4,12 @@ summary: Chronological history of repository and skill changes.

# Changelog

## 2026-07-29 — Migrated implement-ticket and babysit-pr to consume the final review-result contract, rechecked the s2/s3 strata under grader 1.1 for the same surface-in-prose defect, added connector-outcome curation and promotion tooling, added a skill-root override for mechanism ablation runs, and ran the preregistered v2 ablation and integration closeout
## 2026-07-29 — Migrated implement-ticket and babysit-pr to consume the final review-result contract, rechecked the s2/s3 strata under grader 1.1 for the same surface-in-prose defect, added connector-outcome curation and promotion tooling, added a skill-root override for mechanism ablation runs, ran the preregistered v2 ablation and integration closeout, and confirmed the session-continuation-summary verification-only regression with an independent rerun

- docs(review-suite): add a confirming rerun of the session-continuation-summary
verification-only regression (#57 follow-up)
- docs(review-suite): run the preregistered v2 s1 ablation matrix and
integration closeout (#57)
integration closeout (#57) (`b4e061f7847b3fc911a05fe4c8e50218f4f957b7`)
- docs: add the CHANGELOG entry for the skill-root ablation override
(`e2c56f68fe56094a6c92fd4a220539f47d6f9f98`)
- feat(review-suite): add a skill-root override for mechanism ablation runs
Expand Down
138 changes: 138 additions & 0 deletions review-suite/evals/v2/session-continuation-summary-confirming-rerun.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,138 @@
# `session-continuation-summary` confirming rerun (verification-only, post-#57)

This is a small, bounded, owner-authorized confirming rerun of one finding from
`#57` (PR #88, commit `b4e061f`): under the `verification-sufficiency-pass-only`
ablation configuration, `session-continuation-summary` - a clean control case in
`s1-correctness-orchestrator` with `expected_root_cause_ids: []` - regressed to
`3/5` false positives, while every other tested configuration (including v1)
showed `0/5`. `#57`'s own report (`S1-ABLATION-MATRIX.md`) found no
execution-level evidence tying this to the very heavy, unrelated shared-machine
load present during its run (load averages 190-200; no timeouts, failures, or
abnormal latency), but flagged the result as small-sample (5 attempts) and worth
a confirming rerun before anyone treats it as settled. That is the entire scope
of this record: rerun the one case under the one configuration and report
whether it reproduces. It does not modify `S1-ABLATION-MATRIX.md`,
`FROZEN-V2-CONFIGURATION.md`, `CLOSEOUT-REPORT.md`, `gate-manifest.json`,
`DECISION-RECORD.md`, `FAILURE-TAXONOMY.md`, `audits/`, or `baseline/v1/` - it
adds two new files only (this one and its raw report JSON). It does not decide
anything about `#52`/`#53`'s disposition.

## Method (exact reproduction, per `FROZEN-V2-CONFIGURATION.md`'s "Ablation mechanism")

1. Copied this worktree's real `skills/` tree to a fresh local directory.
2. Overwrote `<copy>/review-correctness/SKILL.md` with the content of the
committed
`review-suite/evals/v2/ablation-skill-roots/verification-only/review-correctness/SKILL.md`
(byte-for-byte; no edits).
3. Built a single-case corpus directory mirroring
`review-suite/evals/strata/s1-correctness-orchestrator/` but declaring only
`session-continuation-summary` in `corpus.json` and carrying only that case's
`reviewer/`, `private/expectations/`, and `private/provenance/` files - a
targeted confirming rerun of one case, not a re-score of the 7-case stratum.
`corpus.json`'s other fields (`corpus_version 0.1-s1-populated`,
`grader_version 1.1`, `target_skill`, `target_skill_dependencies`, `stratum`)
are carried verbatim from the real stratum corpus, unedited.
4. Ran `review-suite/scripts/evals/runner.py` against that corpus and skill root
with `--runs 5 --timeout 300` (both matching `FROZEN-V2-CONFIGURATION.md`'s
pin for the two new ablation runs), the unchanged executor
(`python3 review-suite/scripts/evals/claude_executor.py`), and no other
flags.
5. Graded with the grader already pinned in this worktree's `corpus.json`/report
output: `grader_version 1.1` - the current, fixed grader, same version `#57`
itself used and the same one `FROZEN-V2-CONFIGURATION.md`'s owner-authorized
version transition covers.

## Runtime verification

- `claude --version` in this worktree immediately before scoring: `2.1.92`,
matching `FROZEN-V2-CONFIGURATION.md`'s pinned `runtime_cli_version` exactly.
- `model`: `claude-opus-4-6[1m]` (recorded verbatim in the report's
`configuration.executor_models`, no `--model` flag passed - same invocation
shape as the original ablation runs).
- No runtime or model drift from the frozen configuration.

## Result

| Metric | Original (`#57`, verification-only, full 7-case stratum) | This rerun (verification-only, this case only) |
| ---------------------------------------------------------------------- | -------------------------------------------------------- | ---------------------------------------------- |
| Attempts | 5 | 5 |
| False positive attempts | **3** | **3** |
| `verdict_stability` | 0.6 | 0.6 |
| `finding_stability` | 1.0 | 1.0 |
| Evaluation failures (timeout/spawn/runtime/malformed/output-too-large) | 0 | 0 |
| Mean latency | 50.93s (whole stratum) | 60.71s (this case only) |
| Max latency | 87.2s (whole stratum) | 73.997s |
| Cost | $3.866613 (whole 7-case, 35-attempt stratum) | $0.569074 (this case, 5 attempts) |

**This reproduces exactly: 3 of 5 attempts, the same count as the original
run.** Directly observed in each reproducing attempt's raw output during this
run (the committed report JSON carries only aggregate `per_case`/`quality`
figures, not per-attempt finding IDs, so this specific detail is reported from
first-hand observation rather than as something re-derivable from the committed
JSON alone): the false-positive finding in every reproducing attempt was the
same shape as originally reported - finding_id `corr.verify-finalize-called`, a
`blocking` finding that the test covering the finalize-only session path
"asserts only on the return value" and would not catch a regression that deleted
the `finalize()` call or added an accidental `start_agent()` call - raised
against a case whose expectation record carries no expected root cause
(`expected_root_cause_ids: []`, a deliberate clean control). The two
non-reproducing attempts returned `clean`, matching the case's expected verdict.

No attempt in this rerun produced a `timeout`, `spawn_failure`,
`runtime_failure`, `output_too_large`, or `malformed_output` classification
(every failure-rate field is `0.0`); observed latency (mean 60.71s, max 73.997s)
stayed well under the 300s timeout. This machine was not confirmed free of
concurrent load during this rerun, but the result is unchanged from the original
regardless: the regression reproduces at the same rate under a fresh,
independent set of 5 attempts.

## Interpretation

The original finding **reproduces**. This confirming rerun does not support the
hypothesis that the `3/5` false-positive rate was a shared-machine-load artifact
of `#57`'s own run - a second, independent set of 5 attempts, run at the pinned
runtime/model version, against the identical ablation mechanism and case, again
resolves `3/5`. Per instruction, this record states the number plainly (`3/5`,
matching the original `3/5`) rather than characterizing it as "confirming" a
specific causal theory beyond that: it is evidence the regression is a real,
repeatable property of the verification-sufficiency-pass-only configuration on
this case, not disappearing under a clean rerun.

This record does not decide what should happen to the
verification-sufficiency-pass-only configuration, `#52`, or `#53` as a result.
See `S1-ABLATION-MATRIX.md`'s and the closeout report's own settled verdicts and
recommendations for that.

## Suite commit note

This rerun's own `configuration.suite_commit` (recorded verbatim in the raw
report JSON) is this worktree's `HEAD` at run time,
`b4e061f7847b3fc911a05fe4c8e50218f4f957b7` (`#57`/PR #88's own merge commit) -
later than the original ablation run's
`e2c56f68fe56094a6c92fd4a220539f47d6f9f98`, since this rerun necessarily runs
after `#57` merged. The real `skills/` tree's four target-skill-closure
documents other than the overlaid `review-correctness/SKILL.md` are unchanged
between those two commits. `configuration.target_skill_digest` differs from the
original verification-only report's own recorded digest (`6e05fb283aa9ed69` here
vs. `d85efbc6381a2775` there); this is expected and already disclosed by
`FROZEN-V2-CONFIGURATION.md`'s "Ablation mechanism" section, which notes the
committed overlay file was reformatted by `just fmt-md` *after* the original
scored runs launched (rewrapped line breaks only, no textual change,
independently re-diffed there against the committed file). This rerun built its
skill root from the overlay file's current, already-reformatted, committed
content, so its digest reflects the post-reformat bytes; the original run's
digest reflects the pre-reformat bytes. Neither digest difference reflects any
change in reviewer-visible instruction content.

## Cost ceiling

- This ticket's own hard ceiling: **$1.00**.
- Actual spend: **$0.569074** (5 attempts, one case, one configuration) - the
only real-money spend this ticket incurred.

## Raw evidence

The full raw report (including per-attempt cost, latency, grading detail, and
the exact `configuration` block reproduced above) is committed alongside this
record at
[`session-continuation-summary-confirming-rerun.report.json`](session-continuation-summary-confirming-rerun.report.json).
124 changes: 124 additions & 0 deletions review-suite/evals/v2/session-continuation-summary-confirming-rerun.report.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,124 @@
{
"adjudication_required": [],
"attempts": 5,
"baseline_eligible": true,
"configuration": {
"artifacts_retained": true,
"cases": 1,
"corpus_version": "0.1-s1-populated",
"executor": "python3 review-suite/scripts/evals/claude_executor.py",
"executor_models": [
"claude-opus-4-6[1m]"
],
"grader_version": "1.1",
"max_output_bytes": 4000000,
"runs_per_case": 5,
"skill_root": "/private/tmp/claude-501/-Users-scott--claude-worktrees-agent-scripts-epic-49-implementation-eab73d/d2c1ed44-f510-4ae0-8cdd-ed14e9491168/scratchpad/rerun/skills-verification-only",
"stratum": {
"grading_is_signal": true,
"ground_truth": "human-review",
"id": "s1-correctness-orchestrator",
"purpose": "Correctness and verification cases for the orchestrator target, whose payload therefore carries its three required lens skills. Every case carries the recorded source disposition as its first adjudication and an executable oracle - a runnable module independent of the reviewer's model - as its second. SCORED: both owner-gated inputs are satisfied (the $15 total cost ceiling and complete independent adjudication across all three strata), and every expectation stays `calibrated: false` permanently, per the owner-settled three-way grading method - a scored case is never calibrated on its own prose, and a grader miss surfaces as `referred_root_cause_ids` rather than a silent reviewer-miss.",
"scored": true
},
"suite_commit": "b4e061f7847b3fc911a05fe4c8e50218f4f957b7",
"target_skill": "review-code-change",
"target_skill_dependencies": [
"review-solution-simplicity",
"review-correctness",
"review-code-simplicity"
],
"target_skill_digest": "6e05fb283aa9ed69",
"target_skill_documents": [
"review-code-change/SKILL.md",
"review-code-change/references/orchestration-protocol.md",
"review-code-simplicity/SKILL.md",
"review-code-simplicity/references/code-simplicity-rubric.md",
"review-correctness/SKILL.md",
"review-correctness/references/correctness-rubric.md",
"review-solution-simplicity/SKILL.md",
"review-solution-simplicity/references/solution-simplicity-rubric.md"
],
"timeout_seconds": 300.0
},
"failures": {
"blocked_rate": 0.0,
"malformed_output_rate": 0.0,
"output_too_large_rate": 0.0,
"protocol_mismatch_rate": 0.0,
"runtime_failure_rate": 0.0,
"spawn_failure_rate": 0.0,
"timeout_rate": 0.0
},
"graded_attempts": 5,
"grader_version": "1.1",
"latency": {
"count": 5,
"max_seconds": 73.99658524990082,
"mean_seconds": 60.707393758092074,
"min_seconds": 40.967696791980416,
"p50_seconds": 65.64288329100236
},
"per_case": [
{
"attempts": 5,
"case_id": "session-continuation-summary",
"ever_referred_relevant_root_cause_ids": [],
"ever_referred_root_cause_ids": [],
"expected_root_cause_ids": [],
"false_alarm_attempts": 3,
"false_clean_attempts": 0,
"false_positive_attempts": 3,
"finding_stability": 1.0,
"graded_attempts": 5,
"intersection_root_cause_ids": [],
"mean_combined_recall": null,
"mean_recall": null,
"stability_denominator": 5,
"statuses": {
"review_result": 5
},
"union_root_cause_ids": [],
"verdict_stability": 0.6
}
],
"protocol_version": "1.0",
"quality": {
"false_alarm_denominator": 5,
"false_alarm_rate": 0.6,
"false_clean_denominator": 0,
"false_clean_rate": null,
"false_positive_denominator": 5,
"false_positive_rate": 0.6,
"material_finding_recall": null,
"recall_attempts": 0,
"referred_denominator": 5,
"referred_rate": 0.0,
"unique_finding_contribution": []
},
"report_version": "1.0",
"simulation": false,
"stability": {
"mean_finding_stability": 1.0,
"mean_verdict_stability": 0.6,
"per_case_finding_stability": {
"session-continuation-summary": 1.0
},
"per_case_stability_denominator": {
"session-continuation-summary": 5
},
"per_case_verdict_stability": {
"session-continuation-summary": 0.6
},
"stability_denominator": 5
},
"usage": {
"available": true,
"reporting_attempts_cost_usd": 5,
"reporting_attempts_input_tokens": 5,
"reporting_attempts_output_tokens": 5,
"total_cost_usd": 0.5690740000000001,
"total_input_tokens": 183610.0,
"total_output_tokens": 13126.0
}
}
Loading