Skip to content

feat(review-suite)!: remove the unproven verification-sufficiency pass - #106

Merged
shaug merged 2 commits into
mainfrom
claude/93-remove-verification-sufficiency
Jul 30, 2026
Merged

feat(review-suite)!: remove the unproven verification-sufficiency pass#106
shaug merged 2 commits into
mainfrom
claude/93-remove-verification-sufficiency

Conversation

@shaug

@shaug shaug commented Jul 30, 2026

Copy link
Copy Markdown
Owner

Summary

Removes the verification-sufficiency pass and its mandatory
verification_sufficiency_evidence field from review-correctness and the
shared review-result contract, advancing schema_version 1.3 → 1.4. The
traversal (consumer/impact) pass and consumer_impact_evidence — the other
mechanism #52/#53 added — are unchanged, byte-for-byte.

  • skills/review-correctness/SKILL.md: removed the verification-sufficiency
    pass instruction. The underlying question (does a claimed test actually
    exercise the triggering condition it addresses?) remains part of ordinary
    correctness review, just without a separately mandated pass or required
    evidence field. The traversal pass instruction is preserved exactly as
    shipped.
  • review-suite/contracts/review-result.schema.json,
    review-suite/CONTRACT.md, review-suite/scripts/validate.py:
    verification_sufficiency_evidence removed entirely (full removal, not an
    optional field — see the decision record for why); schema_version
    1.3 → 1.4; a stale 1.3 result now fails with a clear migration error.
  • Every canonical fixture, standalone expected result, and orchestration case
    across all six bundled review skills migrated to 1.4 atomically; all six
    bundled references/review-suite/ copies refreshed via just sync-contracts and confirmed byte-identical.
  • verification-sufficiency-guard fixture retired (its sole purpose was the
    removed field's clean-pairing behavior). standalone-verification- sufficiency-gap adapted, not deleted: its underlying defect pattern is
    a genuine correctness catch independent of the removed pass (see
    review-suite/fixtures/missing-test/ for the same pattern caught by
    ordinary review), so its changes_required verdict and blocking finding
    are preserved.
  • stale-claim-release-guard and audit-log-flush-keyword-probe corpus
    cases untouched — still general correctness cases.
  • New decision record: review-suite/evals/v2/VERIFICATION-SUFFICIENCY-REMOVAL.md,
    citing S1-ABLATION-MATRIX.md and discriminating-case-validation.md
    directly.
  • README.md and CHANGELOG.md updated to describe the removal and its
    evidentiary basis.

Why

Two independent, honestly-sourced measurements found no demonstrated value
for this pass:

On top of the absence of demonstrated value, the pass carries a confirmed,
twice-reproduced cost: run in isolation (traversal pass disabled), it
regresses session-continuation-summary — a previously clean control — to
3/5 false positives.

By contrast, the traversal pass does show a real, reproducible gap in
the same evidence (artifact-promotion-environment-shortcut,
mean_combined_recall 1.0 → 0.4) and is out of scope for this removal.

This ticket executes a removal decision already approved by the repository
owner on 2026-07-29; it does not re-litigate the evidence.

Non-goals honored

  • Traversal pass / consumer_impact_evidence: untouched (verified
    byte-identical against base, independently confirmed by a fresh
    review-code-change cycle).
  • stale-claim-release-guard / audit-log-flush-keyword-probe corpus cases:
    untouched as corpus content.
  • review-suite/evals/baseline/v1/, gate-manifest.json,
    DECISION-RECORD.md, FAILURE-TAXONOMY.md, audits/, and every existing
    review-suite/evals/v2/ measurement document (S1-ABLATION-MATRIX.md,
    FROZEN-V2-CONFIGURATION.md, CLOSEOUT-REPORT.md,
    discriminating-case-validation.md, etc.): untouched.
  • Epic Epic: Make review-clean verdicts evidence-backed and empirically evaluated #49 closure: not performed (separate, later action).
  • carve-changesets: not invoked — this candidate fit an ordinary single PR.

Test plan

  • just format — all checks passed, no reformatting needed
  • just lintskills-ref validate passed for all 8 skills; plugin
    packaging validation passed
  • just test — 432 tests across review-suite/scripts/tests (310) and
    all 6 skill-level scripts/tests suites (122), all passing
  • Fresh, isolated review-code-change cycle at the final head — clean
    aggregate verdict, zero findings across all three lenses
    (solution-simplicity, correctness, code-simplicity)

Fixes #93

shaug added 2 commits July 30, 2026 07:02
- Remove the verification-sufficiency pass and its mandatory
  `verification_sufficiency_evidence` field from `skills/review-correctness/SKILL.md`
  and the shared review-result contract; advance `schema_version` `1.3 -> 1.4`.
- Update `review-suite/CONTRACT.md` and `review-suite/scripts/validate.py` to
  match: a `clean` verdict no longer requires or accepts this evidence, and a
  stale `1.3` result now fails with a clear migration-to-`1.4` error.
- Migrate every canonical fixture, standalone expected result, and orchestration
  case to `schema_version` `1.4`, then refresh all six bundled
  `references/review-suite/` copies with `just sync-contracts`.
- Retire the `verification-sufficiency-guard` fixture (its sole purpose was the
  removed field's clean-pairing behavior); adapt, not delete, the
  `standalone-verification-sufficiency-gap` eval case, since its underlying
  defect pattern (a claimed test that misses the actual triggering condition)
  is still a genuine correctness catch expected from ordinary review, per
  `review-suite/fixtures/missing-test/`.
- Add `review-suite/evals/v2/VERIFICATION-SUFFICIENCY-REMOVAL.md`, the decision
  record citing `S1-ABLATION-MATRIX.md` and `discriminating-case-validation.md`
  directly, and describe the removal in `README.md` and `CHANGELOG.md`.

- Two independent, honestly-sourced measurements found no demonstrated value
  for this pass: `stale-claim-release-guard` (#57's ablation matrix) and
  `audit-log-flush-keyword-probe` (#89's harder-case validation) both resolve
  identically at `mean_combined_recall 1.0` whether the pass is enabled or not.
- It also carries a confirmed, twice-reproduced cost: run in isolation
  (traversal pass disabled), it regresses `session-continuation-summary` - a
  clean control - to 3/5 false positives.
- The traversal (consumer/impact) pass and `consumer_impact_evidence` showed a
  real, reproducible gap in the same evidence
  (`artifact-promotion-environment-shortcut`, 1.0 vs 0.4) and are explicitly
  out of scope for this ticket; both are verified byte-identical to their
  #52/#53-shipped state.
- Approved directly by the repository owner (#93), executing a removal
  decision already made rather than re-litigating it.

Fixes #93
@shaug
shaug merged commit 8e250e4 into main Jul 30, 2026
1 check passed
@shaug
shaug deleted the claude/93-remove-verification-sufficiency branch July 30, 2026 14:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Remove the unproven verification-sufficiency pass from review-correctness

1 participant