Evaluate independent correctness discovery and per-finding validation

[shaug]

Parent: #49

Blocked by: #53

Definition status

Provisional until #59 and the measured single-pass improvements from #53. Independent explorers and validators are an optional escalation, not a predetermined architecture. Before unblocking, the evidence record must show a remaining recall or stability failure that one-pass changes do not solve, preregister the expected gain and cost bound, and specify the minimum explorer/validator topology. Otherwise narrow or close this ticket.

Measured — the required evidence does not exist, in either direction

A bounded re-score task (not a numbered ticket) measured this directly, then merged a real grader fix and its own findings under review-suite/evals/v2/: s1-rescore-pre-post-comparison.md, GRADER-1.1-COMPARABILITY.md, s1-rescore-frozen-configuration.json, pre53-s1-correctness-orchestrator-2case.report.json.

The two cases this ticket's gate depends on (dependency-strictness-propagation, stale-claim-release-guard) were re-run fresh under the pre-#53 correctness lens (commit 8e4fdbd) and re-graded, alongside the existing post-#53 run, under a corrected grader (grader.py's finding_surfaces() previously read only structured location fields, missing a symbol named only in prose — fixed in PR #84, a247d47). Result: both cases pass the preregistered gate (mean_combined_recall: 1.0) in both eras — pre-#53 and post-#53 alike, every one of 20 checked attempts individually, not an averaged artifact.

This means the frozen v1 baseline's "confident miss" classification for these two cases was very likely the same grader artifact, not a demonstrated reviewer capability gap — the pre-#53 reviewer already found both defects correctly and consistently, before #53's traversal/verification-sufficiency passes ever existed. v1's own raw attempts no longer exist anywhere to confirm this attempt-by-attempt, but the mechanism, code, and expectations are unchanged, and the reported shape matches exactly.

Consequence for this ticket: the "remaining recall or stability failure" this ticket's own gate requires does not exist — not because #53 solved it, but because it was very likely never a real gap on these two cases in the first place. This is measured evidence pointing toward this ticket's own "narrow or close" fallback, not toward unblocking. Disposition is the owner's call, not decided here.

Goal

Evaluate whether the residual failures approved by #59 justify increasing discovery diversity through independent correctness explorations in fresh contexts, then validating every candidate finding before it can affect the aggregate verdict.

Why

One broad correctness pass is vulnerable to anchoring and search-order effects. Repeating the same prompt in the same context is not independent review, and majority voting can discard a real defect found by only one reviewer. The orchestration needs separate discovery paths and evidence-based adjudication.

Required architecture

Independent discovery

• Run two correctness explorers by default:
  1. a contract-and-tests-first explorer;
  2. an impact/dataflow/state-transition-first explorer.
• Give each explorer the same frozen review packet and candidate identity, but use distinct traversal instructions.
• Run a conditional specialist explorer only if a future evidence-backed revision of this ticket demonstrates a domain whose evidence the two default explorers cannot fully cover; Use the v1 baseline to finalize and preregister the review v2 graph #59's decision record found no baseline case justifying risk-routed specialist modules, and Add consumer/impact-traversal evidence to the shared review contract #52/Add correctness traversal and verification-sufficiency passes #53 do not introduce a risk profile for this ticket to key off of, so this bullet remains hypothetical rather than tied to an existing mechanism.
• Execute each explorer in a fresh isolated context. Do not expose another explorer's findings, verdict, or hidden reasoning.
• Preserve result-blind execution: expected fixture outcomes and connector dispositions must not enter reviewer context.

Candidate union and validation

• Normalize and deduplicate candidate findings by affected contract, location, trigger, and consequence; do not collapse findings merely because they touch the same file.
• Send every material candidate finding to a fresh validator context with only:
  • the frozen review packet and repository evidence;
  • the candidate finding as an untrusted claim;
  • the validator rubric.
• The validator must return confirmed, rejected, or blocked, with a reproducer or trace and an explicit explanation of candidate causality and material impact.
• A confirmed finding survives even if only one explorer discovered it. Agreement changes confidence, not truth.
• Rejected findings remain in evaluation telemetry but cannot affect the user-facing verdict.
• If more than 12 material candidates require validation in one run, return blocked with the count and recommend narrowing or splitting the change; never silently truncate the set.

Runtime and portability requirements

• Put provider-neutral orchestration and schemas in the portable review core.
• Implement equivalent fresh-context behavior for the supported OpenAI and Claude runtime adapters.
• If a runtime cannot establish fresh isolated contexts or identify the exact candidate commit, fail closed with blocked.
• Keep the review workflow read-only.

Aggregate result

The aggregate must record:

• explorer identities and strategies;
• fresh-context execution status;
• candidate provenance and deduplication decisions;
• validator outcome for every candidate;
• unique contribution by explorer or specialist;
• final findings and verdict;
• current candidate/base identity and contract version.

Only confirmed findings may produce changes_required. Missing or failed mandatory exploration/validation produces blocked; it cannot produce clean.

Expected change surface

• skills/review-code-change/ orchestration, contracts, runtime adapters, and tests.
• skills/review-correctness/ strategy entrypoints or references needed for the distinct explorers.
• Real-runtime replay harness integration from Build the result-blind review replay evaluator and corpus contract #50.
• Generated bundles for every touched skill.

Acceptance criteria

• A seeded defect found by only one explorer survives validation and makes the aggregate changes_required.
• Duplicate descriptions of the same defect produce one user-facing finding with complete provenance.
• A plausible but false candidate is rejected by the validator and does not contaminate the verdict.
• Explorer order does not change the logical aggregate result for deterministic fixtures.
• Missing isolation, failed explorer execution, failed validation, candidate overflow, or stale commit identity returns blocked.
• OpenAI and Claude adapters demonstrate equivalent isolation and aggregation semantics.
• Repeated result-blind runtime replay records stability, unique contribution, false positives, latency, and cost.
• just format, just lint, and just test pass.

Non-goals

• Requiring majority agreement before reporting a defect.
• Treating reviewer confidence as a substitute for validation evidence.
• Adding more always-on public lenses.
• Automatically editing the candidate change.