Source two harder discriminating cases for the traversal and verification-sufficiency passes - #92
Merged
Merged
Conversation
…ister their validation ceiling (#89) - Add `artifact-promotion-environment-shortcut`, a harder sibling to `dependency-strictness-propagation`: the untouched sibling decision path is two calls deep and lives in a module named for staging/rollout defaults, not dependency or promotion checking. - Add `audit-log-flush-keyword-probe`, a harder sibling to `stale-claim-release-guard`: the test insufficiency is carried by a mocked session that grants success on a keyword match in the request text, rather than by the test's initial state. - Both cases are sourced from real, adjudicated, public review findings in shaug/atelier (PR 315 and PR 755 respectively), minimized to fictional subjects per #58's sourcing/sanitization discipline, and each ships an executable oracle that independently confirms the stated root cause. - Register both cases in the existing `s1-correctness-orchestrator` corpus (`corpus_version` 0.1 -> 0.2) and freeze the validation configuration (suite commit, grader version, run count, $5.00 cost ceiling) before any scored output is examined. - #57's ablation matrix found neither existing target case demonstrates unique causal contribution to the pass meant to fix it, because both sit within the reviewer's default competence even without the pass. This sources genuinely harder cases so the corpus can actually discriminate, or honestly report that it still cannot.
…ort an asymmetric result (#89) - Fix a construction defect found in `artifact-promotion-environment-shortcut` during its first scored run: the packet never stated `ledger.lookup`'s contract for a missing external ref, so 4/5 verification-only attempts raised an unrelated `None`-fallthrough concern instead of the intended rollout-path finding. State the contract explicitly in the packet and correct the oracle's `_Ledger` to match; re-mint the case's `head_sha`. - Re-run both traversal-case configurations against the fixed packet: both passes together `mean_combined_recall` 1.0 (5/5); verification-only (traversal disabled) 0.4 (2/5, 3/5 false-clean, zero false positives). A real, reproducible, uncontaminated discriminating gap. - Score `audit-log-flush-keyword-probe` both ways: both together 1.0 (5/5); traversal-only (verification-sufficiency disabled) also 1.0 (5/5). No gap, same pattern as the existing `stale-claim-release-guard` target case. - Record both results, the sourcing/sanitization table, and the cost ledger in `review-suite/evals/v2/discriminating-case-validation.md`, with the four trustworthy raw reports committed alongside it. - #89 requires validating discriminating power before treating either new case as corpus evidence, and reporting honestly even when the answer is "no gap" for one of the two. The traversal pass now has one real demonstrated case; the verification-sufficiency pass still does not, even after a genuine, harder, disguised construction attempt. - $3.6657 total against the preregistered $5.00 ceiling (30 attempts across six scored runs, including 10 discarded pre-fix attempts whose real cost still counts against this ticket's spend even though their output was not used as evidence).
…cted retry policy (#89) ## Summary - Replace `_flush_candidate`/`_flush_corrected` with one `_flush(..., *, refresh_on_retry: bool)`, mirroring the sibling oracle's own established `guard_unconditionally` flag idiom instead of duplicating the entire retry-policy body across two near-identical functions. ## Why - review-code-change (code-simplicity lens, strong_recommendation): the new oracle repeated its whole policy body across two functions when the directory it lives in already established a single-function-plus-flag shape for this exact kind of candidate/corrected difference (`stale_claim_release_guard.py`). Oracle polarity is unchanged - re-verified by `python3 -m unittest review-suite.scripts.tests.test_eval_oracles`.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
s1-correctness-orchestratorcases, per Curate the scored corpus and capture the frozen v1 review baseline #58's sourcing/sanitization discipline:artifact-promotion-environment-shortcut— a harder sibling todependency-strictness-propagation: the untouched sibling decision path is two calls deep and lives in a module named for staging/rollout defaults, not dependency or promotion checking.audit-log-flush-keyword-probe— a harder sibling tostale-claim-release-guard: the test insufficiency is carried by a mocked session that grants success on a keyword match in the request text, rather than by the test's initial state.shaug/atelier(PR 315, PR 755), minimized to fictional subjects carrying no business logic, domain identifiers, customer context, credentials, or hidden reasoning. The private connector-review source referenced in Operationalize adjudicated connector outcomes as review regressions #56 was not sought out, named, or derived from anywhere in this work.s1-correctness-orchestratorcorpus (corpus_version0.1 → 0.2).mean_combined_recall1.0 with the pass enabled vs. 0.4 disabled); the verification-sufficiency case shows none (1.0 both ways) — the same null result its predecessor case already showed.ledger.lookupcontract caused an unrelated false-positive concern), then re-score before treating the result as evidence.Why
#57's ablation matrix found that neither
dependency-strictness-propagationnorstale-claim-release-guarddemonstrates unique causal contribution to the pass meant to fix it — both cases sit within the reviewer's default competence even without the pass. This is a corpus-evidence gap, not a decision to remove or defend #52/#53's shipped mechanisms. This PR sources genuinely harder cases to close that evidence gap, or to honestly report that it remains open.What this means for #52/#53 (report only — the disposition decision belongs to the repository owner): the traversal pass now has one real, demonstrated case behind it. The verification-sufficiency pass still does not, even after a genuine, harder, disguised construction attempt — combined with the existing false-positive regression finding in
S1-ABLATION-MATRIX.md, the evidentiary case for this specific pass is weaker after this validation than before it.Validation
just format,just lint,just testall pass (313 review-suite tests + all skill-local suites).python3 review-suite/scripts/evals/audit_corpus.pypasses (7 corpora).python3 -m unittest review-suite.scripts.tests.test_eval_oraclespasses (12/12), confirming both new oracles' polarity (fails on candidate, holds on corrected).review-suite/evals/v2/discriminating-case-validation.mdandreview-suite/evals/v2/discriminating-case-validation-frozen-configuration.md.review-code-changecycle raised onestrong_recommendation(code-simplicity: unify the new oracle's duplicated candidate/corrected functions into the sibling oracle's established single-function-plus-flag idiom); fixed and re-reviewed clean.Non-goals
review-suite/evals/baseline/v1/,gate-manifest.json,DECISION-RECORD.md,FAILURE-TAXONOMY.md,audits/,S1-ABLATION-MATRIX.md,FROZEN-V2-CONFIGURATION.md,CLOSEOUT-REPORT.md, or the Run the preregistered review v2 ablation and integration closeout #57/docs(review-suite): confirm the session-continuation-summary verification-only regression #90 confirming-rerun files.skills/review-correctness/SKILL.md, the traversal pass, or the verification-sufficiency pass.Fixes #89