feat(review-suite): add connector-outcome curation and promotion tooling - #87
Merged
Conversation
## Summary - Add `curation-record.schema.json` and `promotion-decision.schema.json` under `review-suite/evals/contracts/`, plus `curation.py` and `audit_curation.py` (`just audit-review-curation`) to load and validate them, mirroring the existing corpus/expectation/provenance pattern. - Add the mechanical disclosure guardrail: when a curation record's `provenance.source_class` is `private_authorized`, its public-facing `source_description` must match an allow-listed generic phrase in the new `disclosure-guardrail.json`, and validation fails closed on a path-like token, a bare hostname-shaped token, or a configured deny-listed identifier. The shipped deny-list is empty by design. - Enforce disposition semantics (accepted/rejected/deferred/duplicate/ unresolved validate distinctly), reviewer/private text separation, and promotion-decision rules: unresolved and plain-duplicate records can never be promoted, a global rubric change needs two representative positives plus a negative control, and a repository-instruction change may only target an existing repository-owned instruction file (AGENTS.md or CLAUDE.md) - no new shared path-rule subsystem. - Add synthetic positive, negative, duplicate, unresolved, restricted-data, and promotion-decision fixtures under `review-suite/evals/curation/`, including three fixtures proving the disclosure guardrail fails closed (path-like token, bare hostname, deny-listed identifier), plus 39 new tests in `test_eval_curation.py`. - Document the workflow in `review-suite/evals/curation/README.md` and the root `README.md`. ## Why - #56 requires a conservative, auditable workflow for turning adjudicated connector outcomes into regression cases and, only when evidence warrants it, rubric or repository-instruction updates - proven here as infrastructure with synthetic fixtures only, per its own non-goals. - #56's own disclosure boundary requires the mechanical guardrail to be real, tested, fail-closed code, not a comment or convention, because curation discipline alone has already been shown elsewhere in this repository (`review-suite/evals/baseline/v1/LIMITATIONS.md` items 21-24, 29-31) not to reliably catch every leak on its own. - None of this data resembles or is derived from any real private source; `baseline/v1/` and `v2/` are untouched. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…icate_of chain ## Summary - `_promotable_errors` now resolves a `duplicate` record's effective disposition by following `duplicate_of` (via new `_resolve_duplicate_disposition`, with cycle detection) before allowing it to fill a promotion decision's positive or negative slot. - Add two synthetic fixtures (`cache-ttl-duplicate-with-new-surface`, `unresolved-duplicate-claim`) and six regression tests covering: a distinct duplicate of an accepted record used as a negative control, of a rejected record used as a positive case, of an unresolved claim, a multi-hop duplicate chain, and a duplicate cycle. ## Why - Independent correctness review of this candidate found that a distinct duplicate's own disposition is always `duplicate` (never `accepted_*`/`rejected_*`), so the prior check let a duplicate of a rejected false positive support a positive case, or a duplicate of an accepted defect support a negative control - feeding a rejected non-finding into a rubric change as if it were a real defect, or the reverse. Resolving through the chain closes that gap without weakening the existing double-counting protection for a plain duplicate. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
… its membership check ## Summary - `_resolve_duplicate_disposition` is now an iterative loop with a local `seen` set instead of self-recursion carrying a `_seen` frozenset accumulator parameter. Same behavior (including cycle detection), simpler control flow. - `_promotable_errors` now computes one `(effective_id, effective_disposition)` pair after the duplicate-only early returns, then runs the accepted/rejected disposition-membership check exactly once, selecting the existing error-message wording only on whether `effective_id == record_id`. Previously the same membership rule was checked twice: once for a duplicate's resolved disposition, once for a direct record's own disposition. - No behavior change: all 45 existing curation tests pass unmodified, including the multi-hop chain and cycle-detection cases. ## Why - Code-simplicity review of this candidate flagged both as unnecessary local complexity: recursion for a simple bounded linear traversal, and duplicated membership-check logic that could drift out of sync between its two copies. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Builds the intake schema, validator, and promotion workflow for turning
adjudicated connector review outcomes into result-blind regression cases and,
only when evidence warrants it, updates to the global rubric or a
repository-owned instruction file. This is infrastructure only, proven with
synthetic fixtures — it does not source, adjudicate, or import any real
connector finding, and it never touches
review-suite/evals/baseline/v1/orreview-suite/evals/v2/(frozen artifacts from #58/#59).review-suite/evals/contracts/curation-record.schema.json— one adjudicatedconnector claim: disposition (8-value vocabulary), public/private evidence
split, and duplicate/unresolved handling.
review-suite/evals/contracts/promotion-decision.schema.json— thenarrowest-owner decision a group of curation records supports: a corpus case
only, a global rubric change, a repository-instruction change, or nothing.
review-suite/evals/contracts/disclosure-guardrail.json+curation.disclosure_guardrail_errors— the mechanical disclosure guardrail:when a record's
provenance.source_classisprivate_authorized, itspublic-facing
source_descriptionmust match an allow-listed generic phrase,and validation fails closed on a path-like token, a bare hostname-shaped
token, or a configured (empty-by-default) deny-listed identifier.
review-suite/scripts/evals/curation.py+audit_curation.py(
just audit-review-curation) — the loader/validator, mirroring the existingcorpus.py/audit_corpus.pypattern.review-suite/evals/curation/— synthetic positive, negative, duplicate,unresolved, restricted-data, and promotion-decision fixtures, including three
fixtures proving the disclosure guardrail fails closed (path-like token,
bare hostname, deny-listed identifier), plus a README documenting the
workflow.
review-suite/scripts/tests/test_eval_curation.py.Why
Turning real adjudicated connector findings into regression cases or rubric
changes needs a conservative, auditable path so an unresolved or duplicate
claim can never quietly become a gating expectation, and so a disclosure
boundary this repository has already needed elsewhere doesn't rely on
convention alone. This ticket's own body requires the guardrail to be real,
tested, fail-closed code, not a comment — that requirement is what
disclosure_guardrail_errorsand its three failing fixtures exist to satisfy.Test plan
just formatjust lintjust test(311 tests, including the 45 new curation tests)python3 review-suite/scripts/evals/audit_curation.py— curation auditpassed
review-code-change(solution-simplicity,correctness, code-simplicity) clean at this head
Fixes #56