From 79cd99313962b378afb2f1e6d76426b110114a0e Mon Sep 17 00:00:00 2001 From: Scott Haug Date: Tue, 28 Jul 2026 15:16:38 -0700 Subject: [PATCH 1/5] docs: run the three required graph audits against the live #51-#57 graph ## Summary - Add review-suite/evals/v2/audits/scope-completeness-audit.md, dependency-sequencing-audit.md, and shovel-readiness-audit.md, each an independent pass over the live GitHub issue graph after #59's approved #51-#57 body edits and native-edge read-back. - Fix two stale cross-references the scope/completeness pass found in the live tracker: #56 referenced a #52 changed-surface concept that #52's narrowing dropped, and #54 referenced a #52 risk profile that no longer exists anywhere in the graph. Neither ticket's disposition changed. - Record a summary of all three audits' findings in DECISION-RECORD.md, and point both v2/README.md and the top-level evals/README.md at the finished audit files. ## Why - #59 requires all three audits to run independently against the live graph, with findings and fixes recorded separately per pass, before the ticket can close. All three passed clean, with one flagged-but-non-blocking open question (whether #56's blockedBy #52 edge still has a real technical justification after #52's narrowing) reported for the owner rather than unilaterally resolved. --- review-suite/evals/README.md | 2 +- review-suite/evals/v2/DECISION-RECORD.md | 39 +++++- review-suite/evals/v2/README.md | 11 +- .../v2/audits/dependency-sequencing-audit.md | 121 ++++++++++++++++++ .../v2/audits/scope-completeness-audit.md | 85 ++++++++++++ .../evals/v2/audits/shovel-readiness-audit.md | 90 +++++++++++++ 6 files changed, 340 insertions(+), 8 deletions(-) create mode 100644 review-suite/evals/v2/audits/dependency-sequencing-audit.md create mode 100644 review-suite/evals/v2/audits/scope-completeness-audit.md create mode 100644 review-suite/evals/v2/audits/shovel-readiness-audit.md diff --git a/review-suite/evals/README.md b/review-suite/evals/README.md index 72153b7..2a17a7e 100644 --- a/review-suite/evals/README.md +++ b/review-suite/evals/README.md @@ -44,7 +44,7 @@ review-suite/evals/ │ ├── FAILURE-TAXONOMY.md every v1 material outcome, classified │ ├── DECISION-RECORD.md per-mechanism evidence and disposition │ ├── gate-manifest.json preregistered v2 gates, before scoring -│ └── audits/ the three graph audits (later step, empty for now) +│ └── audits/ the three graph audits, each its own pass └── artifacts/ opt-in captured output, not in git review-suite/scripts/evals/ diff --git a/review-suite/evals/v2/DECISION-RECORD.md b/review-suite/evals/v2/DECISION-RECORD.md index 50bef99..16bd025 100644 --- a/review-suite/evals/v2/DECISION-RECORD.md +++ b/review-suite/evals/v2/DECISION-RECORD.md @@ -432,5 +432,40 @@ required by #59's own sequencing (scope/completeness, dependency/sequencing, shovel-readiness) run only after the live `#51`–`#57` body and edge changes below are actually applied, as their own later step. Their findings — including the read-back that confirms or corrects this predicted net effect — belong in -`audits/` once that step completes; no file exists there yet in this repository -candidate, and none is claimed to. +`audits/` once that step completes. + +## Audit results (post-mutation) + +The live `#51`–`#57` body edits and the read-back above were applied, and all +three required audits were run independently against the resulting live graph. +Full findings are in +[`audits/scope-completeness-audit.md`](audits/scope-completeness-audit.md), +[`audits/dependency-sequencing-audit.md`](audits/dependency-sequencing-audit.md), +and [`audits/shovel-readiness-audit.md`](audits/shovel-readiness-audit.md). +Summary: + +- **Scope/completeness:** clean. Two stale cross-references (#56's mention of a + dropped #52 changed-surface concept, and #54's mention of a dropped #52 risk + profile) were found and corrected in the live tracker as part of this pass. No + outcome is duplicated across siblings; none that demonstrates a reviewer + defect is left unowned. +- **Dependency/sequencing:** clean, with one flagged-but-unresolved open + question that does not change the audited outcome: #56's `blockedBy #52` edge + has a weaker technical justification after #52's narrowing than the other + edges in the graph, but no evidence shows it is actively wrong, and severing + it was outside this ticket's settled disposition for #56, so it is reported to + the owner rather than unilaterally changed. The predicted net effect — #51 is + the sole open issue with zero open blockers once #59 closes — is confirmed + against the live graph read directly through GraphQL. +- **Shovel-readiness:** clean. Every retained child (#51, #52, #53, #56) has a + settled observable outcome, change surface, compatibility/failure behavior, + positive and negative tests, validation, non-goals, and a plausible one-PR + boundary (two children carry an explicit note on that boundary rather than an + unstated assumption). Deferred children (#54, #55, #57) are correctly + excluded, per #59's own text that provisional markers persist on children that + do not become unblocked in this round. + +No finding from any of the three audits required a materially different +architecture or a ticket split, so none of #51–#57's dispositions changed as a +result of running them, and no second iteration of this repository candidate or +the live graph was needed. diff --git a/review-suite/evals/v2/README.md b/review-suite/evals/v2/README.md index 36d5ad0..2bfa7b0 100644 --- a/review-suite/evals/v2/README.md +++ b/review-suite/evals/v2/README.md @@ -19,11 +19,12 @@ implementation-ready decision for #51–#57. efficiency bounds, and the rules for invalid runs, missing data, runtime drift, and threshold changes. Committed before any scored v2 output exists; every numeric threshold is marked as a proposal requiring owner confirmation. -- `audits/` — **not yet produced by this repository candidate.** #59's own - sequencing runs the three independent graph audits (scope/completeness, - dependency/sequencing, shovel-readiness) only after the live `#51`–`#57` body - and native-edge mutations are applied as a later, separate step; each pass's - findings will be recorded here, as its own file, once that step completes. +- [`audits/scope-completeness-audit.md`](audits/scope-completeness-audit.md), + [`audits/dependency-sequencing-audit.md`](audits/dependency-sequencing-audit.md), + [`audits/shovel-readiness-audit.md`](audits/shovel-readiness-audit.md) — the + three required graph audits, each run as its own independent pass against the + live `#51`–`#57` graph after the body and native-edge mutations described + above, with findings and fixes recorded separately per pass. ## What this is not diff --git a/review-suite/evals/v2/audits/dependency-sequencing-audit.md b/review-suite/evals/v2/audits/dependency-sequencing-audit.md new file mode 100644 index 0000000..ad78fdb --- /dev/null +++ b/review-suite/evals/v2/audits/dependency-sequencing-audit.md @@ -0,0 +1,121 @@ +# Dependency/sequencing audit + +Run against the live `#51`–`#57` graph, read directly through GraphQL on +2026-07-28 (queries and raw results below). Requirement (#59's own text): native +edges match artifact prerequisites; only genuine leaves are actionable; no child +can implement an unapproved mechanism or integrate a contract that does not +exist. Independent of `scope-completeness-audit.md` and +`shovel-readiness-audit.md`. + +## Live graph read-back + +`gh api graphql` query per issue, `blockedBy`/`blocking`/`parent`/`state`/ +`updatedAt`, 2026-07-28: + +| issue | state | parent | blockedBy (state) | blocking (state) | +| ----- | ----- | ------ | ---------------------- | ---------------------- | +| #51 | OPEN | #49 | #59 (OPEN) | #52 (OPEN) | +| #52 | OPEN | #49 | #51 (OPEN) | #56 (OPEN), #53 (OPEN) | +| #53 | OPEN | #49 | #52 (OPEN) | #54 (OPEN) | +| #54 | OPEN | #49 | #53 (OPEN) | #55 (OPEN) | +| #55 | OPEN | #49 | #54 (OPEN) | #57 (OPEN) | +| #56 | OPEN | #49 | #52 (OPEN) | #57 (OPEN) | +| #57 | OPEN | #49 | #56 (OPEN), #55 (OPEN) | (none) | +| #59 | OPEN | #49 | #58 (CLOSED) | #51 (OPEN) | + +No native edge was added or removed by this ticket. #51's `blockedBy #59` edge +is deliberately left in place rather than deleted once #59 closes, matching this +repository's own established convention: #58's `blockedBy #50` edge is still +present today even though #50 has been closed since before #58 started (verified +in the same query above, and independently re-confirmed via a direct `#58` query +returning `blockedBy: [{number: 50, state: CLOSED}]`). `implement-epic`'s own +readiness rule (`skills/implement-epic/references/github.md`: "Choose an +in-scope child whose native `blockedBy` set has no open issue") keys off open +blockers, not edge presence, so this is consistent with how the epic +orchestrator actually selects work. + +## Predicted vs. actual net effect once #59 closes + +`DECISION-RECORD.md`'s "Net effect on the dependency graph" section predicts: +once #59 closes, #51 has zero *open* blockers (its only blocker, #59, is closed) +and is the only issue in the epic with that property. Checking every other +child's `blockedBy` set above against the current *open* graph: #52 (#51, open), +#53 (#52, open), #54 (#53, open), #55 (#54, open), #56 (#52, open), #57 (#56, +#55, both open). Every one of #52–#57 has at least one open blocker regardless +of #59's state. **Confirmed: #51 is the only genuine leaf once #59 closes; the +predicted net effect matches what the live graph actually produces.** #59 +remains open as of this audit (it is being closed only after this audit and the +shovel-readiness audit both pass, per this ticket's own required sequencing), so +this is a verified prediction about the graph's structure, not yet an +observation of #51 sitting unblocked in production — the structural fact (every +other child's *independent* open blocker) does not depend on #59's own state to +hold. + +## Artifact-prerequisite check + +Confirmed each edge corresponds to a real, stated artifact dependency, not +merely a sequencing convention: + +- **#52 → #51**: #52's body states its `consumer_impact_evidence` addition bumps + `review-result.schema.json` `"1.1 → 1.2"`, explicitly "delivered by #51." Real + prerequisite: #52 cannot define a delta from a schema version #51 hasn't + shipped yet. +- **#53 → #52**: #53's body bumps `"1.2 → 1.3"`, and its traversal pass + populates `consumer_impact_evidence`, a field #52 defines. Real prerequisite. +- **#54 → #53**: #54's own gate text requires "the measured single-pass + improvements from #53" — a result that can only exist after #53 ships and is + scored. Real prerequisite, and correctly *not* pointed at #59 (per + `DECISION-RECORD.md`'s explicit instruction: "Keep it blocked on #53's + eventual evidence, not #59 itself" — confirmed: #54's `blockedBy` is `#53` + only). +- **#55 → #54**: #55's gate depends on "#54 (or its evidence-backed + replacement)" establishing final orchestrator behavior. Real prerequisite. +- **#57 → #55, #56**: #57 requires "migrated caller integration from #55" and + "operational feedback/corpus workflow from #56" as named required inputs. Real + prerequisite for both edges. +- **#56 → #52**: weaker than the others after #52's narrowing. #56's body, + post-fix, no longer claims any technical dependency on #52's + `consumer_impact_evidence` schema — its intake/curation record is explicitly + "compatible with the corpus contract from #50/#58," a different, + already-existing contract family. **This is a genuine open question, not a + clear defect**: no evidence found that #56 *cannot* be implemented before #52 + ships, but no evidence was found that it structurally *must* wait either. Per + this ticket's own instruction to report a discrepancy rather than quietly + reconcile it, this edge is left unchanged (removing it was not part of the + owner's settled disposition for #56, which addressed evidence-backing and + sourcing only, not sequencing), and is flagged here for the owner to confirm + or sever. It does not change the actionable-leaf outcome above either way, + since #52 is itself gated on #51 regardless of whether #56 also waits on it. + +## Unapproved-mechanism check + +Confirmed no child's current body describes implementing a mechanism this +ticket's decision record did not approve, and no child integrates a contract +version that does not yet exist in the sequence above: + +- #51 implements only its own schema `1.0 → 1.1` delta — no forward reference to + a version another ticket owns. +- #52 implements only `1.1 → 1.2`, explicitly built on #51's delivered version, + and explicitly does not add the changed-surface ledger, acceptance trace, or + risk profile the decision record dropped. +- #53 implements only `1.2 → 1.3`, explicitly built on #52's delivered version, + and explicitly does not add any specialist module. +- #54, #55, #57 each state their own gate is not yet satisfied and commit to no + premature implementation; #54's remaining "conditional specialist explorer" + bullet was corrected during the scope/completeness pass to no longer imply a + #52 risk profile that does not exist, and is now explicitly marked + hypothetical. +- #56 integrates only the existing `#50`/`#58` corpus/expectation/provenance + contracts, not any not-yet-shipped review-result schema version. + +No finding here. + +## Result + +**Clean pass, one flagged-but-unresolved open question** (the #56→#52 edge's +weakened-but-not-contradicted justification, reported above for the owner to +confirm or sever) **that does not change the audited outcome**: every native +edge matches a real artifact prerequisite except that one weaker case, only +genuine leaves are actionable, #51 is confirmed as the sole next actionable leaf +once #59 closes, and no child implements an unapproved mechanism or integrates a +contract that does not yet exist in the sequence. diff --git a/review-suite/evals/v2/audits/scope-completeness-audit.md b/review-suite/evals/v2/audits/scope-completeness-audit.md new file mode 100644 index 0000000..6f0ce82 --- /dev/null +++ b/review-suite/evals/v2/audits/scope-completeness-audit.md @@ -0,0 +1,85 @@ +# Scope/completeness audit + +Run against the live `#51`–`#57` graph after the approved body edits and +native-edge read-back on 2026-07-28. Requirement (#59's own text): every +demonstrated failure and required invariant has one owner; no sibling duplicates +it; adjacent product/tooling work is excluded. This is one of three +independently run passes; see `dependency-sequencing-audit.md` and +`shovel-readiness-audit.md` for the other two. + +## Method + +For every material outcome in `../FAILURE-TAXONOMY.md`, confirm it maps to +exactly zero or one owning child ticket (zero only when the decision record +explicitly declines to assign a mechanism), and that no two children claim the +same piece of work. Then scan every child body for cross-references to sibling +scope that could now be stale after #52's narrowing and #53's specialist-routing +removal. + +## Ownership map + +| baseline outcome | owner(s) | notes | +| -------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Deterministic validation/verdict contradiction (no corpus case; reproduction in #51's own body) | #51 only | single owner, schema 1.1 | +| `dependency-strictness-propagation` (consumer/traversal miss) | #52 (schema 1.2) + #53 (traversal pass) | deliberate schema/behavior split, explicitly documented identically in both bodies ("#52 owns only the schema and validator... does not own the lens behavior"; "#53... populates #52's `consumer_impact_evidence`"). Not a duplication. | +| `stale-claim-release-guard` (verification-sufficiency miss) | #53 only (schema 1.3 + behavior) | single owner | +| `process-isolation-assertion` (false-alarm instability) | none (no mechanism proposed) | correctly unowned — recorded as a gate-manifest non-regression floor, not a ticket, since no fix is being built, only continued measurement | +| `optional-tool-probe` (mixed partial) | none (explicitly not actionable) | correctly unowned — decision record states this does not justify a new mechanism and is an open calibration question, not a demonstrated reviewer defect | +| Connector-escape stratum (deferred, zero data) | none for *measuring* it; #56 owns a *separate* new connector-outcome workflow | taxonomy and #56's body both explicitly distinguish the still-unmeasured v1 connector-escape stratum from the new, separately-authorized v2-cycle connector-outcome source; no conflation found | +| `registry-client-layering`, `setup-service-path-gateway`, `metrics-label-formatting-duplication`, `watcher-check-policy-duplication` (grader/corpus ambiguity) | none | correctly unowned — not demonstrated reviewer misses | +| `env-inventory-bullet-format` (corpus discriminating-power limitation) | none | correctly excluded as adjacent corpus-curation work, not review-suite mechanism work; no #51–#57 ticket depends on it | + +No baseline outcome has two owners for the same piece of work, and no outcome +that genuinely demonstrates a reviewer defect is left unowned. + +## Adjacent product/tooling work + +Checked that no child's expected change surface reaches outside the review suite +into unrelated product code, deployment, or repository tooling: + +- #51: `review-suite/contracts/`, `review-suite/scripts/validate.py`, + `review-code-change/SKILL.md` — in scope. +- #52: `review-suite/contracts/review-result.schema.json`, + `review-suite/scripts/validate.py`, `review-suite/CONTRACT.md` — in scope. +- #53: `skills/review-correctness/` and the same shared schema — in scope. +- #56: `review-suite/` evaluation area only — in scope; explicitly excludes + scraping or mutating GitHub threads, and explicitly excludes touching + `review-suite/evals/baseline/v1/`. + +No finding here. + +## Cross-reference staleness sweep + +Grepped every child body for references to sibling scope, since #52's narrowing +and #53's specialist-routing removal changed what those siblings now claim to +own. Two stale references were found and fixed as part of this audit pass (both +are prose-consistency fixes required to keep native edges and prose blocker +references in sync, not scope or disposition changes): + +1. **#56**, "Current boundaries" section, previously read "...and #52 may add + changed-surface instruction evidence." #52's changed-surface ledger was + dropped entirely by its narrowing. Fixed to state #52's actual narrowed scope + (consumer/impact-traversal evidence only). +2. **#54**, "Independent discovery" section, previously read "Run a conditional + specialist explorer when the risk profile introduced by #52 selects a + domain..." — #52 never introduces a risk profile (that apparatus was dropped + in the narrowing), and #53 also drops specialist routing entirely, so the + premise this sentence depended on no longer exists anywhere in the graph. + Fixed to state plainly that this remains a hypothetical, not tied to any + existing mechanism, without changing #54's disposition (still deferred, + unchanged) or its gate condition. + +Both fixes were applied via `updateIssue` before this audit was written, and +both are re-verified present in the live bodies as of this pass (see +`dependency-sequencing-audit.md`'s read-back table for the exact `updatedAt` +timestamps). + +No other stale cross-reference was found in #51, #52, #53, #55, #56, or #57. + +## Result + +**Clean pass, no remaining material findings.** Two prose-consistency +corrections were applied during this pass (listed above) and verified. No +outcome is duplicated across siblings, no outcome that demonstrates a reviewer +defect is left without an owner, and no child's expected change surface reaches +into adjacent product or tooling work outside the review suite. diff --git a/review-suite/evals/v2/audits/shovel-readiness-audit.md b/review-suite/evals/v2/audits/shovel-readiness-audit.md new file mode 100644 index 0000000..8b01c65 --- /dev/null +++ b/review-suite/evals/v2/audits/shovel-readiness-audit.md @@ -0,0 +1,90 @@ +# Shovel-readiness audit + +Run against the live, revised bodies of every retained child (`#51`, `#52`, +`#53`, `#56`) after the approved edits. Requirement (#59's own text): every +actionable/retained child has a settled observable outcome, change surface, +compatibility/failure behavior, positive and negative tests, validation, and +non-goals, plus a one-PR boundary. Independent of `scope-completeness-audit.md` +and `dependency-sequencing-audit.md`. Deferred children (#54, #55, #57) are +excluded from this requirement by design — their own gate condition is +explicitly not yet satisfied, so they are not expected to be shovel-ready yet, +and #59's own acceptance criteria only require provisional markers to be removed +"before they become unblocked," not before this ticket closes. + +## #51 — Make clean verdicts require passing validation and current-head lens evidence + +| requirement | present? | evidence | +| ------------------------------ | -------- | ----------------------------------------------------------------------------------------------------------------------------- | +| settled observable outcome | yes | `clean` impossible without passing validation + fresh 3-lens evidence for the exact candidate | +| change surface | yes | contracts, `validate.py`, `CONTRACT.md`, `review-code-change/SKILL.md`, orchestration cases, bundled copies, README/CHANGELOG | +| compatibility/failure behavior | yes | schema `1.0 → 1.1`, additive, atomic migration via `just sync-contracts`, stale v1.0 rejected with a clear error | +| positive and negative tests | yes | 11 enumerated regression cases covering every required behavior and every stale/invalid input | +| validation | yes | `just format`, `just lint`, `just test`; non-regression requirement that the 15 frozen v1 cases replay byte-identically | +| non-goals | yes | explicit list (no coverage ledgers, no new lenses, no selective-reuse exception, no caller changes) | +| one-PR boundary | yes | single schema/validator/orchestration change with a bounded fixture set; no architectural fan-out | + +**Shovel-ready.** + +## #52 — Evaluate compact review coverage, impact, and risk evidence (narrowed) + +| requirement | present? | evidence | +| ------------------------------ | -------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | +| settled observable outcome | yes | one additive `consumer_impact_evidence` array + validator non-emptiness rule, scoped to consumer/impact traversal only | +| change surface | yes | `review-result.schema.json`, `validate.py`, `CONTRACT.md`, fixtures, bundled copies, README/CHANGELOG | +| compatibility/failure behavior | yes | schema `1.1 → 1.2`, additive, atomic migration, stale-1.1-only aggregate fails with a useful error | +| positive and negative tests | yes | 5 enumerated fixtures covering the sibling-call-site case, the omission case, the no-other-consumers case, and its own negative (missing search evidence) | +| validation | yes | `just format`, `just lint`, `just test` | +| non-goals | yes | explicit list, including the dropped changed-surface ledger, acceptance trace, and risk profile named directly | +| one-PR boundary | yes | this is now a single small schema addition — smaller in scope than #51, clearly one-PR-sized | + +**Shovel-ready.** This is the child the scope/completeness audit found +cross-referenced by a now-corrected stale line in #56; that correction did not +require any change to #52's own body. + +## #53 — Evaluate correctness traversal and verification sufficiency (specialist routing dropped) + +| requirement | present? | evidence | +| ------------------------------ | ---------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| settled observable outcome | yes | exactly two required passes (traversal, verification-sufficiency); specialist routing named as explicitly absent | +| change surface | yes | `skills/review-correctness/SKILL.md` and references, `review-result.schema.json`, `validate.py`, `CONTRACT.md`, fixtures, bundled copies | +| compatibility/failure behavior | yes | schema `1.2 → 1.3`, additive, atomic migration | +| positive and negative tests | yes | two fixture pairs directly shaped from the two confident-miss baseline cases, each with a positive and negative variant, plus a non-regression check on clean controls and rejected connector findings | +| validation | yes | `just format`, `just lint`, `just test` | +| non-goals | yes | explicit list, including "no specialist module of any kind," directly traceable to the dropped mechanism | +| one-PR boundary | yes, with a note | this ticket lands two passes plus one schema bump in one PR — larger than #51/#52 but still a single coherent skill-prompt-plus-schema change, not a fan-out across multiple skills or subsystems. If implementation reveals it is materially larger than predicted, the standard `implement-ticket` size gate and `carve-changesets` remain available as the existing safety valve; no change to this ticket's own scope is needed to accommodate that possibility. | + +**Shovel-ready**, with the one-PR-boundary note above recorded rather than +silently assumed. + +## #56 — Operationalize adjudicated connector outcomes as review regressions + +| requirement | present? | evidence | +| ------------------------------ | ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| settled observable outcome | yes | intake/curation workflow, promotion steps, and the new mechanical disclosure guardrail | +| change surface | yes | corpus-compatible intake schema/validator, fixtures including the new guardrail fixture, docs, README/CHANGELOG | +| compatibility/failure behavior | yes | fail-closed intake validation for every disposition class, including the new source-description guardrail | +| positive and negative tests | yes | full disposition matrix (accepted/rejected/deferred/duplicate/unresolved/restricted-data/promotion-decision) plus the new disallowed-`source_description` negative fixture | +| validation | yes | `just format`, `just lint`, `just test` | +| non-goals | yes | explicit list, extended with "populate, rescore, or otherwise touch #58's frozen v1 baseline or its `connector-escape` stratum entry" | +| one-PR boundary | yes, with a note | this ticket's surface (intake schema, validator, six-plus fixture classes, documentation) was already the largest of the four retained children before this ticket's edits, and the one small guardrail addition here does not materially change that. Same safety valve as #53 applies if implementation proves it oversized. | + +**Shovel-ready**, with the same one-PR-boundary note as #53. + +Also confirmed: the disclosure boundary itself (never name the source +repository) is stated as an acceptance criterion ("The source repository named +in "Definition status" above is never written into any commit, code comment, or +generated artifact by this ticket's own tooling or tests"), not left as prose +alone — this makes the hard constraint independently checkable by whoever +implements and reviews #56, rather than resting on discipline only. + +## Result + +**Clean pass, no material findings.** All four retained children (#51, #52, #53, +#56) have a settled observable outcome, an explicit change surface, stated +compatibility/failure behavior, enumerated positive and negative tests, a +validation gate, explicit non-goals, and a plausible one-PR boundary (two of the +four carry an explicit note about that boundary rather than an unstated +assumption, which is itself the point of running this audit as its own pass). +The three deferred children (#54, #55, #57) are correctly excluded from this +requirement, consistent with #59's own text that provisional markers persist on +children that do not become unblocked in this round. From a314a6fef079abb4a176c39757f005060ee5dbec Mon Sep 17 00:00:00 2001 From: Scott Haug Date: Tue, 28 Jul 2026 15:29:44 -0700 Subject: [PATCH 2/5] fix: sweep issue titles too, and correct a precedent timing claim ## Summary - The scope-completeness audit's cross-reference sweep checked only issue bodies. A follow-up title sweep found #52's and #53's live titles still named exactly the concepts ("risk evidence", "specialist routing") their own bodies say were dropped. Both titles are corrected via updateIssue (title only, no body change), and the audit file now records this second sweep and its findings. - The dependency-sequencing audit's #58/#50 precedent argument previously claimed #50 "closed since before #58 started," which live createdAt/ closedAt data contradicts (#58 was created before #50 closed). Replaced with the accurate timeline; the precedent's substance is unaffected. - DECISION-RECORD.md's audit summary now mentions the two title fixes alongside the two body fixes already recorded. ## Why - review-code-change's pass on this candidate returned `changes_required`: the scope-completeness audit's "no other stale cross-reference" claim was incomplete (titles unchecked), and a supporting factual claim in the dependency-sequencing audit was independently disprovable from GitHub's own timestamps. --- review-suite/evals/v2/DECISION-RECORD.md | 8 +++++--- .../v2/audits/dependency-sequencing-audit.md | 17 ++++++++++------- .../evals/v2/audits/scope-completeness-audit.md | 17 ++++++++++++++++- 3 files changed, 31 insertions(+), 11 deletions(-) diff --git a/review-suite/evals/v2/DECISION-RECORD.md b/review-suite/evals/v2/DECISION-RECORD.md index 16bd025..34e9686 100644 --- a/review-suite/evals/v2/DECISION-RECORD.md +++ b/review-suite/evals/v2/DECISION-RECORD.md @@ -444,9 +444,11 @@ Full findings are in and [`audits/shovel-readiness-audit.md`](audits/shovel-readiness-audit.md). Summary: -- **Scope/completeness:** clean. Two stale cross-references (#56's mention of a - dropped #52 changed-surface concept, and #54's mention of a dropped #52 risk - profile) were found and corrected in the live tracker as part of this pass. No +- **Scope/completeness:** clean. Two stale body cross-references (#56's mention + of a dropped #52 changed-surface concept, and #54's mention of a dropped #52 + risk profile) and two stale titles (#52's and #53's, each still naming a + concept — "risk evidence," "specialist routing" — their own bodies say was + dropped) were found and corrected in the live tracker as part of this pass. No outcome is duplicated across siblings; none that demonstrates a reviewer defect is left unowned. - **Dependency/sequencing:** clean, with one flagged-but-unresolved open diff --git a/review-suite/evals/v2/audits/dependency-sequencing-audit.md b/review-suite/evals/v2/audits/dependency-sequencing-audit.md index ad78fdb..21fce12 100644 --- a/review-suite/evals/v2/audits/dependency-sequencing-audit.md +++ b/review-suite/evals/v2/audits/dependency-sequencing-audit.md @@ -26,13 +26,16 @@ exist. Independent of `scope-completeness-audit.md` and No native edge was added or removed by this ticket. #51's `blockedBy #59` edge is deliberately left in place rather than deleted once #59 closes, matching this repository's own established convention: #58's `blockedBy #50` edge is still -present today even though #50 has been closed since before #58 started (verified -in the same query above, and independently re-confirmed via a direct `#58` query -returning `blockedBy: [{number: 50, state: CLOSED}]`). `implement-epic`'s own -readiness rule (`skills/implement-epic/references/github.md`: "Choose an -in-scope child whose native `blockedBy` set has no open issue") keys off open -blockers, not edge presence, so this is consistent with how the epic -orchestrator actually selects work. +present today even though #50 has been closed since 2026-07-27 (verified in the +same query above, and independently re-confirmed via a direct `#58` query +returning `blockedBy: [{number: 50, state: CLOSED}]`) — #58 itself was created +2026-07-26, before #50 closed, and remained open doing its own work until +2026-07-28, so the edge simply went inert in place rather than being removed +once its blocker closed. `implement-epic`'s own readiness rule +(`skills/implement-epic/references/github.md`: "Choose an in-scope child whose +native `blockedBy` set has no open issue") keys off open blockers, not edge +presence, so this is consistent with how the epic orchestrator actually selects +work. ## Predicted vs. actual net effect once #59 closes diff --git a/review-suite/evals/v2/audits/scope-completeness-audit.md b/review-suite/evals/v2/audits/scope-completeness-audit.md index 6f0ce82..9cba691 100644 --- a/review-suite/evals/v2/audits/scope-completeness-audit.md +++ b/review-suite/evals/v2/audits/scope-completeness-audit.md @@ -74,7 +74,22 @@ both are re-verified present in the live bodies as of this pass (see `dependency-sequencing-audit.md`'s read-back table for the exact `updatedAt` timestamps). -No other stale cross-reference was found in #51, #52, #53, #55, #56, or #57. +No other stale cross-reference was found in #51, #52, #53, #55, #56, or #57 +bodies. + +**Titles were swept separately, and this found two more.** The body sweep above +did not check issue *titles* on its first pass; a follow-up check found that +#52's and #53's live titles still named exactly the concepts their own +(already-corrected) bodies say were dropped: #52's title read "Evaluate compact +review coverage, impact, and risk evidence" — "risk evidence" is the +risk-profile apparatus the narrowing removed — and #53's title read "...and +specialist routing," the mechanism #53's own body says is "dropped entirely." +Both are the same class of staleness as the two body fixes above, just in a +field the first pass didn't check. Fixed via `updateIssue` (title only, no body +change): #52 is now "Add consumer/impact-traversal evidence to the shared review +contract," and #53 is now "Add correctness traversal and +verification-sufficiency passes." Re-checked: no other child's title claims +scope its body no longer supports. ## Result From c62befadc4ddb687ee00b0786e38e695061798a9 Mon Sep 17 00:00:00 2001 From: Scott Haug Date: Tue, 28 Jul 2026 15:43:10 -0700 Subject: [PATCH 3/5] fix: correct the remaining stale #52/#53 title references ## Summary - shovel-readiness-audit.md's #52 and #53 section headings still used the pre-fix issue titles ("Evaluate compact review coverage, impact, and risk evidence", "...and specialist routing"). Updated to match the corrected live titles. - DECISION-RECORD.md's #52 and #53 section headings had the same staleness (they predate the title fix commit); updated to match as well. ## Why - The second review-code-change pass on this candidate found the title fix from the prior commit had not propagated to two more headings inside this same delivered set, reproducing the exact staleness class the audits exist to catch. --- review-suite/evals/v2/DECISION-RECORD.md | 4 ++-- review-suite/evals/v2/audits/shovel-readiness-audit.md | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/review-suite/evals/v2/DECISION-RECORD.md b/review-suite/evals/v2/DECISION-RECORD.md index 34e9686..30d2b3d 100644 --- a/review-suite/evals/v2/DECISION-RECORD.md +++ b/review-suite/evals/v2/DECISION-RECORD.md @@ -76,7 +76,7 @@ tickets carry out the actual change. which no evidence here supports; #57 verifies the invariant holds rather than testing whether removing it changes anything. -## #52 — Evaluate compact review coverage, impact, and risk evidence +## #52 — Add consumer/impact-traversal evidence to the shared review contract **Disposition: narrow.** The full changed-surface ledger, acceptance-trace, risk-profile, and aggregate-coverage apparatus in #52's current body is **not** @@ -167,7 +167,7 @@ consumer/impact-traversal gap. structural complexity. This is the explicit removal condition #57's ablation protocol must check. -## #53 — Evaluate correctness traversal, verification sufficiency, and specialist routing +## #53 — Add correctness traversal and verification-sufficiency passes **Disposition: retain the traversal and verification-sufficiency work; drop specialist routing entirely.** Nothing in the baseline supports specialist diff --git a/review-suite/evals/v2/audits/shovel-readiness-audit.md b/review-suite/evals/v2/audits/shovel-readiness-audit.md index 8b01c65..667eaf2 100644 --- a/review-suite/evals/v2/audits/shovel-readiness-audit.md +++ b/review-suite/evals/v2/audits/shovel-readiness-audit.md @@ -25,7 +25,7 @@ and #59's own acceptance criteria only require provisional markers to be removed **Shovel-ready.** -## #52 — Evaluate compact review coverage, impact, and risk evidence (narrowed) +## #52 — Add consumer/impact-traversal evidence to the shared review contract (narrowed) | requirement | present? | evidence | | ------------------------------ | -------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | @@ -41,7 +41,7 @@ and #59's own acceptance criteria only require provisional markers to be removed cross-referenced by a now-corrected stale line in #56; that correction did not require any change to #52's own body. -## #53 — Evaluate correctness traversal and verification sufficiency (specialist routing dropped) +## #53 — Add correctness traversal and verification-sufficiency passes (specialist routing dropped) | requirement | present? | evidence | | ------------------------------ | ---------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | From bc610fb33f55d9a677c3184bfce66120f3f5e04d Mon Sep 17 00:00:00 2001 From: Scott Haug Date: Tue, 28 Jul 2026 15:58:45 -0700 Subject: [PATCH 4/5] fix: stop describing #57's already-applied prose fix as pending ## Summary - DECISION-RECORD.md's #57 entry no longer says the "approved by #58" cross- reference correction is "still pending... not by this repository candidate." That fix, plus two further #58-instead-of-#59 corrections in the same body (thresholds and gate-manifest references), were already applied to the live tracker after PR #79 merged. The entry now says so, and notes scope-completeness-audit.md independently confirms no stale cross-reference remains in #57's live body. ## Why - The third review-code-change pass on this candidate found the document contradicted itself: one section called a fix "still pending," while this candidate's own audit-results section and the live tracker showed it had already happened before these audits ran. --- review-suite/evals/v2/DECISION-RECORD.md | 20 +++++++++++--------- 1 file changed, 11 insertions(+), 9 deletions(-) diff --git a/review-suite/evals/v2/DECISION-RECORD.md b/review-suite/evals/v2/DECISION-RECORD.md index 30d2b3d..b8c4784 100644 --- a/review-suite/evals/v2/DECISION-RECORD.md +++ b/review-suite/evals/v2/DECISION-RECORD.md @@ -404,15 +404,17 @@ basis and two small guardrails. **Disposition: defer, unchanged.** Downstream of #55 and #56; #57's own "Definition status" already names #59 as its gate for the exact scored-v2 configuration, which this record and `gate-manifest.json` provide. One prose -correction is in scope, still pending as of this record: #57's live body -currently reads "Run the evidence-backed v2 experiment approved by #58" in its -Goal section. #58 curated the frozen v1 corpus and baseline; it is #59 — this -ticket — that approves the v2 mechanism set and gate manifest #57 executes -against. That single stale cross-reference **must be corrected in #57's body -when this ticket's later live issue-body edit step is carried out** — not by -this repository candidate, which changes no live issue body itself, per #59's -own validation/delivery boundary; #57's disposition, required inputs, and -acceptance criteria are otherwise unchanged. +correction was in scope and **has since been applied to the live tracker**: +#57's Goal section previously read "Run the evidence-backed v2 experiment +approved by #58." #58 curated the frozen v1 corpus and baseline; it is #59 — +this ticket — that approves the v2 mechanism set and gate manifest #57 executes +against. That stale cross-reference, plus two further "#58" mentions in #57's +own body with the same defect ("thresholds selected by #58" and "the #58 gate +manifest"), were all corrected via `updateIssue` after this candidate's first +repository PR merged, as part of this ticket's live issue-body edit step; #57's +disposition, required inputs, and acceptance criteria are otherwise unchanged. +`audits/scope-completeness-audit.md` independently confirms no stale +cross-reference remains in #57's live body. ## Net effect on the dependency graph From 97513a066156a1ea3a50b9c18b7b2d5e5cea8b14 Mon Sep 17 00:00:00 2001 From: Scott Haug Date: Tue, 28 Jul 2026 16:13:32 -0700 Subject: [PATCH 5/5] fix: add the updatedAt data scope-completeness-audit.md already cited ## Summary - dependency-sequencing-audit.md's live graph read-back table now includes the updatedAt column its own intro line already claimed to have queried, with the actual re-fetched timestamps for #51-#59, so scope-completeness-audit.md's pointer to "the exact updatedAt timestamps" in this table resolves to real data instead of a table that never had that column. ## Why - The fourth review-code-change pass on this candidate found a citation in scope-completeness-audit.md that pointed at data dependency-sequencing- audit.md did not actually contain, even though the underlying claim (the fixes landed when stated) was independently verifiable and true. --- .../v2/audits/dependency-sequencing-audit.md | 31 ++++++++++++------- 1 file changed, 19 insertions(+), 12 deletions(-) diff --git a/review-suite/evals/v2/audits/dependency-sequencing-audit.md b/review-suite/evals/v2/audits/dependency-sequencing-audit.md index 21fce12..5be98fb 100644 --- a/review-suite/evals/v2/audits/dependency-sequencing-audit.md +++ b/review-suite/evals/v2/audits/dependency-sequencing-audit.md @@ -10,18 +10,25 @@ exist. Independent of `scope-completeness-audit.md` and ## Live graph read-back `gh api graphql` query per issue, `blockedBy`/`blocking`/`parent`/`state`/ -`updatedAt`, 2026-07-28: - -| issue | state | parent | blockedBy (state) | blocking (state) | -| ----- | ----- | ------ | ---------------------- | ---------------------- | -| #51 | OPEN | #49 | #59 (OPEN) | #52 (OPEN) | -| #52 | OPEN | #49 | #51 (OPEN) | #56 (OPEN), #53 (OPEN) | -| #53 | OPEN | #49 | #52 (OPEN) | #54 (OPEN) | -| #54 | OPEN | #49 | #53 (OPEN) | #55 (OPEN) | -| #55 | OPEN | #49 | #54 (OPEN) | #57 (OPEN) | -| #56 | OPEN | #49 | #52 (OPEN) | #57 (OPEN) | -| #57 | OPEN | #49 | #56 (OPEN), #55 (OPEN) | (none) | -| #59 | OPEN | #49 | #58 (CLOSED) | #51 (OPEN) | +`updatedAt`, re-fetched most recently on 2026-07-28 (the `updatedAt` column +below reflects each issue's last edit, including this ticket's own title/body +fixes — it is not a fixed snapshot moment, since those fixes landed at different +times across this ticket's several audit-fix rounds): + +| issue | state | parent | blockedBy (state) | blocking (state) | updatedAt | +| ----- | ----- | ------ | ---------------------- | ---------------------- | -------------------- | +| #51 | OPEN | #49 | #59 (OPEN) | #52 (OPEN) | 2026-07-28T22:07:46Z | +| #52 | OPEN | #49 | #51 (OPEN) | #56 (OPEN), #53 (OPEN) | 2026-07-28T22:25:08Z | +| #53 | OPEN | #49 | #52 (OPEN) | #54 (OPEN) | 2026-07-28T22:25:08Z | +| #54 | OPEN | #49 | #53 (OPEN) | #55 (OPEN) | 2026-07-28T22:09:45Z | +| #55 | OPEN | #49 | #54 (OPEN) | #57 (OPEN) | 2026-07-26T05:18:01Z | +| #56 | OPEN | #49 | #52 (OPEN) | #57 (OPEN) | 2026-07-28T22:09:46Z | +| #57 | OPEN | #49 | #56 (OPEN), #55 (OPEN) | (none) | 2026-07-28T22:08:01Z | +| #59 | OPEN | #49 | #58 (CLOSED) | #51 (OPEN) | 2026-07-26T05:20:17Z | + +For reference, #58's own `updatedAt` at the time of this audit is +2026-07-28T04:52:48Z (its close event) — no field of #58 was touched by this +ticket, consistent with the frozen-baseline boundary. No native edge was added or removed by this ticket. #51's `blockedBy #59` edge is deliberately left in place rather than deleted once #59 closes, matching this