Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion review-suite/evals/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@ review-suite/evals/
│ ├── FAILURE-TAXONOMY.md every v1 material outcome, classified
│ ├── DECISION-RECORD.md per-mechanism evidence and disposition
│ ├── gate-manifest.json preregistered v2 gates, before scoring
│ └── audits/ the three graph audits (later step, empty for now)
│ └── audits/ the three graph audits, each its own pass
└── artifacts/ opt-in captured output, not in git

review-suite/scripts/evals/
Expand Down
65 changes: 52 additions & 13 deletions review-suite/evals/v2/DECISION-RECORD.md
Original file line number Diff line number Diff line change
Expand Up @@ -76,7 +76,7 @@ tickets carry out the actual change.
which no evidence here supports; #57 verifies the invariant holds rather than
testing whether removing it changes anything.

## #52 — Evaluate compact review coverage, impact, and risk evidence
## #52 — Add consumer/impact-traversal evidence to the shared review contract

**Disposition: narrow.** The full changed-surface ledger, acceptance-trace,
risk-profile, and aggregate-coverage apparatus in #52's current body is **not**
Expand Down Expand Up @@ -167,7 +167,7 @@ consumer/impact-traversal gap.
structural complexity. This is the explicit removal condition #57's ablation
protocol must check.

## #53 — Evaluate correctness traversal, verification sufficiency, and specialist routing
## #53 — Add correctness traversal and verification-sufficiency passes

**Disposition: retain the traversal and verification-sufficiency work; drop
specialist routing entirely.** Nothing in the baseline supports specialist
Expand Down Expand Up @@ -404,15 +404,17 @@ basis and two small guardrails.
**Disposition: defer, unchanged.** Downstream of #55 and #56; #57's own
"Definition status" already names #59 as its gate for the exact scored-v2
configuration, which this record and `gate-manifest.json` provide. One prose
correction is in scope, still pending as of this record: #57's live body
currently reads "Run the evidence-backed v2 experiment approved by #58" in its
Goal section. #58 curated the frozen v1 corpus and baseline; it is #59 — this
ticket — that approves the v2 mechanism set and gate manifest #57 executes
against. That single stale cross-reference **must be corrected in #57's body
when this ticket's later live issue-body edit step is carried out** — not by
this repository candidate, which changes no live issue body itself, per #59's
own validation/delivery boundary; #57's disposition, required inputs, and
acceptance criteria are otherwise unchanged.
correction was in scope and **has since been applied to the live tracker**:
#57's Goal section previously read "Run the evidence-backed v2 experiment
approved by #58." #58 curated the frozen v1 corpus and baseline; it is #59 —
this ticket — that approves the v2 mechanism set and gate manifest #57 executes
against. That stale cross-reference, plus two further "#58" mentions in #57's
own body with the same defect ("thresholds selected by #58" and "the #58 gate
manifest"), were all corrected via `updateIssue` after this candidate's first
repository PR merged, as part of this ticket's live issue-body edit step; #57's
disposition, required inputs, and acceptance criteria are otherwise unchanged.
`audits/scope-completeness-audit.md` independently confirms no stale
cross-reference remains in #57's live body.

## Net effect on the dependency graph

Expand All @@ -432,5 +434,42 @@ required by #59's own sequencing (scope/completeness, dependency/sequencing,
shovel-readiness) run only after the live `#51`–`#57` body and edge changes
below are actually applied, as their own later step. Their findings — including
the read-back that confirms or corrects this predicted net effect — belong in
`audits/` once that step completes; no file exists there yet in this repository
candidate, and none is claimed to.
`audits/` once that step completes.

## Audit results (post-mutation)

The live `#51`–`#57` body edits and the read-back above were applied, and all
three required audits were run independently against the resulting live graph.
Full findings are in
[`audits/scope-completeness-audit.md`](audits/scope-completeness-audit.md),
[`audits/dependency-sequencing-audit.md`](audits/dependency-sequencing-audit.md),
and [`audits/shovel-readiness-audit.md`](audits/shovel-readiness-audit.md).
Summary:

- **Scope/completeness:** clean. Two stale body cross-references (#56's mention
of a dropped #52 changed-surface concept, and #54's mention of a dropped #52
risk profile) and two stale titles (#52's and #53's, each still naming a
concept — "risk evidence," "specialist routing" — their own bodies say was
dropped) were found and corrected in the live tracker as part of this pass. No
outcome is duplicated across siblings; none that demonstrates a reviewer
defect is left unowned.
- **Dependency/sequencing:** clean, with one flagged-but-unresolved open
question that does not change the audited outcome: #56's `blockedBy #52` edge
has a weaker technical justification after #52's narrowing than the other
edges in the graph, but no evidence shows it is actively wrong, and severing
it was outside this ticket's settled disposition for #56, so it is reported to
the owner rather than unilaterally changed. The predicted net effect — #51 is
the sole open issue with zero open blockers once #59 closes — is confirmed
against the live graph read directly through GraphQL.
- **Shovel-readiness:** clean. Every retained child (#51, #52, #53, #56) has a
settled observable outcome, change surface, compatibility/failure behavior,
positive and negative tests, validation, non-goals, and a plausible one-PR
boundary (two children carry an explicit note on that boundary rather than an
unstated assumption). Deferred children (#54, #55, #57) are correctly
excluded, per #59's own text that provisional markers persist on children that
do not become unblocked in this round.

No finding from any of the three audits required a materially different
architecture or a ticket split, so none of #51–#57's dispositions changed as a
result of running them, and no second iteration of this repository candidate or
the live graph was needed.
11 changes: 6 additions & 5 deletions review-suite/evals/v2/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,11 +19,12 @@ implementation-ready decision for #51–#57.
efficiency bounds, and the rules for invalid runs, missing data, runtime
drift, and threshold changes. Committed before any scored v2 output exists;
every numeric threshold is marked as a proposal requiring owner confirmation.
- `audits/` — **not yet produced by this repository candidate.** #59's own
sequencing runs the three independent graph audits (scope/completeness,
dependency/sequencing, shovel-readiness) only after the live `#51`–`#57` body
and native-edge mutations are applied as a later, separate step; each pass's
findings will be recorded here, as its own file, once that step completes.
- [`audits/scope-completeness-audit.md`](audits/scope-completeness-audit.md),
[`audits/dependency-sequencing-audit.md`](audits/dependency-sequencing-audit.md),
[`audits/shovel-readiness-audit.md`](audits/shovel-readiness-audit.md) — the
three required graph audits, each run as its own independent pass against the
live `#51`–`#57` graph after the body and native-edge mutations described
above, with findings and fixes recorded separately per pass.

## What this is not

Expand Down
131 changes: 131 additions & 0 deletions review-suite/evals/v2/audits/dependency-sequencing-audit.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,131 @@
# Dependency/sequencing audit

Run against the live `#51`–`#57` graph, read directly through GraphQL on
2026-07-28 (queries and raw results below). Requirement (#59's own text): native
edges match artifact prerequisites; only genuine leaves are actionable; no child
can implement an unapproved mechanism or integrate a contract that does not
exist. Independent of `scope-completeness-audit.md` and
`shovel-readiness-audit.md`.

## Live graph read-back

`gh api graphql` query per issue, `blockedBy`/`blocking`/`parent`/`state`/
`updatedAt`, re-fetched most recently on 2026-07-28 (the `updatedAt` column
below reflects each issue's last edit, including this ticket's own title/body
fixes — it is not a fixed snapshot moment, since those fixes landed at different
times across this ticket's several audit-fix rounds):

| issue | state | parent | blockedBy (state) | blocking (state) | updatedAt |
| ----- | ----- | ------ | ---------------------- | ---------------------- | -------------------- |
| #51 | OPEN | #49 | #59 (OPEN) | #52 (OPEN) | 2026-07-28T22:07:46Z |
| #52 | OPEN | #49 | #51 (OPEN) | #56 (OPEN), #53 (OPEN) | 2026-07-28T22:25:08Z |
| #53 | OPEN | #49 | #52 (OPEN) | #54 (OPEN) | 2026-07-28T22:25:08Z |
| #54 | OPEN | #49 | #53 (OPEN) | #55 (OPEN) | 2026-07-28T22:09:45Z |
| #55 | OPEN | #49 | #54 (OPEN) | #57 (OPEN) | 2026-07-26T05:18:01Z |
| #56 | OPEN | #49 | #52 (OPEN) | #57 (OPEN) | 2026-07-28T22:09:46Z |
| #57 | OPEN | #49 | #56 (OPEN), #55 (OPEN) | (none) | 2026-07-28T22:08:01Z |
| #59 | OPEN | #49 | #58 (CLOSED) | #51 (OPEN) | 2026-07-26T05:20:17Z |

For reference, #58's own `updatedAt` at the time of this audit is
2026-07-28T04:52:48Z (its close event) — no field of #58 was touched by this
ticket, consistent with the frozen-baseline boundary.

No native edge was added or removed by this ticket. #51's `blockedBy #59` edge
is deliberately left in place rather than deleted once #59 closes, matching this
repository's own established convention: #58's `blockedBy #50` edge is still
present today even though #50 has been closed since 2026-07-27 (verified in the
same query above, and independently re-confirmed via a direct `#58` query
returning `blockedBy: [{number: 50, state: CLOSED}]`) — #58 itself was created
2026-07-26, before #50 closed, and remained open doing its own work until
2026-07-28, so the edge simply went inert in place rather than being removed
once its blocker closed. `implement-epic`'s own readiness rule
(`skills/implement-epic/references/github.md`: "Choose an in-scope child whose
native `blockedBy` set has no open issue") keys off open blockers, not edge
presence, so this is consistent with how the epic orchestrator actually selects
work.

## Predicted vs. actual net effect once #59 closes

`DECISION-RECORD.md`'s "Net effect on the dependency graph" section predicts:
once #59 closes, #51 has zero *open* blockers (its only blocker, #59, is closed)
and is the only issue in the epic with that property. Checking every other
child's `blockedBy` set above against the current *open* graph: #52 (#51, open),
#53 (#52, open), #54 (#53, open), #55 (#54, open), #56 (#52, open), #57 (#56,
#55, both open). Every one of #52–#57 has at least one open blocker regardless
of #59's state. **Confirmed: #51 is the only genuine leaf once #59 closes; the
predicted net effect matches what the live graph actually produces.** #59
remains open as of this audit (it is being closed only after this audit and the
shovel-readiness audit both pass, per this ticket's own required sequencing), so
this is a verified prediction about the graph's structure, not yet an
observation of #51 sitting unblocked in production — the structural fact (every
other child's *independent* open blocker) does not depend on #59's own state to
hold.

## Artifact-prerequisite check

Confirmed each edge corresponds to a real, stated artifact dependency, not
merely a sequencing convention:

- **#52 → #51**: #52's body states its `consumer_impact_evidence` addition bumps
`review-result.schema.json` `"1.1 → 1.2"`, explicitly "delivered by #51." Real
prerequisite: #52 cannot define a delta from a schema version #51 hasn't
shipped yet.
- **#53 → #52**: #53's body bumps `"1.2 → 1.3"`, and its traversal pass
populates `consumer_impact_evidence`, a field #52 defines. Real prerequisite.
- **#54 → #53**: #54's own gate text requires "the measured single-pass
improvements from #53" — a result that can only exist after #53 ships and is
scored. Real prerequisite, and correctly *not* pointed at #59 (per
`DECISION-RECORD.md`'s explicit instruction: "Keep it blocked on #53's
eventual evidence, not #59 itself" — confirmed: #54's `blockedBy` is `#53`
only).
- **#55 → #54**: #55's gate depends on "#54 (or its evidence-backed
replacement)" establishing final orchestrator behavior. Real prerequisite.
- **#57 → #55, #56**: #57 requires "migrated caller integration from #55" and
"operational feedback/corpus workflow from #56" as named required inputs. Real
prerequisite for both edges.
- **#56 → #52**: weaker than the others after #52's narrowing. #56's body,
post-fix, no longer claims any technical dependency on #52's
`consumer_impact_evidence` schema — its intake/curation record is explicitly
"compatible with the corpus contract from #50/#58," a different,
already-existing contract family. **This is a genuine open question, not a
clear defect**: no evidence found that #56 *cannot* be implemented before #52
ships, but no evidence was found that it structurally *must* wait either. Per
this ticket's own instruction to report a discrepancy rather than quietly
reconcile it, this edge is left unchanged (removing it was not part of the
owner's settled disposition for #56, which addressed evidence-backing and
sourcing only, not sequencing), and is flagged here for the owner to confirm
or sever. It does not change the actionable-leaf outcome above either way,
since #52 is itself gated on #51 regardless of whether #56 also waits on it.

## Unapproved-mechanism check

Confirmed no child's current body describes implementing a mechanism this
ticket's decision record did not approve, and no child integrates a contract
version that does not yet exist in the sequence above:

- #51 implements only its own schema `1.0 → 1.1` delta — no forward reference to
a version another ticket owns.
- #52 implements only `1.1 → 1.2`, explicitly built on #51's delivered version,
and explicitly does not add the changed-surface ledger, acceptance trace, or
risk profile the decision record dropped.
- #53 implements only `1.2 → 1.3`, explicitly built on #52's delivered version,
and explicitly does not add any specialist module.
- #54, #55, #57 each state their own gate is not yet satisfied and commit to no
premature implementation; #54's remaining "conditional specialist explorer"
bullet was corrected during the scope/completeness pass to no longer imply a
#52 risk profile that does not exist, and is now explicitly marked
hypothetical.
- #56 integrates only the existing `#50`/`#58` corpus/expectation/provenance
contracts, not any not-yet-shipped review-result schema version.

No finding here.

## Result

**Clean pass, one flagged-but-unresolved open question** (the #56→#52 edge's
weakened-but-not-contradicted justification, reported above for the owner to
confirm or sever) **that does not change the audited outcome**: every native
edge matches a real artifact prerequisite except that one weaker case, only
genuine leaves are actionable, #51 is confirmed as the sole next actionable leaf
once #59 closes, and no child implements an unapproved mechanism or integrates a
contract that does not yet exist in the sequence.
Loading
Loading