Skip to content

Spec 282: repeatable whole-repo review skill (/repo-review) — spec through implementation#669

Open
IanMayo wants to merge 13 commits into
mainfrom
claude/code-review-plan-95vovz
Open

Spec 282: repeatable whole-repo review skill (/repo-review) — spec through implementation#669
IanMayo wants to merge 13 commits into
mainfrom
claude/code-review-plan-95vovz

Conversation

@IanMayo

@IanMayo IanMayo commented Jul 10, 2026

Copy link
Copy Markdown
Member

Summary

Delivers the /repo-review skill end-to-end: a repeatable, multi-agent whole-repo audit whose findings must survive adversarial verification before they're reported, plus a /repo-review.fix fast-path from any finding to a fix PR. This PR carries the full speckit trail (spec → plan → research → data-model → contracts → tasks) and the implementation.

The distinguishing rule is verified-only: every candidate finding is checked by a separate adversarial verifier that tries to refute it; refuted and undecidable candidates are counted, not listed. The bet is a report the maintainer can act on without re-checking each claim.

What's implemented

Skill assets (.claude/)

  • commands/repo-review.md — clean-tree gate, run identity, verified-only discipline, write boundary, memory + calibration, graceful degradation
  • commands/repo-review.fix.md — ledger-driven fix→test→PR fast-path (mirrors /bugfix)
  • review/workflow.js — Workflow orchestration: recon → per-cell review→verify pipeline (no barrier) → evidence → synthesis
  • review/playbooks/{constitution,correctness,tech-debt,test-quality}.md — 20 falsifiable heuristics (CC/CB/TD/TQ), the tech-debt one framed as a Load existing result files into Attachments dropdown on plot open #172 regression check
  • review/{tier-map.yaml, severity-rubric.md, report-template.md, ledger.schema.json}
  • review/{ruff-strict.toml, eslint-strict.config.mjs} — report-only strict-lint configs for the evidence phase

Typed helper + tests

  • scripts/review-ledger.py — strict-typed validate / reconcile / record-fix-pr, load/save with schema gate on every read and write, defect-identity reconciliation, deterministic playbook-tuning logic
  • tests/repo_review/ — 36 tests (ledger validation, reconciliation lifecycle, fix-handoff preconditions, playbook/tier-map structural checks, tuning logic). All green; 75% helper coverage (uncovered = argparse wrappers, exercised via real CLI transcripts)

Docs & evidence

  • docs/project_notes/decisions.md — ADR-041 (YAML+JSON-Schema over LinkML, defect-identity reconciliation, verified-only bar)
  • specs/282-repo-review-skill/evidence/ — real test summary, CLI transcripts, valid+corrupt validation output, usage walkthrough, ledger sample, labelled report-format excerpt
  • specs/282-repo-review-skill/media/shipped-post.md — feature blog post

Value-capture design (from the interview)

Report-only, but the loop is closed: /repo-review.fix RR-NNN turns a finding into a fix PR; each theme proposes one permanent guard (lint rule / CI gate / CLAUDE.md instruction / constitution amendment); confirmed Critical/High bugs flow into bugs.md; per-heuristic confirmed/refuted stats drive playbook tuning between runs. The headline success metric is resolution rate by the next run, not finding count.

Verification

  • uv run ruff check . + ruff format --check . — clean
  • uv run pytest — 2206 passed, 2 skipped, 1 xfailed (full Python suite; nothing broke)
  • tests/repo_review/ — 36 passed

Not in this PR (deliberate)

  • Inaugural full multi-agent run (T045 / SC-001) — the skill's behavioural acceptance test, run by the maintainer against the live repo; expensive by design (multi-million tokens) and instrumented, not executed in a build session.
  • Blog publish to debrief.github.io (T054) — separate repo; run /publish-future-post there when ready.

🤖 Generated with Claude Code

https://claude.ai/code/session_01KZAovzDhcm7S5FBB3oT8H4

Speckit specification produced from a structured interview covering review
dimensions (constitution conformance, correctness bugs, tech-debt refresh,
test quality), tiered whole-repo scope, verified-only findings via
adversarial verification, multi-agent execution, evidence tooling, and a
findings ledger enabling delta re-runs.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KZAovzDhcm7S5FBB3oT8H4
@github-actions

github-actions Bot commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

📋 Spec Navigator

Review this PR's specs interactively:
👉 Open in Spec Navigator
Raw spec directories changed in this PR:

@IanMayo IanMayo temporarily deployed to debrief-preview-pr-669 July 10, 2026 13:56 Inactive
Second interview round: fix-session handoff (/repo-review.fix consuming
ledger entries), prevention-per-theme guard proposals, memory integration
(bugs.md + failure-pattern docs), playbook self-calibration, resolution-rate
as the primary value metric, and an operating runbook (triage ritual,
quick-wins week, guard adoption, playbook tuning).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KZAovzDhcm7S5FBB3oT8H4
@IanMayo IanMayo temporarily deployed to debrief-preview-pr-669 July 10, 2026 14:42 Inactive
…quickstart)

/speckit.plan output: implementation plan with constitution gate (PASS, two
justified notes), research decisions R-001..R-010, ledger data model + JSON
Schema contract, report-structure and fix-handoff contracts, quickstart, and
cached blog opener. Updates CLAUDE.md recent-changes.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KZAovzDhcm7S5FBB3oT8H4
@IanMayo IanMayo temporarily deployed to debrief-preview-pr-669 July 10, 2026 15:02 Inactive
/speckit.tasks output. Setup + Foundation (ledger schema + typed helper) →
US1 trustworthy report (MVP) → US2 delta re-runs → US4 fix handoff → US5
themes/guards → US3 evidence tooling → US6 memory/calibration → Polish
(inaugural dogfood run, evidence, ADR, blog, PR). Evidence artifacts and
final /speckit.pr task included.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KZAovzDhcm7S5FBB3oT8H4
@IanMayo IanMayo temporarily deployed to debrief-preview-pr-669 July 10, 2026 20:05 Inactive
claude added 3 commits July 11, 2026 12:26
…playbooks)

Phase 1-2 + partial Phase 3: tier-map.yaml, severity-rubric.md, ledger JSON
Schema asset, typed review-ledger.py helper (load/save/validate/reconcile/
record-fix-pr + tuning-recommendation), pytest suite, constitution + correctness
playbooks. In-progress; remaining playbooks, workflow, and commands to follow.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KZAovzDhcm7S5FBB3oT8H4
All four dimension playbooks (CC/CB/TD/TQ), tier-map and playbook structural
tests, tuning-recommendation tests. Fixes: distinguish dry-run (None) from
finalise ({}) in reconcile; snapshot original entries so new ids aren't
swept into the fixed loop; narrow JSON boundary off bare Any; TYPE_CHECKING
imports. 36 tests pass, ruff clean.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KZAovzDhcm7S5FBB3oT8H4
@IanMayo IanMayo temporarily deployed to debrief-preview-pr-669 July 11, 2026 12:47 Inactive
Apply ruff format to review-ledger.py and test_playbook_structure.py (CI runs
ruff format --check); remove unused RESOLVED_STATUSES; add the report template
(FR-007) with themes/prevention and playbook-tuning sections.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KZAovzDhcm7S5FBB3oT8H4
@IanMayo IanMayo temporarily deployed to debrief-preview-pr-669 July 12, 2026 07:32 Inactive
claude added 3 commits July 12, 2026 07:38
…mands, strict lint configs

Adds the Workflow script (recon → review→verify pipeline → evidence → synthesis),
the main /repo-review command (clean-tree gate, verified-only, write boundary,
memory + calibration, degradation) and /repo-review.fix fast-path (ledger-driven
fix→test→PR), plus report-only strict ruff + eslint configs for the evidence phase.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KZAovzDhcm7S5FBB3oT8H4
…ge, report format)

ADR-041 records the ledger design (YAML+JSON-Schema over LinkML, defect-identity
reconciliation, verified-only bar). Evidence: real pytest summary (36 pass, 75%
helper coverage), real CLI transcripts (validate/reconcile/record-fix-pr),
valid+corrupt validation output, usage walkthrough, and a labelled report-format
excerpt.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KZAovzDhcm7S5FBB3oT8H4
Content Specialist shipped-post.md (cached-opener first three sections verbatim
+ By the Numbers / Lessons Learned / What's Next from evidence). Mark 52 of 54
tasks complete; T045 (inaugural full multi-agent run — maintainer acceptance
test) and T054 (blog publish to debrief.github.io — separate repo) remain.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KZAovzDhcm7S5FBB3oT8H4
@IanMayo IanMayo temporarily deployed to debrief-preview-pr-669 July 12, 2026 07:44 Inactive
@IanMayo IanMayo changed the title Spec 282: repeatable whole-repo review skill (/repo-review) Spec 282: repeatable whole-repo review skill (/repo-review) — spec through implementation Jul 12, 2026
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KZAovzDhcm7S5FBB3oT8H4
@IanMayo IanMayo temporarily deployed to debrief-preview-pr-669 July 12, 2026 07:46 Inactive
From the step-back review of long-term value and criteria:
- Pilot-run guidance (runbook step 0, quickstart, command) before the
  inaugural full run — orchestration is validated behaviourally
- Resolve spec/tier-map divergence: stac-writer confirmed Tier 1 (spec
  amended; it is the ADR-033 silent-data-loss surface)
- Retrospective recall benchmark vs bugs.md ground truth (synthesis step,
  report section, runbook) — the review's only recall signal
- Churn-scoped re-runs (priorSha -> recon weighting; --full for annual
  sweeps) so the cadence survives the token cost (R-011)
- Structural guards: constitution-version drift test (playbook anchored to
  CONSTITUTION.md v1.5 — was citing a stale memory-copy version) and a
  wrapped-async parse test for workflow.js
- Playbook completeness note (why Articles V/X/XI-XIII are unchecked);
  report Next Actions footer; guards proposed-vs-adopted metric; refuted-
  pile sample-audit runbook step. R-011/R-012 recorded in research.md.

39 tests pass; ruff check + format clean.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KZAovzDhcm7S5FBB3oT8H4
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants