Spec 282: repeatable whole-repo review skill (/repo-review) — spec through implementation#669
Open
IanMayo wants to merge 13 commits into
Open
Spec 282: repeatable whole-repo review skill (/repo-review) — spec through implementation#669IanMayo wants to merge 13 commits into
IanMayo wants to merge 13 commits into
Conversation
Speckit specification produced from a structured interview covering review dimensions (constitution conformance, correctness bugs, tech-debt refresh, test quality), tiered whole-repo scope, verified-only findings via adversarial verification, multi-agent execution, evidence tooling, and a findings ledger enabling delta re-runs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KZAovzDhcm7S5FBB3oT8H4
Contributor
📋 Spec NavigatorReview this PR's specs interactively:
|
Second interview round: fix-session handoff (/repo-review.fix consuming ledger entries), prevention-per-theme guard proposals, memory integration (bugs.md + failure-pattern docs), playbook self-calibration, resolution-rate as the primary value metric, and an operating runbook (triage ritual, quick-wins week, guard adoption, playbook tuning). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KZAovzDhcm7S5FBB3oT8H4
…quickstart) /speckit.plan output: implementation plan with constitution gate (PASS, two justified notes), research decisions R-001..R-010, ledger data model + JSON Schema contract, report-structure and fix-handoff contracts, quickstart, and cached blog opener. Updates CLAUDE.md recent-changes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KZAovzDhcm7S5FBB3oT8H4
/speckit.tasks output. Setup + Foundation (ledger schema + typed helper) → US1 trustworthy report (MVP) → US2 delta re-runs → US4 fix handoff → US5 themes/guards → US3 evidence tooling → US6 memory/calibration → Polish (inaugural dogfood run, evidence, ADR, blog, PR). Evidence artifacts and final /speckit.pr task included. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KZAovzDhcm7S5FBB3oT8H4
…playbooks) Phase 1-2 + partial Phase 3: tier-map.yaml, severity-rubric.md, ledger JSON Schema asset, typed review-ledger.py helper (load/save/validate/reconcile/ record-fix-pr + tuning-recommendation), pytest suite, constitution + correctness playbooks. In-progress; remaining playbooks, workflow, and commands to follow. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KZAovzDhcm7S5FBB3oT8H4
…an-95vovz # Conflicts: # CLAUDE.md
All four dimension playbooks (CC/CB/TD/TQ), tier-map and playbook structural
tests, tuning-recommendation tests. Fixes: distinguish dry-run (None) from
finalise ({}) in reconcile; snapshot original entries so new ids aren't
swept into the fixed loop; narrow JSON boundary off bare Any; TYPE_CHECKING
imports. 36 tests pass, ruff clean.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KZAovzDhcm7S5FBB3oT8H4
Apply ruff format to review-ledger.py and test_playbook_structure.py (CI runs ruff format --check); remove unused RESOLVED_STATUSES; add the report template (FR-007) with themes/prevention and playbook-tuning sections. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KZAovzDhcm7S5FBB3oT8H4
…mands, strict lint configs Adds the Workflow script (recon → review→verify pipeline → evidence → synthesis), the main /repo-review command (clean-tree gate, verified-only, write boundary, memory + calibration, degradation) and /repo-review.fix fast-path (ledger-driven fix→test→PR), plus report-only strict ruff + eslint configs for the evidence phase. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KZAovzDhcm7S5FBB3oT8H4
…ge, report format) ADR-041 records the ledger design (YAML+JSON-Schema over LinkML, defect-identity reconciliation, verified-only bar). Evidence: real pytest summary (36 pass, 75% helper coverage), real CLI transcripts (validate/reconcile/record-fix-pr), valid+corrupt validation output, usage walkthrough, and a labelled report-format excerpt. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KZAovzDhcm7S5FBB3oT8H4
Content Specialist shipped-post.md (cached-opener first three sections verbatim + By the Numbers / Lessons Learned / What's Next from evidence). Mark 52 of 54 tasks complete; T045 (inaugural full multi-agent run — maintainer acceptance test) and T054 (blog publish to debrief.github.io — separate repo) remain. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KZAovzDhcm7S5FBB3oT8H4
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KZAovzDhcm7S5FBB3oT8H4
From the step-back review of long-term value and criteria: - Pilot-run guidance (runbook step 0, quickstart, command) before the inaugural full run — orchestration is validated behaviourally - Resolve spec/tier-map divergence: stac-writer confirmed Tier 1 (spec amended; it is the ADR-033 silent-data-loss surface) - Retrospective recall benchmark vs bugs.md ground truth (synthesis step, report section, runbook) — the review's only recall signal - Churn-scoped re-runs (priorSha -> recon weighting; --full for annual sweeps) so the cadence survives the token cost (R-011) - Structural guards: constitution-version drift test (playbook anchored to CONSTITUTION.md v1.5 — was citing a stale memory-copy version) and a wrapped-async parse test for workflow.js - Playbook completeness note (why Articles V/X/XI-XIII are unchecked); report Next Actions footer; guards proposed-vs-adopted metric; refuted- pile sample-audit runbook step. R-011/R-012 recorded in research.md. 39 tests pass; ruff check + format clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KZAovzDhcm7S5FBB3oT8H4
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Delivers the
/repo-reviewskill end-to-end: a repeatable, multi-agent whole-repo audit whose findings must survive adversarial verification before they're reported, plus a/repo-review.fixfast-path from any finding to a fix PR. This PR carries the full speckit trail (spec → plan → research → data-model → contracts → tasks) and the implementation.The distinguishing rule is verified-only: every candidate finding is checked by a separate adversarial verifier that tries to refute it; refuted and undecidable candidates are counted, not listed. The bet is a report the maintainer can act on without re-checking each claim.
What's implemented
Skill assets (
.claude/)commands/repo-review.md— clean-tree gate, run identity, verified-only discipline, write boundary, memory + calibration, graceful degradationcommands/repo-review.fix.md— ledger-driven fix→test→PR fast-path (mirrors/bugfix)review/workflow.js— Workflow orchestration: recon → per-cell review→verify pipeline (no barrier) → evidence → synthesisreview/playbooks/{constitution,correctness,tech-debt,test-quality}.md— 20 falsifiable heuristics (CC/CB/TD/TQ), the tech-debt one framed as a Load existing result files into Attachments dropdown on plot open #172 regression checkreview/{tier-map.yaml, severity-rubric.md, report-template.md, ledger.schema.json}review/{ruff-strict.toml, eslint-strict.config.mjs}— report-only strict-lint configs for the evidence phaseTyped helper + tests
scripts/review-ledger.py— strict-typedvalidate/reconcile/record-fix-pr, load/save with schema gate on every read and write, defect-identity reconciliation, deterministic playbook-tuning logictests/repo_review/— 36 tests (ledger validation, reconciliation lifecycle, fix-handoff preconditions, playbook/tier-map structural checks, tuning logic). All green; 75% helper coverage (uncovered = argparse wrappers, exercised via real CLI transcripts)Docs & evidence
docs/project_notes/decisions.md— ADR-041 (YAML+JSON-Schema over LinkML, defect-identity reconciliation, verified-only bar)specs/282-repo-review-skill/evidence/— real test summary, CLI transcripts, valid+corrupt validation output, usage walkthrough, ledger sample, labelled report-format excerptspecs/282-repo-review-skill/media/shipped-post.md— feature blog postValue-capture design (from the interview)
Report-only, but the loop is closed:
/repo-review.fix RR-NNNturns a finding into a fix PR; each theme proposes one permanent guard (lint rule / CI gate / CLAUDE.md instruction / constitution amendment); confirmed Critical/High bugs flow intobugs.md; per-heuristic confirmed/refuted stats drive playbook tuning between runs. The headline success metric is resolution rate by the next run, not finding count.Verification
uv run ruff check .+ruff format --check .— cleanuv run pytest— 2206 passed, 2 skipped, 1 xfailed (full Python suite; nothing broke)tests/repo_review/— 36 passedNot in this PR (deliberate)
debrief.github.io(T054) — separate repo; run/publish-future-postthere when ready.🤖 Generated with Claude Code
https://claude.ai/code/session_01KZAovzDhcm7S5FBB3oT8H4