Deliver: check-tests Naming Honesty - #100
Merged
Merged
Conversation
Adopted-From: #91 Author: architect Waves: 5 Tasks: 12 Out-of-scope deps: no Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Plan: .agents/plans/check-tests-naming-honesty.md (Status + approved event written by: node harness/cli.js approve) Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…s header Wave 1, Task 1.1 Validated: AC#1 — ran the pre-rename and post-rename script against fixture plans covering present, missing, unresolvable, empty-section, backtick, and both usage-error cases; stdout and exit codes diff clean (identical) Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ation together Wave 1, Task 1.2 Validated: AC#2, AC#3 — both sh() invocations now target scripts/check-tests-present.sh; grep for regression/still-green/greenness vocabulary in spine.js returns no hits; recorded event data (categories/summary) left byte-identical Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Wave 1, Task 1.3 Validated: AC#5 — node --test harness/test/spine.test.js reports tests 31 / pass 31 / fail 0, identical to the pre-rename baseline captured from the branch base Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Wave 2, Task 2.1 Validated: AC#5 — node --test harness/test/resume.test.js, 9/9 pass (same count as before); restored assertion verified to have teeth by inspection of spine.js:199 skip-continue preceding the spine.js:216 verify call
Wave 2, Task 2.2 Validated: AC#5 — scripts/test-script-hardening.sh exits 0 with 3/3 cases (same case count), case #7 back to green; backtick-resolution assertions unchanged
Wave 2, Tasks 2.1-2.2
… references Wave 3, Task 3.1 Validated: AC#3 — grep -n "check-tests" harness/hook-runner.js shows only scripts/check-tests-present.sh; node --check clean
Wave 3, Task 3.2 Validated: AC#4 — line names check-tests-present.sh, describes a presence check, arrow column still aligns at 45 across the tree block
Wave 3, Task 3.3 Validated: AC#4 — both references name check-tests-present.sh and describe a presence check; the execution gap is named as issue #89 at line ~200
Wave 4, Task 4.1 Validated: AC#4 — sh() example names scripts/check-tests-present.sh; feature column re-aligned at col 39 across all three post-check lines (awk index check)
Wave 4, Task 4.2 Validated: AC#4 — post-check list names check-tests-present; diff is 1 line, DET/MODEL split claim byte-identical
Wave 4, Task 4.3 Validated: AC#4 — both live invocation sites call scripts/check-tests-present.sh; smoke-tested the rad-deliver invocation shape against this feature's plan (exit 0); failure prose now states missing files, not failing tests; grep -rn check-tests .claude/commands/ shows only the new name
Wave 5, Task 5.1 Validated: AC#2 — grep -rn "check-tests" .claude/agents/ shows only check-tests-present.sh; AC#4 — scripts/lint-agent-files.sh exits 0 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…in discovery-interactive-evaluation.md Deviation A (architect-approved, out of Files in Scope) Validated: AC#1 — git diff confirms the single table row now names check-tests-present.sh; description text unchanged
…ce guard for check-tests-present.sh Wave 2, Task 2.3 (deferred to tail — guard precondition met only after Wave 4) Validated: AC#1, AC#6 — 6 behavior assertions (present, missing, unresolvable, empty-section, backtick-wrapped, executable) each asserting an explicit exit code, all green; stale-reference guard passes on the current tree and was proven to fail correctly via a temporary docs/__guard-probe.tmp containing the bare pre-rename name (exit 1, message named the probe file and line), probe then deleted and git status confirmed clean
Wave 2, Task 2.3 (deferred to tail — guard precondition met only after Wave 4) Validated: execution log rows appended for Deviation A, Deviation B, and Task 2.3
…to tracked files Wave 2, Task 2.3 (deferred to tail — guard precondition met only after Wave 4) Validated: AC#1, AC#6 — guard now enumerates via git grep (the tracked set git ls-files lists) so it cannot fail on gitignored per-machine files no commit can fix; all 7 assertions green; a tracked probe still fails the guard by name (exit 1) while an identical untracked probe is correctly ignored (exit 0); probe removed and git status confirmed clean
Wave 2, Task 2.3 (deferred to tail — guard precondition met only after Wave 4) Validated: execution log row appended for the guard scoping fix
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…scope check-scope.sh reads only the Files-in-Scope *File* column. The `git mv` destination scripts/check-tests-present.sh was named only in a Change cell, so it read as out-of-scope drift (#99), as did the architect-approved rename of docs/discovery-interactive-evaluation.md. Both paths were surfaced and approved during delivery; this records that approval as declared scope rather than PR-body prose. No delivered behavior changes. The plan body changes, so the approved fingerprint moves and a re-approval follows — transitions.js permits a second approved event when the fingerprint differs. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Records the architect approval of the two paths added to Files in Scope as an approved event carrying the new plan fingerprint (d3247102), so check-approval-integrity.sh passes at the PR head. transitions.js:94-112 permits this: a second approved event is a legitimate re-attestation when the fingerprint differs from the prior approval (3775e8f7), and is blocked only when identical. Status restored to complete -- the plan is delivered; rad approve rewrites the display-only header to approved. Header lines are excluded from the fingerprint by construction, so this does not shift the hash (verified). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
seanrreid
added a commit
that referenced
this pull request
Aug 6, 2026
Author: architect Waves: 5 Tasks: 15 Out-of-scope deps: no Closes the gap where the per-wave gate checks file presence rather than behavior, and where every retry rebuilds an identical prompt. Folds #63's contract substrate (documented usage, tasks[] pass-through, and the wave-attempt optional-key design) into Waves 1-2, per the architect decision recorded on #89. #63 stays open for its read side -- blockedReasonCounts, the /rad-insights subsection, and failed-wave transcript snapshots. Consumes the pre-existing research artifact .agents/research/wave-back-pressure.md in place of a fresh Explore sweep; anchors re-verified against post-#100 main.
seanrreid
added a commit
that referenced
this pull request
Aug 10, 2026
Author: architect Waves: 5 Tasks: 15 Out-of-scope deps: no Closes the gap where the per-wave gate checks file presence rather than behavior, and where every retry rebuilds an identical prompt. Folds #63's contract substrate (documented usage, tasks[] pass-through, and the wave-attempt optional-key design) into Waves 1-2, per the architect decision recorded on #89. #63 stays open for its read side -- blockedReasonCounts, the /rad-insights subsection, and failed-wave transcript snapshots. Consumes the pre-existing research artifact .agents/research/wave-back-pressure.md in place of a fresh Explore sweep; anchors re-verified against post-#100 main.
seanrreid
added a commit
that referenced
this pull request
Aug 10, 2026
Author: architect Waves: 5 Tasks: 15 Out-of-scope deps: no Closes the gap where the per-wave gate checks file presence rather than behavior, and where every retry rebuilds an identical prompt. Folds #63's contract substrate (documented usage, tasks[] pass-through, and the wave-attempt optional-key design) into Waves 1-2, per the architect decision recorded on #89. #63 stays open for its read side -- blockedReasonCounts, the /rad-insights subsection, and failed-wave transcript snapshots. Consumes the pre-existing research artifact .agents/research/wave-back-pressure.md in place of a fresh Explore sweep; anchors re-verified against post-#100 main.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #91.
scripts/check-tests.shnever ran a test. It parsed a plan's## Tests to Writesection and checked that each listed file existed on disk. The script's own header
was already honest; the filename and
harness/spine.js's narration were not — the spineclaimed per-wave regression detection and "confirm the prior work is still green" for a
gate that can pass while every promised test fails.
This changes only the honesty of the surface. Adding real execution is #89 and is
deliberately excluded — file presence catches a distinct failure mode (an agent silently
dropping a promised test) that a green suite would never reveal, so both gates are worth
having.
What changed
scripts/check-tests.sh→scripts/check-tests-present.sh(git mv, clean rename, no shim)harness/spine.js— both call sites, plus five comment blocks rewritten to presence languageharness/hook-runner.js, all harness tests,scripts/test-script-hardening.shREADME.md,docs/how-it-works.md,docs/harness-state-store.md,docs/harness-audit.md.claude/commands/team/specs (live invocation sites — smoke-tested, not just grepped).claude/agents/definitionsscripts/test-check-tests-present.sh— five behavior cases with explicit exitcodes, plus a stale-reference guard over tracked files
Zero behavioral delta: parsing, exit codes, and output text are untouched.
Verification
node --test harness/test/*.test.js)scripts/test-*.shsuites greenscripts/check-tests-present.shon this plan: exit 0, 5 test files presentcontaining the old name fails it; an untracked probe is correctly ignored
Architect-resolved items
check-scope.shexits 1 on two paths. Neither is real drift; both were surfaced andapproved during delivery:
scripts/check-tests-present.shcheck-scope.shreads the Files-in-Scope File column only; thegit mvdestination lived in the prose Change column. The checker structurally cannot express a rename — filed as #99.docs/discovery-interactive-evaluation.mdDeviations from the approved plan
lands, since four live files still held the old path. A plan-sequencing error. Executed
as a tail step; the plan document was left unedited so the approved fingerprint stays valid.
scripts/check-tests.shto the bare filename.As approved, it caught 11 further live references the narrow form would have missed.
git grep). A repo guard should not fail ongitignored, per-machine files that no commit can fix.
.claude/settings.local.jsonallowlisted the old name. Updated in the working treewith architect approval, but it is globally gitignored — so that fix is not in this PR
and will not propagate. Other machines may see one permission prompt.
Known-unfixed, by design
The recorded event data at
harness/spine.js(categories: ['check-tests']and itssummary string) keeps the old name. It is written into
events.jsonland may be matchedby folds, so renaming it is a data change, not a naming change — excluded under the
additive-at-parity rule. Better handled with #63/#97.
Historical records in
plans/,.agents/plans/,.agents/research/, and.agents/architecture/intentionally retain the old name; rewriting them would falsifythe archive. A grep for the old name still returns ~30 hits there by design.
Incidental findings filed
check-scope.shcannot express a renameAlso noticed, not filed:
scripts/test-check-scope.shis mode100644onmain— notexecutable, so a runner invoking it directly gets exit 126 and may silently skip it.
Pre-existing, unrelated to this change.
🤖 Generated with Claude Code