Deliver: Per-Wave Back-Pressure Contract (#89, #90) - #105
Merged
Conversation
Feature-scoped research artifact for making "did this wave verify its own work" a deterministic, event-recorded fact rather than an agent self-report. Source: HumanLayer, "Skill Issue: Harness Engineering for Coding Agents". Two load-bearing findings, both verified against the tree: - scripts/check-tests.sh does not run tests. It checks that test files listed in the plan's "## Tests to Write" section exist on disk. The per-wave gate at spine.js:292-307 is real and correctly wired, but the signal flowing through it is file presence, not behavior. The only thing executing verification is the agent, following prose in the wave prompt and self-classifying the result. - Bounded retries re-send an identical prompt. spine.js:249 calls runWave(wave); cli.js:433 binds planCtx once per deliver. No failure context reaches the next attempt, so the doom-loop breaker detects a loop the retry path structurally guarantees. Carries design recommendations for four decision points (declaration site, executor, failure-feedback seam, event shape), the constraints they must respect (pure gate fold, frozen matrix vocabulary, absolute backward compatibility), and five open questions. Status: pending-plan — feeds /rad-plan, not /rad-design. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Author: architect Waves: 5 Tasks: 15 Out-of-scope deps: no Closes the gap where the per-wave gate checks file presence rather than behavior, and where every retry rebuilds an identical prompt. Folds #63's contract substrate (documented usage, tasks[] pass-through, and the wave-attempt optional-key design) into Waves 1-2, per the architect decision recorded on #89. #63 stays open for its read side -- blockedReasonCounts, the /rad-insights subsection, and failed-wave transcript snapshots. Consumes the pre-existing research artifact .agents/research/wave-back-pressure.md in place of a fresh Explore sweep; anchors re-verified against post-#100 main.
Plan: .agents/plans/wave-back-pressure.md
(Status + approved event written by: node harness/cli.js approve)
Two guardrail flags resolved before approval rather than after, so the
fingerprint is frozen over the tightened plan:
- Task 3.2 now enforces a timeout, mapped to the existing fail-timeout
outcome (surface) rather than fail-tests -- a retry cannot fix a hang.
- Task 4.3 now states priorFailure capture is fail-OPEN, the deliberate
exception to the fail-closed default, which governs gate boundaries.
AC#7 and AC#8 added with test coverage for both.
…ve contract Wave 1, Task 1.1 Validated: AC#6 — return-shape table lists `tasks` and `usage` with optionality stated; result→outcome section notes per-task pass-through
Wave 1, Task 1.2 Validated: AC#6 — toWaveResult delegates outcome to resultToOutcome unchanged (null/empty/malformed/unparsed all map identically); a malformed or absent tasks block omits the key rather than throwing or forcing fail-protocol; 216/216 harness tests pass
…d `usage` Wave 1, Task 1.3 Covers BOTH files named by the task: harness/adapters/agent/command.js and harness/adapters/agent/sdk.js. Each replaces its duplicated local toResult with the shared contract.toWaveResult builder, so both emit the optional `tasks` pass-through and normalized `usage` from one definition. Validated: AC#6 — both adapters exercised end-to-end (stub CLI + injected SDK query): with no usage reported the key is omitted and the budget contribution is 0 (spine's `result.usage?.total ?? 0`); outcomes identical across tasks-present / no-tasks / malformed-tasks; usage still attached and counted when reported (sdk 150, command 10); 216/216 harness tests pass
Wave 1 execution-log entries for tasks 1.1-1.3.
Wave 2, Task 2.1 Validated: AC#5 — typedef declares tasks/verify optional on wave-attempt data; zero fold code changed (doc-only diff), npm test --prefix harness 216/216
…event Wave 2, Task 2.2 Validated: AC#5 — spread so the key is absent (never undefined) when the result carries no tasks; drove deliverSpine on the pre-Wave-2 spine.js and on HEAD with identical tasks-free results (no-usage, with-usage, and a fail-tests retry) and cmp'd the serialized event stream byte-for-byte: identical. npm test --prefix harness 216/216.
Wave 2, Task 2.3 Validated: AC#5 — all six folds (reduce, resumeFrom, totalUsage, outcomeCounts, failReasonCounts, retryCounts) asserted against hand-computed pre-change values on a deep-frozen historical log with no tasks/verify keys, plus an additive check that the same log augmented with both keys folds identically. Mutation-checked: perturbing one expected value fails both tests. npm test --prefix harness 218/218 (216 -> 218, +2 new).
Wave 3, Task 3.1
Validated: AC#1 — parsePlanCtx over a plan with no Verify: yields {}; a plan with one yields exactly that wave's command (deeper #### task headings stay inside the wave; blank value is absent)
Wave 3, Task 3.2 Validated: AC#2, AC#7 — passing command exits 0 with output discarded; failing command passes its own exit code through with a bounded excerpt on stdout (500 lines -> 40; 50KB single line -> 8000 bytes); 'sleep 600' under a 3s timeout is killed in 3s and exits the reserved 124 (distinct from failure), with no leaked child process. Command runs under env -i + the adapters' allow-list (PATH HOME LANG LC_ALL TMPDIR TERM): ANTHROPIC_API_KEY/GITHUB_TOKEN/MY_SECRET exported in the parent are NOT visible to it. cwd asserted (not assumed) through the real sh port binding: resolves to the worktree checkout when RAD_WORKTREE isolation is active. Portable kill-after-N (background child + polling watchdog + marker file) — no timeout(1), verified on bash 3.2 and 5.3.
…he result
Wave 3, Task 3.3
Validated: AC#2, AC#3 — at the existing per-wave gate site, a wave that declares
Verify: now runs scripts/check-verify.sh through the injected sh port (shape
unchanged) and reads its real exit code. A non-zero status demotes the outcome
through resolveOutcome to the EXISTING fail-tests; the reserved timeout status
124 takes the EXISTING fail-timeout instead (matrix action surface, not
revision) because a retry cannot fix a hang. Every attempt that ran a command
records verify: {command, status, passed} — spread, so the key is ABSENT (not
present-and-undefined) when no command ran.
Demonstrated end-to-end against the real check-verify.sh + real matrix: pass ->
success/ok, fail -> fail-tests/doom-loop, sleep 600 -> fail-timeout/surface with
a single attempt. AC#1 parity shown by comparison, not assertion: an empty
waveVerify map appends a byte-for-byte identical event sequence and identical sh
call list to the pre-feature call shape. No eighth outcome token exists; gates.js
and matrix.yaml have zero diff vs origin/main.
Wave 3 Validated: process artifact — no testable surface.
Wave 4, Task 4.1 Validated: AC#4 — npm test --prefix harness 218/218 green; every existing runWave stub ignores the new second argument and behaves exactly as before.
…pt section Wave 4, Task 4.2 Validated: AC#4 — absent priorFailure renders byte-for-byte origin/main's prompt (cmp against a temp module built from origin/main:contract.js, two fixtures plus explicit null/undefined); present priorFailure renders the section with an 88,889 char excerpt truncated to the 4000-char cap. 218/218 suite green.
…ilure Wave 4, Task 4.3 Validated: AC#4, AC#8 — end-to-end spine drive shows attempt 2's prompt carrying attempt 1's outcome, blocking-task status and bounded verify excerpt (attempt 1 and 2 prompts now differ); a forced capture failure still runs attempt 2 with a byte-identical priorFailure-absent prompt and records a capture-failed event carrying wave/attempt/outcome/reason. Capture is fail-OPEN by design — the deliberate exception to CLAUDE.md's fail-closed default, which governs gate and check boundaries, not prompt enrichment. Doom-loop fingerprint and MAX_ATTEMPTS byte-identical to origin/main. 218/218 suite green.
Wave 4 execution-log rows for Tasks 4.1-4.3.
Wave 5, Task 5.1 Validated: AC#1 (absent-Verify parity: two runs deep-equal, no verify key, check-verify.sh never invoked; parsePlanCtx yields an empty map), AC#2 (the declared command reaches check-verify.sh as its single argument, after the presence gate), AC#3 (a failing command demotes success to fail-tests inside the frozen vocabulary), AC#4 (attempt 1 gets priorFailure null; attempt 2 carries outcome, blocking task, and excerpt), AC#7 (exit 124 maps to fail-timeout/surface and is not retried), AC#8 (a throwing capture logs capture-failed and attempt 2 still runs) — node --test on both suites, 56/56 pass Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Wave 5, Task 5.2 Validated: AC#2, AC#7 — 14 assertions, each with an explicit exit code: passing (0, output discarded), failing (own code 7), line-cap and byte-cap truncation each asserted to BITE, usage errors (no arg / two args / blank / malformed RAD_VERIFY_TIMEOUT_SECONDS) all exit 2, stderr-only captured, and a wedged command killed at exit 124. Plus the plan's named highest risk as a permanent regression: a non-allow-listed variable never reaches the executed command, with a negative control proving the check is not vacuous. bash scripts/test-check-verify.sh → ALL PASS in 1.4s; committed mode 100755. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Wave 5, Task 5.3 Validated: AC#1, AC#6 — CLAUDE.md gains a Per-Wave Verification block beside the per-wave Model: docs, stating the opt-in, the byte-for-byte absent-declaration guarantee, the containment check-verify.sh adds, and RAD_VERIFY_TIMEOUT_SECONDS (default 600, malformed = hard exit 2 never a silent default, timeout = exit 124 mapping to fail-timeout/surface because a retry cannot fix a hang). agent-contract.test.js covers tasks pass-through, malformed degradation to OMISSION across eight shapes, outcome-drift parity against resultToOutcome, and the prompt section: absence is byte-identical and the truncation cap is asserted to BITE on both the excerpt and the task error. events.js documents the capture-failed event type in the Event typedef and records why it stays absent from PHASE_BY_TYPE (audit-only, establishes no phase, fold unaffected) — added as a comment, never as a key. node --test harness/test/agent-contract.test.js → 23/23 pass Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Wave 5 execution log: step rows for tasks 5.1-5.3, the coverage table, the absent-declaration parity note, and the two discrepancies raised for Gate 2. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Owner
Author
|
Correction. This PR body states that "RAD has no re-approval path (#39)". That is wrong. PR #45 closed that gap on 2026-06-23. The routing decision still stands, on better grounds than the ones given: Log corrected in #106. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Delivers the approved plan
.agents/plans/wave-back-pressure.md.Closes #89
Closes #90
The defect
RAD already ran a deterministic gate after every wave whose outcome would advance,
demoting
success→fail-teststhrough the matrix. But the signal flowing throughthat gate was
scripts/check-tests-present.sh— a file-presence check.A wave could create every promised test file, have all of them fail, and the gate
passed.
The only thing that actually executed verification was the wave agent, following
prose in the wave prompt and self-classifying the result. The harness never saw the
command, never ran it, never saw its output. That is the trusted-prose pattern the
deliver gate exists to eliminate everywhere else in RAD.
Compounding it (#90): every retry rebuilt an identical prompt, so a retry differed
from its predecessor only by model nondeterminism. RAD had a doom-loop detector for a
loop its own retry path structurally guaranteed.
What changed
A
### Wave Nblock may now carry an optionalVerify:line. When present, the harnessexecutes it via
scripts/check-verify.shthrough the spine's existingshport andreads its real exit code. Non-zero demotes to the existing
fail-tests; a timeoutmaps to the existing
fail-timeout. On failure, the next attempt's prompt carries a## Prior Attempt Failuresection with a bounded excerpt and the blocking task's status— so retries now differ.
### Wave 1 Verify: npm test --prefix harnessOpt-in with a byte-for-byte absent-declaration guarantee: a plan declaring no
Verify:anywhere produces an event sequence byte-identical to a pre-verification run.Demonstrated, not asserted — by driving
origin/main's spine and this branch's spinethrough identical scripted results and
cmp-ing the serialized event streams.Waves 1-2 landed the contract substrate folded from #63 (
tasks[]andusageasdocumented optional contract fields, both new
wave-attemptkeys designed as one shapechange). #63 keeps its read side.
Security — the plan's named highest risk
The plan flagged Task 3.2 as its highest risk: "
check-verify.shmust apply theallow-listed-env treatment or it becomes a credential-leak path." Verified independently
with canary values, not taken on report:
ANTHROPIC_API_KEY,GITHUB_TOKEN,MY_SECRETexported in parentHOME LANG PATH TERM TMPDIRsleep 600under a 3s ceilingexit 7timeout(1)is absent on darwin, socheck-verify.shuses a polling watchdog with amarker file to distinguish a real timeout from a command that chose to exit 143 — a
SIGTERMed process and a self-terminating one are indistinguishable by wait status alone.
Verified under both bash 5.3 and
/bin/bash3.2.The command is arbitrary shell from a human-approved plan — approval is the trust
boundary, unchanged.
Invariants held
harness/gates.jszero diff — verification attaches at the spine/script boundary,never inside the fold.
harness/matrix.yamlzero diff — the 7-outcome vocabulary gained nothing. Grepconfirms exactly the frozen seven tokens.
scripts/check-tests-present.shzero diff — presence and behavior catch differentfailure modes; both gates kept (check-tests.sh does not run tests — it checks that promised test files exist #91).
MAX_ATTEMPTSuntouched — the only matching diff lines areadded comments recording that the capture sits after the fingerprint decision and
never feeds it. This plan makes retries differ; it does not change how many there are.
reduce,resumeFrom,totalUsage,outcomeCounts,failReasonCounts,retryCounts, proven against a fixture lacking both new keys andmutation-checked to confirm the gate is not vacuous.
Verification
scripts/test-check-verify.sh(14 assertions, ~2s, committed
100755) which carries a permanent regression test forthe env-containment boundary.
after two rebases, since rebasing changes the approval commit's hash.
For architect review
Two Program Design signatures proved inaccurate, and in both cases the delivering wave
resolved in favor of the actual constraint rather than the document:
check-verify.sh <feature> <command>— theshport passes exactly one argument(
sh('scripts/check-tests-present.sh', feature)), and AC#2 requires the port shape stayunchanged. Shipped as
check-verify.sh <command>;<feature>was unused.buildWavePrompt(wave, planCtx, priorFailure?)— the provider-neutral contract definesadapter(wave, planCtx)as two-argument. Shipped readingplanCtx.priorFailure, foldedin by
cli.js. A third positional argument would have widened the documented contract.Judgment calls to ratify:
PRIOR_FAILURE_FIELD_MAX_CHARS = 4000. The plan deliberately leftthis number to the delivering wave for architect review. Chosen tighter than the
producer's own bound (40 lines / 8000 bytes) so the prompt stays bounded even for text
that never passed through
check-verify.sh.a timeout — GNU
timeout(1)convention, and it errs conservative: the false positivesurfaces (terminal) rather than retrying, so it cannot burn attempts or hide.
capture-failed, appended only on the fail-open degrade path. Followsthe
hook-failedprecedent. Deliberately absent fromPHASE_BY_TYPEso itestablishes no phase and the fold is unaffected — the same treatment as
architecture-approved/owner-claimed/owner-released— and now documented in both thetypedef union and the deliberate-absence comment.
fail-protocol") conflicts withpre-existing behavior:
resultToOutcome({}) === 'fail-protocol'. Ruled to govern thetaskspass-through, not the outcome mapping. An agent emitting an unparseable resultblock has committed a real protocol violation; softening that would weaken the exact check
this feature strengthens and break AC#1's parity guarantee.
Deliberate fail-open exception (AC#8):
priorFailurecapture is prompt enrichment, nota gate, so a capture failure logs its reason and degrades to today's prompt rather than
blocking. This is the stated exception to CLAUDE.md's fail-closed default, which governs
gate and check boundaries. Demonstrated by forcing the capture to throw — attempt 2 still
ran, with a prompt byte-identical to attempt 1's, and a
capture-failedevent recorded.Delivery notes
Gate 2 initially failed on
.agents/research/wave-back-pressure.md— this branch's ownfirst commit (2026-08-04), untouched by any wave.
.agents/research/was missing fromcheck-scope.sh's exempt prefixes while logs/plans/state were all exempt. It could not befixed from inside this delivery: amending the plan's Files in Scope would invalidate the
approval fingerprint, and RAD has no re-approval path (#39). Fixed separately in #104,
merged, then this branch rebased.
One loop run reported
test-check-tests-present.shfailing; not reproducible standalone orin two subsequent full runs, and that run executed concurrently with the harness suite.
Recorded rather than dismissed.
Full execution log, including every wave's concerns and the architect rulings on each:
.agents/logs/wave-back-pressure-2026-08-10.md