feat: test-writing subagent for the development loop#87
Merged
Conversation
Add the test_plan artifact type to the loop manifest schemas and the artifact_type Postgres enum, require a plan-review approval gate in the default development-loop manifest, and change the agent_plans agent_name default to planner. Seed data gains a demo test_plan artifact. Part of #47. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Shared repository-scoped discovery, text search, and line-range read helpers for stage subagents: commit-pinned provenance, secret/generated/ dependency path exclusions, symlink-escape rejection, and fail-closed query and output bounds. Part of #47. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ed plan digests Relocate the planning tool surface to a declared planner subagent with an isolated deny-all sandbox and repository inspection tools. Plan artifacts gain a stable identity, a canonical SHA-256 digest, and a pinned repository revision enforced by pinnedPlanningAgentOutputSchema. Adopts openai/gpt-5.6-sol with xhigh reasoning per the issue #47 model routing contract, superseding the gpt-5.5 label recorded in ADR 0013. Part of #47. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Declared test-writer sibling subagent (openai/gpt-5.6-terra, xhigh) with an isolated deny-all sandbox. It turns an approved plan into focused failing tests and emits loopworks.test_plan.v1 plus loopworks.red_test_evidence.v1: AC-mapped tests, explicit fixtures, a bounded SHA-256-bound test-only unified patch, and HMAC execution receipts minted only by the guarded run_test_suite tool so unexecuted tests cannot self-attest as red. Includes a golden fixture and Eve eval. Part of #47. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…d evidence Run creation now seeds a plan-review approval bound to the plan row and canonical digest, and provisions separate red-evidence and test-plan artifacts for the test-writing stage. recordDevelopmentLoopPlanArtifact persists digest-valid pinned plans and requests review; an approved plan-review advances the run to test-writing while other resolutions block it. applyDevelopmentLoopTestWritingResult advances to development only for exact plan/approval/digest matches with HMAC-verified expected-failure receipts, claim-guarded against concurrent transitions. Part of #47. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The root Eve runtime now only reads durable run state, delegates one stage to the matching declared subagent, and invokes deterministic control-plane tools (read_run_stage_context, record_plan_artifact, apply_test_writing_result). Root bash and load_skill are disabled; the orchestrator never infers approval from prompts or mutates GitHub. Part of #47. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
ADR 0015 captures the neutral root orchestrator, sibling subagents with isolated sandboxes, typed artifact handoffs, and fail-closed plan-review gating. Architecture and persona scenarios updated to match. Part of #47. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
LOOPWORKS_EVE_TEST_RECEIPT_SECRET is required for the test-writing transition (it fails closed without it); the fixture-mode flag is non-production only. Part of #47. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Plan digests, test-plan digests, and execution-receipt HMACs previously hashed JSON.stringify output, which depends on object key order and only stayed deterministic while every caller digested schema-parsed values (jsonb roundtrips reorder keys). Serialize through a recursive key-sorted canonical JSON encoder so digests are stable for any equivalent value. Part of #47. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Receipts previously bound command, paths, exit, output, and patch digests but not plan identity, so a receipt minted under one approved plan could be replayed for another run with an identical patch. The receipt payload now carries the plan digest: run_test_suite requires it at mint time and both verification paths recompute it from the persisted plan, failing closed on mismatch. Part of #47. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Applying an approved plan review previously re-blocked a run that had already advanced to test-writing, and non-approve resolutions wrote raw approval statuses (bypassed, cancelled, applied) into the plan row. Plan-review side effects now advance only on approved, fail closed to a blocked run with a rejected plan for every other resolution, and skip run/plan mutation entirely for applied. Part of #47. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
recordDevelopmentLoopPlanArtifact unconditionally reset loopRuns.startedAt to the queue time; keep an already-recorded start and fall back to queuedAt, matching the test-writing transition. Part of #47. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #47
Summary
Adds the Eve test-writer as a declared sibling subagent under a new neutral root stage orchestrator (ADR 0015), turning an exact approved plan into focused failing tests, versioned red evidence, and a reusable automated test plan before development begins.
planner,test-writer), and persists results only through deterministic control-plane tools (read_run_stage_context,record_plan_artifact,apply_test_writing_result). Rootbashandload_skillare disabled.agent/subagents/planner/with an isolated deny-all sandbox. Plan artifacts gain a stable identity, canonical SHA-256 digest, and pinned repository revision (pinnedPlanningAgentOutputSchema).openai/gpt-5.6-terra, xhigh): emitsloopworks.test_plan.v1(AC-mapped tests, explicit fixtures, bounded SHA-256-bound test-only unified patch) andloopworks.red_test_evidence.v1with HMAC execution receipts minted only by the guardedrun_test_suitetool — unexecuted tests cannot self-attest as red.plan-reviewapproval bound to the plan row and digest and provisions separate red-evidence and test-plan artifacts. The test-writing stage advances only for exact plan/approval/digest matches with verified expected-failure receipts, claim-guarded against concurrent transitions. All mismatches fail closed.Model routing
Per this issue's Model Routing contract, the root orchestrator and planner move from
openai/gpt-5.5toopenai/gpt-5.6-soland the test-writer usesopenai/gpt-5.6-terra(both xhigh). Two reviewer notes:model: z.literal("openai/gpt-5.6-sol-xhigh"), so any previously persistedagent_plansrows carrying the gpt-5.5 label would fail parsing at the test-writing transition. Acceptable pre-production; a data migration would be needed once real plan rows exist.Review hardening applied on top
Follow-up review of the original implementation found and fixed four issues in this branch:
JSON.stringifyoutput, which is key-order dependent and only stayed deterministic while every caller digested schema-parsed values (jsonb roundtrips reorder keys). Now serialized through a recursive key-sorted canonical JSON encoder.bypassed,applied, …) into the plan row. Side effects now advance only onapproved, fail closed (blocked run, rejected plan) for reject/cancel/expire/bypass, and skip run/plan mutation forapplied.recordDevelopmentLoopPlanArtifactno longer resets an already-recordedstartedAt.LOOPWORKS_EVE_TEST_RECEIPT_SECRETandLOOPWORKS_EVE_TEST_WRITER_FIXTURE_MODEare now documented in.env.exampleand the README.Validation
bun run validate(format, lint, agent-docs, markdownlint, typecheck, vitest, Storybook build, Playwright e2e): PASSbun run build: PASSPersona test IDs
M02, A01, A02, A03, R01 — covered by the updated unit/pglite suites and persona scenario doc updates; browser-lane coverage for the new artifacts rides the existing seeded Playwright lane.
🤖 Generated with Claude Code