Skip to content

Build the carve-changesets test and eval suite - #46

Merged
shaug merged 3 commits into
mainfrom
scott/36-carve-eval-suite
Jul 22, 2026
Merged

Build the carve-changesets test and eval suite#46
shaug merged 3 commits into
mainfrom
scott/36-carve-eval-suite

Conversation

@shaug

@shaug shaug commented Jul 22, 2026

Copy link
Copy Markdown
Owner

TL;DR

Completes the carve-changesets verification suite with runtime-agnostic behavioral evaluation, deterministic integration checks, and direct regressions for the epic's safety and rehydration fixes.

Summary

  • Replace the hardcoded agent CLI evaluation path with a fresh-process stdin/stdout JSON executor contract.
  • Add result-blind judgment cases for decomposition, guardrails, immutable validated plans, merge authority, and non-review-ready input.
  • Preserve the deterministic grader as an objective integration layer for clean-tree, plan, immutable-source, chain, equivalence, and validation invariants.
  • Add direct regression coverage for rehydration edge cases, real PR creation through a stub, merge-and-propagate, base/source push safety, derived live state, and the two-part source-freshness override.
  • Document the new evaluation model and keep the anti-spam action vocabulary derived from its expectation source.

Validation

  • just format
  • just lint
  • just test
  • just test-carve-changesets (95 tests)
  • just eval-carve-changesets (9 behavioral cases and 2 deterministic integration cases)
  • Both carve targets with the agent CLI absent from PATH
  • Repository-owned review: clean on the final head after solution-simplicity, correctness, code-simplicity, and required targeted rechecks

Tickets

Fixes #36

shaug added 3 commits July 21, 2026 19:15
@shaug
test: complete the carve-changesets verification suite
f669d32 
## Summary
- Replace the hardcoded agent CLI eval path with result-blind executor and deterministic integration modes
- Add judgment-focused behavioral cases plus objective equivalence fixtures
- Close the remaining rehydration, mutation-safety, propagation, and live-validation regression gaps

## Why
- Make the carve-changesets evaluation suite runtime-agnostic and consistent with repository peers
- Preserve deterministic helper verification while testing the skill's decisions and terminal states
- Complete the final verification child of epic #28
@shaug
fix: require the two-part source freshness override
eb86123 
## Summary
- Require both source-behind allowance and confirmation flags before preflight proceeds
- Add direct regressions for each lone flag and the valid two-flag override

## Why
- Align the executable safety boundary with the published carve-changesets contract
- Prevent behavioral evals from passing while real preflight accepts incomplete authority
@shaug
refactor: derive the eval action vocabulary from expectations
e30b5f1 
## Summary
- Remove the duplicated action vocabulary from the deterministic fixture executor
- Derive the anti-spam test input from required and forbidden expectation actions

## Why
- Keep one authoritative representation of the behavioral evaluation vocabulary
- Ensure the adversarial grading check expands automatically as expectations evolve
@shaug
shaug merged commit 93e64cc into main Jul 22, 2026
1 check passed
@shaug
shaug deleted the scott/36-carve-eval-suite branch July 22, 2026 05:06
@shaug shaug mentioned this pull request Jul 22, 2026
Closed
10 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Build the carve-changesets test and eval suite

1 participant

@shaug