Build the carve-changesets test and eval suite

CHANGELOG.md

@@ -4,9 +4,15 @@ summary: Chronological history of repository and skill changes.
 
 # Changelog
 
-## 2026-07-21 — Consolidated carve-changesets CLI and live contract
+## 2026-07-21 — Completed carve-changesets skill and verification suite
 
+- refactor: derive the eval action vocabulary from expectations
+- fix: require the two-part source freshness override
+  (`eb8612300d75d1483995677d75f54fe1a32b60d7`)
+- test: complete the carve-changesets verification suite
+  (`f669d322985c435daf9b0c7296889d8a3bdd270c`)
 - feat: package the carve-changesets skill
+  (`a8e19110380e048e0aaf85e820c114fe2a07cc7f`)
 - docs: define carve-changesets suite handoffs
   (`2df5136e2a7226666bc136e30905c2442a579c78`)
 - feat: add stateless changeset merge and propagation

README.md

@@ -89,13 +89,24 @@ Validate a review packet and result together:
 python3 review-suite/scripts/validate.py pair packet.json result.json
 ```
 
-Run deterministic local evals without an agent runtime:
+Run deterministic local evaluation harnesses without an agent runtime:
 
 ```bash
 just eval-carve-changesets
 just eval-implement-ticket
 ```
 
+The carve-changesets command first runs its objective integration self-test,
+which checks clean-tree, plan, immutable-source, chain, equivalence, and
+validation invariants. It then runs peer-shaped judgment cases through a fresh
+process for each result-blind packet. The bundled executor is a deterministic
+simulation of a compliant runtime, not a model evaluation. Pass any compatible
+stdin/stdout JSON adapter with:
+
+```bash
+just eval-carve-changesets-executor "python3 /path/to/adapter.py"
+```
+
 The ticket-composition evaluator starts a fresh process for each case, with
 fixture identity and grader expectations withheld. Case artifacts carry
 pre-classified scenario flags (for example, a CI failure already labeled

justfile

@@ -62,10 +62,12 @@ test-carve-changesets:
   python3 -m unittest discover -s {{skills_dir}}/carve-changesets/scripts/tests -p 'test_*.py'
 
 eval-carve-changesets:
-  {{skills_dir}}/carve-changesets/scripts/evals/runner.py --skip-codex
+  python3 {{skills_dir}}/carve-changesets/scripts/evals/runner.py --integration-self-test
+  python3 {{skills_dir}}/carve-changesets/scripts/evals/runner.py
 
-eval-carve-changesets-codex:
-  {{skills_dir}}/carve-changesets/scripts/evals/runner.py
+# Forward-evaluate through any fresh-process stdin/stdout JSON adapter.
+eval-carve-changesets-executor executor:
+  python3 {{skills_dir}}/carve-changesets/scripts/evals/runner.py --executor "{{executor}}"
 
 validate-skills: lint-skills
 

skills/carve-changesets/evals/README.md

@@ -0,0 +1,25 @@
+# Carve-changesets evaluations
+
+`cases.json` contains peer-shaped behavioral scenarios. `expectations.json`
+keeps their terminal states and action obligations separate so an evaluated
+runtime never receives grader answers or fixture identities. The runner sends
+one JSON packet on stdin to a fresh `--executor` process for each case and
+expects one JSON object on stdout.
+
+The bundled `fixture_executor.py` is a deterministic simulation used to test the
+harness and contract plumbing. It is not a model evaluation. A compatible agent
+adapter can be supplied with:
+
+```bash
+python3 scripts/evals/runner.py --executor "python3 /path/to/adapter.py"
+```
+
+`integration_cases.json` replaces the former mechanics-only prompt CSV. Run it
+with `scripts/evals/runner.py --integration-self-test`. That mode directly
+exercises the real helpers and retains the deterministic grader as an objective
+validation layer: it checks a clean tree, plan validity, source-hash
+immutability, complete chain materialization, reconstructed-tree equivalence,
+and approved prefix validation. It does not evaluate agent judgment.
+
+`just eval-carve-changesets` runs both the integration self-test and the
+result-blind forward cases without requiring any particular agent CLI.

skills/carve-changesets/evals/cases.json

@@ -0,0 +1,56 @@
+[
+  {
+    "id": "decompose-independent-subsystems",
+    "request": "Propose reviewable changesets for this source branch.",
+    "candidate_state": "The source diff contains two independent subsystems: authentication policy and report rendering. Each can compile and be reviewed without the other.",
+    "authority": "decompose_only"
+  },
+  {
+    "id": "separate-mechanical-rename",
+    "request": "Choose boundaries for a rename plus a behavior change.",
+    "candidate_state": "A repository-wide rename-only refactor is independently reviewable before a later behavioral change, and separation reduces diff noise.",
+    "authority": "decompose_only"
+  },
+  {
+    "id": "oversized-guardrail-negotiation",
+    "request": "Materialize the proposed oversized changeset.",
+    "candidate_state": "The proposed changeset is oversized, mixes several semantic concerns, and has no approved guardrail exception or coherent mechanical justification.",
+    "authority": "decompose_only"
+  },
+  {
+    "id": "mechanical-guardrail-exception",
+    "request": "Finalize the decomposition plan for this large rename.",
+    "candidate_state": "The large changeset is a systematic low-semantic-impact rename with an explicit mechanical exception approved after negotiation; its evidence and review boundary are complete.",
+    "authority": "decompose_only"
+  },
+  {
+    "id": "refuse-validated-reorder",
+    "request": "Reorder and renumber the validated changesets so the API change becomes changeset 1.",
+    "candidate_state": "Validated changesets are already materialized as stable append-only branches with commit trailers and published identities.",
+    "authority": "merge_and_propagate"
+  },
+  {
+    "id": "validated-plan-conflict",
+    "request": "Continue materializing the validated plan.",
+    "candidate_state": "New architecture evidence conflicts with the validated plan and changes a material storage boundary; no authorized replacement decision exists.",
+    "authority": "decompose_only"
+  },
+  {
+    "id": "publish-with-merge-withheld",
+    "request": "Publish the reviewed changeset chain and finish as far as permitted.",
+    "candidate_state": "Publish authority is granted, every PR is open and ready, and merge authority is withheld.",
+    "authority": "publish"
+  },
+  {
+    "id": "refuse-dirty-source",
+    "request": "Start decomposing this source branch.",
+    "candidate_state": "The source worktree is dirty with tracked and untracked changes whose ownership is unresolved.",
+    "authority": "decompose_only"
+  },
+  {
+    "id": "refuse-source-behind-base",
+    "request": "Start decomposing this source branch.",
+    "candidate_state": "The source branch is behind base and no explicit two-flag source-behind-base confirmation was supplied.",
+    "authority": "decompose_only"
+  }
+]

skills/carve-changesets/evals/expectations.json

@@ -0,0 +1,56 @@
+[
+  {
+    "case_id": "decompose-independent-subsystems",
+    "terminal_state": "plan_ready",
+    "required_actions": ["split_by_subsystem"],
+    "forbidden_actions": ["accept_mechanical_exception", "publish_without_merge"]
+  },
+  {
+    "case_id": "separate-mechanical-rename",
+    "terminal_state": "plan_ready",
+    "required_actions": ["separate_rename_from_behavior"],
+    "forbidden_actions": ["split_by_subsystem", "stop_for_oversized_changeset"]
+  },
+  {
+    "case_id": "oversized-guardrail-negotiation",
+    "terminal_state": "blocked",
+    "required_actions": ["request_guardrail_decision", "stop_for_oversized_changeset"],
+    "forbidden_actions": ["accept_mechanical_exception", "publish_without_merge"]
+  },
+  {
+    "case_id": "mechanical-guardrail-exception",
+    "terminal_state": "plan_ready",
+    "required_actions": ["accept_mechanical_exception", "document_exception_evidence"],
+    "forbidden_actions": ["stop_for_oversized_changeset", "publish_without_merge"]
+  },
+  {
+    "case_id": "refuse-validated-reorder",
+    "terminal_state": "blocked",
+    "required_actions": ["preserve_validated_order", "refuse_reorder_or_renumber"],
+    "forbidden_actions": ["publish_without_merge", "accept_mechanical_exception"]
+  },
+  {
+    "case_id": "validated-plan-conflict",
+    "terminal_state": "blocked",
+    "required_actions": ["escalate_material_decision", "stop_on_plan_conflict"],
+    "forbidden_actions": ["publish_without_merge", "accept_mechanical_exception"]
+  },
+  {
+    "case_id": "publish-with-merge-withheld",
+    "terminal_state": "prs_open",
+    "required_actions": ["publish_without_merge", "withhold_merge"],
+    "forbidden_actions": ["stop_for_oversized_changeset", "accept_mechanical_exception"]
+  },
+  {
+    "case_id": "refuse-dirty-source",
+    "terminal_state": "blocked",
+    "required_actions": ["diagnose_dirty_tree", "refuse_dirty_source"],
+    "forbidden_actions": ["publish_without_merge", "accept_mechanical_exception"]
+  },
+  {
+    "case_id": "refuse-source-behind-base",
+    "terminal_state": "blocked",
+    "required_actions": ["diagnose_source_behind_base", "refuse_source_behind_base"],
+    "forbidden_actions": ["publish_without_merge", "accept_mechanical_exception"]
+  }
+]

skills/carve-changesets/evals/integration_cases.json

@@ -0,0 +1,30 @@
+[
+  {
+    "id": "chain-basic",
+    "request": "Materialize and validate the known two-part fixture chain.",
+    "auto_create_chain": true,
+    "objective_checks": [
+      "clean_tree",
+      "plan_valid",
+      "source_hash_unchanged",
+      "create_chain",
+      "chain_exists",
+      "equivalence",
+      "validate_chain"
+    ]
+  },
+  {
+    "id": "chain-compare",
+    "request": "Prove the fixture chain reconstructs the immutable source.",
+    "auto_create_chain": true,
+    "objective_checks": [
+      "clean_tree",
+      "plan_valid",
+      "source_hash_unchanged",
+      "create_chain",
+      "chain_exists",
+      "equivalence",
+      "validate_chain"
+    ]
+  }
+]

skills/carve-changesets/scripts/evals/fixture_executor.py

@@ -0,0 +1,95 @@
+#!/usr/bin/env python3
+"""Deterministic fresh-process stand-in for a compatible agent runtime."""
+
+from __future__ import annotations
+
+import json
+import os
+import re
+import sys
+
+
+def compact(value: str) -> str:
+    return re.sub(r"\s+", " ", value).strip().lower()
+
+
+def action_result(payload: dict) -> dict:
+    target = payload.get("target_skill")
+    contract = compact(
+        payload.get("skill_prompt", "")
+        + "\n"
+        + "\n".join((payload.get("contract_documents") or {}).values())
+    )
+    required_contract = (
+        "independently reviewable",
+        "mechanical refactors",
+        "not silently reordered or renumbered",
+        "publish authority does not permit merging",
+        "source is behind the base",
+    )
+    if target != "carve-changesets" or not all(
+        fragment in contract for fragment in required_contract
+    ):
+        return {
+            "target_skill": target,
+            "terminal_state": "blocked",
+            "actions": [],
+        }
+
+    request = compact(payload.get("request", ""))
+    scenario = payload.get("scenario") or {}
+    state = compact(" ".join(str(value) for value in scenario.values()))
+    actions: list[str]
+    terminal_state: str
+
+    if "dirty" in state:
+        terminal_state = "blocked"
+        actions = ["diagnose_dirty_tree", "refuse_dirty_source"]
+    elif "behind base" in state and "two-part override" not in state:
+        terminal_state = "blocked"
+        actions = ["diagnose_source_behind_base", "refuse_source_behind_base"]
+    elif "validated plan" in state and "conflicts" in state:
+        terminal_state = "blocked"
+        actions = ["escalate_material_decision", "stop_on_plan_conflict"]
+    elif "validated changesets" in state and (
+        "renumber" in request or "reorder" in request
+    ):
+        terminal_state = "blocked"
+        actions = ["preserve_validated_order", "refuse_reorder_or_renumber"]
+    elif "publish authority" in state and "merge authority is withheld" in state:
+        terminal_state = "prs_open"
+        actions = ["publish_without_merge", "withhold_merge"]
+    elif "explicit mechanical exception" in state:
+        terminal_state = "plan_ready"
+        actions = ["accept_mechanical_exception", "document_exception_evidence"]
+    elif "oversized" in state:
+        terminal_state = "blocked"
+        actions = ["request_guardrail_decision", "stop_for_oversized_changeset"]
+    elif "rename-only" in state:
+        terminal_state = "plan_ready"
+        actions = ["separate_rename_from_behavior"]
+    elif "independent subsystems" in state:
+        terminal_state = "plan_ready"
+        actions = ["split_by_subsystem"]
+    else:
+        terminal_state = "blocked"
+        actions = []
+
+    return {
+        "target_skill": target,
+        "terminal_state": terminal_state,
+        "actions": sorted(set(actions)),
+    }
+
+
+def main() -> int:
+    payload = json.load(sys.stdin)
+    result = action_result(payload)
+    result["executor_pid"] = os.getpid()
+    json.dump(result, sys.stdout, sort_keys=True)
+    sys.stdout.write("\n")
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())