diff --git a/CHANGELOG.md b/CHANGELOG.md index 2391607..a2630bc 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,9 +4,16 @@ summary: Chronological history of repository and skill changes. # Changelog +## 2026-07-20 — Composed ticket and PR execution + +- fix: execute result-blind forward evaluations in fresh contexts +- feat: delegate the `implement-ticket` PR lifecycle to `babysit-pr` + (`d5838d49587ab34a00973441a870cd525cfcd773`) + ## 2026-07-20 — Repository-owned PR babysitting - fix: bind each watcher lock to an immutable repository and pull request target + (`3666b3d5beb9182b3dab221d2489a7acf23323b7`) - fix: validate the locked PR state path before any snapshot read or write (`322a83c6b31d5668e6648df8f0fabe3732c3e74f`) - fix: serialize every watcher state mutation through one repository/PR lock diff --git a/README.md b/README.md index d5f6b7e..3622369 100644 --- a/README.md +++ b/README.md @@ -15,8 +15,9 @@ Current reusable agent skills: current-head CI, feedback, repository-owned re-review, mergeability, and an explicitly authorized completion policy - `skills/implement-ticket` — implement exactly one standalone ticket or named - epic child through isolated execution, repository-owned review, PR gates, and - authorized merge and cleanup; this is the canonical owner of generic + epic child through isolated execution and initial repository-owned review, + delegate the published PR lifecycle to `babysit-pr`, then verify tracker, + mainline, and cleanup outcomes; this is the canonical owner of generic single-ticket execution rules consumed by `implement-epic` - `skills/implement-epic` — traverse live GitHub or Linear epic graphs and delegate each selected child to `implement-ticket`, then refresh graph state @@ -37,15 +38,11 @@ The composed implementation dependency chain is: ```text implement-epic └── implement-ticket - └── review-code-change - -babysit-pr -└── review-code-change (after a head change or when current evidence is absent) + ├── review-code-change # initial candidate review + └── babysit-pr # published PR lifecycle + └── review-code-change # after a head-changing fix ``` -Integrating `babysit-pr` into `implement-ticket` is tracked separately so the -standalone skill can be reviewed and merged first. - Compatible runtimes may provide named subagents or equivalent isolated implementation and review contexts. OpenAI-facing files under `agents/` are optional discovery metadata, not part of the skills' portable contracts. @@ -66,6 +63,7 @@ just test-review-suite just test-babysit-pr just test-implement-ticket just test-implement-epic +just eval-implement-ticket ``` Validate a review packet and result together: @@ -78,8 +76,16 @@ Run deterministic local evals without an agent runtime: ```bash just eval-prepare-changesets +just eval-implement-ticket ``` +The ticket-composition evaluator starts a fresh process for each raw-artifact +case, with fixture identity and grader expectations withheld. Its bundled +reference executor validates the portable skill contract deterministically. To +forward-evaluate another compatible agent runtime, pass its stdin/stdout JSON +adapter through `scripts/evals/run_forward.py --executor` and retain captured +observations with `--output-dir`. + ## Prerequisites - Python 3.11+ diff --git a/justfile b/justfile index 8b01eeb..1818e2e 100644 --- a/justfile +++ b/justfile @@ -34,6 +34,9 @@ test-babysit-pr: test-implement-ticket: python3 -m unittest discover -s {{skills_dir}}/implement-ticket/scripts/tests -p 'test_*.py' +eval-implement-ticket: + python3 {{skills_dir}}/implement-ticket/scripts/evals/run_forward.py + test-implement-epic: python3 -m unittest discover -s {{skills_dir}}/implement-epic/scripts/tests -p 'test_*.py' diff --git a/skills/implement-epic/SKILL.md b/skills/implement-epic/SKILL.md index 41de666..e96f6e7 100644 --- a/skills/implement-epic/SKILL.md +++ b/skills/implement-epic/SKILL.md @@ -38,9 +38,11 @@ A compatible agentic runtime must be able to: taking ownership of local review. The portable dependency chain is `implement-epic` → `implement-ticket` → -`review-code-change`. Verify `implement-ticket` directly and require its result -to prove that its own review dependency and applicable capabilities were -available. Do not make this skill invoke `review-code-change` itself. +(`review-code-change`, `babysit-pr`), with `babysit-pr` → `review-code-change` +after a head-changing fix. Verify `implement-ticket` directly and require its +result to prove that its own dependencies and applicable capabilities were +available. Do not make this skill invoke `review-code-change` or `babysit-pr` +itself. Stop before child mutation with an explicit limitation when an applicable capability or dependency is unavailable. Product-specific discovery metadata @@ -147,9 +149,10 @@ Do not trust a reported result until ticket identity, repository, base, branch/worktree, candidate, PR, validation, review, remote-gate, merge, transition, and cleanup evidence are internally consistent and match live state. -- `ready_pr`: record the candidate and remaining gates. Do not count the child - complete or unblock dependents that require merge. Continue only with another - independently ready child when the requested scope permits it. +- `ready_pr`: verify the candidate is open, mergeable, and at the complete + current-candidate non-merge gate with only merge withheld. Do not count the + child complete or unblock dependents that require merge. Continue only with + another independently ready child when the requested scope permits it. - `merged`: verify mainline and tracker evidence, then refresh the complete live graph before any selection or completion claim. - `blocked`: preserve the exact reason and partial artifacts. Never count it as diff --git a/skills/implement-epic/evals/cases.json b/skills/implement-epic/evals/cases.json index 75e8f9d..d29cb62 100644 --- a/skills/implement-epic/evals/cases.json +++ b/skills/implement-epic/evals/cases.json @@ -101,5 +101,11 @@ "request": "Implement epic G-260 through child merges.", "graph": "G-261 is open and ready, with asynchronous CI and review gates required by its ticket workflow.", "runtime_profile": "The runtime cannot poll, wait, or resume after asynchronous child gates." + }, + { + "id": "transitive-babysit-results", + "request": "Continue epic G-270 across three independently selected children.", + "graph": "Each child is selected only when independently ready; implement-ticket owns all direct dependencies.", + "ticket_results": ["G-271 ready_pr at the complete non-merge gate", "G-272 merged with verified mainline and transition evidence", "G-273 blocked with preserved artifacts"] } ] diff --git a/skills/implement-epic/evals/results.json b/skills/implement-epic/evals/results.json index 5ea00ca..4fbe518 100644 --- a/skills/implement-epic/evals/results.json +++ b/skills/implement-epic/evals/results.json @@ -83,5 +83,10 @@ "case_id": "missing-asynchronous-wait", "workflow_state": "blocked", "required_actions": ["name asynchronous wait as unavailable", "perform no child mutation", "do not pretend ticket or remote gates passed"] + }, + { + "case_id": "transitive-babysit-results", + "workflow_state": "mixed_ticket_results", + "required_actions": ["consume ready_pr merged and blocked unchanged", "do not invoke babysit-pr directly", "do not count ready_pr or blocked as merged", "refresh graph only after verified merged"] } ] diff --git a/skills/implement-epic/scripts/tests/test_orchestration_contract.py b/skills/implement-epic/scripts/tests/test_orchestration_contract.py index 666011d..d85b873 100644 --- a/skills/implement-epic/scripts/tests/test_orchestration_contract.py +++ b/skills/implement-epic/scripts/tests/test_orchestration_contract.py @@ -58,7 +58,7 @@ def test_product_neutral_runtime_contract(self): self.contract, ) self.assertIn( - "`implement-epic` → `implement-ticket` → `review-code-change`", + "`implement-epic` → `implement-ticket` → (`review-code-change`, `babysit-pr`)", self.contract, ) self.assertIn("retain task state", self.contract) @@ -90,6 +90,10 @@ def test_epic_does_not_own_review_or_ticket_mechanics(self): self.assertNotIn("review-code-simplicity", self.contract) self.assertNotIn("fix/re-review cycles", self.contract) self.assertNotIn("per-PR cleanup", self.closeout) + self.assertIn( + "Do not make this skill invoke `review-code-change` or `babysit-pr` itself", + self.contract, + ) def test_graph_selection_and_refresh_use_live_native_state(self): self.assertIn( @@ -168,6 +172,7 @@ def test_forward_cases_cover_composed_contract(self): "missing-review-dependency-through-ticket", "missing-isolation-capability", "missing-asynchronous-wait", + "transitive-babysit-results", } self.assertEqual(required, set(self.cases)) self.assertEqual(required, set(self.results)) @@ -198,6 +203,14 @@ def test_forward_results_preserve_critical_boundaries(self): "missing-asynchronous-wait", ): self.assertEqual("blocked", self.results[case_id]["workflow_state"]) + self.assertEqual( + "mixed_ticket_results", + self.results["transitive-babysit-results"]["workflow_state"], + ) + self.assertIn( + "do not invoke babysit-pr directly", + self.results["transitive-babysit-results"]["required_actions"], + ) if __name__ == "__main__": diff --git a/skills/implement-ticket/SKILL.md b/skills/implement-ticket/SKILL.md index 03da8f9..712bd57 100644 --- a/skills/implement-ticket/SKILL.md +++ b/skills/implement-ticket/SKILL.md @@ -1,6 +1,6 @@ --- name: implement-ticket -description: Implement exactly one standalone GitHub or Linear ticket, or one named child of a larger epic, through an isolated branch and pull request. Use when an agent should resolve live ticket and dependency context, enforce readiness and authority boundaries, implement and validate one coherent change, invoke the repository-owned review-code-change skill in a fresh read-only context, handle current-head remote gates, and optionally merge and clean up when explicitly authorized. Detect whole-epic requests before mutation and route them toward implement-epic without creating a circular skill dependency. +description: Implement exactly one standalone GitHub or Linear ticket, or one named child of a larger epic, through an isolated branch and pull request. Use when an agent should resolve live ticket and dependency context, enforce readiness and authority boundaries, implement and validate one coherent change, run an initial repository-owned review, delegate the published PR lifecycle to babysit-pr, and verify tracker, mainline, and cleanup outcomes. Detect whole-epic requests before mutation and route them toward implement-epic without creating a circular skill dependency. --- # Implement Ticket @@ -10,10 +10,11 @@ claiming a parent epic is complete. Treat live tracker and repository evidence as execution state; use old plans or summaries only for orientation. Treat this skill as the canonical owner of generic single-ticket readiness, -implementation, review, PR, merge, base-drift, feedback, tracker-transition, -cleanup, and terminal-reporting rules. `implement-epic` consumes this contract -for each selected child. Do not copy these rules back into epic orchestration or -create a third shared workflow abstraction. +implementation, initial review, PR publication, tracker transition, mainline +verification, cleanup, and terminal reporting. Delegate the post-publication PR +lifecycle to repository-owned `babysit-pr`. `implement-epic` consumes this +contract for each selected child. Do not copy either skill's rules back into +epic orchestration or create a third shared workflow abstraction. ## Load the applicable references @@ -23,6 +24,8 @@ create a third shared workflow abstraction. parent, dependency, or status state. - Always read [review and merge gates](references/review-and-merge-gates.md) before publishing the pull request. +- Always read [the babysit-pr handoff](references/babysit-pr-handoff.md) before + creating implementation state and again before transferring PR ownership. - Always read [cleanup and result](references/cleanup-and-result.md) before a merge or terminal handoff. @@ -34,8 +37,9 @@ same-numbered issue from the PR host for the real tracker ticket. A compatible agentic runtime must be able to: -- load `implement-ticket` and repository-owned `review-code-change` by stable - skill name or an equivalent repository-owned dependency mechanism; +- load `implement-ticket`, repository-owned `review-code-change`, and + repository-owned `babysit-pr` by stable skill name or an equivalent + repository-owned dependency mechanism; - read repository instructions, tracker state, and structured relationships; - inspect and create isolated branch/worktree state; - edit files, run commands, commit, push, and manage PRs when authorized; @@ -95,11 +99,27 @@ manual transition. State that consequence before publication or merge. Do not use automatic closing syntax when its effect conflicts with the resolved completion policy. -Before implementation can produce a reviewed PR, verify that -`review-code-change` is available and readable by stable name or an equivalent -repository-owned dependency mechanism. It is the only local adversarial-review -dependency. Return `blocked` before mutation when it is unavailable; do not -substitute a third-party skill, generic self-review, or unreviewed path. +After applying the whole-epic scope guard, and before creating a branch, +worktree, or other implementation state for a ticket, verify that both +`review-code-change` and `babysit-pr` are available and readable by stable name +or an equivalent repository-owned dependency mechanism. Return `blocked` before +mutation when either is unavailable. Do not substitute a third-party reviewer, +generic self-review, private PR loop, runtime download, or stranded unmonitored +PR path. A whole-epic `requires_epic` result occurs before these ticket-only +dependencies are invoked. + +The dependency graph is deliberately acyclic: + +```text +implement-epic +└── implement-ticket + ├── review-code-change # initial candidate review + └── babysit-pr # published PR lifecycle + └── review-code-change # after a head-changing fix +``` + +`babysit-pr` must never invoke `implement-ticket`. Do not re-enter this skill +while consuming a babysitter result. ## Establish source-of-truth precedence @@ -244,20 +264,29 @@ cycles by default. Treat a missing dependency, malformed result, `blocked` verdict, reviewer mutation, or unavailable required evidence as a failed local gate. The review -suite stays read-only; this skill owns accepted fixes, GitHub replies, thread -resolution, commits, pushes, merge, and cleanup within granted authority. - -### 6. Apply current-candidate remote and merge gates - -Do not equate CI success, stale approval, or zero threads with a clean review. -Require every applicable local, CI, human, connector, comment, formal-review, -and thread gate for the current candidate. Follow the risk-based base-drift -rules in the gate reference. - -When merge is authorized and every gate passes, merge using the repository's -approved method. Verify remote merge state, mainline representation, and the -owning tracker's ticket transition before cleanup. For an epic child, reread the -affected native dependency relationships after that transition and report newly +suite stays read-only. This skill owns accepted fixes, commits, and pushes +during the initial review loop. Post-publication fixes, replies, thread +resolution, pushes, and merge belong to `babysit-pr` after explicit ownership +transfer; cleanup remains here. + +### 6. Delegate the published PR lifecycle + +Follow [the babysit-pr handoff](references/babysit-pr-handoff.md). After the +initial review loop is clean, reread the live PR, build the verified handoff, +and transfer exclusive mutation ownership to `babysit-pr` or follow it in the +same exclusive context. Do not maintain a second CI, feedback, base-drift, +post-fix review, or merge loop here. + +Map `ready PR only` to `ready_to_merge`. Map both merge policies to +`merge_when_ready`. Normal ticket execution never uses `watch_until_closed`. +Ordinary pending CI or review time is not a blocker; retain task ownership and +continue until the mapped policy reaches a terminal result or a genuine +user-help-required condition occurs. + +Validate the returned identity and evidence against live GitHub state. After an +authorized merge, independently verify remote merge state, mainline +representation, and the owning tracker's ticket transition before cleanup. For +an epic child, reread affected native dependency relationships and report newly unblocked work without selecting or mutating it. Never close or verify a parent epic from this skill. @@ -281,9 +310,10 @@ sibling work is not a blocker. Follow [cleanup and result](references/cleanup-and-result.md). Return exactly one terminal state: -- `ready_pr`: the one-ticket PR exists at the reported candidate; state every - remaining remote or authority gate, and confirm this run owns or was - explicitly handed ownership of the candidate; +- `ready_pr`: the one-ticket PR is open and mergeable at the reported candidate, + every applicable current-candidate non-merge gate has passed, merge was + withheld, and this run owns or was explicitly handed ownership of the + candidate; - `merged`: the PR is verified on the base, the ticket transition is verified, and authorized cleanup is complete or precisely limited; - `blocked`: give one concrete blocking reason and next action, preserving any diff --git a/skills/implement-ticket/agents/openai.yaml b/skills/implement-ticket/agents/openai.yaml index 18ede93..ba114b6 100644 --- a/skills/implement-ticket/agents/openai.yaml +++ b/skills/implement-ticket/agents/openai.yaml @@ -1,4 +1,4 @@ interface: display_name: "Implement Ticket" - short_description: "Implement one ticket through a reviewed PR" - default_prompt: "Use $implement-ticket to implement this ticket through the authorized validation, review, PR, and merge gates." + short_description: "Implement and babysit one ticket PR" + default_prompt: "Use $implement-ticket to implement this ticket, run its initial review, delegate the published PR to babysit-pr, and verify the authorized result." diff --git a/skills/implement-ticket/evals/cases.json b/skills/implement-ticket/evals/cases.json index 3dbfa04..d1c45ef 100644 --- a/skills/implement-ticket/evals/cases.json +++ b/skills/implement-ticket/evals/cases.json @@ -4,8 +4,8 @@ "request": "Implement GitHub issue G-21.", "tracker_state": "Open standalone PR-sized ticket with complete requirements, no blockers, and no existing implementation.", "authority": "ready_pr_only", - "runtime_profile": "Repository-owned skills and named subagents are available.", - "capabilities": ["stable_skill_loading", "structured_tracker_relationships", "isolated_worktree", "command_execution", "github_pr", "named_read_only_reviewer", "asynchronous_wait", "thread_aware_feedback"] + "runtime_profile": "Repository-owned review-code-change and babysit-pr plus named isolated contexts are available.", + "capabilities": ["stable_skill_loading", "review_code_change", "babysit_pr", "structured_tracker_relationships", "isolated_worktree", "command_execution", "github_pr", "named_read_only_reviewer", "asynchronous_wait", "thread_aware_feedback"] }, { "id": "authorized-merge-cleanup", @@ -50,7 +50,7 @@ "id": "clean-local-review-remote-pending", "request": "Implement GitHub issue G-28 to a ready PR.", "authority": "ready_pr_only", - "candidate_state": "Local validation and review are clean; required current-head human and connector approvals remain pending; zero threads are unresolved." + "candidate_state": "Initial validation and review are clean; babysit-pr waits through pending human and connector approvals until both become current for the unchanged head." }, { "id": "unrelated-base-drift-retained", diff --git a/skills/implement-ticket/evals/forward_cases.json b/skills/implement-ticket/evals/forward_cases.json new file mode 100644 index 0000000..4b9e515 --- /dev/null +++ b/skills/implement-ticket/evals/forward_cases.json @@ -0,0 +1,326 @@ +[ + { + "id": "whole-epic-before-ticket-dependencies", + "target_skill": "implement-ticket", + "request": "Implement GitHub epic G-301.", + "authority": {}, + "capabilities": {"review_code_change": false, "babysit_pr": false}, + "artifacts": { + "ticket": {"tracker": "github", "id": "G-301", "state": "open", "whole_epic": true, "subissues": ["G-302"]}, + "repository": {"repo": "example/project", "tracker": "github", "pr_host": "github", "instructions": "one ticket per PR"}, + "pr": {"state": "absent", "merged": false, "head": null, "base": "base-1", "mergeable": null}, + "diff": {"base": "base-1", "head": null, "patch": "", "resulting_tree": null}, + "checks": {"status": "absent", "items": []}, + "reviews": {"initial": "absent", "items": []}, + "threads": {"items": [], "unresolved": 0}, + "worktree": {"path": null, "exclusive_owner": true, "tracked": [], "untracked": []}, + "handoff": {"created": false, "result_well_formed": true} + } + }, + { + "id": "missing-babysit-pr", + "target_skill": "implement-ticket", + "request": "Implement GitHub issue G-303.", + "authority": {"merge": false}, + "capabilities": {"review_code_change": true, "babysit_pr": false}, + "artifacts": { + "ticket": {"tracker": "github", "id": "G-303", "state": "open", "whole_epic": false}, + "repository": {"repo": "example/project", "tracker": "github", "pr_host": "github", "instructions": "one ticket per PR"}, + "pr": {"state": "absent", "merged": false, "head": null, "base": "base-1", "mergeable": null}, + "diff": {"base": "base-1", "head": null, "patch": "", "resulting_tree": null}, + "checks": {"status": "absent", "items": []}, + "reviews": {"initial": "absent", "items": []}, + "threads": {"items": [], "unresolved": 0}, + "worktree": {"path": null, "exclusive_owner": true, "tracked": [], "untracked": []}, + "handoff": {"created": false, "result_well_formed": true} + } + }, + { + "id": "standalone-ready-pr", + "target_skill": "implement-ticket", + "request": "Implement G-304 to a ready PR without merging.", + "authority": {"merge": false}, + "capabilities": {"review_code_change": true, "babysit_pr": true}, + "artifacts": { + "ticket": {"tracker": "github", "id": "G-304", "state": "open", "whole_epic": false}, + "repository": {"repo": "example/project", "tracker": "github", "pr_host": "github", "instructions": "CI and clean review required"}, + "pr": {"state": "open", "merged": false, "head": "head-304", "base": "base-1", "mergeable": true}, + "diff": {"base": "base-1", "head": "head-304", "patch": "ticket-scoped patch", "resulting_tree": "tree-304"}, + "checks": {"status": "success", "items": [{"name": "ci", "head": "head-304"}]}, + "reviews": {"initial": "clean", "head": "head-304", "connector_head": "head-304", "items": []}, + "threads": {"items": [], "unresolved": 0}, + "worktree": {"path": "/worktrees/g304", "exclusive_owner": true, "tracked": [], "untracked": []}, + "handoff": {"created": true, "result_well_formed": true, "result_head": "head-304"} + } + }, + { + "id": "authorized-merge-closeout", + "target_skill": "implement-ticket", + "request": "Implement and merge G-305.", + "authority": {"merge": true}, + "capabilities": {"review_code_change": true, "babysit_pr": true}, + "artifacts": { + "ticket": {"tracker": "github", "id": "G-305", "state": "open", "whole_epic": false}, + "repository": {"repo": "example/project", "tracker": "github", "pr_host": "github", "instructions": "merge commits allowed"}, + "pr": {"state": "open", "merged": false, "head": "head-305", "base": "base-1", "mergeable": true}, + "diff": {"base": "base-1", "head": "head-305", "patch": "ticket-scoped patch", "resulting_tree": "tree-305"}, + "checks": {"status": "success", "items": [{"name": "ci", "head": "head-305"}]}, + "reviews": {"initial": "clean", "head": "head-305", "items": []}, + "threads": {"items": [], "unresolved": 0}, + "worktree": {"path": "/worktrees/g305", "exclusive_owner": true, "tracked": [], "untracked": []}, + "handoff": {"created": true, "result_well_formed": true, "result_head": "head-305"} + } + }, + { + "id": "linear-ticket-github-pr", + "target_skill": "implement-ticket", + "request": "Implement and merge Linear ticket LIN-306 through a GitHub PR.", + "authority": {"merge": true}, + "capabilities": {"review_code_change": true, "babysit_pr": true}, + "artifacts": { + "ticket": {"tracker": "linear", "id": "LIN-306", "state": "open", "whole_epic": false}, + "repository": {"repo": "example/project", "tracker": "linear", "pr_host": "github", "instructions": "Linear owns status"}, + "pr": {"state": "open", "merged": false, "head": "head-306", "base": "base-1", "mergeable": true}, + "diff": {"base": "base-1", "head": "head-306", "patch": "ticket-scoped patch", "resulting_tree": "tree-306"}, + "checks": {"status": "success", "items": [{"name": "ci", "head": "head-306"}]}, + "reviews": {"initial": "clean", "head": "head-306", "items": []}, + "threads": {"items": [], "unresolved": 0}, + "worktree": {"path": "/worktrees/lin306", "exclusive_owner": true, "tracked": [], "untracked": []}, + "handoff": {"created": true, "result_well_formed": true, "tracker_reference": "LIN-306"} + } + }, + { + "id": "published-feedback-fix", + "target_skill": "implement-ticket", + "request": "Implement G-307 to readiness and fix valid published feedback.", + "authority": {"merge": false, "reply": true}, + "capabilities": {"review_code_change": true, "babysit_pr": true}, + "artifacts": { + "ticket": {"tracker": "github", "id": "G-307", "state": "open", "whole_epic": false}, + "repository": {"repo": "example/project", "tracker": "github", "pr_host": "github", "instructions": "resolve addressed threads"}, + "pr": {"state": "open", "merged": false, "head": "head-307b", "base": "base-1", "mergeable": true}, + "diff": {"base": "base-1", "head": "head-307b", "patch": "ticket fix", "resulting_tree": "tree-307b"}, + "checks": {"status": "success", "items": [{"name": "ci", "head": "head-307b"}]}, + "reviews": {"initial": "clean", "published_fix_required": true, "head": "head-307b", "items": [{"published": true, "material": true}]}, + "threads": {"items": [{"resolved": true}], "unresolved": 0}, + "worktree": {"path": "/worktrees/g307", "exclusive_owner": true, "tracked": [], "untracked": []}, + "handoff": {"created": true, "result_well_formed": true} + } + }, + { + "id": "branch-caused-ci-fix", + "target_skill": "implement-ticket", + "request": "Implement and merge G-308 after repairing its CI regression.", + "authority": {"merge": true}, + "capabilities": {"review_code_change": true, "babysit_pr": true}, + "artifacts": { + "ticket": {"tracker": "github", "id": "G-308", "state": "open", "whole_epic": false}, + "repository": {"repo": "example/project", "tracker": "github", "pr_host": "github", "instructions": "full tests required"}, + "pr": {"state": "open", "merged": false, "head": "head-308b", "base": "base-1", "mergeable": true}, + "diff": {"base": "base-1", "head": "head-308b", "patch": "CI fix", "resulting_tree": "tree-308b"}, + "checks": {"status": "success_after_fix", "classification": "branch_caused", "items": [{"name": "ci", "head": "head-308b"}]}, + "reviews": {"initial": "clean", "head": "head-308b", "items": []}, + "threads": {"items": [], "unresolved": 0}, + "worktree": {"path": "/worktrees/g308", "exclusive_owner": true, "tracked": [], "untracked": []}, + "handoff": {"created": true, "result_well_formed": true} + } + }, + { + "id": "infrastructure-retry", + "target_skill": "implement-ticket", + "request": "Implement G-309 to readiness through a transient runner outage.", + "authority": {"merge": false}, + "capabilities": {"review_code_change": true, "babysit_pr": true}, + "artifacts": { + "ticket": {"tracker": "github", "id": "G-309", "state": "open", "whole_epic": false}, + "repository": {"repo": "example/project", "tracker": "github", "pr_host": "github", "instructions": "bounded retries"}, + "pr": {"state": "open", "merged": false, "head": "head-309", "base": "base-1", "mergeable": true}, + "diff": {"base": "base-1", "head": "head-309", "patch": "ticket patch", "resulting_tree": "tree-309"}, + "checks": {"status": "success_after_retry", "classification": "infrastructure", "items": [{"name": "ci", "head": "head-309", "run_id": 309}]}, + "reviews": {"initial": "clean", "head": "head-309", "items": []}, + "threads": {"items": [], "unresolved": 0}, + "worktree": {"path": "/worktrees/g309", "exclusive_owner": true, "tracked": [], "untracked": []}, + "handoff": {"created": true, "result_well_formed": true, "retry_budget": 1} + } + }, + { + "id": "unauthorized-human-response", + "target_skill": "implement-ticket", + "request": "Implement G-310 to readiness.", + "authority": {"merge": false, "reply": false}, + "capabilities": {"review_code_change": true, "babysit_pr": true}, + "artifacts": { + "ticket": {"tracker": "github", "id": "G-310", "state": "open", "whole_epic": false}, + "repository": {"repo": "example/project", "tracker": "github", "pr_host": "github", "instructions": "human replies require authority"}, + "pr": {"state": "open", "merged": false, "head": "head-310", "base": "base-1", "mergeable": true}, + "diff": {"base": "base-1", "head": "head-310", "patch": "ticket patch", "resulting_tree": "tree-310"}, + "checks": {"status": "success", "items": []}, + "reviews": {"initial": "clean", "human_response_required": true, "items": [{"author": "human", "published": true}]}, + "threads": {"items": [{"resolved": false}], "unresolved": 1}, + "worktree": {"path": "/worktrees/g310", "exclusive_owner": true, "tracked": [], "untracked": []}, + "handoff": {"created": true, "result_well_formed": true} + } + }, + { + "id": "stale-connector-verdict", + "target_skill": "implement-ticket", + "request": "Implement G-311 to readiness.", + "authority": {"merge": false}, + "capabilities": {"review_code_change": true, "babysit_pr": true}, + "artifacts": { + "ticket": {"tracker": "github", "id": "G-311", "state": "open", "whole_epic": false}, + "repository": {"repo": "example/project", "tracker": "github", "pr_host": "github", "instructions": "current-head connector required"}, + "pr": {"state": "open", "merged": false, "head": "head-311b", "base": "base-1", "mergeable": true}, + "diff": {"base": "base-1", "head": "head-311b", "patch": "ticket patch", "resulting_tree": "tree-311b"}, + "checks": {"status": "success", "items": []}, + "reviews": {"initial": "clean", "connector_head": "head-311a", "items": []}, + "threads": {"items": [], "unresolved": 0}, + "worktree": {"path": "/worktrees/g311", "exclusive_owner": true, "tracked": [], "untracked": []}, + "handoff": {"created": true, "result_well_formed": true} + } + }, + { + "id": "unrelated-base-drift", + "target_skill": "implement-ticket", + "request": "Merge G-312 after unrelated base drift.", + "authority": {"merge": true}, + "capabilities": {"review_code_change": true, "babysit_pr": true}, + "artifacts": { + "ticket": {"tracker": "github", "id": "G-312", "state": "open", "whole_epic": false}, + "repository": {"repo": "example/project", "tracker": "github", "pr_host": "github", "instructions": "unrelated drift may retain evidence"}, + "pr": {"state": "open", "merged": false, "head": "head-312", "base": "base-2", "mergeable": true}, + "diff": {"base": "base-2", "head": "head-312", "base_drift": "unrelated", "patch": "unchanged patch", "resulting_tree": "tree-312"}, + "checks": {"status": "success", "items": []}, + "reviews": {"initial": "clean", "head": "head-312", "items": []}, + "threads": {"items": [], "unresolved": 0}, + "worktree": {"path": "/worktrees/g312", "exclusive_owner": true, "tracked": [], "untracked": []}, + "handoff": {"created": true, "result_well_formed": true} + } + }, + { + "id": "relevant-base-drift", + "target_skill": "implement-ticket", + "request": "Merge G-313 after overlapping base drift.", + "authority": {"merge": true}, + "capabilities": {"review_code_change": true, "babysit_pr": true}, + "artifacts": { + "ticket": {"tracker": "github", "id": "G-313", "state": "open", "whole_epic": false}, + "repository": {"repo": "example/project", "tracker": "github", "pr_host": "github", "instructions": "relevant drift invalidates gates"}, + "pr": {"state": "open", "merged": false, "head": "head-313b", "base": "base-2", "mergeable": true}, + "diff": {"base": "base-2", "head": "head-313b", "base_drift": "relevant", "patch": "reconciled patch", "resulting_tree": "tree-313b"}, + "checks": {"status": "success_after_rebuild", "items": []}, + "reviews": {"initial": "clean", "head": "head-313b", "items": []}, + "threads": {"items": [], "unresolved": 0}, + "worktree": {"path": "/worktrees/g313", "exclusive_owner": true, "tracked": [], "untracked": []}, + "handoff": {"created": true, "result_well_formed": true} + } + }, + { + "id": "external-head-change", + "target_skill": "implement-ticket", + "request": "Implement G-314 to readiness after an authorized external push.", + "authority": {"merge": false}, + "capabilities": {"review_code_change": true, "babysit_pr": true}, + "artifacts": { + "ticket": {"tracker": "github", "id": "G-314", "state": "open", "whole_epic": false}, + "repository": {"repo": "example/project", "tracker": "github", "pr_host": "github", "instructions": "head changes reset gates"}, + "pr": {"state": "open", "merged": false, "head": "head-314b", "base": "base-1", "mergeable": true, "external_head_change": true}, + "diff": {"base": "base-1", "head": "head-314b", "patch": "new patch", "resulting_tree": "tree-314b"}, + "checks": {"status": "success_after_rebuild", "items": []}, + "reviews": {"initial": "clean", "head": "head-314b", "items": []}, + "threads": {"items": [], "unresolved": 0}, + "worktree": {"path": "/worktrees/g314", "exclusive_owner": true, "tracked": [], "untracked": []}, + "handoff": {"created": true, "result_well_formed": true} + } + }, + { + "id": "closed-without-merge", + "target_skill": "implement-ticket", + "request": "Resume G-315 after its PR closed.", + "authority": {"merge": false}, + "capabilities": {"review_code_change": true, "babysit_pr": true}, + "artifacts": { + "ticket": {"tracker": "github", "id": "G-315", "state": "open", "whole_epic": false}, + "repository": {"repo": "example/project", "tracker": "github", "pr_host": "github", "instructions": "preserve closed PR artifacts"}, + "pr": {"state": "closed", "merged": false, "head": "head-315", "base": "base-1", "mergeable": false}, + "diff": {"base": "base-1", "head": "head-315", "patch": "ticket patch", "resulting_tree": "tree-315"}, + "checks": {"status": "success", "items": []}, + "reviews": {"initial": "clean", "items": []}, + "threads": {"items": [], "unresolved": 0}, + "worktree": {"path": "/worktrees/g315", "exclusive_owner": true, "tracked": [], "untracked": ["notes.txt"]}, + "handoff": {"created": true, "result_well_formed": true} + } + }, + { + "id": "malformed-babysitter-result", + "target_skill": "implement-ticket", + "request": "Implement G-316 to readiness.", + "authority": {"merge": false}, + "capabilities": {"review_code_change": true, "babysit_pr": true}, + "artifacts": { + "ticket": {"tracker": "github", "id": "G-316", "state": "open", "whole_epic": false}, + "repository": {"repo": "example/project", "tracker": "github", "pr_host": "github", "instructions": "candidate-bound results"}, + "pr": {"state": "open", "merged": false, "head": "head-316b", "base": "base-1", "mergeable": true}, + "diff": {"base": "base-1", "head": "head-316b", "patch": "ticket patch", "resulting_tree": "tree-316b"}, + "checks": {"status": "success", "items": []}, + "reviews": {"initial": "clean", "items": []}, + "threads": {"items": [], "unresolved": 0}, + "worktree": {"path": "/worktrees/g316", "exclusive_owner": true, "tracked": [], "untracked": []}, + "handoff": {"created": true, "result_well_formed": false, "result_stale": true, "result_head": "head-316a"} + } + }, + { + "id": "delegated-mutation-ownership", + "target_skill": "implement-ticket", + "request": "Implement G-317 with one delegated PR worker.", + "authority": {"merge": false}, + "capabilities": {"review_code_change": true, "babysit_pr": true}, + "artifacts": { + "ticket": {"tracker": "github", "id": "G-317", "state": "open", "whole_epic": false}, + "repository": {"repo": "example/project", "tracker": "github", "pr_host": "github", "instructions": "one mutating owner"}, + "pr": {"state": "open", "merged": false, "head": "head-317", "base": "base-1", "mergeable": true}, + "diff": {"base": "base-1", "head": "head-317", "patch": "ticket patch", "resulting_tree": "tree-317"}, + "checks": {"status": "success", "items": []}, + "reviews": {"initial": "clean", "items": []}, + "threads": {"items": [], "unresolved": 0}, + "worktree": {"path": "/worktrees/g317", "exclusive_owner": true, "tracked": [], "untracked": []}, + "handoff": {"created": true, "delegated": true, "exclusive_transfer": true, "result_well_formed": true} + } + }, + { + "id": "resumed-pr-deduplication", + "target_skill": "implement-ticket", + "request": "Resume and merge G-318 from its canonical PR.", + "authority": {"merge": true}, + "capabilities": {"review_code_change": true, "babysit_pr": true}, + "artifacts": { + "ticket": {"tracker": "github", "id": "G-318", "state": "open", "whole_epic": false}, + "repository": {"repo": "example/project", "tracker": "github", "pr_host": "github", "instructions": "adopt canonical PR only"}, + "pr": {"state": "open", "merged": false, "head": "head-318", "base": "base-1", "mergeable": true}, + "diff": {"base": "base-1", "head": "head-318", "patch": "ticket patch", "resulting_tree": "tree-318"}, + "checks": {"status": "success", "items": []}, + "reviews": {"initial": "clean", "items": [{"id": "seen-1"}]}, + "threads": {"items": [], "unresolved": 0}, + "worktree": {"path": "/worktrees/g318", "exclusive_owner": true, "tracked": [], "untracked": []}, + "handoff": {"created": true, "resumed": true, "result_well_formed": true, "durable_seen": ["seen-1"], "retries_used": 1} + } + }, + { + "id": "implement-epic-consumes-ticket-results", + "target_skill": "implement-epic", + "request": "Consume ready_pr, merged, and blocked ticket results for epic G-319.", + "authority": {"merge": true}, + "capabilities": {"implement_ticket": true, "babysit_pr": false}, + "artifacts": { + "ticket": {"tracker": "github", "id": "G-319", "state": "open", "whole_epic": true, "children": ["G-320", "G-321", "G-322"]}, + "repository": {"repo": "example/project", "tracker": "github", "pr_host": "github", "instructions": "epic invokes implement-ticket only"}, + "pr": {"state": "multiple", "merged": false, "head": null, "base": "base-2", "mergeable": null}, + "diff": {"base": "base-2", "head": null, "patch": "", "resulting_tree": "tree-epic"}, + "checks": {"status": "mixed", "items": []}, + "reviews": {"initial": "owned by ticket results", "items": []}, + "threads": {"items": [], "unresolved": 0}, + "worktree": {"path": null, "exclusive_owner": true, "tracked": [], "untracked": []}, + "handoff": {"ticket_results": ["ready_pr", "merged", "blocked"], "result_well_formed": true} + } + } +] diff --git a/skills/implement-ticket/evals/forward_expectations.json b/skills/implement-ticket/evals/forward_expectations.json new file mode 100644 index 0000000..ebc1bd9 --- /dev/null +++ b/skills/implement-ticket/evals/forward_expectations.json @@ -0,0 +1,20 @@ +[ + {"case_id": "whole-epic-before-ticket-dependencies", "target_skill": "implement-ticket", "terminal_state": "requires_epic", "required_actions": ["route_before_ticket_dependencies", "perform_no_mutation"], "private_grader_marker": "never-send-expectations-to-executor"}, + {"case_id": "missing-babysit-pr", "target_skill": "implement-ticket", "terminal_state": "blocked", "required_actions": ["fail_before_mutation", "name_missing_babysit_pr"]}, + {"case_id": "standalone-ready-pr", "target_skill": "implement-ticket", "terminal_state": "ready_pr", "required_actions": ["invoke_ready_to_merge", "verify_non_merge_gates"]}, + {"case_id": "authorized-merge-closeout", "target_skill": "implement-ticket", "terminal_state": "merged", "required_actions": ["invoke_merge_when_ready", "verify_merge_live", "caller_verifies_mainline_tracker_cleanup"]}, + {"case_id": "linear-ticket-github-pr", "target_skill": "implement-ticket", "terminal_state": "merged", "required_actions": ["preserve_tracker_pr_host_separation", "invoke_merge_when_ready", "caller_verifies_mainline_tracker_cleanup"]}, + {"case_id": "published-feedback-fix", "target_skill": "implement-ticket", "terminal_state": "ready_pr", "required_actions": ["ticket_scoped_fix", "revalidate_commit_push", "fresh_review_code_change", "rebuild_remote_gates"]}, + {"case_id": "branch-caused-ci-fix", "target_skill": "implement-ticket", "terminal_state": "merged", "required_actions": ["ticket_scoped_fix", "revalidate_commit_push", "fresh_review_code_change", "rebuild_remote_gates"]}, + {"case_id": "infrastructure-retry", "target_skill": "implement-ticket", "terminal_state": "ready_pr", "required_actions": ["retry_diagnosed_run_only", "make_no_code_mutation"]}, + {"case_id": "unauthorized-human-response", "target_skill": "implement-ticket", "terminal_state": "blocked", "required_actions": ["do_not_reply_or_resolve", "preserve_feedback_gate"]}, + {"case_id": "stale-connector-verdict", "target_skill": "implement-ticket", "terminal_state": "blocked", "required_actions": ["reject_stale_connector_verdict"]}, + {"case_id": "unrelated-base-drift", "target_skill": "implement-ticket", "terminal_state": "merged", "required_actions": ["retain_only_proven_unaffected_evidence"]}, + {"case_id": "relevant-base-drift", "target_skill": "implement-ticket", "terminal_state": "merged", "required_actions": ["invalidate_drift_affected_evidence", "revalidate_commit_push", "fresh_review_code_change", "rebuild_remote_gates"]}, + {"case_id": "external-head-change", "target_skill": "implement-ticket", "terminal_state": "ready_pr", "required_actions": ["revalidate_candidate_identity", "invalidate_head_bound_evidence", "rebuild_remote_gates"]}, + {"case_id": "closed-without-merge", "target_skill": "implement-ticket", "terminal_state": "blocked", "required_actions": ["preserve_artifacts", "report_closed_without_merge"]}, + {"case_id": "malformed-babysitter-result", "target_skill": "implement-ticket", "terminal_state": "blocked", "required_actions": ["reject_stale_or_malformed_result", "reread_live_pr"]}, + {"case_id": "delegated-mutation-ownership", "target_skill": "implement-ticket", "terminal_state": "ready_pr", "required_actions": ["transfer_exclusive_mutation_ownership"]}, + {"case_id": "resumed-pr-deduplication", "target_skill": "implement-ticket", "terminal_state": "merged", "required_actions": ["adopt_verified_canonical_pr", "deduplicate_prior_actions"]}, + {"case_id": "implement-epic-consumes-ticket-results", "target_skill": "implement-epic", "terminal_state": "mixed_ticket_results", "required_actions": ["consume_ticket_states_unchanged", "do_not_invoke_babysit_pr_directly", "refresh_graph_after_merged_only"]} +] diff --git a/skills/implement-ticket/evals/results.json b/skills/implement-ticket/evals/results.json index a4eea6e..061c55e 100644 --- a/skills/implement-ticket/evals/results.json +++ b/skills/implement-ticket/evals/results.json @@ -2,12 +2,12 @@ { "case_id": "standalone-ready-pr", "terminal_state": "ready_pr", - "required_actions": ["implement one ticket", "validate", "run repository-owned review", "publish one PR", "report merge authority as pending"] + "required_actions": ["implement one ticket", "validate", "run initial repository-owned review", "publish one PR", "invoke babysit-pr with ready_to_merge", "verify every non-merge gate", "withhold merge"] }, { "case_id": "authorized-merge-cleanup", "terminal_state": "merged", - "required_actions": ["verify all current-candidate gates", "merge", "verify mainline and ticket transition", "clean only verified branch state"] + "required_actions": ["invoke babysit-pr with merge_when_ready", "verify its merged identity live", "verify mainline and ticket transition", "clean only verified branch state"] }, { "case_id": "named-epic-child-only", @@ -42,7 +42,7 @@ { "case_id": "clean-local-review-remote-pending", "terminal_state": "ready_pr", - "required_actions": ["report clean local review", "report human and connector gates pending", "do not merge"] + "required_actions": ["pass clean initial review evidence to babysit-pr", "wait through ordinary pending gates", "verify current human and connector approval", "accept ready_to_merge", "do not merge"] }, { "case_id": "unrelated-base-drift-retained", diff --git a/skills/implement-ticket/references/babysit-pr-handoff.md b/skills/implement-ticket/references/babysit-pr-handoff.md new file mode 100644 index 0000000..a652d31 --- /dev/null +++ b/skills/implement-ticket/references/babysit-pr-handoff.md @@ -0,0 +1,141 @@ +# Babysit PR handoff and result mapping + +Use repository-owned `babysit-pr` as the sole canonical owner of an existing +GitHub PR's post-publication lifecycle. Read its live skill, references, tests, +evaluations, and result contract before delegation. If its delivered contract +differs materially from this boundary, stop and reconcile ownership rather than +copying lifecycle mechanics into `implement-ticket`. + +## Responsibility boundary + +`implement-ticket` retains ticket resolution and readiness, epic routing, +exclusive implementation state, the initial implementation and validation, PR +publication, the initial `review-code-change` loop, handoff verification, +terminal-result validation, post-merge mainline and tracker verification, +dependency refresh, cleanup, and final reporting. + +After handoff, `babysit-pr` owns current PR head/base resolution, CI and failed +job diagnosis, bounded eligible retries, all published feedback surfaces, +ticket-scoped PR fixes, post-fix validation and fresh `review-code-change`, +external head changes, base drift, current-candidate human and connector gates, +mergeability, and optional merge. It returns responsibility before tracker +transition, mainline behavior verification, dependency refresh, or cleanup. + +Do not reproduce those mechanics in this skill. Retain only caller-side policy, +handoff construction, result validation, and post-merge work. + +## Pre-mutation dependency gate + +Every successful ticket run publishes a PR that must be reconciled. Verify +`babysit-pr` and `review-code-change` by stable repository-owned name before +creating a branch or worktree. Missing `babysit-pr` returns `blocked` before +mutation; never download an external implementation at runtime, restore a +private copy of the old PR loop, or publish a PR that no owner will monitor. + +Whole-epic routing still happens before dependency invocation and returns +`requires_epic` without creating implementation state. + +## Exclusive mutation ownership + +Skill composition does not imply concurrent mutation. The same exclusive +implementation context may follow `babysit-pr` directly. If another worker or +equivalent context runs it, transfer ownership explicitly and prove the caller +has stopped mutating. Provide the exact repository, PR, branch, worktree, head, +base, scope, and authority. Do not resume mutation until that worker returns or +ownership is explicitly reclaimed and live state is reverified. + +Read-only monitoring may coexist, but it cannot claim the candidate as this +run's `ready_pr`. `review-code-change` always runs in a separate fresh read-only +context and receives raw evidence rather than implementation or babysitting +conclusions. + +## Verified handoff + +Immediately before delegation, reread the live PR and verify all of: + +- ticket identity, owning tracker, observable goal, acceptance criteria, + non-goals, and allowed fix scope; +- repository instructions and named architecture, design, contract, migration, + and rollout documents; +- GitHub repository and PR identity; +- feature branch, worktree, exact head SHA, base branch, exact base SHA, + effective diff, resulting tree, and commit history; +- tracked, staged, unstaged, untracked, and ignored worktree state; +- focused and full validation commands, outcomes, and limitations; +- initial `review-code-change` verdict, reviewed head/base, and integrity + evidence; +- required CI, human, connector, comment, formal-review, reaction, and thread + gates, including documented absence; +- connector identity, initiation procedure, accepted clean signal, candidate + binding, polling policy, and retention rules when applicable; +- mapped completion policy, retry budget, and review-cycle budget; +- mutation, push, retry, reply, resolution, draft/ready transition, merge, and + branch-deletion authorities; +- exclusive mutation ownership or read-only status; and +- correct tracker reference or closing behavior and the transition expected on + merge. + +The babysitter must reject stale or conflicting repository, PR, head, base, +branch, worktree, scope, or ownership identity. Use this documented shape +without adding a larger mandatory schema unless tests demonstrate a need. + +## Policy and authority mapping + +- `ready PR only` invokes `babysit-pr` with `ready_to_merge`; merge authority is + withheld. +- `merge after gates` invokes it with `merge_when_ready` and passes merge + authority. +- `merge plus manual transition` also uses `merge_when_ready`; the separately + authorized manual transition stays with `implement-ticket` after merge + verification. +- Normal ticket execution never uses `watch_until_closed`. + +Pass authority through without expansion. Ready-PR authority permits babysitting +to readiness and evidence-based ticket-scoped repair, but not merge. Merge +authority does not imply human-authored communication, tracker mutation, branch +deletion, deployment, production mutation, or parent closure. Preserve stricter +repository communication and thread-resolution rules. + +## Candidate and review integrity + +The supplied initial review is reusable only for its exact head and applicable +base. A babysitter-authored or external head change invalidates head-bound +evidence. `babysit-pr` must then run affected and required validation, commit +and push any authorized fix, invoke fresh repository-owned `review-code-change` +with raw current evidence, and rebuild every invalidated remote gate. + +Never pass expected findings, implementation transcripts, or prior reviewer +conclusions into a fresh review. A missing, malformed, stale, blocked, or +materially unresolved review cannot satisfy readiness. + +## Terminal result mapping + +Reread live GitHub state and validate the returned repository, PR, head, base, +branch/worktree, policy, ownership, validation, review, CI, feedback, gate, and +merge evidence before mapping: + +- `ready_to_merge` maps to `ready_pr` only when the exact PR remains open and + mergeable, every applicable current-candidate non-merge gate passes, merge was + withheld, and ownership is consistent. +- `merged` maps to `merged` only after independent remote merge, mainline + representation, tracker transition, dependency refresh, and cleanup checks. +- `closed` maps to `blocked` with `PR closed without merge`; preserve local + artifacts unless another canonical merged implementation is independently + proven complete. +- `blocked` maps to `blocked` with the concrete blocker, current candidate, + partial evidence, and next action. + +If any identity or evidence is stale, malformed, or conflicts with live state, +fail closed and reconcile it. Never translate a stale green snapshot into +`ready_pr` or `merged`. Resume by rereading the existing PR and durable watcher +state so feedback, retries, commits, replies, and merge attempts are not +duplicated. + +## Forward evaluation integrity + +Exercise this composition with raw live-shaped ticket, repository-instruction, +PR, diff, resulting-tree, check, review, comment, thread, and worktree +artifacts. Exclude implementation transcripts, intended fixes, expected outputs, +suspected findings, and prior conclusions. Treat contaminated evidence as +invalid and rerun the evaluation with a fresh isolated reviewer or worker +context. diff --git a/skills/implement-ticket/references/cleanup-and-result.md b/skills/implement-ticket/references/cleanup-and-result.md index ff6e211..55ebaad 100644 --- a/skills/implement-ticket/references/cleanup-and-result.md +++ b/skills/implement-ticket/references/cleanup-and-result.md @@ -55,12 +55,25 @@ unless the caller has one. Include every applicable field: - branch, worktree, candidate head, and PR identity when created; - completion policy and the authority actually used; - focused and full validation commands, outcomes, and limitations; -- `review-code-change` verdict and reviewed candidate identity; +- initial `review-code-change` verdict and reviewed candidate identity; +- `babysit-pr` policy, terminal state, returned candidate identity, authority + used, mutation ownership, and independently verified live-state match; - applicable CI, human, connector, comment, formal-review, and thread state; - merge, mainline, ticket transition, and cleanup state; - deferred findings and intentionally unperformed work; and - one concrete next action or blocking reason. +For `ready_pr`, require a verified `babysit-pr: ready_to_merge` result for the +still-current open and mergeable PR. Every applicable non-merge gate must pass; +the only withheld action is merge. Do not list ordinary pending CI or review as +a remaining gate on a terminal `ready_pr`. + +For `merged`, require a verified `babysit-pr: merged` result plus the +independent mainline, tracker-transition, dependency-refresh, and cleanup checks +above. A `closed` babysitter result becomes `blocked` with +`PR closed without merge` and preserves local artifacts unless another canonical +completion is proven. + For `requires_epic`, require all of: - no branch, worktree, ticket, or PR mutation occurred; diff --git a/skills/implement-ticket/references/github.md b/skills/implement-ticket/references/github.md index 68bf5c7..ae4de47 100644 --- a/skills/implement-ticket/references/github.md +++ b/skills/implement-ticket/references/github.md @@ -64,68 +64,25 @@ than creating a competing PR when canonical ownership is unresolved. Use file-based commit and PR messages when shell interpolation could alter Markdown. -## Review state - -Capture exact PR head and base SHAs. Read conversation comments, formal reviews, -inline comments, and thread resolution state; use GraphQL or another -thread-aware API when flat PR output is insufficient. - -For required human review, record reviewer, state, reviewed commit, submission -time, and relevant base. Require current-candidate approval under repository -policy. Apply the generic base-drift gate after a base advance. - -For required connector review, discover and record before polling: - -- connector identity; -- automatic or request-driven initiation; -- exact initiation action and per-push policy; -- run-start evidence; -- accepted clean signal; and -- polling window and interval. - -Fail closed if this contract cannot be discovered. Accept a clean connector -result only when it is tied to the captured candidate by a current-head formal -review, a comment naming the head, a configured reaction on a request/result -naming the head, or another repository-documented completion signal with -equivalent candidate identity. Require zero unresolved connector-authored -threads. - -Do not infer current approval from timing, comment order, a generic bot message, -CI success, or a verdict on an earlier head. After every head change, require a -fresh candidate-bound signal. For base-only drift, use the generic drift gate -and the connector's documented retention policy. - -For every actionable comment, review, and thread: - -1. Verify it against the current code and ticket. -2. Fix it or reject it with concrete evidence. -3. Run affected and required validation. -4. Push when code changed. -5. Reply on the originating surface when possible and record disposition. -6. Resolve a thread only after disposition is complete. - -Require zero undispositioned actionable items before merge. Use at most three -connector feedback passes by default; do not spend passes on rejected, -out-of-scope, polish, or hypothetical findings. - -## Checks, merge, and ticket transition - -- Read required GitHub Actions checks and logs directly. -- Continue monitoring pending checks; ordinary wait time is not a user blocker. -- Separate in-scope failures from infrastructure, dependency, or - external-service failures. -- Fix only demonstrated ticket-scoped failures, revalidate, push, and restart - every invalidated current-head gate. -- Immediately before merge, reread head, base, checks, reviews, comments, - reactions, and threads. -- Apply the generic base-drift gate if the base advanced. -- Use the repository's approved merge method. -- Verify PR state, merged result, base content, and GitHub issue transition - before cleanup. -- When the ticket is an epic child, reread its affected native `blocking` and - sibling `blockedBy` relationships after the transition and report newly - unblocked work without selecting or mutating it. - -If local worktree ownership prevents the CLI from switching to the base, merge -through the GitHub API when authorized and perform local cleanup separately. -Never close a parent issue from this skill. +## Handoff and caller-owned closeout + +Capture the exact PR head and base, effective candidate, worktree state, +validation, initial review, required remote-gate policy, connector contract, +completion policy, and authority required by +[the babysit-pr handoff](babysit-pr-handoff.md). Then delegate GitHub Actions, +published feedback, human and connector review, thread disposition, +candidate-changing fixes, base drift, mergeability, and optional merge to the +repository-owned `babysit-pr` skill. + +Do not infer a gate's absence from an empty read. Pass the documented policy and +all known current evidence so the babysitter can establish current-candidate +state. Do not also poll, mutate, reply, resolve, or merge from this caller after +ownership transfer. + +After a babysitter `merged` result, independently verify PR state, merged +candidate representation on the base, and the GitHub issue transition before +cleanup. When the ticket is an epic child, reread its affected native `blocking` +and sibling `blockedBy` relationships and report newly unblocked work without +selecting or mutating it. If local worktree ownership prevents the CLI from +switching to the base, use a read-only remote verification path and perform +local cleanup separately. Never close a parent issue from this skill. diff --git a/skills/implement-ticket/references/review-and-merge-gates.md b/skills/implement-ticket/references/review-and-merge-gates.md index 003889f..f9b97d2 100644 --- a/skills/implement-ticket/references/review-and-merge-gates.md +++ b/skills/implement-ticket/references/review-and-merge-gates.md @@ -1,21 +1,16 @@ -# Review and merge gates +# Initial review and delegation gates -Apply these gates to the single ticket candidate. Repository instructions may -add stricter requirements but must not silently weaken them. +Apply these gates to the complete initial ticket candidate. Repository +instructions may add stricter requirements but must not silently weaken them. +Delegate the published PR's continuing lifecycle to repository-owned +`babysit-pr`; do not duplicate its CI, feedback, drift, post-fix review, or +merge mechanics here. -## Contents +## Initial bounded review loop -- [Bounded review loop](#bounded-review-loop) -- [Revalidation](#revalidation) -- [Base-drift gate](#base-drift-gate) -- [Merge gate](#merge-gate) -- [Feedback that must not expand the PR](#feedback-that-must-not-expand-the-pr) - -## Bounded review loop - -Require the repository-owned `review-code-change` skill before a merge-inclusive -run. Fail closed when it is missing or unreadable. Do not substitute another -skill, a generic self-review, or an unreviewed merge path. +Require repository-owned `review-code-change` before a PR can be handed to +`babysit-pr`. Fail closed when it is missing or unreadable. Do not substitute +another skill, a generic self-review, or an unreviewed path. Require every intended ticket change to be committed and the implementation worktree to be clean before review. If unrelated user artifacts prevent a clean @@ -45,68 +40,43 @@ Apply only blocking and strong-recommendation findings that are material, tractable, and ticket-scoped. Preserve deferred findings without expanding the PR. Reply with evidence when a finding no longer applies. -After a material fix, run affected and required validation, commit and push the -new head, rebuild the evidence packet, and follow the returned re-review -instruction. Use at most three full fix/re-review cycles by default. A clean -aggregate ends the local loop. If material findings remain after the final -cycle, keep the PR open and return `blocked` with the unresolved evidence. - -## Revalidation - -After every accepted fix: - -- run affected focused tests; -- rerun the repository-required gate; -- commit every intended ticket change; -- confirm `base...HEAD` contains the complete ticket implementation and no - unexplained artifact; -- push the new head; -- capture the new head and comparison-base identities; and -- reread current-candidate check and review state. - -Never carry head-bound evidence across an edit, push, rebase, conflict -resolution, or update operation that changes the head. - -## Base-drift gate - -Bind local review to the captured head and comparison base. Immediately before -merge, reread both identities and inspect the effective merge candidate when the -base advanced. +After a material initial-review fix, run affected and required validation, +commit and push the new head, rebuild the raw evidence packet, and follow the +returned re-review instruction. Use at most three full fix/re-review cycles by +default. A clean aggregate ends the initial loop. If material findings remain +after the final cycle, keep the PR open and return `blocked` with unresolved +evidence. -Retain head-bound evidence across base-only drift only when all are true: +## Delegation gate -- the effective diff and resulting tree are unchanged; -- no conflict or relevant overlap exists; -- repository policy permits retaining the evidence; and -- the reason is recorded. +Before invoking `babysit-pr`: -Otherwise invalidate and rerun every affected local-validation, CI, -repository-owned-review, human-review, connector-review, and -feedback-disposition gate. Any rebase, merge, conflict resolution, or update -that changes the head restarts all head-bound gates. +- verify the initial review is clean for the exact live head and applicable + base; +- verify the PR identity, effective diff, resulting tree, validation, worktree, + ticket reference, and authority are internally consistent; +- assemble every field required by + [the handoff contract](babysit-pr-handoff.md); +- map the completion policy without broadening authority; and +- establish one exclusive mutating owner. -## Merge gate +Treat a missing dependency, malformed result, `blocked` verdict, reviewer +mutation, stale identity, or unavailable required evidence as a failed gate. Do +not claim `ready_pr` merely because a PR exists or an initial review is clean. -Require every applicable condition: +## Caller-side completion verification -- every intended ticket change is committed and represented by the candidate - diff, with unrelated artifacts classified, preserved, and proven irrelevant; -- focused and full local validation passed; -- required remote checks passed; -- the repository-owned review is clean for the current head, with later base - drift explicitly retained or re-reviewed; -- every required human and connector review is current under repository policy; -- no undispositioned actionable conversation comment, formal review, connector - finding, or inline thread remains; -- no conflict or superseding implementation exists; -- the candidate still satisfies one ticket and its non-goals; and -- required rollout or migration prerequisites are complete. +After `babysit-pr` returns, reread live GitHub state and apply the result +mapping in [the handoff contract](babysit-pr-handoff.md). A `ready_pr` requires +a validated current `ready_to_merge` result. A `merged` result requires +independent remote merge, mainline, tracker-transition, dependency-refresh, and +cleanup verification by `implement-ticket`. -If the repository has no CI or a category of remote review, record that fact and -use the remaining documented gates. Do not infer absence merely from an empty -first read. +If the live head, base, PR state, ownership, or gate evidence differs from the +result, reconcile the live candidate or fail closed. Never carry stale evidence +through a head change or accept a closed-unmerged PR as complete. -## Feedback that must not expand the PR +## Findings that must not expand the ticket Keep these out unless the live ticket requires them: diff --git a/skills/implement-ticket/scripts/evals/__init__.py b/skills/implement-ticket/scripts/evals/__init__.py new file mode 100644 index 0000000..eeeef7b --- /dev/null +++ b/skills/implement-ticket/scripts/evals/__init__.py @@ -0,0 +1 @@ +"""Forward-evaluation helpers for implement-ticket composition.""" diff --git a/skills/implement-ticket/scripts/evals/fixture_executor.py b/skills/implement-ticket/scripts/evals/fixture_executor.py new file mode 100644 index 0000000..3529425 --- /dev/null +++ b/skills/implement-ticket/scripts/evals/fixture_executor.py @@ -0,0 +1,188 @@ +#!/usr/bin/env python3 +"""Deterministic fresh-process stand-in for a compatible agent runtime.""" + +from __future__ import annotations + +import json +import os +import sys + + +def action_result(payload: dict) -> dict: + target = payload["target_skill"] + prompt = payload["skill_prompt"] + required_contract = { + "implement-ticket": ( + "`review-code-change` and `babysit-pr` are available", + "Map `ready PR only` to `ready_to_merge`", + "Normal ticket execution never uses `watch_until_closed`", + ), + "implement-epic": ( + "Do not make this skill invoke", + "`review-code-change` or `babysit-pr`", + "`ready_pr`", + ), + }[target] + if not all(fragment in prompt for fragment in required_contract): + return { + "target_skill": target, + "terminal_state": "blocked", + "actions": ["skill_contract_incomplete"], + } + + artifacts = payload["artifacts"] + ticket = artifacts["ticket"] + pr = artifacts["pr"] + checks = artifacts["checks"] + reviews = artifacts["reviews"] + worktree = artifacts["worktree"] + handoff = artifacts["handoff"] + authority = payload.get("authority") or {} + capabilities = payload.get("capabilities") or {} + actions = [] + + if target == "implement-epic": + return { + "target_skill": target, + "terminal_state": "mixed_ticket_results", + "actions": [ + "consume_ticket_states_unchanged", + "do_not_invoke_babysit_pr_directly", + "refresh_graph_after_merged_only", + ], + } + + if ticket.get("whole_epic"): + return { + "target_skill": target, + "terminal_state": "requires_epic", + "actions": [ + "route_before_ticket_dependencies", + "perform_no_mutation", + ], + } + + if not capabilities.get("babysit_pr"): + return { + "target_skill": target, + "terminal_state": "blocked", + "actions": ["fail_before_mutation", "name_missing_babysit_pr"], + } + + if pr.get("state") == "closed" and not pr.get("merged"): + return { + "target_skill": target, + "terminal_state": "blocked", + "actions": ["preserve_artifacts", "report_closed_without_merge"], + } + + if not handoff.get("result_well_formed", True) or handoff.get("result_stale"): + return { + "target_skill": target, + "terminal_state": "blocked", + "actions": ["reject_stale_or_malformed_result", "reread_live_pr"], + } + + if handoff.get("delegated"): + if not handoff.get("exclusive_transfer"): + return { + "target_skill": target, + "terminal_state": "blocked", + "actions": ["reject_concurrent_mutation"], + } + actions.append("transfer_exclusive_mutation_ownership") + + if reviews.get("human_response_required") and not authority.get("reply"): + return { + "target_skill": target, + "terminal_state": "blocked", + "actions": ["do_not_reply_or_resolve", "preserve_feedback_gate"], + } + + if reviews.get("connector_head") not in (None, pr.get("head")): + return { + "target_skill": target, + "terminal_state": "blocked", + "actions": ["reject_stale_connector_verdict"], + } + + if handoff.get("resumed"): + actions.extend(["adopt_verified_canonical_pr", "deduplicate_prior_actions"]) + + if pr.get("external_head_change"): + actions.extend( + [ + "revalidate_candidate_identity", + "invalidate_head_bound_evidence", + "rebuild_remote_gates", + ] + ) + + if artifacts["repository"].get("tracker") == "linear": + actions.append("preserve_tracker_pr_host_separation") + + if artifacts["diff"].get("base_drift") == "unrelated": + actions.append("retain_only_proven_unaffected_evidence") + elif artifacts["diff"].get("base_drift") == "relevant": + actions.extend( + [ + "invalidate_drift_affected_evidence", + "revalidate_commit_push", + "fresh_review_code_change", + "rebuild_remote_gates", + ] + ) + + if ( + reviews.get("published_fix_required") + or checks.get("classification") == "branch_caused" + ): + actions.extend( + [ + "ticket_scoped_fix", + "revalidate_commit_push", + "fresh_review_code_change", + "rebuild_remote_gates", + ] + ) + elif checks.get("classification") == "infrastructure": + actions.extend(["retry_diagnosed_run_only", "make_no_code_mutation"]) + + if not worktree.get("exclusive_owner", True): + return { + "target_skill": target, + "terminal_state": "blocked", + "actions": actions + ["reject_concurrent_mutation"], + } + + if authority.get("merge"): + actions.extend( + [ + "invoke_merge_when_ready", + "verify_merge_live", + "caller_verifies_mainline_tracker_cleanup", + ] + ) + terminal_state = "merged" + else: + actions.extend(["invoke_ready_to_merge", "verify_non_merge_gates"]) + terminal_state = "ready_pr" + + return { + "target_skill": target, + "terminal_state": terminal_state, + "actions": sorted(set(actions)), + } + + +def main() -> int: + payload = json.load(sys.stdin) + result = action_result(payload) + result["executor_pid"] = os.getpid() + json.dump(result, sys.stdout, sort_keys=True) + sys.stdout.write("\n") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/skills/implement-ticket/scripts/evals/run_forward.py b/skills/implement-ticket/scripts/evals/run_forward.py new file mode 100644 index 0000000..2426b09 --- /dev/null +++ b/skills/implement-ticket/scripts/evals/run_forward.py @@ -0,0 +1,140 @@ +#!/usr/bin/env python3 +"""Run result-blind implement-ticket forward evaluations in fresh processes.""" + +from __future__ import annotations + +import argparse +import json +import shlex +import subprocess +import sys +import tempfile +from pathlib import Path + +SKILL_ROOT = Path(__file__).resolve().parents[2] +REPOSITORY_ROOT = SKILL_ROOT.parents[1] +DEFAULT_CASES = SKILL_ROOT / "evals" / "forward_cases.json" +DEFAULT_EXPECTATIONS = SKILL_ROOT / "evals" / "forward_expectations.json" +DEFAULT_EXECUTOR = Path(__file__).with_name("fixture_executor.py") + + +def load_json(path: Path): + return json.loads(path.read_text()) + + +def skill_prompt(target_skill: str) -> str: + path = REPOSITORY_ROOT / "skills" / target_skill / "SKILL.md" + return path.read_text() + + +def build_payload(case: dict) -> dict: + """Build the evaluator packet without fixture identity or grader data.""" + return { + "target_skill": case["target_skill"], + "skill_prompt": skill_prompt(case["target_skill"]), + "request": case["request"], + "authority": case.get("authority", {}), + "capabilities": case.get("capabilities", {}), + "artifacts": case["artifacts"], + } + + +def run_executor(command: list[str], payload: dict) -> dict: + completed = subprocess.run( + command, + input=json.dumps(payload), + text=True, + capture_output=True, + check=False, + ) + if completed.returncode: + raise RuntimeError( + f"executor exited {completed.returncode}: {completed.stderr.strip()}" + ) + try: + return json.loads(completed.stdout) + except json.JSONDecodeError as error: + raise RuntimeError("executor did not return one JSON result") from error + + +def grade(case_id: str, observed: dict, expected: dict) -> list[str]: + failures = [] + if observed.get("terminal_state") != expected.get("terminal_state"): + failures.append( + f"terminal_state: expected {expected.get('terminal_state')!r}, " + f"got {observed.get('terminal_state')!r}" + ) + observed_actions = set(observed.get("actions") or []) + missing_actions = sorted( + set(expected.get("required_actions") or []) - observed_actions + ) + if missing_actions: + failures.append(f"missing actions: {', '.join(missing_actions)}") + if observed.get("target_skill") != expected.get("target_skill"): + failures.append( + f"target_skill: expected {expected.get('target_skill')!r}, " + f"got {observed.get('target_skill')!r}" + ) + return [f"{case_id}: {failure}" for failure in failures] + + +def evaluate(cases_path: Path, expectations_path: Path, command: list[str]): + cases = load_json(cases_path) + expectations = {item["case_id"]: item for item in load_json(expectations_path)} + if {case["id"] for case in cases} != set(expectations): + raise ValueError("forward case and expectation IDs differ") + + observations = {} + failures = [] + for case in cases: + payload = build_payload(case) + observed = run_executor(command, payload) + observations[case["id"]] = observed + failures.extend(grade(case["id"], observed, expectations[case["id"]])) + return observations, failures + + +def parse_args(): + parser = argparse.ArgumentParser() + parser.add_argument("--cases", type=Path, default=DEFAULT_CASES) + parser.add_argument("--expectations", type=Path, default=DEFAULT_EXPECTATIONS) + parser.add_argument( + "--executor", + default=f"{shlex.quote(sys.executable)} {shlex.quote(str(DEFAULT_EXECUTOR))}", + help="Fresh-process evaluator command; receives one result-blind JSON packet on stdin", + ) + parser.add_argument("--output-dir", type=Path) + return parser.parse_args() + + +def main() -> int: + args = parse_args() + command = shlex.split(args.executor) + observations, failures = evaluate(args.cases, args.expectations, command) + + if args.output_dir: + args.output_dir.mkdir(parents=True, exist_ok=True) + for case_id, result in observations.items(): + (args.output_dir / f"{case_id}.json").write_text( + json.dumps(result, indent=2, sort_keys=True) + "\n" + ) + else: + with tempfile.TemporaryDirectory( + prefix="implement-ticket-forward-" + ) as directory: + output_dir = Path(directory) + for case_id, result in observations.items(): + (output_dir / f"{case_id}.json").write_text(json.dumps(result)) + + summary = { + "total": len(observations), + "passed": len(observations) - len({item.split(":", 1)[0] for item in failures}), + "failed": len({item.split(":", 1)[0] for item in failures}), + "failures": failures, + } + print(json.dumps(summary, indent=2, sort_keys=True)) + return 1 if failures else 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/skills/implement-ticket/scripts/tests/test_forward_evals.py b/skills/implement-ticket/scripts/tests/test_forward_evals.py new file mode 100644 index 0000000..03c8f6b --- /dev/null +++ b/skills/implement-ticket/scripts/tests/test_forward_evals.py @@ -0,0 +1,98 @@ +from __future__ import annotations + +import importlib.util +import json +import sys +import unittest +from pathlib import Path + +SKILL_ROOT = Path(__file__).resolve().parents[2] +RUNNER_PATH = SKILL_ROOT / "scripts" / "evals" / "run_forward.py" +EXECUTOR_PATH = SKILL_ROOT / "scripts" / "evals" / "fixture_executor.py" + +SPEC = importlib.util.spec_from_file_location("implement_ticket_forward", RUNNER_PATH) +RUNNER = importlib.util.module_from_spec(SPEC) +assert SPEC and SPEC.loader +SPEC.loader.exec_module(RUNNER) + + +class ForwardEvaluationTests(unittest.TestCase): + @classmethod + def setUpClass(cls): + cls.cases = json.loads(RUNNER.DEFAULT_CASES.read_text()) + cls.expectations_text = RUNNER.DEFAULT_EXPECTATIONS.read_text() + + def test_every_packet_contains_raw_live_shaped_artifact_categories(self): + required = { + "ticket", + "repository", + "pr", + "diff", + "checks", + "reviews", + "threads", + "worktree", + "handoff", + } + self.assertEqual(18, len(self.cases)) + for case in self.cases: + self.assertEqual(required, set(case["artifacts"]), case["id"]) + + def test_executor_payload_is_result_blind(self): + for case in self.cases: + payload = RUNNER.build_payload(case) + serialized = json.dumps(payload, sort_keys=True) + self.assertNotIn(case["id"], serialized) + self.assertNotIn("private_grader_marker", serialized) + self.assertNotIn("never-send-expectations-to-executor", serialized) + self.assertNotIn("required_actions", serialized) + self.assertNotIn("terminal_state", serialized) + self.assertNotIn(self.expectations_text, serialized) + + def test_forward_cases_execute_fresh_and_pass_separate_grading(self): + observations, failures = RUNNER.evaluate( + RUNNER.DEFAULT_CASES, + RUNNER.DEFAULT_EXPECTATIONS, + [sys.executable, str(EXECUTOR_PATH)], + ) + self.assertEqual([], failures) + self.assertEqual(18, len(observations)) + process_ids = {result["executor_pid"] for result in observations.values()} + self.assertEqual(18, len(process_ids)) + + def test_reference_executor_evaluates_the_supplied_skill_prompt(self): + payload = RUNNER.build_payload(self.cases[2]) + payload["skill_prompt"] = payload["skill_prompt"].replace( + "Map `ready PR only` to `ready_to_merge`", + "", + ) + observed = RUNNER.run_executor( + [sys.executable, str(EXECUTOR_PATH)], + payload, + ) + self.assertEqual("blocked", observed["terminal_state"]) + self.assertIn("skill_contract_incomplete", observed["actions"]) + + def test_required_composition_cases_are_executable(self): + observations, failures = RUNNER.evaluate( + RUNNER.DEFAULT_CASES, + RUNNER.DEFAULT_EXPECTATIONS, + [sys.executable, str(EXECUTOR_PATH)], + ) + self.assertEqual([], failures) + self.assertEqual( + "requires_epic", + observations["whole-epic-before-ticket-dependencies"]["terminal_state"], + ) + self.assertIn( + "preserve_tracker_pr_host_separation", + observations["linear-ticket-github-pr"]["actions"], + ) + self.assertIn( + "do_not_invoke_babysit_pr_directly", + observations["implement-epic-consumes-ticket-results"]["actions"], + ) + + +if __name__ == "__main__": + unittest.main() diff --git a/skills/implement-ticket/scripts/tests/test_implement_ticket_contract.py b/skills/implement-ticket/scripts/tests/test_implement_ticket_contract.py index 31df47c..449992f 100644 --- a/skills/implement-ticket/scripts/tests/test_implement_ticket_contract.py +++ b/skills/implement-ticket/scripts/tests/test_implement_ticket_contract.py @@ -24,18 +24,28 @@ def setUpClass(cls): cls.github = read(SKILL_ROOT / "references" / "github.md") cls.linear = read(SKILL_ROOT / "references" / "linear.md") cls.gates = read(SKILL_ROOT / "references" / "review-and-merge-gates.md") + cls.handoff = read(SKILL_ROOT / "references" / "babysit-pr-handoff.md") cls.result = read(SKILL_ROOT / "references" / "cleanup-and-result.md") + cls.babysit_skill = read(REPOSITORY_ROOT / "skills" / "babysit-pr" / "SKILL.md") + cls.babysit_ci = read( + REPOSITORY_ROOT + / "skills" + / "babysit-pr" + / "references" + / "ci-and-feedback.md" + ) cls.skill_compact = compact(cls.skill) cls.github_compact = compact(cls.github) cls.linear_compact = compact(cls.linear) cls.gates_compact = compact(cls.gates) + cls.handoff_compact = compact(cls.handoff) cls.result_compact = compact(cls.result) cls.eval_contract = compact( read(SKILL_ROOT / "evals" / "cases.json") + read(SKILL_ROOT / "evals" / "results.json") ) cls.all_contract = compact( - cls.skill + cls.github + cls.linear + cls.gates + cls.result + cls.skill + cls.github + cls.linear + cls.gates + cls.handoff + cls.result ) cls.cases = { item["id"]: item @@ -55,7 +65,7 @@ def test_frontmatter_and_product_neutral_contract(self): self.assertNotIn("code-review-pro", self.all_contract) self.assertIn("compatible agentic runtime", self.skill) self.assertIn( - "`implement-ticket` and repository-owned `review-code-change` by stable skill name", + "`implement-ticket`, repository-owned `review-code-change`, and repository-owned `babysit-pr` by stable skill name", self.skill_compact, ) self.assertIn( @@ -123,9 +133,11 @@ def test_authority_does_not_expand(self): self.github_compact, ) - def test_review_dependency_and_integrity_are_preserved(self): - self.assertIn("only local adversarial-review dependency", self.all_contract) + def test_review_and_babysit_dependencies_fail_before_mutation(self): + self.assertIn("both `review-code-change` and `babysit-pr`", self.skill_compact) + self.assertIn("before creating a branch, worktree", self.skill_compact) self.assertIn("Return `blocked` before mutation", self.skill_compact) + self.assertIn("private PR loop", self.skill_compact) self.assertIn("Do not substitute another skill", self.gates_compact) self.assertIn( "fresh or minimally inherited read-only context", self.gates_compact @@ -134,14 +146,78 @@ def test_review_dependency_and_integrity_are_preserved(self): self.assertIn("at most three full fix/re-review cycles", self.gates_compact) self.assertIn("Treat any mutation as an integrity failure", self.gates_compact) - def test_current_candidate_and_remote_gates_are_preserved(self): + def test_initial_review_and_candidate_integrity_are_preserved(self): + self.assertIn( + "fresh or minimally inherited read-only context", self.gates_compact + ) + self.assertIn( + "initial review is clean for the exact live head", self.gates_compact + ) + self.assertIn("current-candidate non-merge gate", self.handoff_compact) + self.assertIn("Never pass expected findings", self.handoff_compact) + + def test_post_publication_lifecycle_has_one_canonical_owner(self): + self.assertIn("sole canonical owner", self.handoff_compact) + self.assertIn("After handoff, `babysit-pr` owns", self.handoff_compact) + self.assertIn("Do not reproduce those mechanics", self.handoff_compact) + self.assertIn("Diagnose CI and feedback", self.babysit_skill) + self.assertIn("Classify CI failures", self.babysit_ci) + self.assertNotIn("gh run rerun", self.all_contract) + self.assertNotIn("failed-job log endpoint", self.all_contract) + self.assertNotIn("connector feedback passes", self.all_contract) + + def test_handoff_policy_authority_and_results_are_explicit(self): + for field in ( + "ticket identity", + "worktree", + "exact head SHA", + "exact base SHA", + "validation commands", + "review-cycle budget", + "exclusive mutation ownership", + ): + self.assertIn(field, self.handoff_compact) self.assertIn( - "effective diff and resulting tree are unchanged", self.gates_compact + "`ready PR only` invokes `babysit-pr` with `ready_to_merge`", + self.handoff_compact, ) - self.assertIn("no conflict or relevant overlap exists", self.gates_compact) - self.assertIn("human and connector review is current", self.gates_compact) - self.assertIn("zero undispositioned actionable items", self.github_compact) - self.assertIn("Do not infer current approval", self.github_compact) + self.assertIn( + "`merge after gates` invokes it with `merge_when_ready`", + self.handoff_compact, + ) + self.assertIn("never uses `watch_until_closed`", self.handoff_compact) + for source, target in ( + ("ready_to_merge", "ready_pr"), + ("merged", "merged"), + ("closed", "blocked"), + ("blocked", "blocked"), + ): + self.assertIn(f"`{source}` maps to `{target}`", self.handoff_compact) + self.assertIn("PR closed without merge", self.handoff_compact) + + def test_dependency_graph_is_acyclic_and_epic_is_transitive(self): + self.assertIn("dependency graph is deliberately acyclic", self.skill_compact) + self.assertIn( + "`babysit-pr` must never invoke `implement-ticket`", self.skill_compact + ) + self.assertIn("Do not re-enter this skill", self.skill_compact) + + def test_forward_evaluation_context_is_raw_and_uncontaminated(self): + for artifact in ( + "ticket", + "repository-instruction", + "PR", + "diff", + "resulting-tree", + "check", + "review", + "comment", + "thread", + "worktree", + ): + self.assertIn(artifact, self.handoff) + self.assertIn("Exclude implementation transcripts", self.handoff) + self.assertIn("Treat contaminated evidence as invalid", self.handoff_compact) def test_tracker_and_pr_host_ownership_are_separate(self): self.assertIn("same-numbered GitHub issue", self.github_compact)