fix: harden agent skills via iterative adversarial review

.github/workflows/ci.yml

@@ -55,13 +55,13 @@ jobs:
       - name: Run unit tests
         run: |
           found=0
-          for tests in skills/*/scripts/tests; do
+          for tests in skills/*/scripts/tests review-suite/scripts/tests; do
             if [ -d "$tests" ]; then
               found=1
               echo "Running tests in $tests"
               python -m unittest discover -s "$tests" -p 'test_*.py'
             fi
           done
           if [ "$found" -eq 0 ]; then
-            echo "No skill tests found under skills/*/scripts/tests"
+            echo "No tests found under skills/*/scripts/tests or review-suite/scripts/tests"
           fi

CHANGELOG.md

@@ -4,9 +4,65 @@ summary: Chronological history of repository and skill changes.
 
 # Changelog
 
+## 2026-07-20 — Portability, watcher resilience, and Claude adaptation
+
+- refactor: route the clear predicate through `has_failed_pr_checks` — the
+  code-simplicity lens on PR #27 flagged the last inline copy of the
+  failed-PR-check policy inside `is_github_candidate_clear`; all three agreement
+  sites now structurally share one predicate
+- fix: share one failed-PR-check predicate across the watcher — the initial
+  `review-code-change` pass on PR #27 found the retry gate refusing retries that
+  `recommend_actions` recommends for failed-runs-only states; extract
+  `has_failed_pr_checks` and use it in both sites, and match repository case
+  insensitively in state-target validation
+  (`625dae641a9652368f03b6be825f48d9addab056`)
+- fix: close the final low-severity review findings — mirror the clear predicate
+  in `has_failed_pr_checks` so a PR-check-backed failed run never reads as
+  `idle`, case-normalize repositories before deriving state files and locks,
+  match fragment run links, reject `--repo` without an explicit `--pr`, make
+  boolean schema constants reject numeric one, and document `--poll-seconds` and
+  `--max-flaky-retries` (`f79266a390e970cd25cf8af1bed6b9bd9cf154ee`)
+- fix: align retry gating and delegation tooling with review round four — accept
+  cancelled-only check failures in the retry gate so a recommended retry is
+  never refused, grant the review orchestrator the subagent and skill tools its
+  Claude adapter requires, reject `--once` with `--retry-failed-now`, match
+  query-string run links, add a repo digest to default state filenames, keep
+  `diagnose_ci_failure` visible after retry exhaustion, and document zero-check
+  `--stop-when-clear` pairing (`ddb29d0ce0409554cec61ed54b2c6e7ed6d84c6a`)
+- fix: close adversarial-review findings — resolve bundled-validator schemas in
+  both layouts and execute every bundled copy in place, scope failed workflow
+  runs to the PR's own checks so push/schedule failures cannot wedge the
+  watcher, emit `resolve_draft_state`/`resolve_merge_conflict` instead of
+  `idle`, complete `forbidden_actions` on all forward expectations with a
+  vocabulary-spam canary, stop backfilling `target_skill` in the Claude
+  executor, reject `--once --watch`, handle `OSError` cleanly, import bundled
+  validators in review-skill tests, and document eval flag pre-classification,
+  the gh 2.37 floor, and state-file durability
+  (`48b6f614d15d50dae4ba5c63d7b3e3471647dd1a`)
+- fix: close independent-review findings — count cancelled checks and failed
+  runs/jobs in the watcher's clear predicate, run review-suite tests in CI,
+  bundle the dependency-free packet validator into each review skill, make
+  `--stop-when-clear` imply `ready_to_merge` and test every documented CLI
+  invocation, fail closed on empty `gh pr checks` payloads, surface ghost-author
+  comments, move watcher state into a per-user 0700 directory, add
+  forbidden-action forward grading, unify `observed_sequence` tokens, and rename
+  `agents/claude.md` to `agents/claude-code.md` to avoid the case-insensitive
+  CLAUDE.md memory-file collision (`b5bf81b81a6dd521edcdfc561988ca621a566d39`)
+- fix: make skills self-contained and adapt the suite to Claude runtimes —
+  bundle the review-suite contract into each review skill with a
+  `just sync-contracts` target and drift test, use skill-root-relative watcher
+  paths, survive transient watcher failures with bounded backoff, add
+  `--max-polls`/`--stop-when-clear` bounded watch modes and a
+  `confirm_feedback_disposition` action, move eval answer keys out of
+  reviewer-visible input directories, add a Claude headless forward-eval
+  executor, add `agents/claude.md` adapters, `allowed-tools` on review skills,
+  and trigger-oriented skill descriptions, and trim contract tests to
+  load-bearing invariants (`474756bea51237376b81ad7d593eef2d8de273f1`)
+
 ## 2026-07-20 — Composed ticket and PR execution
 
 - fix: execute result-blind forward evaluations in fresh contexts
+  (`f452db4cf47e56b3f8fea560977a3ce98ca26caa`)
 - feat: delegate the `implement-ticket` PR lifecycle to `babysit-pr`
   (`d5838d49587ab34a00973441a870cd525cfcd773`)
 

CLAUDE.md

@@ -0,0 +1,5 @@
+# Claude Code Guidelines
+
+See @AGENTS.md for the repository working model, required checks, skill
+conventions, safety rules, and git workflow. Both files apply equally; keep
+repository guidance in `AGENTS.md` rather than duplicating it here.

README.md

@@ -44,8 +44,19 @@ implement-epic
 ```
 
 Compatible runtimes may provide named subagents or equivalent isolated
-implementation and review contexts. OpenAI-facing files under `agents/` are
-optional discovery metadata, not part of the skills' portable contracts.
+implementation and review contexts. Files under each skill's `agents/` directory
+(`openai.yaml` for OpenAI runtimes, `claude-code.md` for Claude Code) are
+optional discovery and adapter metadata, not part of the skills' portable
+contracts.
+
+Each review skill bundles a verbatim copy of the canonical `review-suite`
+contract and schemas under its own `references/review-suite/` directory so the
+skill remains self-contained when installed elsewhere. Edit only the canonical
+files and refresh the copies with:
+
+```bash
+just sync-contracts
+```
 
 ## Quick Start
 
@@ -79,18 +90,26 @@ just eval-prepare-changesets
 just eval-implement-ticket
 ```
 
-The ticket-composition evaluator starts a fresh process for each raw-artifact
-case, with fixture identity and grader expectations withheld. Its bundled
-reference executor validates the portable skill contract deterministically. To
-forward-evaluate another compatible agent runtime, pass its stdin/stdout JSON
+The ticket-composition evaluator starts a fresh process for each case, with
+fixture identity and grader expectations withheld. Case artifacts carry
+pre-classified scenario flags (for example, a CI failure already labeled
+branch-caused or infrastructure), so the harness grades obligation mapping and
+terminal-state selection — not evidence classification itself. Its bundled
+reference executor is a deterministic simulation of a compliant runtime, not a
+model. To forward-evaluate a real agent runtime, pass its stdin/stdout JSON
 adapter through `scripts/evals/run_forward.py --executor` and retain captured
-observations with `--output-dir`.
+observations with `--output-dir`. A Claude Code headless adapter is bundled:
+
+```bash
+just eval-implement-ticket-claude
+```
 
 ## Prerequisites
 
 - Python 3.11+
 - Git
-- GitHub CLI (`gh`) for PR workflows
+- GitHub CLI (`gh`) 2.37 or newer for PR workflows (the babysit-pr watcher
+  requires `gh pr checks --json`)
 - `skills-ref` on `PATH` for skill validation (optional but recommended)
 
 ## Notes

justfile

@@ -8,6 +8,19 @@ md_targets := "."
 list-skills:
   @find {{skills_dir}} -mindepth 1 -maxdepth 1 -type d -print
 
+# Refresh the review-suite contract copies bundled into each review skill so
+# the skills stay self-contained when installed outside this repository.
+sync-contracts:
+  @for skill in review-code-change review-correctness review-code-simplicity review-solution-simplicity; do \
+    dest="{{skills_dir}}/$skill/references/review-suite"; \
+    mkdir -p "$dest"; \
+    cp review-suite/CONTRACT.md "$dest/CONTRACT.md"; \
+    cp review-suite/contracts/review-packet.schema.json "$dest/review-packet.schema.json"; \
+    cp review-suite/contracts/review-result.schema.json "$dest/review-result.schema.json"; \
+    cp review-suite/scripts/validate.py "$dest/validate.py"; \
+    echo "Synced $dest"; \
+  done
+
 test:
   @found=0; \
   for tests in {{skills_dir}}/*/scripts/tests; do \
@@ -37,6 +50,11 @@ test-implement-ticket:
 eval-implement-ticket:
   python3 {{skills_dir}}/implement-ticket/scripts/evals/run_forward.py
 
+# Real-runtime forward evaluation; requires the `claude` CLI on PATH.
+eval-implement-ticket-claude:
+  python3 {{skills_dir}}/implement-ticket/scripts/evals/run_forward.py \
+    --executor "python3 {{skills_dir}}/implement-ticket/scripts/evals/claude_executor.py"
+
 test-implement-epic:
   python3 -m unittest discover -s {{skills_dir}}/implement-epic/scripts/tests -p 'test_*.py'
 
@@ -97,7 +115,7 @@ lint-skills:
   AGENTSKILLS_DIR=".tools/agentskills"; \
   SKILLS_REF_BIN=""; \
   install_skills_ref() { \
-    echo "Installing skills-ref into .venv from local cache..."; \
+    echo "Installing skills-ref: recreating .venv and cloning agentskills from GitHub (network required)..."; \
     rm -rf .venv; \
     python -m venv .venv; \
     mkdir -p .tools; \

review-suite/CONTRACT.md

@@ -15,6 +15,17 @@ silently redefine them.
 - `fixtures/` contains raw review inputs and separate expected material outcomes
   for deterministic tests and independent forward tests.
 
+Because a skill folder is the unit of distribution, each review skill bundles a
+verbatim copy of this document, both schemas, and the dependency-free
+`validate.py` under its own `references/review-suite/` directory so the skill
+works — including packet and result validation — when installed outside this
+repository. The copies are mechanical mirrors, not forks: edit only the
+canonical files here, refresh the copies with `just sync-contracts`, and rely on
+the bundled-contract test to fail on any drift. References in this document to
+`scripts/validate.py` and `fixtures/` describe this canonical directory; in an
+installed skill, use the bundled `validate.py` beside this file, and expect no
+fixtures.
+
 ## Review packet
 
 A review packet binds the review to one candidate and states what that candidate

review-suite/scripts/tests/test_bundled_contracts.py

@@ -0,0 +1,92 @@
+"""Verify skill-bundled review-suite contract copies match the canonical source.
+
+Each review skill bundles the canonical contract and schemas under
+`references/review-suite/` so the skill remains self-contained when installed
+outside this repository. `just sync-contracts` refreshes the copies; this test
+fails when any copy drifts from the canonical file.
+"""
+
+from __future__ import annotations
+
+import subprocess
+import sys
+import unittest
+from pathlib import Path
+
+REPOSITORY_ROOT = Path(__file__).resolve().parents[3]
+REVIEW_SUITE = REPOSITORY_ROOT / "review-suite"
+BUNDLING_SKILLS = (
+    "review-code-change",
+    "review-correctness",
+    "review-code-simplicity",
+    "review-solution-simplicity",
+)
+CANONICAL_FILES = {
+    "CONTRACT.md": REVIEW_SUITE / "CONTRACT.md",
+    "review-packet.schema.json": REVIEW_SUITE
+    / "contracts"
+    / "review-packet.schema.json",
+    "review-result.schema.json": REVIEW_SUITE
+    / "contracts"
+    / "review-result.schema.json",
+    "validate.py": REVIEW_SUITE / "scripts" / "validate.py",
+}
+
+
+class BundledContractTests(unittest.TestCase):
+    def test_every_review_skill_bundles_identical_contract_copies(self):
+        for skill in BUNDLING_SKILLS:
+            bundle = REPOSITORY_ROOT / "skills" / skill / "references" / "review-suite"
+            for name, canonical in CANONICAL_FILES.items():
+                bundled = bundle / name
+                with self.subTest(skill=skill, file=name):
+                    self.assertTrue(
+                        bundled.exists(),
+                        f"{bundled} is missing; run `just sync-contracts`",
+                    )
+                    self.assertEqual(
+                        canonical.read_bytes(),
+                        bundled.read_bytes(),
+                        f"{bundled} drifted from {canonical}; "
+                        "run `just sync-contracts`",
+                    )
+
+    def test_every_bundled_validator_executes_in_its_installed_layout(self):
+        """The bundled validate.py must run from references/review-suite/.
+
+        Byte-identity alone once shipped a copy that resolved its schemas
+        against the canonical directory layout and crashed everywhere else;
+        execute each copy in place against a known-good fixture pair.
+        """
+        fixture = REVIEW_SUITE / "fixtures" / "repository-convention-clean"
+        for skill in BUNDLING_SKILLS:
+            bundled = (
+                REPOSITORY_ROOT
+                / "skills"
+                / skill
+                / "references"
+                / "review-suite"
+                / "validate.py"
+            )
+            with self.subTest(skill=skill):
+                completed = subprocess.run(
+                    [
+                        sys.executable,
+                        str(bundled),
+                        "pair",
+                        str(fixture / "packet.json"),
+                        str(fixture / "expected.json"),
+                    ],
+                    capture_output=True,
+                    text=True,
+                    check=False,
+                )
+                self.assertEqual(
+                    0,
+                    completed.returncode,
+                    f"{bundled} failed: {completed.stderr.strip()}",
+                )
+
+
+if __name__ == "__main__":
+    unittest.main()

review-suite/scripts/tests/test_contracts.py

@@ -82,6 +82,15 @@ def test_unknown_enum_is_rejected(self):
         self.packet["validation"][0]["status"] = "skipped"
         self.assertTrue(VALIDATOR.validate_packet(self.packet))
 
+    def test_boolean_const_rejects_numeric_one(self):
+        # Python's `1 == True` must not let numeric values satisfy
+        # `"const": true`.
+        self.packet["candidate"]["diff"]["complete"] = 1
+        self.assertIn(
+            "$.candidate.diff.complete: expected constant True",
+            VALIDATOR.validate_packet(self.packet),
+        )
+
     def test_unavailable_validation_requires_reason(self):
         self.packet["validation"][0] = {
             "name": "tests",

review-suite/scripts/validate.py

@@ -10,10 +10,27 @@
 from pathlib import Path
 from typing import Any
 
-ROOT = Path(__file__).resolve().parents[1]
+HERE = Path(__file__).resolve().parent
+
+
+def _schema_file(name: str) -> Path:
+    """Locate a schema in either supported layout.
+
+    Canonical layout: review-suite/scripts/validate.py with schemas under
+    review-suite/contracts/. Bundled layout (installed review skills):
+    references/review-suite/validate.py with the schemas beside it.
+    """
+    for candidate in (HERE / name, HERE.parent / "contracts" / name):
+        if candidate.is_file():
+            return candidate
+    raise FileNotFoundError(
+        f"Cannot locate {name} beside {HERE} or in {HERE.parent / 'contracts'}"
+    )
+
+
 SCHEMAS = {
-    "packet": ROOT / "contracts" / "review-packet.schema.json",
-    "result": ROOT / "contracts" / "review-result.schema.json",
+    "packet": _schema_file("review-packet.schema.json"),
+    "result": _schema_file("review-result.schema.json"),
 }
 
 BLOCKABLE_PACKET_ERROR_PATTERNS = (
@@ -74,8 +91,11 @@ def validate_schema(value: Any, schema: dict[str, Any], at: str = "$") -> list[s
     if expected_type and not _is_type(value, expected_type):
         return [f"{at}: expected {expected_type}"]
 
-    if "const" in schema and value != schema["const"]:
-        errors.append(f"{at}: expected constant {schema['const']!r}")
+    if "const" in schema:
+        const = schema["const"]
+        # `1 == True` in Python; a boolean constant must reject numeric 1/1.0.
+        if value != const or isinstance(const, bool) != isinstance(value, bool):
+            errors.append(f"{at}: expected constant {const!r}")
     if "enum" in schema and value not in schema["enum"]:
         errors.append(f"{at}: expected one of {schema['enum']!r}")
     if isinstance(value, str):