Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 3 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,14 +36,12 @@ schema_version: 1
id: output_file_exists
type: deterministic
description: Verify that output.json was generated.
target_paths:
- output.json
script: |
import json, sys
from pathlib import Path
ctx = json.loads(Path(sys.argv[1]).read_text())
target = ctx["targets"][0]
assert target["exists"], f"{target['path']} not found"
output = Path(ctx["project_root"]) / "output.json"
assert output.exists(), "output.json not found"
```

### Harness judge check
Expand All @@ -53,10 +51,8 @@ schema_version: 1
id: summary_is_accurate
type: harness_judge
description: The generated summary accurately reflects source data.
target_paths:
- summary.txt
- source_data.json
instructions: |
Read summary.txt and source_data.json.
Compare the summary against the source data.
Score 1 if accurate, 0 if it contains fabricated claims.
```
Expand Down
48 changes: 15 additions & 33 deletions docs/check-format.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,12 +14,11 @@ All check types share these fields:
| `id` | string | Yes | Unique identifier (`[a-zA-Z0-9_-]` only) |
| `type` | string | Yes | One of: `deterministic`, `harness_judge` |
| `description` | string | Yes | Human-readable description |
| `target_paths` | list[string] | No | Files/directories the check operates on |
| `tags` | list[string] | No | Tags for filtering (future use) |

## Deterministic checks

Run a Python script that asserts conditions about target files.
Run a Python script that asserts conditions about project files.

| Field | Type | Required | Description |
|---|---|---|---|
Expand All @@ -30,11 +29,10 @@ Exactly one of `script` or `script_path` must be set.

### How it works

1. Target paths are resolved relative to the project root
2. A `context.json` file is written with check metadata and resolved targets
3. The script runs as `python <script> <context.json>`
4. Exit code 0 = passed, non-zero = failed
5. Infrastructure errors (missing script, OS execution failure) = error
1. A `context.json` file is written with check metadata and project root
2. The script runs as `python <script> <context.json>`
3. Exit code 0 = passed, non-zero = failed
4. Infrastructure errors (missing script, OS execution failure) = error

### context.json shape

Expand All @@ -44,15 +42,7 @@ Exactly one of `script` or `script_path` must be set.
"description": "Check description",
"project_root": "/absolute/project/root",
"source_path": "/absolute/path/to/check.yaml",
"output_dir": "/absolute/path/to/output/checks/my_check",
"targets": [
{
"path": "relative/path.txt",
"resolved_path": "/absolute/resolved/path.txt",
"exists": true,
"is_dir": false
}
]
"output_dir": "/absolute/path/to/output/checks/my_check"
}
```

Expand All @@ -63,38 +53,32 @@ schema_version: 1
id: no_todo_comments
type: deterministic
description: No TODO markers in Python source files.
target_paths:
- src
script: |
import json, sys
from pathlib import Path
ctx = json.loads(Path(sys.argv[1]).read_text())
for t in ctx["targets"]:
p = Path(t["resolved_path"])
if p.is_dir():
for f in p.rglob("*.py"):
text = f.read_text()
if "TODO" in text:
sys.exit(1)
src = Path(ctx["project_root"]) / "src"
for f in src.rglob("*.py"):
if "TODO" in f.read_text():
sys.exit(1)
```

## Harness judge checks

Send target file content to the configured harness agent with evaluation instructions.
Invoke the configured harness agent with evaluation instructions. The agent can read project files on its own.

| Field | Type | Required | Description |
|---|---|---|---|
| `instructions` | string | Yes | Evaluation criteria for the judge |
| `model` | string | No | Override the harness model for this check |
| `target_paths` | list[string] | Yes | Must be non-empty |

### How it works

1. Target files are read and included in a judging prompt
1. A judging prompt is built from the check description and instructions
2. The configured harness agent subprocess receives that prompt
3. The final verdict must include `{"score": 0|1, "reason": "..."}`
4. Score 1 = passed, 0 = failed, parse error = error
5. Harness spawn failures, timeouts, or unreadable targets = error
5. Harness spawn failures or timeouts = error

### Example

Expand All @@ -103,11 +87,9 @@ schema_version: 1
id: readme_has_install_steps
type: harness_judge
description: README clearly explains how to install the package.
target_paths:
- README.md
instructions: |
Does the README give a new user enough information to install
and run the package locally? Score 1 if yes, 0 if no.
Read README.md. Does it give a new user enough information to
install and run the package locally? Score 1 if yes, 0 if no.
```

## Auto-discovery
Expand Down
39 changes: 12 additions & 27 deletions skills/eval-banana/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,8 +35,6 @@ schema_version: 1 # Always 1. Required. No default.
id: my_check_id # Unique across the project. Pattern: [a-zA-Z0-9_-]+
type: deterministic # One of: deterministic, harness_judge
description: Human-readable # Required. Non-empty.
target_paths: # Files/dirs the check operates on. Resolved from project_root.
- path/to/file.json # Optional. Content is passed to the script/judge when provided.
tags: [fast, critical] # Optional list of free-form tags.
```

Expand All @@ -51,17 +49,16 @@ schema_version: 1
id: output_has_result_key
type: deterministic
description: output.json exists and contains a 'result' key.
target_paths:
- output.json
script: |
import json, sys
from pathlib import Path

ctx = json.loads(Path(sys.argv[1]).read_text())
target = ctx["targets"][0]
if not target["exists"]:
project_root = Path(ctx["project_root"])
output = project_root / "output.json"
if not output.exists():
sys.exit(1)
data = json.loads(Path(target["resolved_path"]).read_text())
data = json.loads(output.read_text())
if "result" not in data:
sys.exit(1)
```
Expand All @@ -78,22 +75,12 @@ The script is invoked as `python <script> <context.json>`. Read `sys.argv[1]` to
"description": "output.json exists and contains a 'result' key.",
"project_root": "/abs/path/to/project",
"source_path": "/abs/path/to/project/eval_checks/my_check.yaml",
"output_dir": "/abs/path/.../.eval-banana/results/<run_id>/checks/output_has_result_key",
"targets": [
{
"path": "output.json",
"resolved_path": "/abs/path/to/project/output.json",
"exists": true,
"is_dir": false
}
]
"output_dir": "/abs/path/.../.eval-banana/results/<run_id>/checks/output_has_result_key"
}
```

Key points:
- `targets` entries align 1:1 with `target_paths` in the YAML, in order.
- `resolved_path` is absolute — use it directly, don't re-resolve from `path`.
- `exists` and `is_dir` are pre-checked for convenience.
- `project_root` is absolute — use it to locate any file in the project.
- The subprocess runs with `cwd = project_root`, so relative paths in the script also resolve from there.

### Deterministic failure mapping
Expand All @@ -105,32 +92,30 @@ Key points:

## Writing a `harness_judge` check

Invokes the configured harness agent with instructions. The agent must eventually emit JSON: `{"score": 0|1, "reason": "one sentence"}`. When `target_paths` are provided, their content is included in the prompt. When omitted, the agent can read files on its own.
Invokes the configured harness agent with instructions. The agent can read files on its own. It must eventually emit JSON: `{"score": 0|1, "reason": "one sentence"}`.

```yaml
schema_version: 1
id: readme_explains_install
type: harness_judge
description: README gives a new user enough info to install the package.
target_paths:
- README.md
instructions: |
Does the README give a new user enough information to install
and run the package locally (environment setup, install command,
and how to invoke it)? Score 1 if yes, 0 if anything critical
is missing.
Read README.md. Does it give a new user enough information to
install and run the package locally (environment setup, install
command, and how to invoke it)? Score 1 if yes, 0 if anything
critical is missing.
```

Guidelines for good instructions:
- State the exact condition for score 1 and score 0.
- Be binary — avoid "mostly", "partially", etc.
- Reference concrete things to look for.
- Tell the agent which files to read.
- Keep it short. Long instructions confuse the judge.
- Do not ask for scores outside {0, 1} — the parser rejects anything else as `error`.

Optional fields:
- `model: gpt-5.4` — override the default harness model for this one check.
- Multiple `target_paths` — all files are concatenated with separators in the prompt.

## Auto-discovery rules

Expand Down
60 changes: 21 additions & 39 deletions skills/eval-banana/references/examples.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,18 +20,18 @@ schema_version: 1
id: changelog_exists_and_nonempty
type: deterministic
description: CHANGELOG.md exists and is at least 200 chars.
target_paths:
- CHANGELOG.md
script: |
import json, sys
import sys
from pathlib import Path

ctx = json.loads(Path(sys.argv[1]).read_text())
target = ctx["targets"][0]
if not target["exists"]:
print(f"missing: {target['path']}", file=sys.stderr)
ctx_path = Path(sys.argv[1])
import json
ctx = json.loads(ctx_path.read_text())
changelog = Path(ctx["project_root"]) / "CHANGELOG.md"
if not changelog.exists():
print(f"missing: CHANGELOG.md", file=sys.stderr)
sys.exit(1)
content = Path(target["resolved_path"]).read_text(encoding="utf-8")
content = changelog.read_text(encoding="utf-8")
if len(content) < 200:
print(f"only {len(content)} chars", file=sys.stderr)
sys.exit(1)
Expand All @@ -44,15 +44,12 @@ schema_version: 1
id: output_matches_schema
type: deterministic
description: output.json has required top-level keys and correct types.
target_paths:
- output.json
script: |
import json, sys
from pathlib import Path

ctx = json.loads(Path(sys.argv[1]).read_text())
target = ctx["targets"][0]
data = json.loads(Path(target["resolved_path"]).read_text())
data = json.loads((Path(ctx["project_root"]) / "output.json").read_text())

required = {"id": str, "created_at": str, "items": list}
for key, expected_type in required.items():
Expand All @@ -74,20 +71,17 @@ schema_version: 1
id: no_print_statements_in_src
type: deterministic
description: No print() calls in src/ (should use logging).
target_paths:
- src
script: |
import json, re, sys
from pathlib import Path

ctx = json.loads(Path(sys.argv[1]).read_text())
src_dir = Path(ctx["targets"][0]["resolved_path"])
src_dir = Path(ctx["project_root"]) / "src"
pattern = re.compile(r"\bprint\(")
offenders = []
for py_file in src_dir.rglob("*.py"):
text = py_file.read_text(encoding="utf-8")
for line_no, line in enumerate(text.splitlines(), 1):
# Skip comments and docstrings (naive but fine for most cases)
if line.strip().startswith("#"):
continue
if pattern.search(line):
Expand All @@ -104,14 +98,12 @@ schema_version: 1
id: users_csv_has_enough_rows
type: deterministic
description: users.csv has at least 100 rows (excluding header).
target_paths:
- data/users.csv
script: |
import csv, json, sys
from pathlib import Path

ctx = json.loads(Path(sys.argv[1]).read_text())
path = Path(ctx["targets"][0]["resolved_path"])
path = Path(ctx["project_root"]) / "data" / "users.csv"
with path.open() as f:
reader = csv.reader(f)
next(reader, None) # Skip header
Expand All @@ -128,10 +120,8 @@ schema_version: 1
id: readme_has_quickstart
type: harness_judge
description: README contains a quickstart that a new user can follow in under 5 minutes.
target_paths:
- README.md
instructions: |
Look for a "Quick start" or "Getting started" section in the README.
Read README.md. Look for a "Quick start" or "Getting started" section.
Score 1 ONLY if it contains: (1) an install command, (2) a minimum
config step if required, and (3) at least one example command to run.
Score 0 if any of these are missing or unclear.
Expand All @@ -144,10 +134,8 @@ schema_version: 1
id: error_messages_are_helpful
type: harness_judge
description: Error messages in errors.log are helpful and professional.
target_paths:
- errors.log
instructions: |
Read the error messages in the log. Score 1 if they: (a) explain what
Read errors.log. Score 1 if the error messages: (a) explain what
went wrong in plain language, (b) suggest what to do next, and (c) do
NOT expose stack traces or internal paths to end users. Score 0 if
any message is cryptic, blames the user, or leaks internals.
Expand All @@ -160,14 +148,11 @@ schema_version: 1
id: summary_matches_source
type: harness_judge
description: The generated summary accurately reflects the source data.
target_paths:
- summary.md
- source_data.json
instructions: |
Compare the claims in summary.md against the facts in source_data.json.
Score 1 if every numeric claim, name, and date in the summary can be
verified from the source data. Score 0 if ANY claim is fabricated,
hallucinated, or contradicts the source.
Read summary.md and source_data.json. Compare the claims in the summary
against the facts in the source data. Score 1 if every numeric claim,
name, and date in the summary can be verified from the source data.
Score 0 if ANY claim is fabricated, hallucinated, or contradicts the source.
```

## Harness judge: multi-file comparison
Expand All @@ -177,12 +162,9 @@ schema_version: 1
id: docs_consistent_with_code
type: harness_judge
description: API docs describe the actual endpoints implemented in routes.py.
target_paths:
- docs/api.md
- src/routes.py
instructions: |
For each endpoint documented in docs/api.md, check if it exists in
src/routes.py. Score 1 if every documented endpoint is implemented
AND every public endpoint in routes.py is documented. Score 0 if
there is any drift between the two files.
Read docs/api.md and src/routes.py. For each endpoint documented in the
API docs, check if it exists in routes.py. Score 1 if every documented
endpoint is implemented AND every public endpoint in routes.py is
documented. Score 0 if there is any drift between the two files.
```
Loading
Loading