skillbench

Evaluate and auto-optimize agent skills — an 8-dimension rubric, median-of-3 denoising, a noise-band ratchet, and git-isolated hill-climbing.

中文文档 · Quickstart · Host adapters

---

What it is

skillbench is a single SKILL.md-shaped methodology (plus one tiny standard-library helper script) that an agent loads to score a skill and iteratively improve it.

It sits in the gap between two existing project categories:

Category	Example scope	What it does	What it doesn't
"How to write skills" tutorials	authoring guides, best-practice collections	Teaches humans/agents to write skills	Doesn't measure a given skill's quality
Skill security scanners	permission/red-flag linters	Flags unsafe skills	Doesn't say if a skill is good to use
skillbench (this)	evaluate + auto-optimize	Scores how good/usable a skill is, then hill-climbs it	Not a tutorial, not a security scanner

Inspired by Karpathy's autoresearch loop: evaluate → improve → re-test → keep or roll back.

Core mechanics

1. 8-dimension rubric (100 pts). Six structure dimensions (60 pts: frontmatter, workflow clarity, edge-case coverage, checkpoints, instruction specificity, resource integration) + two effect dimensions (40 pts: overall architecture, live behavior).
2. Median-of-3 denoising. LLM scoring has ±1–2 pts of built-in noise. Every round is scored N=3 times in independent contexts; the median (not the mean) is the round score. Range > 4 triggers a 4th scoring pass; still unstable → flagged, no ratchet call.
3. Noise-band ratchet. A change is kept only if median_new − median_old > noise band (default 2). Changes inside the band count as "flat" (not kept, not a regression). This is the fix for naive "strictly-higher" ratchets that mistake noise for progress.
4. Git isolation. Optimization runs on a temp working copy (mktemp -d + git init), never the host's production dir. It uses git revert (not reset --hard) and exports a optimize.diff for human review before any write-back. In-place edits require an explicit --in-place --yes plus a second confirmation.

Three evaluation modes

A strict "with-skill vs without-skill" A/B is impossible in most hosts (not every host can toggle a single skill at runtime). So the "live behavior" dimension is explicitly three-tiered, and every run does a host-capability self-check first and records the chosen tier.

Mode	Definition	Condition	Score handling
live	Host supports runtime skill toggle: run same prompt with/without the skill, compare directly	Host has per-run skill inject/remove	Full weight
approx	Paste full SKILL.md into agent A's prompt, agent B is baseline; compare	Sub-agents available, no skill toggle	Full weight + token-bias note
dry_run	Static reasoning: read the skill, simulate a typical prompt's execution path	No sub-agent / offline / constrained	Score × 0.7 discount, labeled dry_run

Priority is live > approx > dry_run, with honest auto-downgrade recorded in the report.

approx is field-tested, not just designed. We re-evaluated both dogfood skills in approx mode (sub-agent A gets the full SKILL.md pasted, agent B is baseline, same prompts, N=3 median scoring). Results vs dry_run:

Skill	dry_run median (×0.7 on dim 8)	approx median (full weight)	Δ	Ranking flip?
verify-before-claim	74.4	82.0	+7.6	no
openclaw-mem	63.65	70.0	+6.35	no

Both deltas fall inside the range explained by removing the dim-8 ×0.7 discount (≤7.5 plus small per-dimension variation), and the ranking between skills is stable across tiers — i.e. the tier system shifts levels as designed without reshuffling order. Full comparison in examples/approx-vs-dryrun-comparison.md.

Dogfood example (real, de-identified)

We ran skillbench on two of our own skills in dry_run mode (N=3, noise band 2):

Skill	Baseline median	Optimized median	Δ	Ratchet decision
verify-before-claim	74.4	76.4	+2.0	revert_within_noise
openclaw-mem	63.65	65.65	+2.0	revert_within_noise

Both improvements landed exactly on the noise band (+2.0) and were correctly not kept. This is the honest, useful result: the ratchet refused to bank a change it couldn't distinguish from scoring noise. See examples/ for the full de-identified score dumps.

Limitations (read these)

• Scoring is subjective. The rubric is LLM-scored; the median-of-3 + noise band reduces but does not eliminate subjectivity. Treat scores as directional, not absolute.
• dry_run is a discount, not a substitute. Static reasoning about behavior (× 0.7) is weaker evidence than a real live/approx run. Prefer higher tiers when the host allows.
• v0.1 scope is deliberately small (YAGNI). SKILL.md methodology + one helper script. No CLI, no Web UI, no CI integration yet — those come once there are real users.
• Ratchet can be conservative. A genuine +1.5 improvement gets classified as "flat". That's the intended trade-off: we prefer missing a small gain over banking noise.

Repo layout

skillbench/
  SKILL.md            # the methodology (load this into your agent)
  scripts/
    score_median.py   # stdlib-only: median/range aggregation + ratchet decision
  README.md           # you are here
  README.zh-CN.md     # Chinese version
  QUICKSTART.md       # 5-minute walkthrough
  LICENSE             # MIT
  examples/           # de-identified real score dumps
  docs/
    hosts.md          # host adapter notes (Claude Code / OpenClaw / generic agent)

License

MIT — see LICENSE.