Autonomous AI skill optimizer — an AI coach that scores and auto-improves your agent skills.
You have a bunch of SKILL.md files (instruction manuals for your AI agent). How good they are directly affects how well your AI performs. source-command-darwin is an automated coach that:
- Reads all your SKILL.md files
- Scores each one on a 100-point scale (8 dimensions)
- Picks the worst-scoring ones
- Has the AI rewrite them to be better
- Scores the new version — higher? → keep it. Lower? → auto-revert.
- Repeats until you stop it
Think of it as: a spell-checker, but for your AI agent's brain. It doesn't change what your skills do — it makes how they're written objectively better.
Agent skills (prompts-as-code) are powerful but hard to maintain:
| Pain | How source-command-darwin Fixes It |
|---|---|
| No objective quality metric | 8-dimension rubric, 100-point scale |
| Structural problems are invisible | Static analysis catches missing triggers, vague steps, no fallbacks |
| Improvements are fragile | Git ratchet: commit good changes, auto-revert bad ones |
| No version control discipline | One dimension per round, sub-agent evaluation, human confirmation |
Phase 0: Initialize
└─ Scan target skills, create git branch, init results.tsv
Phase 0.5: Design Test Prompts
└─ For each skill, generate 2-3 realistic user prompts
Phase 1: Baseline Evaluation
└─ Score each skill on all 8 dimensions (100-point scale)
Phase 2: Optimize Loop (repeat per skill)
├─ Identify weakest dimension
├─ Generate improvement (edit SKILL.md)
├─ Re-evaluate (sub-agent or dry run)
├─ Score improved? → git commit (keep)
└─ Score worse? → git revert (rollback)
Phase 3: Summary Report
└─ Generate comparison table + improvement highlights
Before: price-action-trading 62/100 (vague steps, missing edge cases)
After: price-action-trading 81/100 (concrete parameters, fallback paths added)
Before: market-analysis 75/100 (no checkpoint design)
After: market-analysis 88/100 (user confirmation gates added)
| Principle | What It Means |
|---|---|
| Single editable asset | Each round only touches one SKILL.md |
| Dual evaluation | Structure score (static analysis) + Effect score (run test prompts) |
| Ratchet mechanism | Only keep improvements, auto-rollback regressions |
| Independent scoring | Effect evaluation runs in a sub-agent to avoid self-grading bias |
| Human in the loop | Pause after each skill optimization for user confirmation |
| # | Dimension | Weight | What It Measures |
|---|---|---|---|
| 1 | Frontmatter Quality | 8 | name, description (what + when + trigger words), ≤1024 chars |
| 2 | Workflow Clarity | 15 | Steps are explicit, ordered, with clear input/output per step |
| 3 | Edge Case Coverage | 10 | Handles exceptions, has fallback paths, error recovery |
| 4 | Checkpoint Design | 7 | User confirmation at key decisions, prevents runaway autonomy |
| 5 | Instruction Specificity | 15 | No vague language, concrete parameters/formats/examples |
| 6 | Resource Integration | 5 | References/scripts/assets correctly linked and reachable |
| 7 | Overall Architecture | 15 | Clear hierarchy, no redundancy, no gaps, consistent style |
| # | Dimension | Weight | What It Measures |
|---|---|---|---|
| 8 | Live Test Performance | 25 | Run 2-3 test prompts; compare output quality with and without the skill |
Scoring: Each dimension rated 1-10, multiplied by weight. Total = Σ(dimension × weight) / 10. Max = 100.
- An AI coding agent that supports skills (OpenAI Codex, Claude Code, Cursor, etc.)
- Your existing skills in
SKILL.mdformat - Git initialized in your skills directory
Optimize all my skills
→ Runs Phases 0-3 on every skill found. → Starts with baseline scores, then optimizes the lowest-scoring ones.
Optimize the price-action-trading skill
→ Targets only the specified skill (Phases 0.5-2).
Evaluate all my skills — score only, don't modify
→ Runs Phases 0.5-1 (design test prompts + baseline scoring). No edits.
Show skill optimization history
→ Reads and displays results.tsv.
timestamp commit skill old_score new_score status dimension note eval_mode
2026-03-31T10:00 baseline my-skill - 72 baseline - Initial evaluation full_test
2026-03-31T10:05 a1b2c3d my-skill 72 79 keep Edge Cases Added fallback paths full_test
2026-03-31T10:10 b2c3d4e my-skill 79 76 revert Specificity Over-constrained dry_runeval_mode:full_test(ran sub-agent test) ordry_run(simulated reasoning)status:keep(improvement) orrollback(regression)
=== Optimization Summary ===
| Skill | Before | After | Change |
|-----------------|--------|-------|--------|
| my-skill-alpha | 62 | 81 | +19 |
| my-skill-beta | 75 | 85 | +10 |
| Average | 68 | 83 | +15 |
Key improvements:
1. [alpha] Added edge case handling — test output quality improved significantly
2. [beta] Restructured workflow — baseline comparison shows clear advantage
| Priority | Type | Example | Action |
|---|---|---|---|
| P0 | Effect issues | Test output deviates from intent | Check for misleading instructions |
| P1 | Structural | Missing frontmatter triggers, no step structure | Restructure to linear flow |
| P2 | Specificity | Vague steps ("process the image") | Add concrete operations and parameters |
| P3 | Readability | Long paragraphs, repetitive content | Split into tables, deduplicate |
Each round targets the highest priority remaining issue. One dimension per round.
- Don't change what a skill does — only optimize how it's written and executed
- Don't introduce new dependencies — no adding scripts or references the skill didn't have
- One dimension per round — avoids multi-variable attribution confusion
- Keep file size reasonable — optimized SKILL.md should not exceed 150% of original size
- Respect existing style — maintain the author's voice and conventions
- Everything is revertable — all changes on a git branch;
git revert, neverreset --hard - Independent evaluation — effect dimension must be tested by a sub-agent, never self-graded
# Copy to Codex skills directory
Copy-Item .\SKILL.md "$env:USERPROFILE\.codex\skills\source-command-darwin\SKILL.md"
# Or symlink (recommended for development)
New-Item -ItemType SymbolicLink `
-Path "$env:USERPROFILE\.codex\skills\source-command-darwin" `
-Target (Get-Location).Pathcp SKILL.md ~/.claude/skills/source-command-darwin/SKILL.mdPlace SKILL.md wherever your agent expects skill definitions. The skill is agent-agnostic — it only needs the ability to read/write files and run git commands.
| autoresearch | source-command-darwin |
|---|---|
program.md (goals + constraints) |
This SKILL.md (rubric + rules) |
train.py (code to optimize) |
Each SKILL.md being optimized |
val_bpb (validation loss) |
8-dimension weighted score (incl. live test) |
git ratchet (keep only improvements) |
Same — git commit on improvement, revert on regression |
test set |
test-prompts.json per skill |
Key difference: human confirmation before finalizing each skill. autoresearch is fully autonomous; skill quality is too nuanced for pure automation.
Q: Does this work for non-English skills? A: Yes. The rubric evaluates structure and output quality, not language.
Q: How long does a full optimization take? A: ~2-5 minutes per skill (evaluation + 1-3 optimization rounds). Budget 30-60 minutes for 10 skills.
Q: Can I customize the rubric? A: Absolutely. Adjust weights, add dimensions, or change scoring criteria to match your priorities.
Q: What if the agent keeps reverting everything? A: Your skills are already well-optimized, or test prompts aren't discriminating enough. Try more challenging prompts or focus on P0 (effect) issues.
- Karpathy's autoresearch — the core "evaluate → improve → ratchet" loop
- The Codex / Claude Code skill ecosystem — making prompts-as-code a real paradigm