Skip to content

v0.5: Aggressive SKILL.md simplification via multi-LLM pipeline #58

Description

@earino

Goal

Tighten SKILL.md from ~1,560 lines to ~900–1,000 lines (target ~40% cut) while preserving every load-bearing prescription that came from a real failure. No new infrastructure; no per-host generation; just systematic compression of redundancy and reference material that duplicates inline content.

Why this, and why now

PR #57 (templating) got killed because measurement showed the cross-host enumerations were already compact (3.5–5% size reduction, not the 15% estimated). The bigger insight was that the qwen3.6-35b failure modes we fixed in v0.4.4 — mkdir before cat >, heredoc-with-quoted-delimiter, "pass $PROMPT AS-IS", piping through extract-fit-fields, subagent_type cross-host — were all prescriptiveness-at-ambiguity-points failures. None were "the doc is too long, model lost attention." Different bug class.

This means trimming pure redundancy cannot regress weak-model reliability for the failure modes we know about — those failures live at choice points, and we're not touching the prescriptions at choice points. It also frees headroom inside OpenCode's 51,200-byte tool_output cap (currently ~96% full at 78KB), which was the actual back-half-of-SKILL.md truncation bug from samples-issue42/session-ses_2359.md.

Where the redundancy is

Honest catalog from a careful read of the current SKILL.md:

Section Lines What it is
"Usage analytics (telemetry)" reference ~250 Re-renders the inline "$TEL" calls already present in Phase 0/1/3/5/7/9
"Debugging this skill" playbook ~150 Self-contained; belongs in docs/DEBUGGING.md with a 5-line pointer in SKILL.md
"Re-rendering PDFs after manual edits" ~80 Repeats the path prologue and PDF render commands; can fold into Phase 8
Phase 0: 9 expanded question shapes ~80 Same 2-option + Other pattern with different content; one example + a field list does it
War-story parentheticals ~60 "(observed under qwen3.6-35b on OpenCode, run ses_236d — Qwen rewrote …)" appears ~15 times; one-line "why" pointers preserve the rule
Prose filler ~40 "It's important that...", "Note that...", redundant "important" / "must" combinations

Total estimated cut: ~660 lines. SKILL.md goes from 1,560 → ~900.

Methodology: multi-LLM pipeline

Single-author rewrites have author bias (the writer defends their own prose). Multi-LLM pipeline polls multiple authors AND multiple reviewers AND the actual weak-model audience.

Phase 1 — Parallel independent drafts

Three independent rewrites, each model gets the SAME brief: "tighten SKILL.md while keeping every `do X / NOT Y` prescription; this runs on weak local models like qwen3.6-35b on OpenCode."

  • Claude Opus 4.7 (1M context) — drafted in-session by the maintainer's Claude Code agent.
  • GPT-5 Pro — Eduardo pastes SKILL.md + brief into ChatGPT Pro.
  • Kimi K2 — Eduardo pastes SKILL.md + brief into Kimi (its 1M context fits the original + multiple drafts side-by-side).

The interesting signal isn't whose draft is best. It's what all three independently cut. Sentences that all three remove are filler with high confidence. Sentences only one cuts get scrutiny.

Phase 2 — Cross-model adversarial review

Each draft handed to a model that didn't write it, with brief: "compare to the original SKILL.md (also pasted). Name every prescription that's missing or now ambiguous. Cite specific original sentences."

  • Claude's draft → reviewed by GPT-5
  • GPT-5's draft → reviewed by Claude (in-session)
  • Kimi's draft → reviewed by either

Different blindspots than the writer. If the reviewer flags a missing prescription, the writer's draft restores it.

Phase 3 — Weak-model audience polling

Strongest reliability gate, cheap. Take the synthesized draft and feed it to qwen3.6-35b on the maintainer's local Ollama:

"Given this SKILL.md, walk me through what you'd do for /resumasher /tmp/job.md."

Pass = qwen names the right phases, reads the right files (resume, jd, cache), dispatches sub-agents with the right shape, fires the right telemetry events. Fail = qwen stalls, asks ambiguous questions, improvises, or skips a phase. A failure here means we cut a load-bearing prescription. Restore and re-poll.

This is the actual regression gate. Way cheaper than a full end-to-end run (no PDF render, no sub-agent dispatch, no waiting on cloud LLM calls).

Phase 4 — Real end-to-end run on weak model

After the trimmed draft passes the audience poll, do a full /resumasher run on a fixture (GOLDEN_FIXTURES/resume.md + a JD) under qwen3.6-35b on OpenCode. PDFs render, fit fields populate, telemetry fires, no improvisation. If this passes, ship.

Phase 5 — Pytest structural gate (continuous)

pytest tests/ -q runs after every commit. The existing weak-model guards (tests/test_skill_md_weak_model_guards.py) already pin the most critical prescriptions (Phase 0 mkdir, Phase 2 heredoc) — they catch accidental drops without manual review. Same for the OpenCode shim and Phase 7 prescription tests. Drift here = restore the prescription before the commit lands.

Branch and PR plan

  • Cut feature/skill-md-shrink off main.
  • Land cuts in the order from the redundancy catalog above (telemetry reference first — biggest, lowest risk; debugging playbook → docs/DEBUGGING.md next; etc.). One commit per cut so a regression is easy to bisect.
  • After each commit: pytest. After the cumulative shrink draft is ready: weak-model audience poll. Before merge: full end-to-end on qwen.
  • Target single PR with multiple commits, not a series of small PRs — the cuts compound and reviewing them as a unit is easier than reviewing five "drop-50-lines" PRs.

Acceptance criteria

  • SKILL.md ≤ 1,000 lines (currently 1,560)
  • Every prescription in tests/test_skill_md_weak_model_guards.py and tests/test_opencode_command_shim.py still passes
  • Three-model parallel drafts collected; consensus cuts identified; non-consensus cuts justified inline
  • qwen3.6-35b audience poll passes (correct phases + files + dispatch shape)
  • qwen3.6-35b full end-to-end run produces resume.pdf + cover-letter.pdf + interview-prep.pdf with valid fit assessment
  • No regression in the 476-test pytest suite

Stretch — extract DEBUGGING.md and PDF-rerender into separate skills

Both sections are self-contained workflows triggered by separate user intents (debugging an issue vs. re-rendering after edits). Could live as docs/DEBUGGING.md + docs/RE-RENDER.md with SKILL.md pointers. Lighter than splitting them into Claude Code "child skills" but achieves the same separation-of-concerns. Worth it if SKILL.md still feels heavy after the in-tree shrink lands.

Related

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions