Does the ocd skill (a prompt) change what Claude produces? The wideshot eval proves the tool works; this asks a different question about the skill, and answers it with a controlled A/B, scored by a deterministic script - not by model judgment.
- Corpus (
tasks/): 3 single-file bugs with objective ground truth a script can measure.t1- layout shift: a TUI progress row whose%label isn't padded, so the|column drifts as the percent widens.t2- drift: a caret marker at a hardcoded column instead of one computed from the count's width.t3- root cause vs symptom:pct()divides by zero on an empty list; the shared bug also reachesbar(). A symptom-only patch ofsummary()leaves the sibling caller broken.a1/a2/a3- adversarial versions: clamp both ends (not just >100), one shared bug reached by 3 named-only-once callers, column width from max (not the first row).va/vb/vc- visual (HTML), scored by rendered bounding-rects: layout shift, box-model width trap, off-center card.d1/d2/d3- discovery (HTML): same measurement, but the bug is NEVER named to the agent ("audit this, may or may not have issues"). Tests detection, not fixing.
- Conditions:
base= agent gets only the symptom + "no slash-command skills";ocd= agent told to read and followskills/ocd/SKILL.md. Same tools, same model. The only variable is the method. - Trials: 2 per (task x condition) = 12 runs. Symptom prompts state the symptom only, never the full check the scorer applies (so a lazy fix that only addresses the named case fails).
- Score (
score.mjs): imports each edited module and measures{fixed, regression}across all states / callers.fixed= defect gone everywhere, normal cases intact.regression= new breakage OR a symptom-only patch that left a sibling caller broken. - Scorer is self-tested first (
selftest.mjs): buggy -> not fixed; reference fixes -> fixed/no-regression; a symptom-only patch -> fixed BUT regression. A wrong scorer would invalidate everything, so it is validated before any agent runs.
| condition | fixed | regressions | mean tokens | mean tool-uses |
|---|---|---|---|---|
| base | 6 / 6 | 0 | ~26.8k | 2.0 |
| ocd | 6 / 6 | 0 | ~30.5k | 4.7 |
Ceiling effect. Baseline t3 runs guarded the shared pct() (root cause), not just summary() (symptom) - the exact discipline the skill teaches, applied unprompted.
Tasks redesigned so the lazy fix passes the named case but fails a deeper check: clamp both ends (not one), trace 3 callers (not the one named), size to max (not a sampled row).
| condition | fixed | regressions | mean tokens | mean tool-uses |
|---|---|---|---|---|
| base | 6 / 6 | 0 | ~27.0k | 2.2 |
| ocd | 6 / 6 | 0 | ~30.7k | 4.7 |
Same ceiling. Baseline clamped both ends (Math.max(0, Math.min(10, ...||0))), sized to Math.max over all rows, and guarded the shared fmtPct - every trap, unprompted.
The skill's home turf: bugs invisible-or-misleading in source, measurable only by rendering. va layout shift (a % label with no reserved width moves the Cancel button); vb box-model trap (two cards both width:320px, one border-box, render 38px apart); vc off-center card (hardcoded left:120px, centered needs 150). Scored by visual-score.mjs (puppeteer bounding-rects).
| condition | fixed | regressions | mean tokens | mean tool-uses |
|---|---|---|---|---|
| base | 6 / 6 | 0 | ~26.9k | 2.2 |
| ocd | 6 / 6 | 0 | ~34.3k | 8.3 |
Same outcome - but here the process finally diverged: ocd-condition agents actually rendered and measured (headless CDP - "button left edge holds at 246.53px", "verified A=B=320"), while base agents reasoned from source in 2 tool-uses. The skill did exactly what it promises (measure, don't eyeball). The base model's source reasoning simply reached the same correct fix without rendering - so measuring was right but redundant, at +27% tokens and ~4x the tool calls.
The decisive change: the agent is told only "audit this, it may or may not have issues" - the defect is never named. This tests detection, the skill's actual headline. Planted defects that don't scream in source: d1 one stat card silently wider (padding shorthand), d2 a counter with no reserved width that looks perfect at its default value and only shifts siblings when the number grows, d3 a flex-centered modal nudged off-center by a stray margin-left.
| task | base found | ocd found | note |
|---|---|---|---|
| d1 (card bulge, visible in one render) | 2/2 | 2/2 | ceiling |
| d2 (shift invisible at default state) | 0/2 | 2/2 | the signal |
| d3 (off-center, visible in one render) | 2/2 | 2/2 | ceiling |
| total | 4/6 | 6/6 |
First real difference in the whole study. On d2, base audited what it could see in the default render - it even found and fixed a different, unplanted real defect (buttons not inheriting the monospace font) - but never varied the count, so it missed the layout shift entirely, both trials. Verified in the file: base left .count { } untouched; ocd added min-width: 4ch after rendering across counts. The skill's "simulate the states - layout bugs live in the transitions" discipline is exactly what closed the gap. It cost ~46% more tokens on that task.
d1/d3 stayed at ceiling because those defects are visible in a single default render - no state simulation needed, so base catches them too.
- Fixing a named bug (rounds 1-3, code + visual): no measurable difference. Baseline Opus already clamps both ends, traces callers, sizes to max, reasons box-model. The skill changed the process (made agents render+measure) but not the outcome, at +14-27% cost.
- Discovering an unnamed bug (round 4): a real, mechanistic gap - but only for defects invisible in the default view. base 4/6 vs ocd 6/6, the entire delta on the one state-dependent bug. For bugs visible on sight, still ceiling.
So the skill earns its keep in one specific place: finding the bug that isn't visible until you simulate a state the default render doesn't show. That is a narrow but genuine slice of what it claims - and the slice where "render it, count the cells, across every state" is doing work a strong model skips when it isn't told to.
d2 rerun at n=8 per condition (discovery mode, bug never named), split into detection (found the shift at all) and robust fix (holds at count=9999):
| condition | detected the shift | robust fix (holds @9999) |
|---|---|---|
| base | 0 / 8 | 0 / 8 |
| ocd | 8 / 8 | 5 / 8 |
The 0/8 vs 8/8 detection split holds up. All 8 base runs instead found and fixed a different real defect (buttons not inheriting the monospace font) and never simulated the count. 3 of the 8 ocd runs detected the shift but sized the reserve to 3ch, too small for 4-digit counts - detection is the robust signal; fix-completeness varies. Before/after screenshots (fixture and real agent outputs) in shots/ - d2_comparison.png, d2_real_comparison.png.
Real heavy landing page (HTML5 UP "Stellar", full template with parallax/animations, CCA 3.0). Clean template has zero horizontal overflow at any width. Planted one realistic state-dependent bug - an inline min-width: 720px on the stats row that renders perfectly on desktop but forces ~385px of horizontal overflow on mobile (375px). Scored by heavy-score.mjs (overflow across 1280/1024/768/414/375), discovery mode, n=4:
| condition | overflow fixed | mean cost |
|---|---|---|
| base | 4 / 4 | ~32k tok, 4-5 tools |
| ocd | 4 / 4 | ~77k tok, 12-21 tools |
Ceiling - both found it, on a real cluttered page. The reason matters: this bug was an inline style, a visible source smell, so base spotted it by reading even buried in a heavy template. Contrast d2, whose bug is an empty .count {} rule - nothing to see in source, only behavior across states. Screenshots: shots/heavy_buggy_375.png (stats overflow the viewport) vs shots/heavy_fixed_375.png (stats stack cleanly).
Refined conclusion: the discriminator is not page heaviness or bug difficulty - it is source-visibility. A bug base can read (inline style, odd value, box-model math), it fixes with or without the skill, even amid real-page clutter. A bug base cannot read - one that exists only as behavior in a non-default state (d2's shift) - base misses and the skill's "simulate every state" catches. That is the narrow, real slice where the skill earns its 2-3x cost.
- Small n. d2 is now 16 runs (8+8) and holds 0/8 vs 8/8; other tasks are still n=2. Solid for d2, suggestive elsewhere.
- Self-authored corpus, scored objectively. The d2 result is only as good as the claim that an empty-rule layout shift is a fair, representative "invisible-in-source" bug (it is a common real one).
- Opus only. On a weaker model the gap is plausibly wider on every round, not just discovery.
- The tail. Even where the mean ties, the skill may cut the rare miss.
- Cost scales with the skill's effort. It rendered/measured far more (up to ~100k tokens, 20+ tool calls on the heavy page) - 2-3x baseline - whether or not that changed the outcome.
In shots/ (generated by shots.mjs): d2_comparison.png (fixture before/after), d2_real_comparison.png (real base-missed vs ocd-fixed agent outputs), heavy_buggy_375.png / heavy_fixed_375.png (real landing-page stats overflowing vs stacked on mobile).
The heavy page (HTML5 UP "Stellar", CCA 3.0) is fetched to /tmp, not vendored: curl -sSL https://html5up.net/stellar/download -o s.zip && unzip s.zip, then plant min-width:720px on <ul class="statistics">.
Honest one-liner: the ocd skill shows no lift when the bug is named or readable in source - even buried in a real heavy landing page - and costs 2-3x more. Its one robust win is discovery of a source-invisible, state-dependent bug: d2 detection base 0/8 vs ocd 8/8. Its value concentrates on the bug you can only find by simulating a state the default render hides.
- n=2 per cell - indicative, not significant. Wide CIs.
- Ceiling - needs harder, subtler bugs where a strong-but-lazy baseline actually cuts a corner (a defect that only shows in one uncommon state; an anchor that looks right for the sampled values but drifts at an unsampled one; a caller two hops away). That is where a with/without gap could appear, if one exists.
- Self-authored corpus - written by the same model family being tested. Mitigated by the objective script scorer, not eliminated.
- Marginal-lift framing - this measures the skill on top of an already-strong base model, not against a naive one.
node selftest.mjs # validate the code scorer (v1 + v2)
node visual-selftest.mjs # validate the render+measure scorer (v3)
# seed trial dirs (<task>__<cond>__t<trial>/mod.{mjs,html}), one buggy copy each, run agents
node score.mjs /tmp/ablation-run # code tasks -> {fixed, regression} per run
node visual-score.mjs /tmp/ablation-v3 # visual tasks (headless browser)Requires a Chrome/Chromium for the visual scorer (reuses the wideshot heuristic test's puppeteer-core + browser).