test: eval + bench for wideshot heuristic, ocd skill ablation study

README.md

@@ -41,6 +41,25 @@ Code invariants:
 - Tests and typecheck green, one runnable check for new logic
 - No dead code, stale comments, or debug residue
 
+## Does it actually help? (measured)
+
+Fair skepticism: a strong model already fixes plenty of UI bugs on its own.
+So the skill was A/B tested - the same audit run with it and without it, scored by a script that renders the output and measures pixels, not by opinion.
+
+The headline result is on the exact bug class this plugin exists for: a layout shift that only appears in a state the default view never shows.
+An agent is handed a toolbar and told only "audit this" - the bug is never named - across 8 trials per condition:
+
+![ocd vs baseline on a hidden-state layout shift](eval/ablation/shots/d2_real_comparison.png)
+
+- **Without the skill: 0/8 found it.** Every run fixed a *different*, visible issue (the buttons' font) and never simulated the changing count, so the shift sailed through.
+- **With the skill: 8/8 found it.** Its "simulate every state" rule renders the count at 3 and at 9999, sees the buttons drift, and reserves the width.
+
+That is the skill doing the one thing a fast read cannot: catching the bug you only see when you render a state that isn't on screen.
+
+Honest scope, because the same study measured it: when a bug is named, visible on sight, or readable in the source - even buried in a real, heavy landing page - a strong base model already fixes it, and the skill adds no correctness there while costing 2-3x more tokens.
+Its edge is specifically the source-invisible, state-dependent bug.
+Full write-up, all five rounds, both render-and-measure scorers, and every number including the ties: [`eval/ablation`](eval/ablation).
+
 ## Install
 
 As a plugin (recommended, gets updates):
@@ -88,6 +107,25 @@ The natural move after changing a button is to screenshot the button - a frame i
 It uses whatever browser tooling the session already has, or falls back to a bundled ~50-line puppeteer-core script that reuses your installed Chrome (no browser download, <2s per shot).
 The heuristic handles shadow DOM, inner scroll containers, `display:contents`, and `position:fixed`, and returns why it picked the frame.
 
+### Verified, not asserted
+
+Those handling claims are backed by an eval + bench, not just prose - fittingly, given the skill's own "measurements, not impressions" rule.
+The suite lives in [`skills/wideshot/scripts/test`](skills/wideshot/scripts/test) (12 correctness cases in real headless Chrome, plus a perf bench): `npm test`, `npm run bench`.
+
+The heuristic is a *container-selection* step that runs before a screenshot, so the fair comparison is against the framing other tools give you.
+Measured on one engine (puppeteer-core + Chrome), same page and target, so the only variable is what gets framed (Apple M5, arm64 - ratios matter, not absolute ms):
+
+| framing strategy | time (median) | output | note |
+|------------------|---------------|--------|------|
+| element-only (Playwright / shot-scraper / capture-website `--element`) | 25.1 ms | 40 x 30 px | no siblings -> alignment/shift unverifiable |
+| **wideshot** | 24.5 ms | 920 x 50 px | frames the pane, same speed as element-only |
+| full-page | 41.9 ms | 1280 x 4400 px | captures everything, 1.7x slower, unfocused |
+
+The heuristic itself adds < 0.1 ms (p95), below the `performance.now()` floor.
+So wideshot costs the same as a plain element shot but returns a frame you can actually verify in - where element-only deleted the siblings and full-page buried the change.
+Every other tool screenshots exactly the selector you hand it or the whole page; only wideshot walks to the meaningful container.
+Full matrix and how-to in the [eval README](skills/wideshot/scripts/test/README.md).
+
 ## Structure
 
 ```
@@ -98,6 +136,7 @@ skills/
   ocd/SKILL.md               the audit-and-fix pass
   wideshot/SKILL.md          context-framed screenshots for visual verification
   wideshot/scripts/          container heuristic + fallback capture tool
+  wideshot/scripts/test/     eval (12 cases) + perf bench for the heuristic
 ```
 
 Skills under `skills/` are auto-discovered by the plugin.

eval/ablation/README.md

@@ -0,0 +1,129 @@
+# ocd skill ablation
+
+Does the ocd *skill* (a prompt) change what Claude produces? The wideshot eval proves the *tool* works; this asks a different question about the *skill*, and answers it with a controlled A/B, scored by a deterministic script - not by model judgment.
+
+## Method
+
+- **Corpus** (`tasks/`): 3 single-file bugs with objective ground truth a script can measure.
+  - `t1` - layout shift: a TUI progress row whose `%` label isn't padded, so the `|` column drifts as the percent widens.
+  - `t2` - drift: a caret marker at a hardcoded column instead of one computed from the count's width.
+  - `t3` - root cause vs symptom: `pct()` divides by zero on an empty list; the shared bug also reaches `bar()`. A symptom-only patch of `summary()` leaves the sibling caller broken.
+  - `a1`/`a2`/`a3` - adversarial versions: clamp both ends (not just >100), one shared bug reached by 3 named-only-once callers, column width from max (not the first row).
+  - `va`/`vb`/`vc` - visual (HTML), scored by rendered bounding-rects: layout shift, box-model width trap, off-center card.
+  - `d1`/`d2`/`d3` - discovery (HTML): same measurement, but the bug is NEVER named to the agent ("audit this, may or may not have issues"). Tests detection, not fixing.
+- **Conditions**: `base` = agent gets only the symptom + "no slash-command skills"; `ocd` = agent told to read and follow `skills/ocd/SKILL.md`. Same tools, same model. The only variable is the method.
+- **Trials**: 2 per (task x condition) = 12 runs. Symptom prompts state the symptom only, never the full check the scorer applies (so a lazy fix that only addresses the named case fails).
+- **Score** (`score.mjs`): imports each edited module and measures `{fixed, regression}` across all states / callers. `fixed` = defect gone everywhere, normal cases intact. `regression` = new breakage OR a symptom-only patch that left a sibling caller broken.
+- **Scorer is self-tested first** (`selftest.mjs`): buggy -> not fixed; reference fixes -> fixed/no-regression; a symptom-only patch -> fixed BUT regression. A wrong scorer would invalidate everything, so it is validated before any agent runs.
+
+## Results
+
+### Round 1 - easy tasks (`t1`-`t3`, n=2)
+
+| condition | fixed | regressions | mean tokens | mean tool-uses |
+|-----------|-------|-------------|-------------|----------------|
+| base      | 6 / 6 | 0           | ~26.8k      | 2.0            |
+| ocd       | 6 / 6 | 0           | ~30.5k      | 4.7            |
+
+Ceiling effect. Baseline `t3` runs guarded the shared `pct()` (root cause), not just `summary()` (symptom) - the exact discipline the skill teaches, applied unprompted.
+
+### Round 2 - adversarial tasks (`a1`-`a3`, n=2)
+
+Tasks redesigned so the lazy fix passes the *named* case but fails a deeper check: clamp both ends (not one), trace 3 callers (not the one named), size to max (not a sampled row).
+
+| condition | fixed | regressions | mean tokens | mean tool-uses |
+|-----------|-------|-------------|-------------|----------------|
+| base      | 6 / 6 | 0           | ~27.0k      | 2.2            |
+| ocd       | 6 / 6 | 0           | ~30.7k      | 4.7            |
+
+Same ceiling. Baseline clamped **both** ends (`Math.max(0, Math.min(10, ...||0))`), sized to `Math.max` over all rows, and guarded the shared `fmtPct` - every trap, unprompted.
+
+### Round 3 - visual tasks, scored by headless-browser rects (`va`-`vc`, n=2)
+
+The skill's home turf: bugs invisible-or-misleading in source, measurable only by rendering. `va` layout shift (a `%` label with no reserved width moves the Cancel button); `vb` box-model trap (two cards both `width:320px`, one `border-box`, render 38px apart); `vc` off-center card (hardcoded `left:120px`, centered needs 150). Scored by `visual-score.mjs` (puppeteer bounding-rects).
+
+| condition | fixed | regressions | mean tokens | mean tool-uses |
+|-----------|-------|-------------|-------------|----------------|
+| base      | 6 / 6 | 0           | ~26.9k      | 2.2            |
+| ocd       | 6 / 6 | 0           | ~34.3k      | 8.3            |
+
+Same outcome - but here the **process** finally diverged: ocd-condition agents actually **rendered and measured** (headless CDP - "button left edge holds at 246.53px", "verified A=B=320"), while base agents reasoned from source in 2 tool-uses. The skill did exactly what it promises (measure, don't eyeball). The base model's source reasoning simply reached the same correct fix without rendering - so measuring was right but redundant, at +27% tokens and ~4x the tool calls.
+
+### Round 4 - DISCOVERY: bug not named (`d1`-`d3`, n=2)
+
+The decisive change: the agent is told only "audit this, it may or may not have issues" - the defect is never named. This tests *detection*, the skill's actual headline. Planted defects that don't scream in source: `d1` one stat card silently wider (padding shorthand), `d2` a counter with no reserved width **that looks perfect at its default value and only shifts siblings when the number grows**, `d3` a flex-centered modal nudged off-center by a stray `margin-left`.
+
+| task | base found | ocd found | note |
+|------|-----------|-----------|------|
+| d1 (card bulge, visible in one render) | 2/2 | 2/2 | ceiling |
+| **d2 (shift invisible at default state)** | **0/2** | **2/2** | **the signal** |
+| d3 (off-center, visible in one render) | 2/2 | 2/2 | ceiling |
+| **total** | **4/6** | **6/6** | |
+
+**First real difference in the whole study.** On `d2`, base audited what it could see in the default render - it even found and fixed a *different, unplanted* real defect (buttons not inheriting the monospace font) - but never varied the count, so it missed the layout shift entirely, both trials. Verified in the file: base left `.count { }` untouched; ocd added `min-width: 4ch` after rendering across counts. The skill's "simulate the states - layout bugs live in the transitions" discipline is exactly what closed the gap. It cost ~46% more tokens on that task.
+
+`d1`/`d3` stayed at ceiling because those defects are visible in a single default render - no state simulation needed, so base catches them too.
+
+### Bottom line (48 runs, 12 bug types)
+
+- **Fixing a *named* bug (rounds 1-3, code + visual): no measurable difference.** Baseline Opus already clamps both ends, traces callers, sizes to max, reasons box-model. The skill changed the process (made agents render+measure) but not the outcome, at +14-27% cost.
+- **Discovering an *unnamed* bug (round 4): a real, mechanistic gap - but only for defects invisible in the default view.** base 4/6 vs ocd 6/6, the entire delta on the one state-dependent bug. For bugs visible on sight, still ceiling.
+
+So the skill earns its keep in one specific place: **finding the bug that isn't visible until you simulate a state the default render doesn't show.** That is a narrow but genuine slice of what it claims - and the slice where "render it, count the cells, across every state" is doing work a strong model skips when it isn't told to.
+
+### Round 5 - hardening d2 (n=8) + a real heavy page
+
+**d2 rerun at n=8 per condition** (discovery mode, bug never named), split into *detection* (found the shift at all) and *robust fix* (holds at count=9999):
+
+| condition | detected the shift | robust fix (holds @9999) |
+|-----------|--------------------|--------------------------|
+| base      | **0 / 8**          | 0 / 8                    |
+| ocd       | **8 / 8**          | 5 / 8                    |
+
+The 0/8 vs 8/8 detection split holds up. All 8 base runs instead found and fixed a *different* real defect (buttons not inheriting the monospace font) and never simulated the count. 3 of the 8 ocd runs detected the shift but sized the reserve to `3ch`, too small for 4-digit counts - detection is the robust signal; fix-completeness varies. Before/after screenshots (fixture and real agent outputs) in `shots/` - `d2_comparison.png`, `d2_real_comparison.png`.
+
+**Real heavy landing page** (HTML5 UP "Stellar", full template with parallax/animations, CCA 3.0). Clean template has zero horizontal overflow at any width. Planted one realistic state-dependent bug - an inline `min-width: 720px` on the stats row that renders perfectly on desktop but forces ~385px of horizontal overflow on mobile (375px). Scored by `heavy-score.mjs` (overflow across 1280/1024/768/414/375), discovery mode, n=4:
+
+| condition | overflow fixed | mean cost |
+|-----------|----------------|-----------|
+| base      | **4 / 4**      | ~32k tok, 4-5 tools |
+| ocd       | **4 / 4**      | ~77k tok, 12-21 tools |
+
+**Ceiling - both found it, on a real cluttered page.** The reason matters: this bug was an *inline style*, a visible source smell, so base spotted it by reading even buried in a heavy template. Contrast d2, whose bug is an *empty `.count {}` rule* - nothing to see in source, only behavior across states. Screenshots: `shots/heavy_buggy_375.png` (stats overflow the viewport) vs `shots/heavy_fixed_375.png` (stats stack cleanly).
+
+**Refined conclusion: the discriminator is not page heaviness or bug difficulty - it is source-visibility.** A bug base can *read* (inline style, odd value, box-model math), it fixes with or without the skill, even amid real-page clutter. A bug base cannot read - one that exists only as behavior in a non-default state (d2's shift) - base misses and the skill's "simulate every state" catches. That is the narrow, real slice where the skill earns its 2-3x cost.
+
+## Caveats that still stand
+
+- **Small n.** d2 is now 16 runs (8+8) and holds 0/8 vs 8/8; other tasks are still n=2. Solid for d2, suggestive elsewhere.
+- **Self-authored corpus**, scored objectively. The d2 result is only as good as the claim that an empty-rule layout shift is a fair, representative "invisible-in-source" bug (it is a common real one).
+- **Opus only.** On a weaker model the gap is plausibly wider on every round, not just discovery.
+- **The tail.** Even where the mean ties, the skill may cut the rare miss.
+- **Cost scales with the skill's effort.** It rendered/measured far more (up to ~100k tokens, 20+ tool calls on the heavy page) - 2-3x baseline - whether or not that changed the outcome.
+
+## Screenshots
+
+In `shots/` (generated by `shots.mjs`): `d2_comparison.png` (fixture before/after), `d2_real_comparison.png` (real base-missed vs ocd-fixed agent outputs), `heavy_buggy_375.png` / `heavy_fixed_375.png` (real landing-page stats overflowing vs stacked on mobile).
+
+The heavy page (HTML5 UP "Stellar", CCA 3.0) is fetched to `/tmp`, not vendored: `curl -sSL https://html5up.net/stellar/download -o s.zip && unzip s.zip`, then plant `min-width:720px` on `<ul class="statistics">`.
+
+Honest one-liner: **the ocd skill shows no lift when the bug is named or readable in source - even buried in a real heavy landing page - and costs 2-3x more. Its one robust win is discovery of a source-invisible, state-dependent bug: d2 detection base 0/8 vs ocd 8/8. Its value concentrates on the bug you can only find by simulating a state the default render hides.**
+
+## Honest limitations
+
+- **n=2 per cell** - indicative, not significant. Wide CIs.
+- **Ceiling** - needs harder, subtler bugs where a strong-but-lazy baseline actually cuts a corner (a defect that only shows in one uncommon state; an anchor that looks right for the sampled values but drifts at an unsampled one; a caller two hops away). That is where a with/without gap could appear, if one exists.
+- **Self-authored corpus** - written by the same model family being tested. Mitigated by the objective script scorer, not eliminated.
+- **Marginal-lift framing** - this measures the skill on top of an already-strong base model, not against a naive one.
+
+## Run it
+
+```bash
+node selftest.mjs                    # validate the code scorer (v1 + v2)
+node visual-selftest.mjs             # validate the render+measure scorer (v3)
+# seed trial dirs (<task>__<cond>__t<trial>/mod.{mjs,html}), one buggy copy each, run agents
+node score.mjs        /tmp/ablation-run   # code tasks  -> {fixed, regression} per run
+node visual-score.mjs /tmp/ablation-v3    # visual tasks (headless browser)
+```
+
+Requires a Chrome/Chromium for the visual scorer (reuses the wideshot heuristic test's puppeteer-core + browser).

eval/ablation/fixtures/dfixedD1.html

@@ -0,0 +1,15 @@
+<!doctype html>
+<html><head><meta charset="utf-8"><style>
+  body { margin: 0; font: 14px sans-serif; padding: 20px; }
+  .stats { display: flex; gap: 16px; }
+  .card { width: 160px; padding: 16px; border: 2px solid #ccc; border-radius: 8px; background: #fafafa; }
+  .card.mid { padding: 16px; }
+  .num { font-size: 24px; font-weight: 600; }
+  .lbl { color: #888; font-size: 12px; }
+</style></head><body>
+  <div class="stats">
+    <div class="card" id="c1"><div class="num">42</div><div class="lbl">Active</div></div>
+    <div class="card mid" id="c2"><div class="num">7</div><div class="lbl">Pending</div></div>
+    <div class="card" id="c3"><div class="num">128</div><div class="lbl">Done</div></div>
+  </div>
+</body></html>

eval/ablation/fixtures/dfixedD2.html

@@ -0,0 +1,21 @@
+<!doctype html>
+<html><head><meta charset="utf-8"><style>
+  body { margin: 0; font: 14px monospace; padding: 20px; }
+  .toolbar { display: inline-flex; align-items: center; gap: 12px; border: 1px solid #ddd; padding: 8px 12px; border-radius: 6px; }
+  .count { display: inline-block; min-width: 4ch; text-align: right; }
+  .actions { display: flex; gap: 8px; }
+  .btn { border: 1px solid #333; background: #fff; padding: 4px 8px; border-radius: 4px; }
+</style></head><body>
+  <div class="toolbar">
+    <span>Items</span>
+    <span class="count" id="count">3</span>
+    <div class="actions">
+      <button class="btn" id="clear">Clear</button>
+      <button class="btn">Export</button>
+    </div>
+  </div>
+  <script>
+    const c = new URLSearchParams(location.search).get('count') || '3';
+    document.getElementById('count').textContent = c;
+  </script>
+</body></html>

eval/ablation/fixtures/dfixedD3.html

@@ -0,0 +1,15 @@
+<!doctype html>
+<html><head><meta charset="utf-8"><style>
+  body { margin: 0; font: 14px sans-serif; }
+  .overlay { position: fixed; inset: 0; background: rgba(0,0,0,.3); display: flex; align-items: center; justify-content: center; }
+  .modal { width: 320px; background: #fff; border-radius: 8px; padding: 24px; box-shadow: 0 8px 24px rgba(0,0,0,.2); }
+  h2 { margin: 0 0 8px; font-size: 18px; }
+  p { margin: 0; color: #555; }
+</style></head><body>
+  <div class="overlay" id="overlay">
+    <div class="modal" id="modal">
+      <h2>Delete project?</h2>
+      <p>This action cannot be undone.</p>
+    </div>
+  </div>
+</body></html>

eval/ablation/fixtures/fixed1.mjs

@@ -0,0 +1,5 @@
+export function renderRow(pct, status) {
+  const filled = Math.round(pct / 10);
+  const bar = '#'.repeat(filled) + '.'.repeat(10 - filled);
+  return `[${bar}] ${String(pct).padStart(3)}% | ${status}`;
+}

eval/ablation/fixtures/fixed2.mjs

@@ -0,0 +1,6 @@
+export function renderStats(count) {
+  const line = `Total: ${count} items`;
+  const caretCol = 'Total: '.length + String(count).length;
+  const marker = ' '.repeat(caretCol) + '^';
+  return line + '\n' + marker;
+}

eval/ablation/fixtures/fixed3.mjs

@@ -0,0 +1,11 @@
+export function pct(done, total) {
+  if (total === 0) return 0;
+  return Math.round((done / total) * 100);
+}
+export function bar(done, total) {
+  const f = Math.round(pct(done, total) / 10);
+  return '#'.repeat(f) + '.'.repeat(10 - f);
+}
+export function summary(done, total) {
+  return `${pct(done, total)}% ${bar(done, total)}`;
+}

eval/ablation/fixtures/fixedA1.mjs

@@ -0,0 +1,4 @@
+export function bar(pct) {
+  const f = Math.max(0, Math.min(10, Math.round(pct / 10)));
+  return '#'.repeat(f) + '.'.repeat(10 - f);
+}

eval/ablation/fixtures/fixedA2.mjs

@@ -0,0 +1,13 @@
+export function fmtPct(done, total) {
+  if (total === 0) return '0%';
+  return Math.round((done / total) * 100) + '%';
+}
+export function statusLine(t) {
+  return `${t.name}: ${fmtPct(t.done, t.total)}`;
+}
+export function tooltip(t) {
+  return `${fmtPct(t.done, t.total)} complete`;
+}
+export function ariaLabel(t) {
+  return `progress ${fmtPct(t.done, t.total)}`;
+}

eval/ablation/fixtures/fixedA3.mjs

@@ -0,0 +1,4 @@
+export function table(counts) {
+  const w = Math.max(...counts.map(c => String(c).length));
+  return counts.map(c => `item ${String(c).padStart(w)}`).join('\n');
+}