From 9902e4afb68438ec27f58446ad7a86d44bb20ea2 Mon Sep 17 00:00:00 2001 From: Jerry Xiao Date: Sat, 4 Jul 2026 15:46:27 -0700 Subject: [PATCH] test: add wideshot heuristic eval/bench and ocd skill ablation - skills/wideshot/scripts/test: 12-case correctness eval + perf bench for pick-container.js, run in real headless Chrome (jsdom has no layout engine, so its rects are 0x0 and every geometric assertion would be vacuous). - eval/ablation: controlled A/B study of the ocd skill vs baseline, scored by deterministic render-and-measure scripts, not model judgment. 5 rounds, 48+ subagent runs. Robust finding: no lift when a bug is named or readable in source (even on a real heavy landing page), 2-3x cost; the one real win is discovery of a source-invisible, state-dependent layout shift (base 0/8 vs ocd 8/8), with before/after screenshots. - README: wideshot perf/comparison table and a "Does it actually help? (measured)" section presenting the ablation evidence with honest scope. --- README.md | 39 + eval/ablation/README.md | 129 +++ eval/ablation/fixtures/dfixedD1.html | 15 + eval/ablation/fixtures/dfixedD2.html | 21 + eval/ablation/fixtures/dfixedD3.html | 15 + eval/ablation/fixtures/fixed1.mjs | 5 + eval/ablation/fixtures/fixed2.mjs | 6 + eval/ablation/fixtures/fixed3.mjs | 11 + eval/ablation/fixtures/fixedA1.mjs | 4 + eval/ablation/fixtures/fixedA2.mjs | 13 + eval/ablation/fixtures/fixedA3.mjs | 4 + eval/ablation/fixtures/symptom3.mjs | 13 + eval/ablation/fixtures/symptomA1.mjs | 6 + eval/ablation/fixtures/symptomA2.mjs | 15 + eval/ablation/fixtures/vfixedVA.html | 18 + eval/ablation/fixtures/vfixedVB.html | 10 + eval/ablation/fixtures/vfixedVC.html | 10 + eval/ablation/fixtures/wrongA3.mjs | 6 + eval/ablation/heavy-score.mjs | 51 + eval/ablation/results-d2hard.json | 114 ++ eval/ablation/results-heavy.json | 10 + eval/ablation/results-v2.json | 86 ++ eval/ablation/results-v3.json | 86 ++ eval/ablation/results-v4.json | 86 ++ eval/ablation/results.json | 86 ++ eval/ablation/score.mjs | 108 ++ eval/ablation/selftest.mjs | 41 + eval/ablation/shots.mjs | 26 + eval/ablation/shots/d2_buggy_count3.png | Bin 0 -> 5749 bytes eval/ablation/shots/d2_buggy_count9999.png | Bin 0 -> 6456 bytes eval/ablation/shots/d2_compare.html | 25 + eval/ablation/shots/d2_comparison.png | Bin 0 -> 87741 bytes eval/ablation/shots/d2_fixed_count3.png | Bin 0 -> 5929 bytes eval/ablation/shots/d2_fixed_count9999.png | Bin 0 -> 6456 bytes eval/ablation/shots/d2_real_compare.html | 26 + eval/ablation/shots/d2_real_comparison.png | Bin 0 -> 90434 bytes eval/ablation/shots/heavy_buggy_1280.png | Bin 0 -> 156658 bytes eval/ablation/shots/heavy_buggy_375.png | Bin 0 -> 139327 bytes eval/ablation/shots/heavy_fixed_375.png | Bin 0 -> 179236 bytes eval/ablation/shots/real_base_3.png | Bin 0 -> 5950 bytes eval/ablation/shots/real_base_9999.png | Bin 0 -> 6521 bytes eval/ablation/shots/real_ocd_3.png | Bin 0 -> 6056 bytes eval/ablation/shots/real_ocd_9999.png | Bin 0 -> 6521 bytes eval/ablation/tasks/a1.mjs | 7 + eval/ablation/tasks/a2.mjs | 15 + eval/ablation/tasks/a3.mjs | 6 + eval/ablation/tasks/d1.html | 16 + eval/ablation/tasks/d2.html | 22 + eval/ablation/tasks/d3.html | 16 + eval/ablation/tasks/t1.mjs | 7 + eval/ablation/tasks/t2.mjs | 9 + eval/ablation/tasks/t3.mjs | 13 + eval/ablation/tasks/va.html | 20 + eval/ablation/tasks/vb.html | 12 + eval/ablation/tasks/vc.html | 11 + eval/ablation/visual-score.mjs | 124 ++ eval/ablation/visual-selftest.mjs | 25 + skills/wideshot/scripts/test/README.md | 61 + skills/wideshot/scripts/test/bench.mjs | 118 ++ skills/wideshot/scripts/test/harness.mjs | 67 ++ .../wideshot/scripts/test/package-lock.json | 1016 +++++++++++++++++ skills/wideshot/scripts/test/package.json | 13 + .../scripts/test/pick-container.test.mjs | 157 +++ 63 files changed, 2820 insertions(+) create mode 100644 eval/ablation/README.md create mode 100644 eval/ablation/fixtures/dfixedD1.html create mode 100644 eval/ablation/fixtures/dfixedD2.html create mode 100644 eval/ablation/fixtures/dfixedD3.html create mode 100644 eval/ablation/fixtures/fixed1.mjs create mode 100644 eval/ablation/fixtures/fixed2.mjs create mode 100644 eval/ablation/fixtures/fixed3.mjs create mode 100644 eval/ablation/fixtures/fixedA1.mjs create mode 100644 eval/ablation/fixtures/fixedA2.mjs create mode 100644 eval/ablation/fixtures/fixedA3.mjs create mode 100644 eval/ablation/fixtures/symptom3.mjs create mode 100644 eval/ablation/fixtures/symptomA1.mjs create mode 100644 eval/ablation/fixtures/symptomA2.mjs create mode 100644 eval/ablation/fixtures/vfixedVA.html create mode 100644 eval/ablation/fixtures/vfixedVB.html create mode 100644 eval/ablation/fixtures/vfixedVC.html create mode 100644 eval/ablation/fixtures/wrongA3.mjs create mode 100644 eval/ablation/heavy-score.mjs create mode 100644 eval/ablation/results-d2hard.json create mode 100644 eval/ablation/results-heavy.json create mode 100644 eval/ablation/results-v2.json create mode 100644 eval/ablation/results-v3.json create mode 100644 eval/ablation/results-v4.json create mode 100644 eval/ablation/results.json create mode 100644 eval/ablation/score.mjs create mode 100644 eval/ablation/selftest.mjs create mode 100644 eval/ablation/shots.mjs create mode 100644 eval/ablation/shots/d2_buggy_count3.png create mode 100644 eval/ablation/shots/d2_buggy_count9999.png create mode 100644 eval/ablation/shots/d2_compare.html create mode 100644 eval/ablation/shots/d2_comparison.png create mode 100644 eval/ablation/shots/d2_fixed_count3.png create mode 100644 eval/ablation/shots/d2_fixed_count9999.png create mode 100644 eval/ablation/shots/d2_real_compare.html create mode 100644 eval/ablation/shots/d2_real_comparison.png create mode 100644 eval/ablation/shots/heavy_buggy_1280.png create mode 100644 eval/ablation/shots/heavy_buggy_375.png create mode 100644 eval/ablation/shots/heavy_fixed_375.png create mode 100644 eval/ablation/shots/real_base_3.png create mode 100644 eval/ablation/shots/real_base_9999.png create mode 100644 eval/ablation/shots/real_ocd_3.png create mode 100644 eval/ablation/shots/real_ocd_9999.png create mode 100644 eval/ablation/tasks/a1.mjs create mode 100644 eval/ablation/tasks/a2.mjs create mode 100644 eval/ablation/tasks/a3.mjs create mode 100644 eval/ablation/tasks/d1.html create mode 100644 eval/ablation/tasks/d2.html create mode 100644 eval/ablation/tasks/d3.html create mode 100644 eval/ablation/tasks/t1.mjs create mode 100644 eval/ablation/tasks/t2.mjs create mode 100644 eval/ablation/tasks/t3.mjs create mode 100644 eval/ablation/tasks/va.html create mode 100644 eval/ablation/tasks/vb.html create mode 100644 eval/ablation/tasks/vc.html create mode 100644 eval/ablation/visual-score.mjs create mode 100644 eval/ablation/visual-selftest.mjs create mode 100644 skills/wideshot/scripts/test/README.md create mode 100644 skills/wideshot/scripts/test/bench.mjs create mode 100644 skills/wideshot/scripts/test/harness.mjs create mode 100644 skills/wideshot/scripts/test/package-lock.json create mode 100644 skills/wideshot/scripts/test/package.json create mode 100644 skills/wideshot/scripts/test/pick-container.test.mjs diff --git a/README.md b/README.md index e74c037..e4f944b 100644 --- a/README.md +++ b/README.md @@ -41,6 +41,25 @@ Code invariants: - Tests and typecheck green, one runnable check for new logic - No dead code, stale comments, or debug residue +## Does it actually help? (measured) + +Fair skepticism: a strong model already fixes plenty of UI bugs on its own. +So the skill was A/B tested - the same audit run with it and without it, scored by a script that renders the output and measures pixels, not by opinion. + +The headline result is on the exact bug class this plugin exists for: a layout shift that only appears in a state the default view never shows. +An agent is handed a toolbar and told only "audit this" - the bug is never named - across 8 trials per condition: + +![ocd vs baseline on a hidden-state layout shift](eval/ablation/shots/d2_real_comparison.png) + +- **Without the skill: 0/8 found it.** Every run fixed a *different*, visible issue (the buttons' font) and never simulated the changing count, so the shift sailed through. +- **With the skill: 8/8 found it.** Its "simulate every state" rule renders the count at 3 and at 9999, sees the buttons drift, and reserves the width. + +That is the skill doing the one thing a fast read cannot: catching the bug you only see when you render a state that isn't on screen. + +Honest scope, because the same study measured it: when a bug is named, visible on sight, or readable in the source - even buried in a real, heavy landing page - a strong base model already fixes it, and the skill adds no correctness there while costing 2-3x more tokens. +Its edge is specifically the source-invisible, state-dependent bug. +Full write-up, all five rounds, both render-and-measure scorers, and every number including the ties: [`eval/ablation`](eval/ablation). + ## Install As a plugin (recommended, gets updates): @@ -88,6 +107,25 @@ The natural move after changing a button is to screenshot the button - a frame i It uses whatever browser tooling the session already has, or falls back to a bundled ~50-line puppeteer-core script that reuses your installed Chrome (no browser download, <2s per shot). The heuristic handles shadow DOM, inner scroll containers, `display:contents`, and `position:fixed`, and returns why it picked the frame. +### Verified, not asserted + +Those handling claims are backed by an eval + bench, not just prose - fittingly, given the skill's own "measurements, not impressions" rule. +The suite lives in [`skills/wideshot/scripts/test`](skills/wideshot/scripts/test) (12 correctness cases in real headless Chrome, plus a perf bench): `npm test`, `npm run bench`. + +The heuristic is a *container-selection* step that runs before a screenshot, so the fair comparison is against the framing other tools give you. +Measured on one engine (puppeteer-core + Chrome), same page and target, so the only variable is what gets framed (Apple M5, arm64 - ratios matter, not absolute ms): + +| framing strategy | time (median) | output | note | +|------------------|---------------|--------|------| +| element-only (Playwright / shot-scraper / capture-website `--element`) | 25.1 ms | 40 x 30 px | no siblings -> alignment/shift unverifiable | +| **wideshot** | 24.5 ms | 920 x 50 px | frames the pane, same speed as element-only | +| full-page | 41.9 ms | 1280 x 4400 px | captures everything, 1.7x slower, unfocused | + +The heuristic itself adds < 0.1 ms (p95), below the `performance.now()` floor. +So wideshot costs the same as a plain element shot but returns a frame you can actually verify in - where element-only deleted the siblings and full-page buried the change. +Every other tool screenshots exactly the selector you hand it or the whole page; only wideshot walks to the meaningful container. +Full matrix and how-to in the [eval README](skills/wideshot/scripts/test/README.md). + ## Structure ``` @@ -98,6 +136,7 @@ skills/ ocd/SKILL.md the audit-and-fix pass wideshot/SKILL.md context-framed screenshots for visual verification wideshot/scripts/ container heuristic + fallback capture tool + wideshot/scripts/test/ eval (12 cases) + perf bench for the heuristic ``` Skills under `skills/` are auto-discovered by the plugin. diff --git a/eval/ablation/README.md b/eval/ablation/README.md new file mode 100644 index 0000000..d48ce65 --- /dev/null +++ b/eval/ablation/README.md @@ -0,0 +1,129 @@ +# ocd skill ablation + +Does the ocd *skill* (a prompt) change what Claude produces? The wideshot eval proves the *tool* works; this asks a different question about the *skill*, and answers it with a controlled A/B, scored by a deterministic script - not by model judgment. + +## Method + +- **Corpus** (`tasks/`): 3 single-file bugs with objective ground truth a script can measure. + - `t1` - layout shift: a TUI progress row whose `%` label isn't padded, so the `|` column drifts as the percent widens. + - `t2` - drift: a caret marker at a hardcoded column instead of one computed from the count's width. + - `t3` - root cause vs symptom: `pct()` divides by zero on an empty list; the shared bug also reaches `bar()`. A symptom-only patch of `summary()` leaves the sibling caller broken. + - `a1`/`a2`/`a3` - adversarial versions: clamp both ends (not just >100), one shared bug reached by 3 named-only-once callers, column width from max (not the first row). + - `va`/`vb`/`vc` - visual (HTML), scored by rendered bounding-rects: layout shift, box-model width trap, off-center card. + - `d1`/`d2`/`d3` - discovery (HTML): same measurement, but the bug is NEVER named to the agent ("audit this, may or may not have issues"). Tests detection, not fixing. +- **Conditions**: `base` = agent gets only the symptom + "no slash-command skills"; `ocd` = agent told to read and follow `skills/ocd/SKILL.md`. Same tools, same model. The only variable is the method. +- **Trials**: 2 per (task x condition) = 12 runs. Symptom prompts state the symptom only, never the full check the scorer applies (so a lazy fix that only addresses the named case fails). +- **Score** (`score.mjs`): imports each edited module and measures `{fixed, regression}` across all states / callers. `fixed` = defect gone everywhere, normal cases intact. `regression` = new breakage OR a symptom-only patch that left a sibling caller broken. +- **Scorer is self-tested first** (`selftest.mjs`): buggy -> not fixed; reference fixes -> fixed/no-regression; a symptom-only patch -> fixed BUT regression. A wrong scorer would invalidate everything, so it is validated before any agent runs. + +## Results + +### Round 1 - easy tasks (`t1`-`t3`, n=2) + +| condition | fixed | regressions | mean tokens | mean tool-uses | +|-----------|-------|-------------|-------------|----------------| +| base | 6 / 6 | 0 | ~26.8k | 2.0 | +| ocd | 6 / 6 | 0 | ~30.5k | 4.7 | + +Ceiling effect. Baseline `t3` runs guarded the shared `pct()` (root cause), not just `summary()` (symptom) - the exact discipline the skill teaches, applied unprompted. + +### Round 2 - adversarial tasks (`a1`-`a3`, n=2) + +Tasks redesigned so the lazy fix passes the *named* case but fails a deeper check: clamp both ends (not one), trace 3 callers (not the one named), size to max (not a sampled row). + +| condition | fixed | regressions | mean tokens | mean tool-uses | +|-----------|-------|-------------|-------------|----------------| +| base | 6 / 6 | 0 | ~27.0k | 2.2 | +| ocd | 6 / 6 | 0 | ~30.7k | 4.7 | + +Same ceiling. Baseline clamped **both** ends (`Math.max(0, Math.min(10, ...||0))`), sized to `Math.max` over all rows, and guarded the shared `fmtPct` - every trap, unprompted. + +### Round 3 - visual tasks, scored by headless-browser rects (`va`-`vc`, n=2) + +The skill's home turf: bugs invisible-or-misleading in source, measurable only by rendering. `va` layout shift (a `%` label with no reserved width moves the Cancel button); `vb` box-model trap (two cards both `width:320px`, one `border-box`, render 38px apart); `vc` off-center card (hardcoded `left:120px`, centered needs 150). Scored by `visual-score.mjs` (puppeteer bounding-rects). + +| condition | fixed | regressions | mean tokens | mean tool-uses | +|-----------|-------|-------------|-------------|----------------| +| base | 6 / 6 | 0 | ~26.9k | 2.2 | +| ocd | 6 / 6 | 0 | ~34.3k | 8.3 | + +Same outcome - but here the **process** finally diverged: ocd-condition agents actually **rendered and measured** (headless CDP - "button left edge holds at 246.53px", "verified A=B=320"), while base agents reasoned from source in 2 tool-uses. The skill did exactly what it promises (measure, don't eyeball). The base model's source reasoning simply reached the same correct fix without rendering - so measuring was right but redundant, at +27% tokens and ~4x the tool calls. + +### Round 4 - DISCOVERY: bug not named (`d1`-`d3`, n=2) + +The decisive change: the agent is told only "audit this, it may or may not have issues" - the defect is never named. This tests *detection*, the skill's actual headline. Planted defects that don't scream in source: `d1` one stat card silently wider (padding shorthand), `d2` a counter with no reserved width **that looks perfect at its default value and only shifts siblings when the number grows**, `d3` a flex-centered modal nudged off-center by a stray `margin-left`. + +| task | base found | ocd found | note | +|------|-----------|-----------|------| +| d1 (card bulge, visible in one render) | 2/2 | 2/2 | ceiling | +| **d2 (shift invisible at default state)** | **0/2** | **2/2** | **the signal** | +| d3 (off-center, visible in one render) | 2/2 | 2/2 | ceiling | +| **total** | **4/6** | **6/6** | | + +**First real difference in the whole study.** On `d2`, base audited what it could see in the default render - it even found and fixed a *different, unplanted* real defect (buttons not inheriting the monospace font) - but never varied the count, so it missed the layout shift entirely, both trials. Verified in the file: base left `.count { }` untouched; ocd added `min-width: 4ch` after rendering across counts. The skill's "simulate the states - layout bugs live in the transitions" discipline is exactly what closed the gap. It cost ~46% more tokens on that task. + +`d1`/`d3` stayed at ceiling because those defects are visible in a single default render - no state simulation needed, so base catches them too. + +### Bottom line (48 runs, 12 bug types) + +- **Fixing a *named* bug (rounds 1-3, code + visual): no measurable difference.** Baseline Opus already clamps both ends, traces callers, sizes to max, reasons box-model. The skill changed the process (made agents render+measure) but not the outcome, at +14-27% cost. +- **Discovering an *unnamed* bug (round 4): a real, mechanistic gap - but only for defects invisible in the default view.** base 4/6 vs ocd 6/6, the entire delta on the one state-dependent bug. For bugs visible on sight, still ceiling. + +So the skill earns its keep in one specific place: **finding the bug that isn't visible until you simulate a state the default render doesn't show.** That is a narrow but genuine slice of what it claims - and the slice where "render it, count the cells, across every state" is doing work a strong model skips when it isn't told to. + +### Round 5 - hardening d2 (n=8) + a real heavy page + +**d2 rerun at n=8 per condition** (discovery mode, bug never named), split into *detection* (found the shift at all) and *robust fix* (holds at count=9999): + +| condition | detected the shift | robust fix (holds @9999) | +|-----------|--------------------|--------------------------| +| base | **0 / 8** | 0 / 8 | +| ocd | **8 / 8** | 5 / 8 | + +The 0/8 vs 8/8 detection split holds up. All 8 base runs instead found and fixed a *different* real defect (buttons not inheriting the monospace font) and never simulated the count. 3 of the 8 ocd runs detected the shift but sized the reserve to `3ch`, too small for 4-digit counts - detection is the robust signal; fix-completeness varies. Before/after screenshots (fixture and real agent outputs) in `shots/` - `d2_comparison.png`, `d2_real_comparison.png`. + +**Real heavy landing page** (HTML5 UP "Stellar", full template with parallax/animations, CCA 3.0). Clean template has zero horizontal overflow at any width. Planted one realistic state-dependent bug - an inline `min-width: 720px` on the stats row that renders perfectly on desktop but forces ~385px of horizontal overflow on mobile (375px). Scored by `heavy-score.mjs` (overflow across 1280/1024/768/414/375), discovery mode, n=4: + +| condition | overflow fixed | mean cost | +|-----------|----------------|-----------| +| base | **4 / 4** | ~32k tok, 4-5 tools | +| ocd | **4 / 4** | ~77k tok, 12-21 tools | + +**Ceiling - both found it, on a real cluttered page.** The reason matters: this bug was an *inline style*, a visible source smell, so base spotted it by reading even buried in a heavy template. Contrast d2, whose bug is an *empty `.count {}` rule* - nothing to see in source, only behavior across states. Screenshots: `shots/heavy_buggy_375.png` (stats overflow the viewport) vs `shots/heavy_fixed_375.png` (stats stack cleanly). + +**Refined conclusion: the discriminator is not page heaviness or bug difficulty - it is source-visibility.** A bug base can *read* (inline style, odd value, box-model math), it fixes with or without the skill, even amid real-page clutter. A bug base cannot read - one that exists only as behavior in a non-default state (d2's shift) - base misses and the skill's "simulate every state" catches. That is the narrow, real slice where the skill earns its 2-3x cost. + +## Caveats that still stand + +- **Small n.** d2 is now 16 runs (8+8) and holds 0/8 vs 8/8; other tasks are still n=2. Solid for d2, suggestive elsewhere. +- **Self-authored corpus**, scored objectively. The d2 result is only as good as the claim that an empty-rule layout shift is a fair, representative "invisible-in-source" bug (it is a common real one). +- **Opus only.** On a weaker model the gap is plausibly wider on every round, not just discovery. +- **The tail.** Even where the mean ties, the skill may cut the rare miss. +- **Cost scales with the skill's effort.** It rendered/measured far more (up to ~100k tokens, 20+ tool calls on the heavy page) - 2-3x baseline - whether or not that changed the outcome. + +## Screenshots + +In `shots/` (generated by `shots.mjs`): `d2_comparison.png` (fixture before/after), `d2_real_comparison.png` (real base-missed vs ocd-fixed agent outputs), `heavy_buggy_375.png` / `heavy_fixed_375.png` (real landing-page stats overflowing vs stacked on mobile). + +The heavy page (HTML5 UP "Stellar", CCA 3.0) is fetched to `/tmp`, not vendored: `curl -sSL https://html5up.net/stellar/download -o s.zip && unzip s.zip`, then plant `min-width:720px` on `