e2e: differential testing harness for public extensions - #7
Merged
Conversation
Also repo hygiene: remove committed test_transform binary, add .gitmodules for LatexToCalc submodule, fix dead README link, reconcile API stat inconsistencies, trim doc boilerplate
Capture raw tabs.query/tabs.sendMessage before the wrap walk so the shim's own background ping-relay polling never appears in traces (mirrors the rawFetch pattern). Preserve removeListener/hasListener on wrapped event objects via a WeakMap from the caller's original callback to the wrapped one, so removeListener(origCb) actually works post-shim instead of silently no-opping. Covering tests added to shim.test.ts for both; confirmed red on the pre-fix shim.js and green after.
Wires corpus/probes/convert/run together and runs the differential harness against real extensions (LatexToCalc, OneNote Web Clipper) for the first time, fixing what that exposed: - Background-context ctx labels now collapse to a canonical "background" on both sides via a shim ctx-override baked in at injection time, instead of the raw (and browser-specific) background file name — Chrome's MV3 service worker and Firefox's converted event-page background were tagged with different ctx strings, so diffTraces' per-ctx LCS bucketing never matched them up and every background event looked like a divergence. - normalizeTrace strips the `temporary` key, which only ever appears because the Firefox driver installs via installAddon(path, true) for testing; a real signed install never sets it. - chromeDriver/firefoxDriver launch failures now close the context/quit the driver before rethrowing, so a partial launch failure doesn't leak a browser process across the corpus loop. Both corpus entries pass with 0 unallowed divergences; OneNote's allowed_diffs are traced to two documented root causes (an unguarded Chrome-only offscreen API call in OneNote's own source — filed as #2 — and native tabs.onUpdated event-count variance across browsers), not accepted on faith.
Chrome: headless:true alone never surfaced the extension's service
worker within the wait timeout. --headless=new passed as an explicit
arg (with headless left false so Playwright doesn't also inject its
own headless flag) starts the service worker reliably — verified via
E2E_INTEGRATION=1 drivers.integration.test.ts and a full corpus run,
both headless and producing identical results to the headed runs.
Firefox: opts.addArguments("-headless") — temp add-on install and
prefs.js uuid recovery both work unchanged headless.
Review found the tabs.* allowed_diffs entries were papering over real
normalizer gaps rather than fixing them. Fixes, in diff.ts's
normalizeTrace:
- tabs.on*:fired events carry ids positionally (e.g. onUpdated's
tabId, changeInfo, tab), not under an ID_KEY-named object key, so
they never normalized across sides. Now mapped through the same
mapId as object-keyed ids.
- Numeric timestamps (Tab.lastAccessed) were never scrubbed --
EPOCH/ISO only matched strings. Added isEpochLike() to catch
10/13-digit (optionally fractional) numbers.
- The Chrome-vs-Firefox native tabs.Tab shape difference is now fixed
at the source: any Tab-shaped object is projected down to
{url, title, status, index, active} instead of being allowlisted
wholesale per API.
allowed_diffs also gains an "<api-glob>#<substring>" form (isAllowed
now takes the whole event, not just the api string), so runtime.error,
net.fetch, and runtime.sendMessage in corpus.json are pinned to the
one call site each was actually triaged against, instead of
allowlisting every call to that API.
probes.ts: contentProbe now checks whether content_scripts.matches
actually covers the fixture origin before reporting "ran" (OneNote's
only match onenote.officeapps.live.com and was injecting nothing);
pingProbe now fails the entry on chrome/firefox asymmetry and reports
"skipped" (not "ran") when neither side's content script answers.
run.ts factors probe failures into report.pass and prints probe notes
in the summary line.
Scoped the `temporary`-key strip to runtime.onInstalled:fired only,
not any key at any depth of any event.
Re-ran the corpus with the four now-unnecessary tabs.* allowed_diffs
entries deleted to see what was empirically still needed: two
genuinely distinct, evidence-backed residuals remain (an offscreen-
cascade tour tab that Firefox never creates at all, and a real
browser-native event-shape/timing difference on the harness's own
probe-driven tab -- Firefox fires Reader-Mode-only changeInfo updates
Chrome has no equivalent for). Both extensions PASS with 0 unallowed
divergences on repeated headless runs, no retries needed.
Six focused follow-ups from re-review: - diff.ts mapId: frameId:0 / tabId:-1 are WebExtensions spec sentinels, not dynamic ids -- remapping them consumed an <id:N> slot and skewed every real id after. Left as literal 0/-1 (already directly comparable across browsers). - diff.ts isEpochLike: bare 10-digit integers (e.g. a real Chrome tab id, 1141107017) are indistinguishable from a 10-digit seconds-epoch timestamp. Tightened to only scrub an exact 13-digit bare integer, or a fractional number in the 10-13-digit magnitude range (no legitimate id is ever fractional). String-form EPOCH regex unchanged. - diff.ts: corrected a stale comment claiming favIconUrl was part of the Tab projection; it isn't. - probes.ts pingProbe: now checks contentScriptsCoverUrl before opening any pages (same check contentProbe uses), instead of opening pages and only recognizing "no content script" after the fact. For OneNote this removes the harness's own fixture-tab opens from its trace entirely. - probes.ts: exported matchPatternCoversUrl, added tests/probes.test.ts covering scheme-wildcard scoping, *.host subdomain matching without over-matching lookalike domains, path globbing, and exact-host matching. Re-ran the corpus after the pingProbe fix to re-derive what's empirically still needed (not guessed): OneNote's residual dropped from 22 to 8 unallowed divergences, all chrome-only and all part of the single offscreen-cascade tour tab. Re-pinned with tour-tab substrings where the args were distinctive (tabs.create#getting-started, tabs.onUpdated:fired#getting-started, tabs.query#lastFocusedWindow); tabs.create:resolve and tabs.onCreated:fired left name-only with an explicit note that their args are empty (url/title not yet set) at fire/resolve time. Corrected the tabs.onUpdated:fired _note, which had misattributed most of the (now-eliminated) fixture-tab-source divergences to a permanent engine difference that was real but on the wrong source. Both extensions PASS with 0 unallowed divergences on two consecutive headless runs, no retries needed.
…ics, gates) - run.ts fails the extension (unless quarantined) when either trace is empty or nothing matched across all contexts, reported distinctly (vacuous/vacuityReason in report.json, console line, chrome=N firefox=M matched=K); assessVacuity + countMatchedEvents added to diff.ts with unit tests - --only with no matching corpus id now prints known ids and exits 1 - add typecheck script + CI step, drop --passWithNoTests now that vacuity tests exist - preserve the first attempt's report/traces/screenshots under attempt1/ before the retry's rmSync wipes them, so CI-uploaded results still carry forensics for a reproduced failure - quarantine the OneNote corpus entry (no fixture host for its content_scripts origin yet, tracked in chrome2moz#5); still runs and reports - settle 600ms before closing browsers so the shim's timer-based flush lands the trace tail - file chrome2moz#6 for the shim-ordering issue (spy shim runs before the converter's own compat shims) and reference it from the spec's Known limits section
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
A differential e2e testing harness (
e2e/) that verifies converted extensions behave equivalently in Firefox to the originals in Chrome:chrome.*/browser.*API call to a local telemetry serverallowed_diffssupportsapi#substringpinning so expected divergence is scoped to investigated call sites, with per-pattern rationale incorpus.jsonCI (
.github/workflows/e2e.yml) runs typecheck + unit tests + the full corpus on every PR.Found along the way
chrome.offscreen/getContextscall (Extensions using chrome.offscreen without feature-detection break entirely on Firefox (e.g. OneNote Web Clipper) #2)commandsin either browser (spike-verified) — commands probe ships as skippedFollow-ups filed
#3 (Plan 2: web snapshot corpus + record/replay), #4 (Plan 3: three-way baseline, README badge, coverage, clipboard), #5 (harness backlog), #6 (shim ordering)
Design spec:
docs/superpowers/specs/2026-07-29-e2e-differential-testing-design.md(kept in sync with shipped behavior). Also includes repo hygiene: removed a committed 5.7MB test binary, added.gitmodulesfor LatexToCalc, fixed stat inconsistencies and a dead README link.First-CI-run watchlist