Performance: Render Bench Harness - #93
Merged
Merged
Conversation
The benchmark suite lived inside the app it was measuring, and the app kept contaminating the measurements. An animating spinner inflated every figure in proportion to DOM size; the results panel rendered inside the measured route and so measured itself; switching fixtures went through the router, which will not re-render the route you are already on, so repeat samples needed an "/idle" route to bounce through and a click handler that navigated could silently pre-empt the sampler and hang a run on sample 1. Those are not bugs to fix one at a time — they follow from where the suite was mounted. It now lives in apps/playgrounds/solid/render-bench, where a fixture is a value rather than a route. Removing the suite leaves createTestStore's `navigate` parameter dead — its only use was the "/idle" bouncing above — so it goes too, along with its argument at the one call site. The Tokens, Mutations, Queries and Routing tests deliberately stay: the environment that corrupted the timings is the same environment that validates those. The principle is recorded in SchemaTests.schema.ts. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ess app packages/schema-system/benchmarks ran headless against happy-dom. That is the right tool for a fast regression filter, but it cannot answer the question the benchmarks were being asked. happy-dom has no layout engine, so paint is invisible, and its JS CSSOM implementation overstates flush by roughly 2.7x versus a real browser — a change that looked like +6% there measured +160% in a browser, because the cost lived in what the per-prop effects went on to do rather than in the walk itself. apps/playgrounds/solid/render-bench keeps both. The headless benchmarks move across unchanged as the fast filter, with their standing explicitly downgraded: a regression there means stop, a win there is only a hypothesis, and no figure from that file is ever published. Alongside them is a browser harness that measures build, flush and paint against a real layout engine and reports the median of five warm samples. What the harness adds over what it replaces: - A ladder of rungs rendering identical output through progressively fewer layers — raw DOM, plain Solid, Solid over design-system components, and the same again binding props as DOM properties rather than attributes. The difference between adjacent rungs is that layer's cost. - tests/ladder.test.tsx, which asserts the rungs stay equivalent. A silent divergence would make every published ratio wrong while still looking plausible, so this runs in CI as a correctness test. - A second ladder on page-shaped content — 50 feed posts, five component types, three levels of nesting — because attribution derived from trivial uniform cards is fairly open to the charge that it does not generalise. - A `minimal` fixture that measures the frame floor rather than leaving it to be guessed at. Paint is bracketed by a double requestAnimationFrame and so cannot report less than one frame interval of waiting. Benchmarks are kept out of CI (vitest.bench.config.ts, separate from vitest.config.ts) — timings on shared runners are noise and must not decide whether a merge is allowed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
One memo per prop looks wasteful for a node whose props are all static literals, and consolidating them into a single shared memo per node is the obvious fix. It was tried and measured slower — roughly +6% headless and worse in a browser, because the per-read indirection and the object allocation it introduced outweighed every memo it removed. That result is invisible in the code, so the next person to read this loop will have the same idea and spend the same time disproving it. Recorded in place, along with the one direction that has not been tried: resolving static props at template-install time. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Prompted by a colleague asking how much more WE costs to run relative to Solid and raw DOM. Answering that honestly needed more than a ratio, because the obvious way to measure it is misleading in both directions. Headline: WE templates cost about 25% more JavaScript work than hand-writing the same page with the same components, and about 3.4x a hand-rolled DOM page that has none of the design system's theming, states, encapsulation or accessibility. Four fifths of the full stack cost is the design system rather than the template layer — so a team weighing adoption is mostly weighing the design system, which they would also pay for by hand. The report leads with the whole-stack comparison, because that is what a reader evaluating adoption actually wants, and treats the design-system/template split as supporting attribution rather than the opening claim. Two things it is careful about: - Roughly 33ms of every total is the frame floor, not work. A double requestAnimationFrame on a 60Hz display spans up to two frame intervals — 33.3ms predicted, 33.2ms measured against a two-element fixture. Totals are therefore compressed and JS work is the figure that scales. An earlier reading of an update burst at "33ms" was this floor; the real cost is 0.2ms. - Every figure is a median of three runs in a clean browser profile, with the hardware stated, run-to-run ranges given rather than single numbers, and the noisiest rows flagged as such. It also records what the measurements do not explain — templates flushing faster than hand-written components, and a property-versus-attribute binding gap that moves with the fixture. A recommendation about the latter was withdrawn once it failed to reproduce on realistic content. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The report claimed the design-system layer "moves a great deal with content", citing 65.0ms on the simple ladder against 36.9ms on the realistic one. That compared raw milliseconds across fixtures with different element counts — 1,201 against 801 — and read a size difference as a content one. Worse, it invited the reasonable reaction that the simpler fixture was somehow costing more. Normalised, the two nearly agree, and the two available normalisations disagree in direction: the simple fixture is 17% more expensive per element and 14% cheaper per prop. That is the signature of no real per-unit effect, the fixtures simply differing in both element count and props per element. Both residuals are inside the 17–27% run-to-run spread of these particular rows. Re-measuring would not have helped — the gap is deterministic, not noise, and reproduces at any sample count. Now states the size caveat before quoting the two totals rather than after, carries element counts in the layer table so the difference is visible at a glance, and adds a normalised per-element/per-prop table. The design-system-to-template ratio is also now given as 5.3x and 3.1x rather than "3–5x", which clipped the top of its own range, and is labelled a property of the pages measured rather than a constant. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…e report Verified every countable claim against the source rather than trusting the prose. Eight were wrong or unclear: - Component-type counts were understated. The realistic fixture uses seven component types and the simple one three, not "five" and "two". Counted from the hand-written controls. - The +25% headline sat directly above a table of the simple fixture, from which a reader divides +19%. The 25% is the realistic feed's figure; the summary now says so before the table rather than leaving the mismatch to be discovered. - Two ways of computing the tax were being mixed silently. Dividing the median JS-work figures gives +32% realistic and +19% simple; the per-run medians quoted throughout give +25% and +16%. Both appear in the document and a reviewer would find the contradiction immediately. Finding 4 now states both, and why the paired per-run figure is the one used — each run compares rungs under the same session and thermal state, whereas dividing medians combines runs and an unstable denominator inflates the result. - The realistic ladder's design-system figure subtracts the simple ladder's 0.8ms Solid baseline, because that ladder has no plain-Solid rung. Small at this size, but it is an approximation and was presented as a measurement. - The normalisation table's prop counts (4,000 / 1,950) and finding 3's binding counts (2,400 / 1,450) describe the same fixtures and were unreconciled. The latter is the subset landing on custom elements as HTML attributes; now said. - "653 call sites" could not be reproduced — the scope it was counted under is lost. Replaced with over 800 we-* call sites, which a grep reproduces. - "78 design-system props" is 79 non-state props, counted from the key arrays with spreads resolved. - "Four fifths" was used for two different quantities: flush as a share of the design system's cost (81%, correct) and the design system as a share of the whole stack (84%, understated). The latter is now given as a percentage. All derived figures re-checked arithmetically and reconciled with the raw-data tables. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
✅ Deploy Preview for coasys-we ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Move the benchmarks out of the app, and publish what they measure
Summary
WE had no trustworthy figure for what its template system costs to render, relative either to Solid
or to raw DOM. Producing one turned out to require fixing the measurement first: the benchmark suite
lived inside the app it was measuring, and the app kept contaminating the results. An animating
spinner inflated every figure in proportion to DOM size. The results panel rendered inside the
measured route and so measured itself. Switching fixtures went through the router — which will not
re-render the route you are already on — so repeat samples needed an
/idleroute to bouncethrough, and a click handler that navigated could silently pre-empt the sampler and hang a run on
its first sample. None of those are bugs to fix one at a time; they follow from where the suite was
mounted.
This branch moves the benchmarks into a standalone app where a fixture is a value rather than a
route, adds a ladder of hand-written controls so each layer's cost can be attributed rather than
guessed at, enforces the ladder's equivalence in CI, and publishes the resulting measurements as
docs/architecture/performance.md. No shipped runtime behaviour changes beyond the removal ofthe in-app benchmark route itself — the one design-system optimisation this work produced was
already merged separately in #92.
The headline: WE templates cost ~25% more JavaScript work than hand-writing the same page with
the same components, and 3.4× a hand-rolled DOM page that has none of the design system's
theming, states, encapsulation or accessibility. 84% of the full-stack cost is the design system
rather than the template layer.
Changes
New —
apps/playgrounds/solid/render-benchA standalone app with no AD4M, no stores, no app shell and no embedded apps. None of those are things
the renderer depends on, and a team adopting WE brings their own shell rather than ours — so the
harness doubles as a portability guard: it cannot build if the renderer or design system ever
acquires an AD4M dependency.
src/runner.ts— phases, median-of-N,timeRender,timeUpdates. Documents at length whythere is no router, because that was the source of most of the old harness's failure modes. Paint
is bracketed by a double
requestAnimationFrameand therefore cannot report less than one frameinterval; the doc comment says so, so the number is not read as work.
src/controls.tsx— the hand-written rungs: raw DOM, plain Solid, Solid over design-systemcomponents, and the same again binding props as DOM properties rather than attributes. That last
rung exists so the comparison is not open to the charge that its control was written badly.
RawDomCardsreturns a real element rather than JSX so construction lands in the measured Buildphase instead of being charged to Mount.
src/fixtures.ts— both ladders. The realistic one (50 feed posts, seven component types,three levels of nesting, per-card varying content) exists because attribution derived from trivial
uniform cards is fairly open to the charge that it does not generalise.
tests/ladder.test.tsx— asserts every rung renders identical content and mounts identicalcustom elements. The schema fixtures and the hand-written controls are separate implementations, so
a silent divergence would invalidate every published ratio while still looking entirely plausible.
Runs in CI as a correctness test.
bench/headless.bench.tsx— the previous package's benchmarks, moved across with theirstanding explicitly downgraded. happy-dom has no layout engine and overstates flush by ~2.7×, so a
regression there means stop, a win there is only a hypothesis, and no figure from that file is ever
published. Kept out of CI via a separate
vitest.bench.config.ts, because timings on sharedrunners are noise and must not decide whether a merge is allowed.
Removed — the in-app benchmark suite
SchemaBenchmark.schema.ts,BenchmarkTimer.tsx, its registry entries, and ~400 lines fromtestStore.ts. Removing the suite leftcreateTestStore'snavigateparameter dead — its only usewas the
/idlebouncing described above — so it goes too, along with its argument at the single callsite, which in turn stranded an unused
shellRouteStorebinding.SchemaTests.schema.tsgains a note recording why the Tokens, Mutations, Queries and Routing testsdeliberately stay in-app: the environment that corrupted the timings is the same environment that
validates those.
Removed —
packages/schema-system/benchmarksSuperseded by the harness above;
setProperty.probe.tsand the tsconfig move across (git tracks bothas renames).
SchemaRenderer.tsx— a measured dead end, recorded in placeComment only. Consolidating the per-prop memos into one shared memo per node is the obvious
optimisation for a node whose props are all static literals. It was tried and measured slower —
~+6% headless, worse in a browser — because the per-read indirection and object allocation outweighed
every memo removed. That result is invisible in the code, so the next reader would have the same idea
and spend the same time disproving it. The one untried direction (resolving static props at
template-install time) is noted alongside.
docs/architecture/performance.mdThe deliverable, indexed in
docs/README.md. Structured to answer the reader's questions in theorder they ask them: how much slower, will users notice, what does it buy, where does the cost go.
Three things it is deliberately careful about:
requestAnimationFrameon a 60 Hzdisplay spans up to two frame intervals — 33.3ms predicted, 33.2ms measured against a two-element
fixture. Totals are therefore compressed and JS work is the figure that scales. An earlier reading
of an update burst at "33ms" was this floor; the real cost is 0.2ms.
medians used throughout give +25% / +16%. Both appear in the document, so it states both and
defends the paired per-run figure rather than leaving a reviewer to find the contradiction.
no comparison against React or Vue, and the design-system rungs — the denominator of every tax
figure — are the least stable rows in the suite.
It also records two findings it cannot explain (templates flush faster than hand-written code; the
prop:advantage appears on one fixture and not the other) and flags them as not citable. Arecommendation to expose
prop:variants in the generated types was withdrawn once it failed toreplicate on realistic content.
Known follow-ups
the simple ladder than the realistic one.
~80non-state design-system props are registered for attribute reflection where roughly sevenneed it. Identified during this work, unimplemented; the largest remaining design-system lever.
tests/ladder.test.tsxcovers therendering contract from outside, but the primitives themselves have no unit tests to extend.
the reason the shell's initial parse is larger than it needs to be.
packages/app-framework(unusedQueryAdapter/ModelClass,one import-sort). Present on
devbefore this branch — confirmed by stashing — and left alonerather than folded into unrelated commits.
Test plan
Verified on this branch:
pnpm -r build— cleanpnpm -r typecheck— clean@we/playground-render-bench— 11 tests passing (ladder equivalence +setProperty)@we/schema-shared— 448 passing@we/schema-solid— 38 passing@we/playground-portable-slice— 7 passingprettier --checkclean on all touched fileseslint— no new errors; the 3 remaining confirmed pre-existing by stashing and re-runningvite build+preview) in an incognito Chrome 148 window, kept focused throughoutratios and differences)
and 3), prop and attribute-binding counts,
we-*call sites, non-state design-system prop countNot verified:
as a limitation rather than implying the ratios travel
remaining test schemas are covered by typecheck and schema validation only