Cross-backend golden-image (visual regression) test harness

[mattleibow]

Fixes #4295

Summary

This PR adds a holistic golden-image (visual regression) test harness to the main test suite. It renders a set of deterministic scenes through every available SkiaSharp backend (renderer) and diffs the result against committed golden PNGs with a strict per-renderer tolerance. Each (renderer × scene) pair is one xUnit theory cell.

The harness lives under tests/Tests/SkiaSharp/Visual/ and runs in-process inside the test runners we already ship — no separate render-host app, no Playwright/adb/simctl orchestration. Because the portable code is linked by the shared SkiaSharp.Tests project, the same matrix compiles and runs in SkiaSharp.Tests.Console (desktop), SkiaSharp.Tests.Devices (MAUI: Android/iOS/Mac Catalyst), and SkiaSharp.Tests.Wasm (browser).

Backends that need an extra NuGet package (Vulkan → SharpVk, Direct3D → Vortice) live in SkiaSharp's existing satellite host projects (SkiaSharp.Vulkan.Tests.Console, SkiaSharp.Direct3D.Tests.Console) instead of the base host, so that dependency never reaches the MAUI/WASM builds. They share the same engine via a small VisualMatrixTestsBase and a ~15-line driver class — see Project structure.

Important

This is a draft handoff. The framework, discipline, seeding tooling, CI provisioning, and the full macOS golden set are done and verified locally. What remains is mechanical and can only be done on CI/other platforms: harvest and commit the non-macOS goldens, and (optionally) add the device/WASM GPU renderers. See Remaining work below.

Why this exists

The in-flight Graphite backend PR (#3968) carries its own ad-hoc visual harness. This PR supersedes that infra with a clean, reusable one on main, so #3968 can rebase and only add its Graphite renderer classes + golden PNGs — no new test/CI/infra. See The Graphite seam.

---

What's in the box

Architecture

ISkiaScene.Draw(canvas) ─▶ IRenderer.RenderAsync ─▶ byte[] RGBA8888/Premul
                                                        │
              emit ##SKIA-GOLDEN-IMAGE## marker (PNG) into the test results (TRX)
                                                        │
   GoldenStore.TryLoad: {renderer}.{platform} ▸ {renderer}
                                                        │
                              SKPixelComparer.Compare(golden, actual, tolerance)
                                                        │
   pass  │  FAIL (out of tolerance, OR unseeded — captured PNG is in the TRX)  │  Skip (backend genuinely absent)

Piece	Type	Role
Scene	ISkiaScene	Deterministic draw op; same bytes every run on a backend
Renderer	IRenderer	Renders a scene through one backend → RGBA8888/premul pixels
Catalogs	SceneCatalog, RendererCatalog	Reflection-discover every public parameterless scene/renderer
Engine	VisualMatrixTestsBase	Shared per-cell pipeline (render → emit → compare-or-fail); reused by every host
Matrix	VisualMatrixTests	[Theory] over the catalog product in the base assembly; calls the engine
Comparison	SKPixelComparer (extended)	Tolerance-aware per-channel diff + colored diff image
Tolerance	GoldenTolerance	Per-renderer + per-(renderer, scene) tolerance
Golden I/O	GoldenStore	Resolves/loads goldens (read-only); encodes captures to PNG

Scenes (5)

DiagonalLines, FilledCircle, RedRoundedRectOnWhite, GradientBlend, Text (uses a bundled Roboto font from tests/Content/fonts — no system fonts, so it's deterministic).

Renderers (4 today)

Renderer	Where it runs	How it gets its context
raster	everywhere (shared)	CPU, always available
ganesh-metal	macOS Console + iOS/Mac Catalyst device hosts (shared, Apple-gated)	in-process Metal
ganesh-gl	Console only (desktop)	reuses the existing tests/Tests/SkiaSharp/GlContexts/ abstraction (CGL/GLX/WGL)
ganesh-vulkan	SkiaSharp.Vulkan.Tests.Console satellite (Linux/Windows)	headless SharpVk; bring-up failure → skip, EntryPoint/MissingMethod → fail

Project structure — base host vs. satellites

The base SkiaSharp.Tests.Console must not take a SharpVk/Vortice dependency: it is the host the MAUI-device and WASM builds consume, and pulling a GPU-vendor package into it would bloat or break those. SkiaSharp already ships dedicated satellite projects for exactly these backends, both already built and run by tests-netcore:

Project	Adds	Visual renderers it hosts
SkiaSharp.Tests.Console (base)	—	raster, ganesh-gl (reuses in-repo GlContexts/), ganesh-metal (Apple-gated)
SkiaSharp.Vulkan.Tests.Console	SharpVk	ganesh-vulkan (+ future graphite-vulkan)
SkiaSharp.Direct3D.Tests.Console	Vortice	direct3d (future)

The shared VisualMatrixTests [Theory] only runs in the assembly that compiles it (the base host), driving the renderers auto-discovered there. A satellite adds a thin VulkanVisualTests : VisualMatrixTestsBase that iterates RendererCatalog.NamesIn(Assembly.GetExecutingAssembly()) × scenes — i.e. only the renderers compiled into that satellite — so raster/GL/Metal are never double-run, and a new Vulkan-family renderer (e.g. Graphite's) joins automatically with no test-class edit. ganesh-gl/ganesh-metal stay in the base host because they need no extra package (GL reuses the existing GlContexts/ abstraction; Metal is in-process P/Invoke).

Golden storage & lookup — two layers, generalizing over platform only

tests/Content/Goldens/
  {renderer}.{platform}/{scene}.png   ← per-platform override (this OS/driver diverges)
  {renderer}/{scene}.png              ← the renderer's golden, shared across platforms

GoldenStore.TryLoad resolves the per-platform path first, then the shared {renderer}/ path. {platform} is a short VisualPlatform.Tag (macos/windows/linux/android/ios/maccatalyst/tvos/browser).

• The fallback never crosses renderers — a missing GPU golden is never satisfied by the CPU baseline (that aliases GPU-vs-CPU antialiasing and hides regressions). The "shared" layer is per renderer, at {renderer}/.
• {renderer}/ is the cross-platform share for the common case (CPU raster + software GL are deterministic across OSes/arches); a genuinely divergent OS/driver gets a {renderer}.{platform}/ override that wins.

Goldens live under tests/Content/Goldens/ so they ride the existing Content pipeline (Console copies them next to the binary; Devices/Wasm embed them as resources). No per-project globbing.

Failure discipline (the property this harness exists to guarantee)

A cell skips only when the backend is genuinely absent — IRenderer.IsAvailable == false, or RenderAsync throws RendererUnavailableException (runtime probe found no device/driver/context). Everything else is a hard failure, including:

• a render that throws anything else (incl. EntryPointNotFoundException/MissingMethodException from a broken binding),
• pixels out of tolerance against a golden that exists,
• an unseeded cell — the backend ran and produced pixels but no golden is committed. This is a fail, not a skip (a green here would be a coverage hole). It's safe to enforce because the captured PNG is already in the TRX to harvest (next section).

There is no path that downgrades a real regression to a skip or a warning.

Seeding goldens — harvest from the test results (TRX)

There is no in-process record mode and no environment variable. Every cell emits its rendered PNG into the test log on pass and fail as a single XML-safe line:

##SKIA-GOLDEN-IMAGE## path={renderer}.{platform}/{scene}.png size=WxH base64=<png bytes>

The TRX is the one output channel that exists uniformly on every host — desktop, MAUI device, and WASM — including the device/browser hosts where the filesystem is sandboxed/embedded and an in-process write-to-source-tree is impossible. So seeding is:

# 1. run with a TRX report (from the test build output dir):
./SkiaSharp.Tests --filter-trait "Category=Visual" --report-trx --report-trx-filename visual.trx
# 2. harvest the markers into tests/Content/Goldens and commit (from repo root):
python3 scripts/infra/tests/extract-visual-goldens.py path/to/visual.trx --dry-run   # preview
python3 scripts/infra/tests/extract-visual-goldens.py path/to/visual.trx             # write
git add tests/Content/Goldens && git commit

A new cell fails as unseeded on its first run; after harvest+commit it compares strictly and goes green. To share a golden across platforms, move byte-identical PNGs up to {renderer}/; the harvest then skips re-creating the per-platform copy whenever an existing {renderer}/ golden is byte-identical, so the promotion sticks.

CI wiring (scripts/azure-templates-stages-test.yml)

• Software GPU on the Linux .NET Core agent: installs xvfb mesa-utils libgl1-mesa-dri mesa-vulkan-drivers vulkan-tools, starts Xvfb, and pins GL/Vulkan to Mesa's software rasterizers (LIBGL_ALWAYS_SOFTWARE=1 + llvmpipe + lavapipe VK_ICD_FILENAMES). This makes ganesh-gl/ganesh-vulkan actually render on a headless agent so their captures can be harvested. Fail-safe: if any piece is missing, context creation throws RendererUnavailableException and the cell skips — a provisioning gap never turns into a red build. (Bonus: the existing GRContextTest/GRGlInterfaceTest GL tests now also exercise llvmpipe.)
• Existing PublishTestResults + the testlogs_* artifacts carry the .trx files back for harvesting — no extra collection step.
• The matrix runs inside the existing stages (tests-netcore, tests-android, tests-ios, tests-maccatalyst, tests-wasm). There is no dedicated visual stage and no tests-visual build target.

---

What's verified vs. not

Verified locally (macOS Console, net10 arm64):

• SkiaSharp.Tests.Console + shared SkiaSharp.Tests.csproj build clean.
• Base SkiaSharp.Tests.Console matrix → 15 pass / 0 skip (raster + ganesh-gl + ganesh-metal × 5 scenes).
• SkiaSharp.Vulkan.Tests.Console satellite → 5 ganesh-vulkan cells skip cleanly (no Vulkan ICD on the box), driven by the thin VulkanVisualTests. --filter-trait "Category=Visual" works both directions in both hosts.
• extract-visual-goldens.py against a real TRX: all 15 markers parsed with correct two-layer paths; harvest is byte-idempotent (re-harvest → git status clean); the byte-aware shared-skip works (a promoted raster/FilledCircle.png makes the per-platform copy skip while GPU cells with different bytes still write).

NOT verifiable on this macOS box (needs CI / other platforms):

• Linux Mesa/lavapipe provisioning actually lighting up ganesh-gl/ganesh-vulkan.
• Whether a single {renderer}/ raster golden is portable across arches within tolerance, or whether some scenes need per-platform raster goldens.
• Windows / Android / iOS / Mac Catalyst / WASM cells (all currently unseeded → will fail until their goldens are harvested on their agents — by design).

---

Remaining work (handoff checklist)

The framework is complete. The rest is seeding goldens per platform (mechanical, but must run on each platform's CI agent) and optionally adding device/WASM GPU renderers.

• Seed desktop GPU goldens from CI. Run the matrix on the Linux and Windows agents (provisioning is already wired for Linux). The GPU cells will fail as unseeded on the first run — that's expected. Download each lane's testlogs_* artifact, run extract-visual-goldens.py against it, review, and commit ganesh-gl.linux/, ganesh-vulkan.linux/, ganesh-gl.windows/ (and ganesh-vulkan.windows/ if a Windows ICD is present).
• Decide raster portability. After Linux/Windows raster cells run, check whether their captures match the committed raster.macos/ bytes within (2, 0.002). If yes, promote raster.macos/* → raster/* (and let the harvest's byte-aware skip keep them shared). If a scene diverges, leave it per-platform.
• Windows software-Vulkan provisioning (optional). The Linux apt path is done; Windows currently runs on whatever ICD is present (fail-safe). Add a software ICD if deterministic Windows Vulkan is wanted.
• P2 — device/WASM GPU renderers (the only coverage gap vs Dev/graphite backend #3968). Add, gated and skip-safe:
  • ganesh-gles on Android (raw P/Invoke EGL/GLES, OS-gated),
  • ganesh-vulkan on Android,
  • WebGL2 on WASM (OffscreenCanvas interop).
    These were left out deliberately so an unverified device P/Invoke can't cause a false red on a host that can't be tested locally. Each is a new IRenderer class + golden folder only — no harness changes. Seed their goldens via the same TRX harvest on the device/browser lanes.
• Tighten tolerances for the software-driver GPU cells once their goldens are stable (they can move toward the deterministic end). Use per-(renderer, scene) overrides in GoldenTolerance.ByRendererScene for individually divergent cells rather than loosening a whole renderer.
• direct3d cell (P3, optional) — add in the separate Direct3D console project; CatalogReflection discovers renderers from the entry assembly too.

---

The Graphite seam

#3968 rebases onto this by adding renderer classes and golden PNGs only — no test/csproj/CI changes. Concretely it adds:

• tests/VulkanTests/Visual/GraphiteVulkanRenderer.cs in the SkiaSharp.Vulkan.Tests.Console satellite (beside GaneshVulkanRenderer); the satellite's VulkanVisualTests discovers it via RendererCatalog.NamesIn(thisAssembly), so it joins that matrix with no test-class edit,
• Visual/Renderers/GraphiteMetalRenderer.cs (shared + Apple-gated, beside GaneshMetalRenderer, auto-discovered by the base VisualMatrixTests),
• Content/Goldens/graphite-*.{platform}/*.png (seeded per platform by harvesting its CI TRX).

The catalogs auto-discover both. The Graphite renderer files take a small (~5-line) rebase edit: implement SkiaSharp.Tests.Visual.IRenderer, acquire the GPU context from the shared TestConfig/GlContexts providers (or the satellite's existing VkContext/GRSharpVkBackendContext) instead of #3968's VulkanLoader/WglLoader/EglLoader, drop the inline ComputeDiff (the harness compares), and drop the out-of-process host sessions + VisualFactAttribute opt-in gate (the matrix runs by default).

---

File guide

Harness core (tests/Tests/SkiaSharp/Visual/)

• ISkiaScene.cs, IRenderer.cs, RendererUnavailableException.cs — the seam.
• SceneCatalog.cs, RendererCatalog.cs, CatalogReflection.cs — reflection discovery.
• Tests/VisualMatrixTestsBase.cs — the reusable per-cell engine: emits the ##SKIA-GOLDEN-IMAGE## marker, compares, enforces discipline. Shared by every host.
• Tests/VisualMatrixTests.cs — the base host's thin [Theory] over every auto-discovered renderer × scene.
• GoldenStore.cs — two-layer resolve/load (read-only) + PNG encode.
• GoldenTolerance.cs — raster (2, 0.002), GPU (12, 0.02), per-cell overrides.
• VisualPlatform.cs — Tag (the {platform} segment).
• RendererPixels.cs, GpuRenderGate.cs — pixel normalization + a GPU serialization lock.
• Scenes/*.cs (5), Renderers/*.cs (raster, ganesh-metal), Renderers/Desktop/*.cs (ganesh-gl — excluded from the shared csproj, Console-only).
• RendererCatalog.NamesIn(Assembly) — filters the catalog to renderers declared in one assembly; the seam a satellite uses to run only its own renderers.

Satellite host (tests/VulkanTests/Visual/, compiled into SkiaSharp.Vulkan.Tests.Console)

• GaneshVulkanRenderer.cs — moved here from the base host so SharpVk stays out of the shared test code.
• VulkanVisualTests.cs — the ~15-line VisualMatrixTestsBase driver running NamesIn(thisAssembly) × scenes.

Comparison — tests/Tests/Utils/SKPixelComparer.cs: tolerance-aware Compare(...channelTolerance) (+ MaxChannelDelta) and GenerateDifferenceImage (red = over tolerance, amber = minor).

Tooling — scripts/infra/tests/extract-visual-goldens.py: harvests markers from TRX → writes goldens, with a path-safety guard, --dry-run, and the byte-aware shared-skip.

CI — scripts/azure-templates-stages-test.yml: Linux software-GPU provisioning.

csproj — SkiaSharp.Tests.csproj excludes Visual/Renderers/Desktop/** (same as GlContexts/*); base SkiaSharp.Tests.Console.csproj includes everything but takes no GPU-vendor package. SkiaSharp.Vulkan.Tests.Console.csproj already references SharpVk + the base host and compiles ..\VulkanTests\**, so the moved renderer needed no csproj change.

Docs — documentation/dev/golden-image-tests.md: the full durable design doc (architecture, discipline, seeding, CI, extending, the Graphite seam).

---

How to run locally

dotnet cake --target=externals-download            # C#-only change → pre-built natives are fine
dotnet build tests/SkiaSharp.Tests.Console/SkiaSharp.Tests.Console.csproj -c Release
cd tests/SkiaSharp.Tests.Console/bin/Release/net*/
./SkiaSharp.Tests --filter-trait "Category=Visual"        # run only the visual matrix
./SkiaSharp.Tests --filter-not-trait "Category=Visual"    # run everything else

On macOS the base host shows 15 pass / 0 skip. The ganesh-vulkan cells live in the Vulkan satellite:

dotnet build tests/SkiaSharp.Vulkan.Tests.Console/SkiaSharp.Vulkan.Tests.Console.csproj -c Release
cd tests/SkiaSharp.Vulkan.Tests.Console/bin/Release/net*/
./SkiaSharp.Vulkan.Tests --filter-trait "Category=Visual"   # 5 ganesh-vulkan cells; skip without an ICD

🤖 Generated with Copilot CLI

[github-actions]

📦 Try the packages from this PR

Warning

Do not run these scripts without first reviewing the code in this PR.

Step 1 — Download the packages

bash / macOS / Linux:

curl -fsSL https://raw.githubusercontent.com/mono/SkiaSharp/main/scripts/get-skiasharp-pr.sh | bash -s -- 4236

PowerShell / Windows:

iex "& { $(irm https://raw.githubusercontent.com/mono/SkiaSharp/main/scripts/get-skiasharp-pr.ps1) } 4236"

Step 2 — Add the local NuGet source

dotnet nuget add source ~/.skiasharp/hives/pr-4236/packages --name skiasharp-pr-4236

More options

Option	Description
--successful-only / -SuccessfulOnly	Only use successful builds
--force / -Force	Overwrite previously downloaded packages
--list / -List	List available artifacts without downloading
--build-id ID / -BuildId ID	Download from a specific build

Or download manually from Azure Pipelines — look for the nuget artifact on the build for this PR.

Remove the source when you're done:

dotnet nuget remove source skiasharp-pr-4236

[github-actions]

📖 Documentation Preview

The documentation for this PR has been deployed and is available at:

🔗 View Staging Site
🔗 View Staging Docs
🔗 View Staging Gallery (Blazor)
🔗 View Staging Gallery (Uno Platform)
🔗 View Staging SkiaFiddle

This preview will be updated automatically when you push new commits to this PR.

---

This comment is automatically updated by the documentation staging workflow.

[mattleibow]

🤝 Resume notes for whoever takes this over

Short orientation + tribal knowledge that isn't obvious from the PR body. The durable design doc is documentation/dev/golden-image-tests.md — read it first.

Current state (framework done; seeding is what's left)

• Harness, two-layer golden lookup, TRX-harvest seeding, and CI wiring are in place. Failure PNGs are published on every test lane: scripts/azure-templates-stages-test.yml runs extract-visual-goldens.py as an always() post-test step that decodes the TRX markers into the testlogs_* artifact.
• Committed goldens today: ganesh-gl.macos/ + ganesh-metal.macos/ (4 geometric scenes each), shared raster/ (4 geometric — verified portable across macOS/Windows/WASM within the raster tolerance), and per-platform Text (raster.windows/Text.png, raster.browser/Text.png).
• Verified locally on macOS just now: base Console visual matrix = 12 pass / 3 fail. The 3 failures are {raster, ganesh-gl, ganesh-metal} × Text failing as unseeded — by design. Commit b29a8302d23 removed the old macOS Text goldens because they were silently blank (the bundled subset font didn't map the old "Skia"/"Sharp" glyphs; the scene now draws "HELLO"/"WORLD!"). They need a real-text reseed (see below). Unseeded = hard FAIL is intentional, not a regression.

Critical-path next action: seed goldens (the only real work left)

Every non-macOS cell, plus the 3 macOS Text cells, currently has no golden and will fail-as-unseeded on first run. That failure is the signal to harvest:

1. Let the test lane run (visual cells fail-as-unseeded, emitting their PNG into the TRX).
2. Download that lane's testlogs_* artifact.
3. python3 scripts/infra/tests/extract-visual-goldens.py <path-to.trx> → writes tests/Content/Goldens/**. Triage with --failures-out DIR to dump actual/golden/diff PNGs and tell an unseeded cell (seed it) apart from a mismatch (investigate — never blind-harvest a regression).
4. Eyeball the PNGs, commit, re-run → green.

Still to seed: macOS Text (raster/gl/metal — harvestable from a macOS lane or even a macOS dev box, which is how the existing macOS goldens were made); Linux raster+gl+vulkan; Windows raster+gl (+vulkan if an ICD is present); Android / iOS / MacCatalyst / WASM raster + their GPU cells.

Gotchas that cost time

• No Vulkan ICD on the macOS dev box → ganesh-vulkan cells skip locally and you cannot reproduce Linux/Windows/device/WASM here. Real GPU verification only happens on CI (Linux agents are provisioned with Mesa GL + Lavapipe Vulkan; see the CI provisioning steps).
• gh pr edit and any GraphQL gh call are SAML-blocked with the available token. Edit the PR via REST instead: gh api -X PATCH repos/mono/SkiaSharp/pulls/4236 -F body=@file; set milestone/labels with gh api -X PATCH repos/mono/SkiaSharp/issues/{n} (issues + PRs share the /issues/ endpoint).
• Always diff against origin/main, not the local main ref (the stale local main shows a spurious docs submodule change that isn't real).
• This is a C#/test-only change → bootstrap with dotnet cake --target=externals-download. Do not rebuild natives.

Run the matrix locally (macOS)

export PATH="/usr/local/share/dotnet:$PATH"
dotnet build tests/SkiaSharp.Tests.Console/SkiaSharp.Tests.Console.csproj -c Release
cd tests/SkiaSharp.Tests.Console/bin/Release/net10.0
export DYLD_LIBRARY_PATH="$PWD:$DYLD_LIBRARY_PATH"
./SkiaSharp.Tests --filter-trait "Category=Visual"

Vulkan satellite: build SkiaSharp.Vulkan.Tests.Console, then ./SkiaSharp.Vulkan.Tests --filter-trait "Category=Visual".

Architecture cheat-sheet

• Shared CPU/GL cells live in tests/Tests/SkiaSharp/Visual/** (project SkiaSharp.Tests) and compile into every host (Console / Devices / Wasm).
• Package-dependent GPU renderers live in satellite hosts so the base Console stays dependency-clean: Vulkan → SkiaSharp.Vulkan.Tests.Console (SharpVk), Direct3D → SkiaSharp.Direct3D.Tests.Console (Vortice). Each is a thin VisualMatrixTestsBase subclass driving RendererCatalog.NamesIn(thisAssembly) × scenes.

The Graphite seam (the whole point of this PR)

When dev/graphite-backend (#3968) rebases onto this, it adds only:

• graphite-vulkan: one IRenderer dropped into tests/VulkanTests/Visual/ → auto-joins the Vulkan satellite via RendererCatalog.NamesIn.
• graphite-metal: one shared Apple-gated IRenderer → auto-joins the base matrix.
• a golden folder per renderer.

Zero test-class / csproj / CI edits. The only rebase friction is a ~5-line edit pointing those renderers at our shared context-provider names (documented in the design doc).