docs(bench): v0.17 — SSIMULACRA2 metric + tuned NVENC settings close most of the GPU->CPU gap

CHANGELOG.md

@@ -68,6 +68,7 @@ All notable changes to Pelorus are documented here. The format is
 
 - **`vf_pelorus_analyze_vulkan`** auto-ROI (`roi=1`) banding detection is now **two-scale** (ADR-0133, CAMBI alignment): alongside the existing per-tile variance scale, a coarse inter-tile mean-luma-gradient scale flags shallow ramps (sky/shadow) that band across many tiles but fall below any single tile's variance floor. A 0x10→0x30 ramp (CAMBI 0.625) that previously flagged 0 tiles now flags 27, and an A/B encode confirmed lower output CAMBI with no regression on textured content. Filter-internal; no interop ABI change. Docs: `docs/metrics/analyze.md`.
 - **Evaluated + rejected: wide-band reach for deband** (multi-radius / extended `range`). Designed by a 5-agent judge panel and built behind an opt-in `reach` AVOption, then measured: the pre-encode CAMBI gain (bench v0.15) is real but **does not survive the encoder** — wider range is monotonically *worse* post-`hevc_nvenc` at every quality (cq10/18/28), because the encoder re-bands the over-smoothed gentle ramp into wider bands. Code reverted; `range` stays capped at 31. Completes v0.15's deferred durability gate with a negative. ADR-0141, bench v0.16.
+- **Benchmark methodology: SSIMULACRA2 is now the primary metric** (bench-results v0.17). VMAF-NEG is retired — it zeroes the enhancement credit (it was the anti-gaming guard for the filter lane) and so *hid* the biggest measured lever. On the same encodes where VMAF-NEG showed "no gain," SS2 reveals that **tuned NVENC settings** (`-preset p7 -tune hq -spatial-aq 1 -temporal-aq 1 -aq-strength 8 -rc-lookahead 32 -multipass fullres -b_ref_mode middle`) beat the conservative FFmpeg defaults by ~**+5 to +10 SS2** at iso-bitrate — a large, universal, free gain (the default is genuinely bad). With tuned settings the GPU→CPU gap shrinks to **~2 SS2** (tuned `hevc_nvenc` vs `x265 -preset slow`; `av1_nvenc` vs `SVT-AV1` similar), which the Pelorus Vulkan filters close on top at GPU speed. Methodology now: CQ-locked ladders (VBR overshoots), SS2 primary + VMAF secondary (the VMAF-up/SS2-down split is the gaming flag), CPU encode as the gold-standard gap reference. Feeds the `tune=auto` design (ADR-0142). Docs: `docs/development/bench-results.md`, `benchmarking.md`.
 
 ### Fixed
 

changelog.d/changed/0142-ss2-metric-tuned-settings.md

@@ -0,0 +1 @@
+- **Benchmark methodology: SSIMULACRA2 is now the primary metric** (bench-results v0.17). VMAF-NEG is retired — it zeroes the enhancement credit (it was the anti-gaming guard for the filter lane) and so *hid* the biggest measured lever. On the same encodes where VMAF-NEG showed "no gain," SS2 reveals that **tuned NVENC settings** (`-preset p7 -tune hq -spatial-aq 1 -temporal-aq 1 -aq-strength 8 -rc-lookahead 32 -multipass fullres -b_ref_mode middle`) beat the conservative FFmpeg defaults by ~**+5 to +10 SS2** at iso-bitrate — a large, universal, free gain (the default is genuinely bad). With tuned settings the GPU→CPU gap shrinks to **~2 SS2** (tuned `hevc_nvenc` vs `x265 -preset slow`; `av1_nvenc` vs `SVT-AV1` similar), which the Pelorus Vulkan filters close on top at GPU speed. Methodology now: CQ-locked ladders (VBR overshoots), SS2 primary + VMAF secondary (the VMAF-up/SS2-down split is the gaming flag), CPU encode as the gold-standard gap reference. Feeds the `tune=auto` design (ADR-0142). Docs: `docs/development/bench-results.md`, `benchmarking.md`.

docs/development/bench-results.md

@@ -650,6 +650,53 @@ before it shipped. The only thing that could revive it is a measured post-encode
 BD-rate win on real 10-bit banding-prone content (the corpus gap) — which the
 mechanism above predicts against.
 
+## v0.17 — the metric was wrong + the tuned-settings lever: GPU→CPU gap is small and closeable (SSIMULACRA2)
+
+The session's payoff, and a methodology correction. Pelorus's goal is **smaller +
+higher-quality GPU (hardware) encodes approaching CPU (software) at GPU speed.**
+Measuring that with VMAF/VMAF-NEG hid most of the signal.
+
+**1. SSIMULACRA2 is the primary metric; VMAF-NEG was the wrong tool.** VMAF
+saturates at 100 on clean content and VMAF-**NEG** zeroes the enhancement credit
+(it was the anti-gaming guard for the *filter* lane — a sharpening filter could
+inflate VMAF, so NEG clamped it). But it also blinds the metric to genuine AQ /
+bit-allocation gains. On the *same* encodes where VMAF-NEG said "no gain," SS2
+(no saturation, robust to sharpening-gaming) shows large differences. The set is
+now **SS2 primary + regular VMAF secondary** (VMAF-up / SS2-down = a gaming flag);
+NEG retired. Multi-metric is fine — the result is what counts.
+
+**2. The NVENC *default* is garbage; tuned settings are the biggest, free lever.**
+`hevc_nvenc -preset p5` with no AQ scored SS2 **15.9** on noisy content vs the
+tuned config's **22.8** (+6.8). The tuned baseline:
+`-preset p7 -tune hq -spatial-aq 1 -temporal-aq 1 -aq-strength 8 -rc-lookahead 32
+-multipass fullres -b_ref_mode middle`. Clean iso-bitrate (SS2): tuned beats
+default by ~**+5 to +10 SS2** — *invisible* to VMAF (±0.5; it's detail-weighted
+and can't see the flat-region AQ win). Pushing harder (AQ15 / lookahead 53) does
+**not** help — the settings lever is tapped at the config above.
+
+**3. The real GPU→CPU gap is small and closeable.** Clean content, CQ-locked,
+SS2 vs bitrate:
+
+| codec | GPU (tuned nvenc) vs CPU (sw) | iso-bitrate gap |
+|---|---|---|
+| HEVC | tuned `hevc_nvenc` vs `x265 -preset slow` | CPU ahead **~1.8–2.5 SS2** |
+| AV1 | tuned `av1_nvenc` vs `SVT-AV1 -preset 4` | CPU ahead **~2.5 SS2** |
+
+The earlier "huge gap" was an artifact of the garbage default + noisy content +
+VBR overshoot (the multipass NVENC config overshoots `-b:v` by ~50%, so CQ-locked
+ladders are required for honest iso-bitrate). With tuned settings the gap is ~2
+SS2 — and that residual is exactly what the **Pelorus Vulkan filters** (denoise on
+grain, per-content reductive) close on top, at GPU speed (NVENC p7+multipass is
+still far faster than x265-slow, so the GPU speed advantage holds).
+
+**4. The program (`tune=auto`, ADR-0142).** Tuned settings = the universal
+baseline every content template starts from; Pelorus Vulkan filters = the
+per-content layer that closes the residual and goes further on impaired content
+(denoise reclaims grain bitrate; the reductive law from v0.2/v0.3). Caveats to
+carry: CQ-lock for iso-bitrate, hardware AV1 (`av1_nvenc`) is much weaker than
+hardware HEVC so its residual gap is larger, and keep the GPU pipeline faster
+than CPU (the whole point).
+
 ## Open / next
 
 1. **SVT-AV1 ROI on real content**: the v0.10 synthetic gain is modest; re-run

docs/development/benchmarking.md

@@ -16,12 +16,20 @@ encoder, same preset, same CQ**:
   pipeline: `ffmpeg -i src -init_hw_device vulkan -vf
   "format,hwupload,pelorus_deband_vulkan=…,hwdownload,format" -c:v hevc_nvenc -cq N`.
 
-For each CQ in a ladder we record (bitrate, VMAF) → two RD curves → **BD-rate**
-(`scripts/bench/bd_rate.py`, Bjøntegaard): negative % = Pelorus needs fewer bits
-for equal VMAF = a win. For deband we also record **CAMBI** (the
-Contrast-Aware Multiscale Banding Index, vmafx `--feature cambi`): lower = less
-banding. Optionally a CPU encode (x265 / SVT-AV1) is included as the
-gold-standard reference, to show how much of the GPU↔CPU gap Pelorus closes.
+For each CQ in a **CQ-locked** ladder (never `-b:v` VBR — the tuned-NVENC
+multipass config overshoots a VBR target by ~50%, breaking iso-bitrate) we record
+(bitrate, **SSIMULACRA2**) → two RD curves → **BD-rate** (`scripts/bench/bd_rate.py`,
+Bjøntegaard): negative % = fewer bits for equal quality = a win.
+
+**SSIMULACRA2 is the primary metric** (vmafx `--feature ssimulacra2`): it does not
+saturate on clean content and is robust to sharpening-gaming, so it both *credits*
+genuine AQ / bit-allocation gains *and* *catches* artifact-adding tricks. Regular
+**VMAF** (v1.0.16, `-m path=…vmaf_v1.0.16_3d0h.json`) is a secondary cross-check —
+a VMAF-up / SS2-down split is the gaming flag. **VMAF-NEG is retired** (it zeroes
+the enhancement credit, blinding the metric to real AQ gains; see bench-results
+v0.17). For deband also record **CAMBI** (lower = less banding). A CPU encode
+(`x265 -preset slow` / `SVT-AV1 -preset 4`) is the gold-standard reference for the
+GPU↔CPU gap — Pelorus' job is to close it **while staying faster than CPU**.
 
 ## Pinned corpus