diff --git a/CHANGELOG.md b/CHANGELOG.md index aeb5c06..6601f00 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -68,6 +68,7 @@ All notable changes to Pelorus are documented here. The format is - **`vf_pelorus_analyze_vulkan`** auto-ROI (`roi=1`) banding detection is now **two-scale** (ADR-0133, CAMBI alignment): alongside the existing per-tile variance scale, a coarse inter-tile mean-luma-gradient scale flags shallow ramps (sky/shadow) that band across many tiles but fall below any single tile's variance floor. A 0x10→0x30 ramp (CAMBI 0.625) that previously flagged 0 tiles now flags 27, and an A/B encode confirmed lower output CAMBI with no regression on textured content. Filter-internal; no interop ABI change. Docs: `docs/metrics/analyze.md`. - **Evaluated + rejected: wide-band reach for deband** (multi-radius / extended `range`). Designed by a 5-agent judge panel and built behind an opt-in `reach` AVOption, then measured: the pre-encode CAMBI gain (bench v0.15) is real but **does not survive the encoder** — wider range is monotonically *worse* post-`hevc_nvenc` at every quality (cq10/18/28), because the encoder re-bands the over-smoothed gentle ramp into wider bands. Code reverted; `range` stays capped at 31. Completes v0.15's deferred durability gate with a negative. ADR-0141, bench v0.16. +- **Benchmark methodology: SSIMULACRA2 is now the primary metric** (bench-results v0.17). VMAF-NEG is retired — it zeroes the enhancement credit (it was the anti-gaming guard for the filter lane) and so *hid* the biggest measured lever. On the same encodes where VMAF-NEG showed "no gain," SS2 reveals that **tuned NVENC settings** (`-preset p7 -tune hq -spatial-aq 1 -temporal-aq 1 -aq-strength 8 -rc-lookahead 32 -multipass fullres -b_ref_mode middle`) beat the conservative FFmpeg defaults by ~**+5 to +10 SS2** at iso-bitrate — a large, universal, free gain (the default is genuinely bad). With tuned settings the GPU→CPU gap shrinks to **~2 SS2** (tuned `hevc_nvenc` vs `x265 -preset slow`; `av1_nvenc` vs `SVT-AV1` similar), which the Pelorus Vulkan filters close on top at GPU speed. Methodology now: CQ-locked ladders (VBR overshoots), SS2 primary + VMAF secondary (the VMAF-up/SS2-down split is the gaming flag), CPU encode as the gold-standard gap reference. Feeds the `tune=auto` design (ADR-0142). Docs: `docs/development/bench-results.md`, `benchmarking.md`. ### Fixed diff --git a/changelog.d/changed/0142-ss2-metric-tuned-settings.md b/changelog.d/changed/0142-ss2-metric-tuned-settings.md new file mode 100644 index 0000000..c2e12e1 --- /dev/null +++ b/changelog.d/changed/0142-ss2-metric-tuned-settings.md @@ -0,0 +1 @@ +- **Benchmark methodology: SSIMULACRA2 is now the primary metric** (bench-results v0.17). VMAF-NEG is retired — it zeroes the enhancement credit (it was the anti-gaming guard for the filter lane) and so *hid* the biggest measured lever. On the same encodes where VMAF-NEG showed "no gain," SS2 reveals that **tuned NVENC settings** (`-preset p7 -tune hq -spatial-aq 1 -temporal-aq 1 -aq-strength 8 -rc-lookahead 32 -multipass fullres -b_ref_mode middle`) beat the conservative FFmpeg defaults by ~**+5 to +10 SS2** at iso-bitrate — a large, universal, free gain (the default is genuinely bad). With tuned settings the GPU→CPU gap shrinks to **~2 SS2** (tuned `hevc_nvenc` vs `x265 -preset slow`; `av1_nvenc` vs `SVT-AV1` similar), which the Pelorus Vulkan filters close on top at GPU speed. Methodology now: CQ-locked ladders (VBR overshoots), SS2 primary + VMAF secondary (the VMAF-up/SS2-down split is the gaming flag), CPU encode as the gold-standard gap reference. Feeds the `tune=auto` design (ADR-0142). Docs: `docs/development/bench-results.md`, `benchmarking.md`. diff --git a/docs/development/bench-results.md b/docs/development/bench-results.md index 36749e5..f9f150c 100644 --- a/docs/development/bench-results.md +++ b/docs/development/bench-results.md @@ -650,6 +650,53 @@ before it shipped. The only thing that could revive it is a measured post-encode BD-rate win on real 10-bit banding-prone content (the corpus gap) — which the mechanism above predicts against. +## v0.17 — the metric was wrong + the tuned-settings lever: GPU→CPU gap is small and closeable (SSIMULACRA2) + +The session's payoff, and a methodology correction. Pelorus's goal is **smaller + +higher-quality GPU (hardware) encodes approaching CPU (software) at GPU speed.** +Measuring that with VMAF/VMAF-NEG hid most of the signal. + +**1. SSIMULACRA2 is the primary metric; VMAF-NEG was the wrong tool.** VMAF +saturates at 100 on clean content and VMAF-**NEG** zeroes the enhancement credit +(it was the anti-gaming guard for the *filter* lane — a sharpening filter could +inflate VMAF, so NEG clamped it). But it also blinds the metric to genuine AQ / +bit-allocation gains. On the *same* encodes where VMAF-NEG said "no gain," SS2 +(no saturation, robust to sharpening-gaming) shows large differences. The set is +now **SS2 primary + regular VMAF secondary** (VMAF-up / SS2-down = a gaming flag); +NEG retired. Multi-metric is fine — the result is what counts. + +**2. The NVENC *default* is garbage; tuned settings are the biggest, free lever.** +`hevc_nvenc -preset p5` with no AQ scored SS2 **15.9** on noisy content vs the +tuned config's **22.8** (+6.8). The tuned baseline: +`-preset p7 -tune hq -spatial-aq 1 -temporal-aq 1 -aq-strength 8 -rc-lookahead 32 +-multipass fullres -b_ref_mode middle`. Clean iso-bitrate (SS2): tuned beats +default by ~**+5 to +10 SS2** — *invisible* to VMAF (±0.5; it's detail-weighted +and can't see the flat-region AQ win). Pushing harder (AQ15 / lookahead 53) does +**not** help — the settings lever is tapped at the config above. + +**3. The real GPU→CPU gap is small and closeable.** Clean content, CQ-locked, +SS2 vs bitrate: + +| codec | GPU (tuned nvenc) vs CPU (sw) | iso-bitrate gap | +|---|---|---| +| HEVC | tuned `hevc_nvenc` vs `x265 -preset slow` | CPU ahead **~1.8–2.5 SS2** | +| AV1 | tuned `av1_nvenc` vs `SVT-AV1 -preset 4` | CPU ahead **~2.5 SS2** | + +The earlier "huge gap" was an artifact of the garbage default + noisy content + +VBR overshoot (the multipass NVENC config overshoots `-b:v` by ~50%, so CQ-locked +ladders are required for honest iso-bitrate). With tuned settings the gap is ~2 +SS2 — and that residual is exactly what the **Pelorus Vulkan filters** (denoise on +grain, per-content reductive) close on top, at GPU speed (NVENC p7+multipass is +still far faster than x265-slow, so the GPU speed advantage holds). + +**4. The program (`tune=auto`, ADR-0142).** Tuned settings = the universal +baseline every content template starts from; Pelorus Vulkan filters = the +per-content layer that closes the residual and goes further on impaired content +(denoise reclaims grain bitrate; the reductive law from v0.2/v0.3). Caveats to +carry: CQ-lock for iso-bitrate, hardware AV1 (`av1_nvenc`) is much weaker than +hardware HEVC so its residual gap is larger, and keep the GPU pipeline faster +than CPU (the whole point). + ## Open / next 1. **SVT-AV1 ROI on real content**: the v0.10 synthetic gain is modest; re-run diff --git a/docs/development/benchmarking.md b/docs/development/benchmarking.md index 90fa56d..8a854d9 100644 --- a/docs/development/benchmarking.md +++ b/docs/development/benchmarking.md @@ -16,12 +16,20 @@ encoder, same preset, same CQ**: pipeline: `ffmpeg -i src -init_hw_device vulkan -vf "format,hwupload,pelorus_deband_vulkan=…,hwdownload,format" -c:v hevc_nvenc -cq N`. -For each CQ in a ladder we record (bitrate, VMAF) → two RD curves → **BD-rate** -(`scripts/bench/bd_rate.py`, Bjøntegaard): negative % = Pelorus needs fewer bits -for equal VMAF = a win. For deband we also record **CAMBI** (the -Contrast-Aware Multiscale Banding Index, vmafx `--feature cambi`): lower = less -banding. Optionally a CPU encode (x265 / SVT-AV1) is included as the -gold-standard reference, to show how much of the GPU↔CPU gap Pelorus closes. +For each CQ in a **CQ-locked** ladder (never `-b:v` VBR — the tuned-NVENC +multipass config overshoots a VBR target by ~50%, breaking iso-bitrate) we record +(bitrate, **SSIMULACRA2**) → two RD curves → **BD-rate** (`scripts/bench/bd_rate.py`, +Bjøntegaard): negative % = fewer bits for equal quality = a win. + +**SSIMULACRA2 is the primary metric** (vmafx `--feature ssimulacra2`): it does not +saturate on clean content and is robust to sharpening-gaming, so it both *credits* +genuine AQ / bit-allocation gains *and* *catches* artifact-adding tricks. Regular +**VMAF** (v1.0.16, `-m path=…vmaf_v1.0.16_3d0h.json`) is a secondary cross-check — +a VMAF-up / SS2-down split is the gaming flag. **VMAF-NEG is retired** (it zeroes +the enhancement credit, blinding the metric to real AQ gains; see bench-results +v0.17). For deband also record **CAMBI** (lower = less banding). A CPU encode +(`x265 -preset slow` / `SVT-AV1 -preset 4`) is the gold-standard reference for the +GPU↔CPU gap — Pelorus' job is to close it **while staying faster than CPU**. ## Pinned corpus