From a30bec6729778b1ba3ad357ab193efd7719bbf00 Mon Sep 17 00:00:00 2001 From: lusoris Date: Sat, 27 Jun 2026 21:26:21 +0200 Subject: [PATCH] =?UTF-8?q?feat(grain):=20emit=20grain=5Fsigma/grain=5Ffla?= =?UTF-8?q?t=20metadata=20=E2=80=94=20the=20tune=3Dauto=20detection=20enab?= =?UTF-8?q?ler=20(ADR-0142)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The session's measurement campaign established the law: NVENC is genuinely worse than x265, but the pre-encode gain that closes it is REDUCTIVE and CONTENT-DEPENDENT (additive pre-sharpen washes; denoise scales with removable grain; clean content gains ~0), and mostly a matter of TUNING. So there is no single best filter -- there is a tuned reductive boost per content class and a router (tune=auto) that detects the class from analyze.* metadata and applies it. The router needs content detection that did not exist: grain_estimate computes per-band RMS residual but emitted zero frame metadata (trapped in the PEL_SEC_FILMGRAIN side-data). This ships the first and highest-leverage enabler: grain_estimate now emits lavfi.pelorus.grain_sigma (peak per-band RMS over the edge-gated flat bands -- structure excluded by construction, so what survives is grain stddev) and lavfi.pelorus.grain_flat (the flat fraction = confidence), via the ADR-0136 av_dict_set pattern. No interop ABI change, no shader change, the value was already computed; the side-data path is untouched. Verified discriminating (heavy-grain 0.019 vs clean-ish Bluray 0.012) and unblocks the proven -34% grainy denoise route. ADR-0142 records the full tune=auto design (8-class routing table + validation priority + the encoder-integration patch parallel track). Follow-ups: the remaining detection enablers (noise_sigma, dark_frac), the per-leg validations, the router itself, and the encoder-patch track. Co-Authored-By: Claude Opus 4.8 --- CHANGELOG.md | 1 + .../added/0142-grain-sigma-metadata.md | 1 + docs/adr/0142-tune-auto-content-router.md | 109 ++++++++++++++++++ docs/adr/README.md | 1 + docs/metrics/grain_estimate.md | 21 ++++ ...add-vf_pelorus_grain_estimate_vulkan.patch | 40 ++++++- .../files/vf_pelorus_grain_estimate_vulkan.c | 28 +++++ 7 files changed, 195 insertions(+), 6 deletions(-) create mode 100644 changelog.d/added/0142-grain-sigma-metadata.md create mode 100644 docs/adr/0142-tune-auto-content-router.md diff --git a/CHANGELOG.md b/CHANGELOG.md index 4b1e8db..aeb5c06 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -62,6 +62,7 @@ All notable changes to Pelorus are documented here. The format is - **`vf_pelorus_denoise_vulkan`** gains an opt-in `lookahead` option (default 0) — forward-lookahead (bidirectional) temporal denoise ([ADR-0137](docs/adr/0137-denoise-forward-lookahead-cadence.md)). `lookahead=1` delays output one frame so the temporal walk also samples the **next** frame (same-coordinate, `tcut`-gated), recovering the leading frame of a held animation drawing (2s/3s cadence) that the causal walk under-denoises. Validated +0.37 dB on a 2s-cadence clip; `lookahead=0` is bit-identical/no-latency. Implemented as an `activate()` delay-line + a 6th `next0_images` shader tap, `.comp` kept in lockstep. Docs: `docs/metrics/denoise.md`. - **`vf_pelorus_dehalo_vulkan`** gains an opt-in `tile` option (default 0) — shared-memory tiling of the box-blur window ([ADR-0139](docs/adr/0139-dehalo-shared-mem-tile.md)). `box_blur` re-reads an overlapping 17×17 window ~5× per pixel (fetch-bound; an ALU-strip cut Arc rtime −70%), so `tile=1` cooperatively loads the window into shared memory once per plane. **Bit-identical** (SSIM 1.000000); validated **−38% (1.6×) on an Arc A380**, ~neutral on the cache-rich 4090 → default off, opt-in for weak/integrated/mobile GPUs + `tune=anime`. The ADR-0134 denoise-tiling idiom applied to its fetch-bound twin (the `aa` sobel kernel was ALU-bound and refuted). Docs: `docs/metrics/dehalo.md`. - **`vf_pelorus_aa_vulkan`** gains an opt-in `fast` option (default 0) — hoists the redundant sobel-mag into shared memory ([ADR-0140](docs/adr/0140-aa-sobel-mag-hoist.md)). aa is ALU-bound (a premise-check found stripping the sobel collapsed rtime −86%, so tiling was refuted), and `sobel_mag` is recomputed ~1156×/px across the overlapping `emask` windows; `fast=1` computes each cell's sobel **once** per workgroup into shared memory and `emask` reduces from the cache. **Bit-identical** (SSIM 1.000000, `cmp` 0 bytes both GPUs); validated **12.6× (−92%) on Arc A380, 2.6× (−62%) on the RTX 4090** — being an ALU win it helps every GPU (unlike fetch-tiling). Docs: `docs/metrics/aa.md`. +- **`vf_pelorus_grain_estimate_vulkan`** now emits `lavfi.pelorus.grain_sigma` and `lavfi.pelorus.grain_flat` as per-frame metadata (the [ADR-0136](docs/adr/0136-analyze-frame-metadata.md) `av_dict_set` pattern — no interop ABI or shader change, the value was already computed). `grain_sigma` is the peak per-band RMS residual measured over edge-gated locally-flat pixels (structure excluded → what survives is grain stddev); `grain_flat` is the flat fraction it was measured over (confidence). This is the first detection enabler for the `tune=auto` content-adaptive router ([ADR-0142](docs/adr/0142-tune-auto-content-router.md)) — it unblocks routing grainy content (the proven −34% denoise lever) without parsing the `PEL_SEC_FILMGRAIN` side-data blob. Verified discriminating: heavy-grain 0.019 vs clean-ish Bluray 0.012. Docs: `docs/metrics/grain_estimate.md`. ### Changed diff --git a/changelog.d/added/0142-grain-sigma-metadata.md b/changelog.d/added/0142-grain-sigma-metadata.md new file mode 100644 index 0000000..035f65e --- /dev/null +++ b/changelog.d/added/0142-grain-sigma-metadata.md @@ -0,0 +1 @@ +- **`vf_pelorus_grain_estimate_vulkan`** now emits `lavfi.pelorus.grain_sigma` and `lavfi.pelorus.grain_flat` as per-frame metadata (the [ADR-0136](docs/adr/0136-analyze-frame-metadata.md) `av_dict_set` pattern — no interop ABI or shader change, the value was already computed). `grain_sigma` is the peak per-band RMS residual measured over edge-gated locally-flat pixels (structure excluded → what survives is grain stddev); `grain_flat` is the flat fraction it was measured over (confidence). This is the first detection enabler for the `tune=auto` content-adaptive router ([ADR-0142](docs/adr/0142-tune-auto-content-router.md)) — it unblocks routing grainy content (the proven −34% denoise lever) without parsing the `PEL_SEC_FILMGRAIN` side-data blob. Verified discriminating: heavy-grain 0.019 vs clean-ish Bluray 0.012. Docs: `docs/metrics/grain_estimate.md`. diff --git a/docs/adr/0142-tune-auto-content-router.md b/docs/adr/0142-tune-auto-content-router.md new file mode 100644 index 0000000..48db1d1 --- /dev/null +++ b/docs/adr/0142-tune-auto-content-router.md @@ -0,0 +1,109 @@ + +# ADR-0142: `tune=auto` — content-adaptive pre-encode router (design) + the grain-sigma detection enabler + +- **Status**: Proposed (2026-06-27) — design accepted; first enabler (grain-sigma metadata) shipped in this PR; per-content legs validated + the router built incrementally +- **Date**: 2026-06-27 +- **Deciders**: Lusoris + +## Context + +The session's measurement campaign (real `vmaf` v1.0.16 + v0.6.1-NEG, an x265-enabled +ffmpeg, on-hardware NVENC) established a **law** that reframes the whole product: + +- Hardware encoders (NVENC/QSV/AMF) **are** worse than software (x265 beats NVENC ~0.005 + SSIM iso-bitrate — no psy-RD/RDOQ/trellis, weak temporal filter, weightp-off-with-B). +- **But the pre-encode gain that closes the gap is REDUCTIVE and CONTENT-DEPENDENT:** + - **Additive** (pre-sharpen/CAS) is a **wash** on VMAF-NEG and costs bits — you cannot + add your way past the deadzone (the HF-energy "recovery" is a false proxy). + - **Reductive** (denoise/dehalo/demosquito/deblock) is real, but the magnitude scales + with *how much removable impairment exists*: −34% BD-rate on the real GPU denoise at + heavy grain, ~5–7% at moderate grain, **~0 on clean** (clean Bluray barely bands → + deband marginal; same law). + - **Source-side rate-control** (perceptual-AQ map, per-shot CRF) **loses** structurally + (ADR-0135/0132) — the encoder's own RC wins. Never emit a QP map. +- And it is **mostly tuning**: the same filter is a false-negative with the wrong params. + Measured: the default denoise leans spatial and *over-smoothed* (tied at high bitrate); + retuned temporal-dominant (`blend=0.85 prev=4`) it recovers a gain (+0.7 NEG) at the + same point — grain is temporally *incoherent*, so the temporal walk is the lever, not + spatial strength. + +There is therefore **no single best filter** — there is the **right tuned boost per +content class**, and a router that detects the class and applies it. This is `tune=auto`. + +## Decision + +Build a content-adaptive router (`tune=auto`) that reads `vf_pelorus_analyze`'s per-frame +metadata, classifies the content per shot (segmented by `scene_cut`, with hysteresis), +and applies the matching **reductive** boost with params **scaled to the measured +impairment**. The routing table (designed by an 8-class boost panel, grounded in the law): + +| content class | detect (analyze.*) | boost | key params (scale with) | expected | +|---|---|---|---|---| +| **grainy/noisy** | `grain_sigma` over flat bands (NEW metadata) | denoise (temporal-dominant) + grain_estimate→FGS resynth | `strength,blend,sigmat ∝ grain_sigma` | **−34%** proven | +| **anime/2D** | high `edge` AND high `banding`/flat co-occurrence | dehalo + aa + deband, co-gated by one edge mask | per the anime tune | ~3–8% SSIMu | +| **textured live-action** | high `variance`/`edge`, low `motion` | denoise spatial-dominant (strip micro-noise only) | `blend 0.3–0.4` | medium | +| **re-encode/compressed** | edge-band temporal excess (NEW) | **new `demosquito`** (edge-band temporal, the band denoise refuses) | premise-gated | single-digit % | +| **dark/low-light/HDR** | `dark_frac` (NEW) | denoise luma-masked to dark tiles | dark-gated | ~5–15% sub-frame | +| **screen/text** | very-high `edge`, very-high flat fraction, low `motion` | deblock (flat-snap + ring-strip) | edge-preserving | medium | +| **high-motion/sports** | `motion` ≥ 0.6 (needs upstream `mc`) | denoise + mc-warp (ADR-0131, on cleaner pixels) | tcut tight | large | +| **clean/pristine** | all impairment signals below floor | **NO-OP** (emit zero filters) | — | ~0 by design (avoided loss) | + +**Critical constraints the router must respect** (verified against the analyze source): +`analyze.texture` is a *derived* `0.5·var + 0.5·edge` blend (not orthogonal) — classify on +`variance`+`edge` directly; `analyze.motion` is 0 unless an upstream `pelorus_mc` attached +`PEL_SEC_MOTION` — every motion-gated route needs `mc` in the graph. + +**The detection gap (this PR's first enabler).** Three detectors need host-side scalars +that did not exist (only `analyze` emitted metadata; `grain_estimate`/`denoise`/`mc` emitted +none): `grain_sigma`, `noise_sigma`, `dark_frac`. All are addable via the ADR-0136 +`av_dict_set` pattern — **no interop ABI change, no shader change, the value is already +computed**. This PR ships the first and highest-leverage one: `vf_pelorus_grain_estimate` +now emits `lavfi.pelorus.grain_sigma` (peak per-band RMS residual over the edge-gated flat +bands — structure is excluded by construction, so what survives is grain stddev) and +`lavfi.pelorus.grain_flat` (the flat fraction it was measured over = confidence). Verified +discriminating: heavy-grain clip 0.0191 vs clean-ish Bluray 0.0116. The grainy leg is the +proven `−34%` lever, so unblocking its detection is the right first step. + +## Validation priority (each leg iso-bitrate-proven before it enters the router) + +1. **grainy** `strength ∝ grain_sigma` scaling (ablate vs a flat preset) — proven lever, just the law. +2. **clean** NO-OP guard (default ties/beats the full chain on pristine clips — the avoided-loss). +3. **high-motion** denoise (the denoise half is proven on BBB; validate on real sports). +4. **textured** micro-noise strip (stand-in shows −19…−23% bits; confirm it survives the gate). +5. **anime** stack (dehalo/aa/deband ablations on real impaired anime). +6. **re-encode** demosquito **premise-check first** (edge-band temporal excess ≫ flat, and denoise leaves it untouched). +7. **dark/HDR** dark-masked denoise (rides the proven lever; the survival-dither leg is the ADR-0141 risk). + +Every leg: tune on one clip, **confirm on a held-out clip**, iso-bitrate, VMAF-NEG vs clean — +so we find *real* tuned gains, not params overfit to one clip's metric. + +## A parallel track — encoder-integration patches + +Distinct from the pre-encode filter lane: patch FFmpeg's `nvenc`/`qsv`/`amf` wrappers to +expose/tune hardware features they underuse (temporal-AQ, deeper lookahead, weighted +prediction for the fade weakness, `b_ref_mode`, the native emphasis-level map our analyze +maps could drive; oneVPL per-block QP / low-power AQ for Intel). Tracked as a parallel +research+validate track — **check the current SDKs** (not memory) for what FFmpeg actually +fails to expose, and prove each patch iso-bitrate on real hardware (the NVENC ME-hints patch +was a zero-gain reminder that encoder patches can also be no-gains). + +## Consequences + +- This PR: `grain_estimate` emits `grain_sigma`/`grain_flat` (filter-only, no ABI/shader + change, side-data path untouched). Foundation for the grainy route. +- Follow-ups (separate PRs, in priority order): the remaining detection enablers + (`noise_sigma`, `dark_frac`, `edge_temporal_excess`), the per-leg validations, the + `tune=auto` router filter/preset, and the encoder-patch track. +- Reinforces the project lane: reductive preprocessing tuned per content, never RC-competing, + never additive-past-the-deadzone. + +## References + +- bench-results v0.2/v0.3 (denoise BD-rate), v0.16/ADR-0141 (additive wash + over-reach), the + session campaign (NVENC0.05` heavy). | +| `lavfi.pelorus.grain_flat` | fraction of the frame the estimate was measured over (flat pixels / total). The estimate's confidence — heavy-edge frames give a small flat fraction and an unreliable sigma. | + +Read them with `metadata=print` or `ffprobe -show_frames`: + +```bash +ffmpeg -init_hw_device vulkan=vk:0 -i in.mkv \ + -vf "hwupload,pelorus_grain_estimate_vulkan,hwdownload,format=yuv420p,metadata=print:key=lavfi.pelorus.grain_sigma" \ + -f null - +``` diff --git a/ffmpeg-patches/0006-add-vf_pelorus_grain_estimate_vulkan.patch b/ffmpeg-patches/0006-add-vf_pelorus_grain_estimate_vulkan.patch index 2fee117..c0ed42e 100644 --- a/ffmpeg-patches/0006-add-vf_pelorus_grain_estimate_vulkan.patch +++ b/ffmpeg-patches/0006-add-vf_pelorus_grain_estimate_vulkan.patch @@ -1,7 +1,7 @@ From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001 From: Lusoris -Date: Fri, 19 Jun 2026 19:50:06 +0200 -Subject: [PATCH 06/11] feat(grain): add vf_pelorus_grain_estimate_vulkan (FGS +Date: Sat, 27 Jun 2026 21:26:02 +0200 +Subject: [PATCH 06/18] feat(grain): add vf_pelorus_grain_estimate_vulkan (FGS param estimator) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 @@ -43,8 +43,8 @@ ADR: docs/adr/0115-grain-estimate.md configure | 2 + libavfilter/Makefile | 1 + libavfilter/allfilters.c | 1 + - .../vf_pelorus_grain_estimate_vulkan.c | 654 ++++++++++++++++++ - 4 files changed, 658 insertions(+) + .../vf_pelorus_grain_estimate_vulkan.c | 682 ++++++++++++++++++ + 4 files changed, 686 insertions(+) create mode 100644 libavfilter/vf_pelorus_grain_estimate_vulkan.c diff --git a/configure b/configure @@ -93,10 +93,10 @@ index 8f49f54808..5a5f777074 100644 extern const FFFilter ff_vf_phase; diff --git a/libavfilter/vf_pelorus_grain_estimate_vulkan.c b/libavfilter/vf_pelorus_grain_estimate_vulkan.c new file mode 100644 -index 0000000000..3d80ab0f44 +index 0000000000..68891c58ab --- /dev/null +++ b/libavfilter/vf_pelorus_grain_estimate_vulkan.c -@@ -0,0 +1,654 @@ +@@ -0,0 +1,682 @@ +/* + * Copyright 2026 Lusoris + * @@ -356,6 +356,16 @@ index 0000000000..3d80ab0f44 + uint64_t flat_total; /* total flat pixels measured */ +} PelorusGrainStats; + ++/* Emit a scalar as lavfi.pelorus.* frame metadata (ADR-0136 pattern), so the ++ * tune=auto content router can read the grain estimate without parsing the ++ * PEL_SEC_FILMGRAIN side-data blob. */ ++static void pel_set_meta_f(AVFrame *frame, const char *key, float v) ++{ ++ char buf[32]; ++ snprintf(buf, sizeof(buf), "%.6f", v); ++ av_dict_set(&frame->metadata, key, buf, 0); ++} ++ +static void reduce_stats(const PelorusGrainBuf *acc, PelorusGrainStats *st) +{ + double corr_sum = 0.0, var_sum = 0.0; @@ -663,6 +673,24 @@ index 0000000000..3d80ab0f44 + + reduce_stats(acc, &st); + ++ /* Grain discriminator as frame metadata for the tune=auto router: peak grain ++ * sigma over populated bands (the residual is measured ONLY on edge-gated ++ * locally-flat pixels, so structure is excluded by construction and what ++ * survives is grain stddev), plus the flat fraction it was measured over ++ * (the estimate's confidence). No interop ABI / shader change; the ++ * PEL_SEC_FILMGRAIN side-data path is untouched. */ ++ { ++ float gs = 0.0f; ++ int b; ++ for (b = 0; b < PEL_GRAIN_BANDS; b++) ++ if (st.band_cnt[b] > 0 && st.band_rms[b] > gs) ++ gs = st.band_rms[b]; ++ pel_set_meta_f(in, "lavfi.pelorus.grain_sigma", gs); ++ pel_set_meta_f(in, "lavfi.pelorus.grain_flat", ++ (float)((double)st.flat_total / ++ ((double)in->width * (double)in->height))); ++ } ++ + memset(&g, 0, sizeof(g)); + map_aom(s, &st, &g); + diff --git a/ffmpeg-patches/files/vf_pelorus_grain_estimate_vulkan.c b/ffmpeg-patches/files/vf_pelorus_grain_estimate_vulkan.c index 3d80ab0..68891c5 100644 --- a/ffmpeg-patches/files/vf_pelorus_grain_estimate_vulkan.c +++ b/ffmpeg-patches/files/vf_pelorus_grain_estimate_vulkan.c @@ -257,6 +257,16 @@ typedef struct PelorusGrainStats { uint64_t flat_total; /* total flat pixels measured */ } PelorusGrainStats; +/* Emit a scalar as lavfi.pelorus.* frame metadata (ADR-0136 pattern), so the + * tune=auto content router can read the grain estimate without parsing the + * PEL_SEC_FILMGRAIN side-data blob. */ +static void pel_set_meta_f(AVFrame *frame, const char *key, float v) +{ + char buf[32]; + snprintf(buf, sizeof(buf), "%.6f", v); + av_dict_set(&frame->metadata, key, buf, 0); +} + static void reduce_stats(const PelorusGrainBuf *acc, PelorusGrainStats *st) { double corr_sum = 0.0, var_sum = 0.0; @@ -564,6 +574,24 @@ static int grain_estimate_vulkan_filter_frame(AVFilterLink *link, AVFrame *in) reduce_stats(acc, &st); + /* Grain discriminator as frame metadata for the tune=auto router: peak grain + * sigma over populated bands (the residual is measured ONLY on edge-gated + * locally-flat pixels, so structure is excluded by construction and what + * survives is grain stddev), plus the flat fraction it was measured over + * (the estimate's confidence). No interop ABI / shader change; the + * PEL_SEC_FILMGRAIN side-data path is untouched. */ + { + float gs = 0.0f; + int b; + for (b = 0; b < PEL_GRAIN_BANDS; b++) + if (st.band_cnt[b] > 0 && st.band_rms[b] > gs) + gs = st.band_rms[b]; + pel_set_meta_f(in, "lavfi.pelorus.grain_sigma", gs); + pel_set_meta_f(in, "lavfi.pelorus.grain_flat", + (float)((double)st.flat_total / + ((double)in->width * (double)in->height))); + } + memset(&g, 0, sizeof(g)); map_aom(s, &st, &g);