From 3ecf2234708a61f8120fa2e6fc60b7f07636716d Mon Sep 17 00:00:00 2001 From: opariffazman Date: Tue, 23 Jun 2026 23:50:11 +0800 Subject: [PATCH 01/26] docs: add Phase 1 AI sharpness/resolution design spec Artifact Reduction + Super Resolution (Maxine VFX, no new co-version) + mipmapped downscale + real fps counter. Independent toggles, no auto GPU management. Phase 2 (fps interpolation via Optical Flow FRUC) tracked as #13. Co-Authored-By: Claude Opus 4.8 (1M context) --- ...n-screen-ai-sharpness-resolution-design.md | 202 ++++++++++++++++++ 1 file changed, 202 insertions(+) create mode 100644 docs/superpowers/specs/2026-06-23-camera-on-screen-ai-sharpness-resolution-design.md diff --git a/docs/superpowers/specs/2026-06-23-camera-on-screen-ai-sharpness-resolution-design.md b/docs/superpowers/specs/2026-06-23-camera-on-screen-ai-sharpness-resolution-design.md new file mode 100644 index 0000000..8001ce9 --- /dev/null +++ b/docs/superpowers/specs/2026-06-23-camera-on-screen-ai-sharpness-resolution-design.md @@ -0,0 +1,202 @@ +# Camera-on-Screen — Design Spec: AI Sharpness & Resolution (Phase 1) + +**Date:** 2026-06-23 +**Status:** Approved (design); pending implementation plan +**Depends on:** M3 (AI Green Screen) + M4 (Eye Contact) — both merged to `main`. Reuses the +worker-thread effect chain (`aigs.{h,cpp}` pattern), the capability probe, and the live-param push. +**Parent spec:** `docs/superpowers/specs/2026-06-20-camera-on-screen-design.md`. +**Phase 2 (separate, deferred):** fps interpolation — GitHub issue #13. + +## 1. Goal + +Make the overlay image sharper and higher-resolution than the camera hardware delivers, using +NVIDIA Maxine AI. Two distinct user-visible wins: + +1. **Cleaner image** — remove the MJPG/H.264 compression artifacts the webcam emits at + 1080p30 (the source of the "softer than the native camera app" complaint). +2. **More effective resolution** — AI-upscale the source so the overlay stays sharp when shown + large, not just at widget size. + +The overlay display size **varies** (small widget ↔ near-fullscreen), so the design must hold up +at any scale: clean + sharpen at the source for small sizes, real upscaled detail for large sizes. + +## 2. Scope + +**In scope (Phase 1):** +- **Artifact Reduction** (Maxine VFX `NVVFX_FX_ARTIFACT_REDUCTION`) — removes compression + artifacts, in-place at source resolution, no dimension change. +- **Super Resolution** (Maxine VFX `NVVFX_FX_SUPER_RES`) — AI upscale, off / 1.5× / 2×, writes a + larger output buffer. +- **Mipmapped downscale** in the overlay present path — fixes bilinear-minification softness when + the overlay is shown smaller than the source. Independent of Maxine; ships first. +- **Real fps counter** — replace the hardcoded `cos_get_status` `30.0` stub with a measured value + so the user can see each effect's cost. +- Independent on/off toggles per effect; capability gating; live-param push; JSON persistence. + +**Out of scope:** +- **fps interpolation (Phase 2)** — NVIDIA Optical Flow FRUC, separate SDK, co-version risk. + GitHub issue #13. Gated on a standalone co-version smoke test before any integration. +- **Automatic GPU-budget management** — no scheduler, no mutual exclusion, no auto-drop. The user + toggles effects and watches the fps readout (explicit decision). +- **Maxine Video Noise Removal / Upscale (non-AI) / relighting** — not needed for this goal. +- **3×/4× Super Res** — capped at 2× (see §7). + +## 3. Why this is low-risk: no new co-version + +Super Resolution and Artifact Reduction live in the **same Maxine VideoFX SDK** already integrated +for green screen (VFX 1.2.0.0). Same `NVVideoEffects.dll` dispatcher, same per-feature-DLL model, +**same TensorRT 10.9 / CUDA 12.x runtime**. Adding them is **not** a new co-version — unlike Phase 2's +Optical Flow SDK. They compile behind the existing `COS_HAS_MAXINE` flag; without the SDK they build +as passthrough stubs, exactly like green screen. + +## 4. Worker-thread effect chain + +`capture.cpp WorkerLoop`, per frame (each stage gated on its atomic enable flag, lazy Start/Stop +identical to the current green-screen block): + +``` +decode (RGB32 BGRA) + → [Artifact Reduction] in-place, source res + → Eye Contact (existing) + → Green Screen (existing) + → [Super Resolution] → LARGER output buffer, new w/h + → publish frame (w/h = SR output size) +``` + +**Order rationale:** +- Artifact Reduction runs **first** so the AI effects downstream consume a clean frame (better + matte and landmarks). +- Super Resolution runs **last** so the expensive upscale happens once on the final composite, and + the green-screen matte + gaze landmarks compute at native 1080p (cheaper, and the models are + tuned for that resolution). + +Maxine effects remain worker-thread-local (CUDA/NvVFX thread affinity); the UI only flips atomic +enable flags; status crosses threads via atomics + leaf-lock (never nested under `g_state.mtx` / +`g_lifecycleMtx`) — unchanged invariants. + +## 5. New shim effect modules + +Two files each, lifecycle shape copied from `aigs.{h,cpp}` +(`Start()/Stop()/IsReady()/ProcessFrame()/LastError()`, worker-thread-local): + +- `native/shim/artifactreduction.{h,cpp}` — `NVVFX_FX_ARTIFACT_REDUCTION`, mode 1 + (strong / for-compressed). Operates in place on the BGRA scratch buffer at source resolution. + **No dimension change.** +- `native/shim/superres.{h,cpp}` — `NVVFX_FX_SUPER_RES`, mode 1, upscale factor from params + (1.5× or 2×). Allocates/owns a **larger** output buffer; `ProcessFrame` returns the new width and + height to the worker, which publishes them. + +Both behind `COS_HAS_MAXINE` → CI-safe passthrough stub when the SDK is absent. The exact +feature-DLL names and Maxine property keys (`NVVFX_STRENGTH`, `NVVFX_MODE`, the upscale-factor key) +are verified against the VFX 1.2.0.0 headers at implementation time. + +## 6. Dimension-change plumbing (the only structurally new part) + +Super Res makes published frames larger than 1080p, which ripples through the consumer: + +- **`MainWindow.xaml.cs:25` `_frameBuf`** is a fixed `1920*1080*4` array — would overflow. + **Pre-size it to 4K** (`3840*2160*4` ≈ 33 MB). One constant change; no dynamic realloc logic. + *(ponytail: pre-size beats a resize-on-demand path; 33 MB is negligible.)* This is the hard cap + on SR output and is the reason SR is capped at 2× of 1080p. +- **Swap chain** — `OverlayWindow.PresentFrame` **already** re-pins the back buffer to the incoming + frame size (`if (_bufW != width || _bufH != height)`). SR's larger frame re-pins once on first + arrival. No change needed. +- **Frame C-ABI** already returns dynamic `w`/`h` from `get_frame`. No contract change for size. + +## 7. Super Res factor cap + +Capped at **2×** (1080p → 2160p). Rationale: 2× of 1080p hits exactly 4K, matching the pre-sized +`_frameBuf`; 3×/4× would need a bigger buffer and far more GPU for diminishing perceptual gain at +overlay sizes. Exposed as off / 1.5× / 2×. Higher factors are a later change if ever needed. + +## 8. C-ABI parity (load-bearing — x64 byte-for-byte) + +Both `shim.h` (C) and `PInvokeShim` (`[StructLayout(Sequential)]` mirror) change together: + +- **`CosParams`** — add `int artifactReductionEnabled`, `int superResEnabled`, `int superResScale` + (`0` = off, else factor ×10: `15` = 1.5×, `20` = 2×). +- **`CosCaps`** — add `int artifactReductionAvailable`, `int superResAvailable` gates. Update the + documented struct size (currently two `int` gates + `char detail[512]` = 520 bytes; each new gate + adds 4 bytes — keep the comment and the C#-side size in sync). +- **`cos_query_capabilities`** — extend the probe to create + load each new effect, exactly as it + already does for green screen and gaze, and set the new gates from the result. + +## 9. Real fps counter + +Replace the hardcoded `30.0` in `cos_get_status`: +- The capture worker increments an atomic frame count and stamps `std::chrono::steady_clock` at + publish. +- `cos_get_status` computes fps over a rolling window (or simple EWMA) under the existing status + leaf-lock — never nested under `g_state.mtx`. +- The existing C# `Fps` property (already polled every pump tick → `PollStatus` → `GetStatus`) now + shows the real value. No new plumbing on the managed side. + +## 10. C# / MVVM (Core + App) + +- **`MainViewModel`** — new observable props `ArtifactReductionEnabled`, `SuperResEnabled`, + `SuperResScale`. Their `On…Changed` partials call `ApplyLiveParams()` → + `Orchestrator.ApplyParams(BuildParams())` → `shim.SetParams`, gated on `IsRunning` + + `EffectsAvailable`, identical to the green-screen props. `ApplyParams` already forces effects off + when `EffectsAvailable` is false — the new flags inherit that. +- **`BuildParams()`** — copy the three new fields into `ShimParams`/`CosParams`. +- **`ToAppConfig` / `AppConfig`** — persist the three fields in `config.json` (anything not copied + reverts to default — must add explicitly). +- **XAML** — two toggles + a Super Res factor combo, bound `Mode=OneWay` to the availability gates + (greyed until the deferred probe lands). The status line shows the now-real fps. + +## 11. Mip downscale (present path — ships first, standalone) + +Replace the `CopyResource` blit in `OverlayWindow.PresentFrame` (`OverlayWindow.cs:328`) with a +textured fullscreen-quad draw: +- frame texture created with `D3D11_RESOURCE_MISC_GENERATE_MIPS`, +- `GenerateMips()` after upload, +- draw a fullscreen quad into the back buffer with a **trilinear** sampler. + +Mip/trilinear minification averages source texels properly → sharp at any shrink ratio, fixing the +bilinear-undersampling softness. Independent of the Maxine work; lands first and is independently +valuable. (Upscale magnification blur at large sizes is what Super Res addresses — the two are +complementary.) + +## 12. Bundler / installer + +- `scripts/assemble-maxine-stage.ps1` + `native/shim/bundle/maxine-manifest.psd1` — add the + SuperRes + ArtifactReduction feature DLLs and their model files to the curated stage. Same VFX + runtime, so **no new TRT/CUDA DLLs** — only the two feature DLLs + models. +- Re-run `native/shim/smoke/trace_closure.cpp` against the produced bundle to capture the new + load closure; update the manifest `Dlls` list from its output. +- `verify-bundle.ps1` runtime probe (`COS_*` unset) must still load all effects. + +## 13. Performance stance + +Independent toggles, real fps readout, **no auto-management** (explicit user decision). Stacking +four AI inferences (Artifact Reduction + Eye Contact + Green Screen + Super Res) will not sustain +high fps on an RTX 3090; the user watches the fps readout and disables to taste. Document this in +the UI note + CLAUDE.md. No scheduler, no mutual exclusion (YAGNI). + +## 14. Testing + +- **Core unit tests** — `BuildParams` / `ToAppConfig` round-trip the three new fields; + `ApplyParams` forces them off when `EffectsAvailable == false` (extend existing green-screen tests). +- **Native smoke** — extend `cos_query_capabilities` probe coverage; an AR/SR `Start` + + `ProcessFrame` smoke on a synthetic frame returns success and (for SR) a larger output size + (mirrors `aigs_smoke.cpp`). +- **fps counter self-check** — feed N frames over a known elapsed time, assert reported fps ≈ N/elapsed. +- **Human gate** — visual sharpness confirmation via a real recorder (Windows.Graphics.Capture / + OBS), the inherent verification gate (`docs/superpowers/verification/`). + +## 15. Incremental landing order + +Each step is independently shippable and independently valuable: + +1. **Mip downscale** (present path) — standalone sharpness win, no Maxine dependency. +2. **Real fps counter** — needed to tune everything after. +3. **Artifact Reduction** — clean source, no dimension change (low risk). +4. **Super Resolution** + 4K `_frameBuf` pre-size — the dimension-change step, landed last. + +## 16. Open questions / verify-at-impl + +- Exact VFX 1.2.0.0 feature-DLL filenames + property keys for SuperRes / ArtifactReduction. +- Whether Super Res mode 1 (compressed) or mode 0 looks better on this webcam's MJPG output — + decide visually during the human gate. +- Confirm trilinear-quad present path keeps the `WS_EX_NOREDIRECTIONBITMAP` clean-capture behavior + (it should — still a DComp flip-model swap chain, only the blit changes). From 60cb6a3cb79cf829964b5b3ab5e597d25b0ea8e1 Mon Sep 17 00:00:00 2001 From: opariffazman Date: Wed, 24 Jun 2026 00:05:25 +0800 Subject: [PATCH 02/26] =?UTF-8?q?docs:=20self-review=20fixes=20=E2=80=94?= =?UTF-8?q?=20defer=20present-path=20minification=20from=20Phase=201?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Found inconsistency: the mip-downscale "swap CopyResource for a quad" cannot fix minification while the back buffer stays pinned to frame res — DComp does the final bilinear minification and ignores the mips. The real fix needs a window-sized back buffer + shader scale (mirror/zoom/MPO invariants), a present-path redesign. Descoped to deferred §11; Phase 1 = Artifact Reduction + Super Res + real fps counter. Tighten CosCaps size 520->528. Co-Authored-By: Claude Opus 4.8 (1M context) --- ...n-screen-ai-sharpness-resolution-design.md | 62 ++++++++++++------- 1 file changed, 39 insertions(+), 23 deletions(-) diff --git a/docs/superpowers/specs/2026-06-23-camera-on-screen-ai-sharpness-resolution-design.md b/docs/superpowers/specs/2026-06-23-camera-on-screen-ai-sharpness-resolution-design.md index 8001ce9..0ea99ba 100644 --- a/docs/superpowers/specs/2026-06-23-camera-on-screen-ai-sharpness-resolution-design.md +++ b/docs/superpowers/specs/2026-06-23-camera-on-screen-ai-sharpness-resolution-design.md @@ -17,8 +17,11 @@ NVIDIA Maxine AI. Two distinct user-visible wins: 2. **More effective resolution** — AI-upscale the source so the overlay stays sharp when shown large, not just at widget size. -The overlay display size **varies** (small widget ↔ near-fullscreen), so the design must hold up -at any scale: clean + sharpen at the source for small sizes, real upscaled detail for large sizes. +The overlay display size **varies** (small widget ↔ near-fullscreen). Phase 1 cleans compression +artifacts (helps all sizes) and adds AI-upscaled detail (helps large sizes, where there is little +or no downscale). The remaining **small-size softness** — bilinear minification in the present path +— is a separate present-path redesign and is **deferred** (see §2 and §11), not solved by the +Maxine effects here. ## 2. Scope @@ -27,13 +30,16 @@ at any scale: clean + sharpen at the source for small sizes, real upscaled detai artifacts, in-place at source resolution, no dimension change. - **Super Resolution** (Maxine VFX `NVVFX_FX_SUPER_RES`) — AI upscale, off / 1.5× / 2×, writes a larger output buffer. -- **Mipmapped downscale** in the overlay present path — fixes bilinear-minification softness when - the overlay is shown smaller than the source. Independent of Maxine; ships first. - **Real fps counter** — replace the hardcoded `cos_get_status` `30.0` stub with a measured value so the user can see each effect's cost. - Independent on/off toggles per effect; capability gating; live-param push; JSON persistence. **Out of scope:** +- **Present-path minification fix (deferred — see §11).** The small-size bilinear-minification + softness needs a present-path redesign (window-sized back buffer + mipped shader scale), which + collides with load-bearing invariants (pinned-to-frame-res back buffer, mirror/zoom in the DComp + transform, the MPO hit-test gotcha). Separated from the Maxine effects to keep Phase 1 low-risk. + To be filed as its own issue if pursued. - **fps interpolation (Phase 2)** — NVIDIA Optical Flow FRUC, separate SDK, co-version risk. GitHub issue #13. Gated on a standalone co-version smoke test before any integration. - **Automatic GPU-budget management** — no scheduler, no mutual exclusion, no auto-drop. The user @@ -115,9 +121,10 @@ Both `shim.h` (C) and `PInvokeShim` (`[StructLayout(Sequential)]` mirror) change - **`CosParams`** — add `int artifactReductionEnabled`, `int superResEnabled`, `int superResScale` (`0` = off, else factor ×10: `15` = 1.5×, `20` = 2×). -- **`CosCaps`** — add `int artifactReductionAvailable`, `int superResAvailable` gates. Update the - documented struct size (currently two `int` gates + `char detail[512]` = 520 bytes; each new gate - adds 4 bytes — keep the comment and the C#-side size in sync). +- **`CosCaps`** — add `int artifactReductionAvailable`, `int superResAvailable` gates. The struct + grows from two `int` gates + `char detail[512]` = **520 bytes** to **four** `int` gates + + `detail[512]` = **528 bytes**; update the size comment in `shim.h` and the `[StructLayout]` C# + mirror together. - **`cos_query_capabilities`** — extend the probe to create + load each new effect, exactly as it already does for green screen and gaze, and set the new gates from the result. @@ -144,18 +151,30 @@ Replace the hardcoded `30.0` in `cos_get_status`: - **XAML** — two toggles + a Super Res factor combo, bound `Mode=OneWay` to the availability gates (greyed until the deferred probe lands). The status line shows the now-real fps. -## 11. Mip downscale (present path — ships first, standalone) +## 11. Present-path minification softness (DEFERRED — not built in Phase 1) -Replace the `CopyResource` blit in `OverlayWindow.PresentFrame` (`OverlayWindow.cs:328`) with a -textured fullscreen-quad draw: -- frame texture created with `D3D11_RESOURCE_MISC_GENERATE_MIPS`, -- `GenerateMips()` after upload, -- draw a fullscreen quad into the back buffer with a **trilinear** sampler. +Recorded here so the analysis is not lost; this is **out of scope** for Phase 1 (§2). -Mip/trilinear minification averages source texels properly → sharp at any shrink ratio, fixing the -bilinear-undersampling softness. Independent of the Maxine work; lands first and is independently -valuable. (Upscale magnification blur at large sizes is what Super Res addresses — the two are -complementary.) +**Root cause (confirmed in code):** the back buffer is pinned to the **frame** resolution +(`PresentFrame` — `CopyResource(back, _frameTex)` is 1:1, `_frameTex` is `MipLevels = 1`), and the +frame-res → window scaling is done by the **DComp visual transform** (`UpdateScale` → `SetTransform`), +which samples **bilinear** with no mip control. A 1080p → ~480px widget downscale therefore +undersamples → softness/aliasing. + +**Why it is not a drop-in blit swap:** putting mips on `_frameTex` and drawing a quad into the +*current* frame-res back buffer changes nothing — DComp still does the final minification afterward +and ignores our mips. A correct fix requires moving the scale into our own D3D draw: +- back buffer becomes **window-sized** (drops the pinned-to-frame-res / `CopyResource` 1:1 invariant), +- draw a fullscreen quad sampling a **mipped** `_frameTex` (`GENERATE_MIPS` + `GenerateMips()`) with + a **trilinear** sampler, +- DComp transform becomes **identity**, and **mirror + center-zoom** move out of the DComp + `Matrix3x2` into the quad's transform, +- re-verify `WS_EX_NOREDIRECTIONBITMAP` clean capture and the size-dependent **MPO hit-test** gotcha + still hold with a window-sized back buffer. + +This touches load-bearing present-path invariants, so it is its own milestone. Note: Super Res does +**not** substitute for it — the upscaled detail is also thrown away by the DComp bilinear downscale +at small sizes. ## 12. Bundler / installer @@ -188,15 +207,12 @@ the UI note + CLAUDE.md. No scheduler, no mutual exclusion (YAGNI). Each step is independently shippable and independently valuable: -1. **Mip downscale** (present path) — standalone sharpness win, no Maxine dependency. -2. **Real fps counter** — needed to tune everything after. -3. **Artifact Reduction** — clean source, no dimension change (low risk). -4. **Super Resolution** + 4K `_frameBuf` pre-size — the dimension-change step, landed last. +1. **Real fps counter** — needed to tune everything after. +2. **Artifact Reduction** — clean source, no dimension change (low risk). +3. **Super Resolution** + 4K `_frameBuf` pre-size — the dimension-change step, landed last. ## 16. Open questions / verify-at-impl - Exact VFX 1.2.0.0 feature-DLL filenames + property keys for SuperRes / ArtifactReduction. - Whether Super Res mode 1 (compressed) or mode 0 looks better on this webcam's MJPG output — decide visually during the human gate. -- Confirm trilinear-quad present path keeps the `WS_EX_NOREDIRECTIONBITMAP` clean-capture behavior - (it should — still a DComp flip-model swap chain, only the blit changes). From 695238cc12d904d4c44d7201ebb03395a880eefc Mon Sep 17 00:00:00 2001 From: opariffazman Date: Wed, 24 Jun 2026 00:18:39 +0800 Subject: [PATCH 03/26] docs: Phase 1 AI sharpness/resolution implementation plan MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 9 tasks: real fps counter, shared VFX path resolver, Artifact Reduction + Super Resolution effects, ABI growth (CosParams +3 / CosCaps 528B), worker chain wiring, Core+managed plumbing with xUnit coverage, App UI + 4K buffer, bundler. TDD steps with full code; minification fix deferred per spec §11. Co-Authored-By: Claude Opus 4.8 (1M context) --- ...26-06-24-ai-sharpness-resolution-phase1.md | 1467 +++++++++++++++++ 1 file changed, 1467 insertions(+) create mode 100644 docs/superpowers/plans/2026-06-24-ai-sharpness-resolution-phase1.md diff --git a/docs/superpowers/plans/2026-06-24-ai-sharpness-resolution-phase1.md b/docs/superpowers/plans/2026-06-24-ai-sharpness-resolution-phase1.md new file mode 100644 index 0000000..3dba841 --- /dev/null +++ b/docs/superpowers/plans/2026-06-24-ai-sharpness-resolution-phase1.md @@ -0,0 +1,1467 @@ +# AI Sharpness & Resolution (Phase 1) Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Add Maxine VFX **Artifact Reduction** + **Super Resolution** effects and a **real fps counter** to the overlay pipeline, giving a sharper/higher-resolution image than the 1080p30 webcam delivers. + +**Architecture:** Two new VFX effects run on the existing capture worker thread, mirroring `aigs.cpp` (green screen): Artifact Reduction first (cleans MJPG artifacts, no size change), Super Resolution last (upscales, larger output buffer). Both live in the same VFX 1.2.0.0 runtime already used for green screen — **no new co-version**. The C ABI grows three `CosParams` fields + two `CosCaps` gates; the managed MVVM chain gains toggles, capability gates, persistence, and live-param push, exactly like the green-screen props. + +**Tech Stack:** C++ (Media Foundation + Maxine VFX, `COS_HAS_MAXINE`), C# .NET 8 (Core + WinUI 3 App), CommunityToolkit.Mvvm, xUnit. + +**Spec:** `docs/superpowers/specs/2026-06-23-camera-on-screen-ai-sharpness-resolution-design.md`. Phase 2 (fps interpolation) = GitHub issue #13. Present-path minification fix is **deferred** (spec §11) — not in this plan. + +## Global Constraints + +- **Builds & tests must be pristine — 0 warnings** (CI enforces `/warnaserror` + `TreatWarningsAsErrors`). +- **Build the native shim BEFORE the App** (App copies `native/shim/x64/$(Configuration)/CameraOnScreen.Shim.dll`). Build the **SDK config last** before running (deploy-the-right-shim). +- **C ABI struct parity is byte-for-byte on x64.** `CosStatus`/`CosParams`/`CosCaps` in `shim.h` and their `[StructLayout(Sequential)]` mirrors in `PInvokeShim` must match. After ABI change verify exports: `dumpbin /exports` must still list exactly the **9** `cos_*` functions. +- **Maxine code is behind `COS_HAS_MAXINE`**; without the SDK the effect compiles as a passthrough stub (never ready). New VFX effects use the **same VFX runtime** as green screen — do NOT introduce a new TensorRT/CUDA runtime. +- **Effects run on the capture worker thread only.** The UI flips atomic enable flags; status crosses threads via atomics + **leaf locks** (never nested under `g_state.mtx` / `g_lifecycleMtx`). +- **Super Res output is capped at 2×** → max frame 3840×2160. The managed `_frameBuf` is pre-sized to that. +- Shim build (PowerShell): `& "C:/Program Files (x86)/Microsoft Visual Studio/2022/BuildTools/MSBuild/Current/Bin/MSBuild.exe" native/shim/shim.vcxproj /p:Configuration=Debug /p:Platform=x64` +- App build: `dotnet build src/CameraOnScreen.App/CameraOnScreen.App.csproj -t:Rebuild` +- Core tests: `dotnet test tests/CameraOnScreen.Core.Tests/CameraOnScreen.Core.Tests.csproj` (single: append `--filter "FullyQualifiedName~Name"`). + +--- + +## File Structure + +**New files:** +- `native/shim/fps_counter.h` — header-only rolling FPS estimator (no Win32/Maxine deps; unit-testable). +- `native/shim/vfx_paths.h` / `vfx_paths.cpp` — shared VFX SDK path resolver + proxy pointer, reused by the two new effects (green-screen `aigs.cpp` keeps its own copy untouched). +- `native/shim/artifactreduction.h` / `artifactreduction.cpp` — `ArtifactReduction` effect (VFX `NVVFX_FX_ARTIFACT_REDUCTION`), in-place, no size change. +- `native/shim/superres.h` / `superres.cpp` — `SuperRes` effect (VFX `NVVFX_FX_SUPER_RES`), larger output buffer. +- `native/shim/smoke/fps_smoke.cpp` — asserts on the FPS estimator (no GPU). +- `native/shim/smoke/effects_smoke.cpp` — dev-box smoke: AR + SR `Start`+`ProcessFrame` on a synthetic frame (needs SDK+GPU). + +**Modified files:** +- `native/shim/capture.h` / `capture.cpp` — fps counter, AR/SR enable atomics + Set methods + active/error state, worker-chain insertion. +- `native/shim/shim.h` — `CosParams` +3 fields, `CosCaps` +2 gates. +- `native/shim/shim.cpp` — real fps in `cos_get_status`; wire AR/SR in `cos_set_params` + `cos_query_capabilities`. +- `native/shim/shim.vcxproj` — compile the four new `.cpp` files. +- `src/CameraOnScreen.Core/Native/Contracts.cs` — `ShimParams` +3, `ShimCapabilities` +2. +- `src/CameraOnScreen.Core/Native/FakeShim.cs` — new capability gates. +- `src/CameraOnScreen.Core/Config/Models.cs` — `EffectSettings` +3. +- `src/CameraOnScreen.Core/Orchestration/Orchestrator.cs` — AR/SR availability + gating in `ApplyParams`. +- `src/CameraOnScreen.Core/ViewModels/MainViewModel.cs` — props, `BuildParams`, `LoadFrom`, `ToAppConfig`, live-push partials. +- `src/CameraOnScreen.App/Native/PInvokeShim.cs` — struct mirrors + `SetParams`/`QueryCapabilities` mapping. +- `src/CameraOnScreen.App/MainWindow.xaml` — two toggles + scale combo. +- `src/CameraOnScreen.App/MainWindow.xaml.cs` — `_frameBuf` pre-size to 4K. +- `scripts/assemble-maxine-stage.ps1` + `native/shim/bundle/maxine-manifest.psd1` — bundle AR/SR feature DLLs + models. + +--- + +## Task 1: Real fps counter (native) + +Replaces the hardcoded `30.0` stub so the user can see effect cost. Fully isolated: no ABI change, no managed change (the managed side already plumbs `Fps`). + +**Files:** +- Create: `native/shim/fps_counter.h` +- Create: `native/shim/smoke/fps_smoke.cpp` +- Modify: `native/shim/capture.h`, `native/shim/capture.cpp`, `native/shim/shim.cpp` + +**Interfaces:** +- Produces: `class FpsCounter { double Sample(double nowSec, uint64_t totalFrames); void Reset(); }` and `double Capture::MeasuredFps() const;` + +- [ ] **Step 1: Write the failing test** — `native/shim/smoke/fps_smoke.cpp` + +```cpp +// Standalone smoke for FpsCounter. Build: cl /EHsc /I.. fps_smoke.cpp +#include +#include +#include +#include "../fps_counter.h" + +static bool near(double a, double b) { return std::fabs(a - b) < 1e-6; } + +int main() { + FpsCounter c; + assert(near(c.Sample(0.0, 0), 0.0)); // first sample primes, no estimate yet + assert(near(c.Sample(1.0, 30), 30.0)); // 30 frames in 1.0s -> 30 fps + assert(near(c.Sample(1.2, 45), 30.0)); // 0.2s < min interval -> keep last estimate + assert(near(c.Sample(2.0, 90), 56.25)); // 45 frames over 0.8s (since last accepted at t=1.0) + c.Reset(); + assert(near(c.Sample(5.0, 999), 0.0)); // reset re-primes + std::puts("fps_smoke OK"); + return 0; +} +``` + +- [ ] **Step 2: Run it to verify it fails** + +Run: `cl /EHsc /I native/shim native/shim/smoke/fps_smoke.cpp /Fe:build/fps_smoke.exe` (from a VS Developer prompt) +Expected: FAIL to compile — `fps_counter.h` does not exist. + +- [ ] **Step 3: Implement `native/shim/fps_counter.h`** + +```cpp +#pragma once +#include + +// Rolling FPS estimate. SINGLE-CONSUMER (the status poll thread) — not internally +// synchronized. Feed a monotonically increasing cumulative frame count plus a +// monotonic clock reading in seconds. Recomputes over the elapsed window, but only +// refreshes once at least kMinInterval has passed so a fast poll can't divide by ~0. +class FpsCounter { +public: + double Sample(double nowSec, uint64_t totalFrames) { + if (!started_) { started_ = true; lastSec_ = nowSec; lastFrames_ = totalFrames; return fps_; } + const double dt = nowSec - lastSec_; + if (dt < kMinInterval) return fps_; // too soon: keep the last estimate + fps_ = static_cast(totalFrames - lastFrames_) / dt; + lastSec_ = nowSec; + lastFrames_ = totalFrames; + return fps_; + } + void Reset() { started_ = false; fps_ = 0.0; lastSec_ = 0.0; lastFrames_ = 0; } +private: + static constexpr double kMinInterval = 0.5; // refresh at most ~2x/sec + bool started_ = false; + double lastSec_ = 0.0; + double fps_ = 0.0; + uint64_t lastFrames_ = 0; +}; +``` + +- [ ] **Step 4: Run the smoke to verify it passes** + +Run: `cl /EHsc /I native/shim native/shim/smoke/fps_smoke.cpp /Fe:build/fps_smoke.exe && build\fps_smoke.exe` +Expected: prints `fps_smoke OK`, exit 0. + +- [ ] **Step 5: Wire the counter into capture** — `native/shim/capture.h` + +Add to the public section of `class Capture`: + +```cpp + // Measured frames-per-second over a rolling window (status polling). Thread-safe to + // call from the single status-poll thread; 0 until the first interval elapses. + double MeasuredFps() const; +``` + +- [ ] **Step 6: Implement counter state + increment** — `native/shim/capture.cpp` + +At the top includes add: + +```cpp +#include +#include "fps_counter.h" +``` + +In `struct CaptureState` add: + +```cpp + std::atomic framesProduced{0}; // bumped by worker after each published frame + FpsCounter fpsCounter; // read only by MeasuredFps (status-poll thread) +``` + +In `WorkerLoop`, immediately after `g_state.hasNewFrame = true;` (the line at the end of the publish block, ~line 362) add: + +```cpp + g_state.framesProduced.fetch_add(1, std::memory_order_release); +``` + +In `Capture::Start`, inside the `g_state.mtx` block that resets frame state (next to `g_state.hasNewFrame = false;`) add: + +```cpp + g_state.framesProduced.store(0, std::memory_order_release); + g_state.fpsCounter.Reset(); +``` + +At the end of the file add the method: + +```cpp +double Capture::MeasuredFps() const { + using clock = std::chrono::steady_clock; + const double nowSec = + std::chrono::duration(clock::now().time_since_epoch()).count(); + const uint64_t frames = g_state.framesProduced.load(std::memory_order_acquire); + return g_state.fpsCounter.Sample(nowSec, frames); +} +``` + +- [ ] **Step 7: Use it in `cos_get_status`** — `native/shim/shim.cpp` + +Replace: + +```cpp + out->fps = g_running ? 30.0 : 0.0; // still a stub count (documented) +``` + +with: + +```cpp + out->fps = g_running ? g_capture.MeasuredFps() : 0.0; +``` + +- [ ] **Step 8: Build the shim and verify clean + exports unchanged** + +Run (PowerShell): +``` +& "C:/Program Files (x86)/Microsoft Visual Studio/2022/BuildTools/MSBuild/Current/Bin/MSBuild.exe" native/shim/shim.vcxproj /p:Configuration=Debug /p:Platform=x64 +dumpbin /exports native/shim/x64/Debug/CameraOnScreen.Shim.dll | Select-String cos_ +``` +Expected: 0 warnings; exactly 9 `cos_*` exports. + +- [ ] **Step 9: Commit** + +```bash +git add native/shim/fps_counter.h native/shim/smoke/fps_smoke.cpp native/shim/capture.h native/shim/capture.cpp native/shim/shim.cpp +git commit -m "feat(shim): real measured fps in cos_get_status (replaces 30.0 stub)" +``` + +--- + +## Task 2: Shared VFX path resolver + +Extract the VFX SDK path + proxy-pointer logic so the two new effects reuse it. `aigs.cpp` keeps its own copy untouched (do not destabilize working green screen). `g_nvVFXSDKPath` stays **defined** in `aigs.cpp`; this module `extern`s and writes it. + +**Files:** +- Create: `native/shim/vfx_paths.h`, `native/shim/vfx_paths.cpp` +- Modify: `native/shim/shim.vcxproj` + +**Interfaces:** +- Produces: `namespace vfx { bool ResolveSdkPaths(std::string& binDir, std::string& modelDir, std::string& err); void PointProxiesAt(const std::string& binDir); }` + +- [ ] **Step 1: Create `native/shim/vfx_paths.h`** + +```cpp +#pragma once +#include + +// Shared VFX-SDK runtime discovery for the VFX-based effects (Artifact Reduction, +// Super Resolution). Mirrors the resolver in aigs.cpp; kept separate so the proven +// green-screen path is not touched. Model subdir is models\vfx in the bundled layout. +namespace vfx { +// Fills binDir (holds NVVideoEffects.dll) and modelDir. Order: COS_VFX_RUNTIME_DIR +// (flat) -> COS_VFX_SDK_DIR (legacy \bin) -> \maxine (bundled). false if none. +bool ResolveSdkPaths(std::string& binDir, std::string& modelDir, std::string& err); +// Points the VFX proxy global (g_nvVFXSDKPath, defined in aigs.cpp) at binDir. +void PointProxiesAt(const std::string& binDir); +} +``` + +- [ ] **Step 2: Create `native/shim/vfx_paths.cpp`** (logic copied verbatim from `aigs.cpp` ResolveSdkPaths/PointProxiesAt) + +```cpp +#include "vfx_paths.h" +#ifdef COS_HAS_MAXINE +#include +#include "paths.h" + +// Defined in aigs.cpp; the VFX proxy reads it before LoadLibrary. +extern char* g_nvVFXSDKPath; + +namespace vfx { +bool ResolveSdkPaths(std::string& binDir, std::string& modelDir, std::string& err) { + char buf[1024] = {0}; + DWORD n = GetEnvironmentVariableA("COS_VFX_RUNTIME_DIR", buf, sizeof(buf)); + if (n > 0 && n < sizeof(buf)) { + std::string root(buf, n); + if (!root.empty() && (root.back() == '\\' || root.back() == '/')) root.pop_back(); + binDir = root; modelDir = root + "\\models"; return true; + } + n = GetEnvironmentVariableA("COS_VFX_SDK_DIR", buf, sizeof(buf)); + if (n > 0 && n < sizeof(buf)) { + std::string root(buf, n); + if (!root.empty() && (root.back() == '\\' || root.back() == '/')) root.pop_back(); + binDir = root + "\\bin"; modelDir = root + "\\bin\\models"; return true; + } + std::string appDir = ShimModuleDir(); + if (!appDir.empty()) { + std::string maxine = appDir + "\\maxine"; + if (DirExists(maxine)) { binDir = maxine; modelDir = maxine + "\\models\\vfx"; return true; } + } + err = "VFX runtime not found: set COS_VFX_RUNTIME_DIR or bundle maxine\\ beside the app"; + return false; +} +void PointProxiesAt(const std::string& binDir) { + static std::string s_bin; + s_bin = binDir; + g_nvVFXSDKPath = const_cast(s_bin.c_str()); +} +} +#else +namespace vfx { +bool ResolveSdkPaths(std::string&, std::string&, std::string& err) { err = "Maxine SDK not built in"; return false; } +void PointProxiesAt(const std::string&) {} +} +#endif +``` + +- [ ] **Step 3: Add to `native/shim/shim.vcxproj`** + +Inside the `` that holds the other `` entries (e.g. `aigs.cpp`), add: + +```xml + +``` + +- [ ] **Step 4: Build the shim (clean)** + +Run the shim build command. Expected: 0 warnings (file compiles; not yet referenced — that is fine). + +- [ ] **Step 5: Commit** + +```bash +git add native/shim/vfx_paths.h native/shim/vfx_paths.cpp native/shim/shim.vcxproj +git commit -m "feat(shim): shared VFX SDK path resolver for new effects" +``` + +--- + +## Task 3: Artifact Reduction effect + +VFX effect that removes compression artifacts in place at source resolution. Mirrors `aigs.cpp` but the output is a full BGR image, not a matte. No dimension change. + +**Files:** +- Create: `native/shim/artifactreduction.h`, `native/shim/artifactreduction.cpp` +- Create/extend: `native/shim/smoke/effects_smoke.cpp` +- Modify: `native/shim/shim.vcxproj` + +**Interfaces:** +- Consumes: `vfx::ResolveSdkPaths`, `vfx::PointProxiesAt` (Task 2). +- Produces: `class ArtifactReduction { static bool Probe(std::string&); bool Start(); void Stop(); bool ProcessFrame(uint8_t* bgra, int w, int h); bool IsReady() const; const std::string& LastError() const; }` + +> **Verify at impl:** confirm the effect selector macro `NVVFX_FX_ARTIFACT_REDUCTION` and that AR input/output images are `NVCV_BGR`/`NVCV_U8`/`NVCV_CHUNKY` against the VFX 1.2.0.0 headers (`nvVideoEffects.h`). Mode key is `NVVFX_MODE` (0 or 1). These match the green-screen pattern; only the selector, the output format (full BGR, not `NVCV_A`), and the write-back differ. + +- [ ] **Step 1: Create `native/shim/artifactreduction.h`** + +```cpp +#pragma once +#include +#include + +// Wraps the Maxine VFX Artifact Reduction effect. CPU-copy: a BGRA frame is uploaded +// to the GPU as BGR, cleaned, downloaded, and written back over the same BGRA buffer +// (RGB replaced, alpha left untouched). No size change. All methods no-throw; failure +// via IsReady()/LastError(). Without COS_HAS_MAXINE this is a never-ready stub. +class ArtifactReduction { +public: + ArtifactReduction(); + ~ArtifactReduction(); + static bool Probe(std::string& detail); + bool Start(); + void Stop(); + bool ProcessFrame(uint8_t* bgra, int width, int height); + bool IsReady() const { return ready_; } + const std::string& LastError() const { return lastError_; } +private: + bool ready_ = false; + std::string lastError_; + void* impl_ = nullptr; +}; +``` + +- [ ] **Step 2: Create `native/shim/artifactreduction.cpp`** + +```cpp +#include "artifactreduction.h" + +#ifdef COS_HAS_MAXINE +#define NOMINMAX +#include +#include +#include +#include +#include "nvCVStatus.h" +#include "nvCVImage.h" +#include "nvVideoEffects.h" // NVVFX_FX_ARTIFACT_REDUCTION (verify symbol) +#include "vfx_paths.h" + +struct ArImpl { + NvVFX_Handle effect = nullptr; + CUstream stream = nullptr; + std::string modelDir; + NvCVImage srcGpu{}; // BGR u8 GPU (input) + NvCVImage dstGpu{}; // BGR u8 GPU (output) + NvCVImage dstCpu{}; // BGR u8 CPU (downloaded) + NvCVImage stage{}; // BGRA u8 GPU (transfer staging) + int w = 0, h = 0; + bool loaded = false; +}; + +ArtifactReduction::ArtifactReduction() = default; +ArtifactReduction::~ArtifactReduction() { Stop(); } + +bool ArtifactReduction::Probe(std::string& detail) { + std::string binDir, modelDir, err; + if (!vfx::ResolveSdkPaths(binDir, modelDir, err)) { detail = err; return false; } + vfx::PointProxiesAt(binDir); + NvVFX_Handle eff = nullptr; + if (NvVFX_CreateEffect(NVVFX_FX_ARTIFACT_REDUCTION, &eff) != NVCV_SUCCESS || !eff) { + detail = "NvVFX_CreateEffect(ArtifactReduction) failed (DLL/SDK load?)"; return false; + } + NvVFX_SetString(eff, NVVFX_MODEL_DIRECTORY, modelDir.c_str()); + NvVFX_SetU32(eff, NVVFX_MODE, 0u); + NvVFX_SetU32(eff, NVVFX_MAX_INPUT_WIDTH, 1920u); + NvVFX_SetU32(eff, NVVFX_MAX_INPUT_HEIGHT, 1080u); + NvCV_Status load = NvVFX_Load(eff); + NvVFX_DestroyEffect(eff); + if (load != NVCV_SUCCESS) { detail = "NvVFX_Load(ArtifactReduction) failed (models missing?)"; return false; } + detail = "ArtifactReduction available"; + return true; +} + +bool ArtifactReduction::Start() { + Stop(); + ArImpl* impl = new (std::nothrow) ArImpl(); + if (!impl) { lastError_ = "out of memory"; return false; } + std::string binDir, err; + if (!vfx::ResolveSdkPaths(binDir, impl->modelDir, err)) { lastError_ = err; delete impl; return false; } + vfx::PointProxiesAt(binDir); + if (NvVFX_CudaStreamCreate(&impl->stream) != NVCV_SUCCESS) { lastError_ = "CudaStreamCreate failed"; delete impl; return false; } + if (NvVFX_CreateEffect(NVVFX_FX_ARTIFACT_REDUCTION, &impl->effect) != NVCV_SUCCESS || !impl->effect) { + lastError_ = "CreateEffect failed"; delete impl; return false; + } + NvVFX_SetString(impl->effect, NVVFX_MODEL_DIRECTORY, impl->modelDir.c_str()); + NvVFX_SetCudaStream(impl->effect, NVVFX_CUDA_STREAM, impl->stream); + NvVFX_SetU32(impl->effect, NVVFX_MODE, 1u); // mode 1 = stronger / for compressed input + impl_ = impl; ready_ = true; lastError_.clear(); + return true; +} + +void ArtifactReduction::Stop() { + auto* impl = static_cast(impl_); + if (!impl) { ready_ = false; return; } + NvCVImage_Dealloc(&impl->srcGpu); + NvCVImage_Dealloc(&impl->dstGpu); + NvCVImage_Dealloc(&impl->dstCpu); + NvCVImage_Dealloc(&impl->stage); + if (impl->effect) NvVFX_DestroyEffect(impl->effect); + if (impl->stream) NvVFX_CudaStreamDestroy(impl->stream); + delete impl; impl_ = nullptr; ready_ = false; +} + +namespace { +NvCV_Status EnsureImages(ArImpl* impl, int w, int h) { + if (impl->loaded && impl->w == w && impl->h == h) return NVCV_SUCCESS; + NvCVImage_Dealloc(&impl->srcGpu); + NvCVImage_Dealloc(&impl->dstGpu); + NvCVImage_Dealloc(&impl->dstCpu); + NvCVImage_Dealloc(&impl->stage); + NvCV_Status s; + s = NvCVImage_Alloc(&impl->srcGpu, w, h, NVCV_BGR, NVCV_U8, NVCV_CHUNKY, NVCV_GPU, 1); if (s != NVCV_SUCCESS) return s; + s = NvCVImage_Alloc(&impl->dstGpu, w, h, NVCV_BGR, NVCV_U8, NVCV_CHUNKY, NVCV_GPU, 1); if (s != NVCV_SUCCESS) return s; + s = NvCVImage_Alloc(&impl->dstCpu, w, h, NVCV_BGR, NVCV_U8, NVCV_CHUNKY, NVCV_CPU, 1); if (s != NVCV_SUCCESS) return s; + s = NvCVImage_Alloc(&impl->stage, w, h, NVCV_BGRA,NVCV_U8, NVCV_CHUNKY, NVCV_GPU, 1); if (s != NVCV_SUCCESS) return s; + s = NvVFX_SetImage(impl->effect, NVVFX_INPUT_IMAGE, &impl->srcGpu); if (s != NVCV_SUCCESS) return s; + s = NvVFX_SetImage(impl->effect, NVVFX_OUTPUT_IMAGE, &impl->dstGpu); if (s != NVCV_SUCCESS) return s; + s = NvVFX_SetU32(impl->effect, NVVFX_MAX_INPUT_WIDTH, static_cast(w)); if (s != NVCV_SUCCESS) return s; + s = NvVFX_SetU32(impl->effect, NVVFX_MAX_INPUT_HEIGHT, static_cast(h)); if (s != NVCV_SUCCESS) return s; + s = NvVFX_Load(impl->effect); if (s != NVCV_SUCCESS) return s; + impl->w = w; impl->h = h; impl->loaded = true; + return NVCV_SUCCESS; +} +} // namespace + +bool ArtifactReduction::ProcessFrame(uint8_t* bgra, int w, int h) { + auto* impl = static_cast(impl_); + if (!impl || !impl->effect || !bgra || w <= 0 || h <= 0) return false; + if (NvCV_Status es = EnsureImages(impl, w, h); es != NVCV_SUCCESS) { + lastError_ = std::string("EnsureImages/Load failed: ") + NvCV_GetErrorStringFromCode(es); ready_ = false; return false; + } + NvCVImage src{}; + NvCVImage_Init(&src, w, h, w * 4, bgra, NVCV_BGRA, NVCV_U8, NVCV_CHUNKY, NVCV_CPU); + if (NvCVImage_Transfer(&src, &impl->srcGpu, 1.0f, impl->stream, &impl->stage) != NVCV_SUCCESS) { lastError_ = "upload failed"; return false; } + if (NvVFX_Run(impl->effect, 0) != NVCV_SUCCESS) { lastError_ = "NvVFX_Run failed"; return false; } + if (NvCVImage_Transfer(&impl->dstGpu, &impl->dstCpu, 1.0f, impl->stream, nullptr) != NVCV_SUCCESS) { lastError_ = "download failed"; return false; } + if (NvVFX_CudaStreamSynchronize(impl->stream) != NVCV_SUCCESS) { lastError_ = "stream sync failed"; return false; } + // Write cleaned BGR back over the BGRA buffer; leave alpha untouched (forced 0xFF upstream). + const uint8_t* d = static_cast(impl->dstCpu.pixels); + const int dpitch = impl->dstCpu.pitch; + for (int y = 0; y < h; ++y) { + const uint8_t* drow = d + static_cast(dpitch) * y; + uint8_t* prow = bgra + static_cast(w) * 4 * y; + for (int x = 0; x < w; ++x) { + prow[x * 4 + 0] = drow[x * 3 + 0]; + prow[x * 4 + 1] = drow[x * 3 + 1]; + prow[x * 4 + 2] = drow[x * 3 + 2]; + } + } + return true; +} + +#else +ArtifactReduction::ArtifactReduction() = default; +ArtifactReduction::~ArtifactReduction() = default; +bool ArtifactReduction::Probe(std::string& detail) { detail = "Maxine SDK not built in"; return false; } +bool ArtifactReduction::Start() { lastError_ = "Maxine SDK not built in"; ready_ = false; return false; } +void ArtifactReduction::Stop() { ready_ = false; } +bool ArtifactReduction::ProcessFrame(uint8_t*, int, int) { return false; } +#endif +``` + +- [ ] **Step 3: Add `artifactreduction.cpp` to `native/shim/shim.vcxproj`** + +```xml + +``` + +- [ ] **Step 4: Add an AR smoke** — append to `native/shim/smoke/effects_smoke.cpp` (create it) + +```cpp +// Dev-box smoke (needs VFX SDK + RTX GPU). Build with the shim's include/lib setup. +// Verifies AR can start and process a synthetic 1280x720 frame. +#include +#include +#include +#include "../artifactreduction.h" + +int main() { + std::string detail; + if (!ArtifactReduction::Probe(detail)) { std::printf("AR unavailable: %s\n", detail.c_str()); return 0; } + ArtifactReduction ar; + assert(ar.Start()); + std::vector frame(1280 * 720 * 4, 128); + bool ok = ar.ProcessFrame(frame.data(), 1280, 720); + std::printf("AR ProcessFrame: %s (%s)\n", ok ? "ok" : "fail", ar.LastError().c_str()); + assert(ok); + ar.Stop(); + std::puts("effects_smoke AR OK"); + return 0; +} +``` + +- [ ] **Step 5: Build the shim (clean) and run the AR smoke on the dev box (SDK config)** + +Run the shim build with `COS_VFX_SDK_DIR` set (SDK config). Expected: 0 warnings. Then build+run `effects_smoke.cpp` (dev box): prints `effects_smoke AR OK`. (If no SDK/GPU, `Probe` returns the stub message and the smoke exits 0 — acceptable on CI-style hosts.) + +- [ ] **Step 6: Commit** + +```bash +git add native/shim/artifactreduction.h native/shim/artifactreduction.cpp native/shim/smoke/effects_smoke.cpp native/shim/shim.vcxproj +git commit -m "feat(shim): Maxine VFX Artifact Reduction effect" +``` + +--- + +## Task 4: Super Resolution effect + +VFX effect that upscales the frame (1.5× or 2×) into a **larger output buffer**. Runs last in the chain. + +**Files:** +- Create: `native/shim/superres.h`, `native/shim/superres.cpp` +- Modify: `native/shim/smoke/effects_smoke.cpp`, `native/shim/shim.vcxproj` + +**Interfaces:** +- Consumes: `vfx::ResolveSdkPaths`, `vfx::PointProxiesAt`. +- Produces: `class SuperRes { static bool Probe(std::string&); bool Start(int scaleX10); void Stop(); bool ProcessFrame(const uint8_t* bgra, int w, int h, std::vector& out, int& outW, int& outH); bool IsReady() const; const std::string& LastError() const; }` where `scaleX10` ∈ {15, 20}. + +> **Verify at impl:** confirm `NVVFX_FX_SUPER_RES`, and how the upscale factor is selected — the VFX SuperRes effect infers scale from the output image size relative to the input (set both images), plus `NVVFX_MODE` (0/1). If a dedicated scale property (`NVVFX_SCALE`) is required by 1.2.0.0, set it. Output dims must be one of the SDK-supported factors of the input. + +- [ ] **Step 1: Create `native/shim/superres.h`** + +```cpp +#pragma once +#include +#include +#include + +// Wraps the Maxine VFX Super Resolution effect. Upscales a BGRA frame by 1.5x or 2x +// into a freshly sized BGRA output buffer (alpha = 0xFF). Without COS_HAS_MAXINE this +// is a never-ready stub. scaleX10: 15 => 1.5x, 20 => 2x. +class SuperRes { +public: + SuperRes(); + ~SuperRes(); + static bool Probe(std::string& detail); + bool Start(int scaleX10); + void Stop(); + // Reads w*h BGRA from 'bgra', writes outW*outH BGRA into 'out'. Returns false on failure + // (out untouched). outW/outH are w*scale, h*scale. + bool ProcessFrame(const uint8_t* bgra, int w, int h, std::vector& out, int& outW, int& outH); + bool IsReady() const { return ready_; } + const std::string& LastError() const { return lastError_; } +private: + bool ready_ = false; + int scaleX10_ = 20; + std::string lastError_; + void* impl_ = nullptr; +}; +``` + +- [ ] **Step 2: Create `native/shim/superres.cpp`** + +```cpp +#include "superres.h" + +#ifdef COS_HAS_MAXINE +#define NOMINMAX +#include +#include +#include +#include +#include "nvCVStatus.h" +#include "nvCVImage.h" +#include "nvVideoEffects.h" // NVVFX_FX_SUPER_RES (verify symbol) +#include "vfx_paths.h" + +struct SrImpl { + NvVFX_Handle effect = nullptr; + CUstream stream = nullptr; + std::string modelDir; + NvCVImage srcGpu{}; // BGR u8 GPU (input, w x h) + NvCVImage dstGpu{}; // BGR u8 GPU (output, outW x outH) + NvCVImage dstCpu{}; // BGR u8 CPU (downloaded) + NvCVImage stage{}; // BGRA u8 GPU (upload staging, w x h) + int w = 0, h = 0, ow = 0, oh = 0; + bool loaded = false; +}; + +static void ScaledDims(int w, int h, int scaleX10, int& ow, int& oh) { + ow = (w * scaleX10) / 10; + oh = (h * scaleX10) / 10; +} + +SuperRes::SuperRes() = default; +SuperRes::~SuperRes() { Stop(); } + +bool SuperRes::Probe(std::string& detail) { + std::string binDir, modelDir, err; + if (!vfx::ResolveSdkPaths(binDir, modelDir, err)) { detail = err; return false; } + vfx::PointProxiesAt(binDir); + NvVFX_Handle eff = nullptr; + if (NvVFX_CreateEffect(NVVFX_FX_SUPER_RES, &eff) != NVCV_SUCCESS || !eff) { + detail = "NvVFX_CreateEffect(SuperRes) failed (DLL/SDK load?)"; return false; + } + NvVFX_SetString(eff, NVVFX_MODEL_DIRECTORY, modelDir.c_str()); + NvVFX_SetU32(eff, NVVFX_MODE, 1u); + NvVFX_SetU32(eff, NVVFX_MAX_INPUT_WIDTH, 1920u); + NvVFX_SetU32(eff, NVVFX_MAX_INPUT_HEIGHT, 1080u); + NvCV_Status load = NvVFX_Load(eff); + NvVFX_DestroyEffect(eff); + if (load != NVCV_SUCCESS) { detail = "NvVFX_Load(SuperRes) failed (models missing?)"; return false; } + detail = "SuperRes available"; + return true; +} + +bool SuperRes::Start(int scaleX10) { + Stop(); + scaleX10_ = (scaleX10 == 15) ? 15 : 20; // only 1.5x / 2x supported here + SrImpl* impl = new (std::nothrow) SrImpl(); + if (!impl) { lastError_ = "out of memory"; return false; } + std::string binDir, err; + if (!vfx::ResolveSdkPaths(binDir, impl->modelDir, err)) { lastError_ = err; delete impl; return false; } + vfx::PointProxiesAt(binDir); + if (NvVFX_CudaStreamCreate(&impl->stream) != NVCV_SUCCESS) { lastError_ = "CudaStreamCreate failed"; delete impl; return false; } + if (NvVFX_CreateEffect(NVVFX_FX_SUPER_RES, &impl->effect) != NVCV_SUCCESS || !impl->effect) { + lastError_ = "CreateEffect failed"; delete impl; return false; + } + NvVFX_SetString(impl->effect, NVVFX_MODEL_DIRECTORY, impl->modelDir.c_str()); + NvVFX_SetCudaStream(impl->effect, NVVFX_CUDA_STREAM, impl->stream); + NvVFX_SetU32(impl->effect, NVVFX_MODE, 1u); + impl_ = impl; ready_ = true; lastError_.clear(); + return true; +} + +void SuperRes::Stop() { + auto* impl = static_cast(impl_); + if (!impl) { ready_ = false; return; } + NvCVImage_Dealloc(&impl->srcGpu); + NvCVImage_Dealloc(&impl->dstGpu); + NvCVImage_Dealloc(&impl->dstCpu); + NvCVImage_Dealloc(&impl->stage); + if (impl->effect) NvVFX_DestroyEffect(impl->effect); + if (impl->stream) NvVFX_CudaStreamDestroy(impl->stream); + delete impl; impl_ = nullptr; ready_ = false; +} + +namespace { +NvCV_Status EnsureImages(SrImpl* impl, int w, int h, int ow, int oh) { + if (impl->loaded && impl->w == w && impl->h == h && impl->ow == ow && impl->oh == oh) return NVCV_SUCCESS; + NvCVImage_Dealloc(&impl->srcGpu); + NvCVImage_Dealloc(&impl->dstGpu); + NvCVImage_Dealloc(&impl->dstCpu); + NvCVImage_Dealloc(&impl->stage); + NvCV_Status s; + s = NvCVImage_Alloc(&impl->srcGpu, w, h, NVCV_BGR, NVCV_U8, NVCV_CHUNKY, NVCV_GPU, 1); if (s != NVCV_SUCCESS) return s; + s = NvCVImage_Alloc(&impl->dstGpu, ow, oh, NVCV_BGR, NVCV_U8, NVCV_CHUNKY, NVCV_GPU, 1); if (s != NVCV_SUCCESS) return s; + s = NvCVImage_Alloc(&impl->dstCpu, ow, oh, NVCV_BGR, NVCV_U8, NVCV_CHUNKY, NVCV_CPU, 1); if (s != NVCV_SUCCESS) return s; + s = NvCVImage_Alloc(&impl->stage, w, h, NVCV_BGRA,NVCV_U8, NVCV_CHUNKY, NVCV_GPU, 1); if (s != NVCV_SUCCESS) return s; + s = NvVFX_SetImage(impl->effect, NVVFX_INPUT_IMAGE, &impl->srcGpu); if (s != NVCV_SUCCESS) return s; + s = NvVFX_SetImage(impl->effect, NVVFX_OUTPUT_IMAGE, &impl->dstGpu); if (s != NVCV_SUCCESS) return s; + s = NvVFX_Load(impl->effect); if (s != NVCV_SUCCESS) return s; + impl->w = w; impl->h = h; impl->ow = ow; impl->oh = oh; impl->loaded = true; + return NVCV_SUCCESS; +} +} // namespace + +bool SuperRes::ProcessFrame(const uint8_t* bgra, int w, int h, std::vector& out, int& outW, int& outH) { + auto* impl = static_cast(impl_); + if (!impl || !impl->effect || !bgra || w <= 0 || h <= 0) return false; + int ow, oh; ScaledDims(w, h, scaleX10_, ow, oh); + if (NvCV_Status es = EnsureImages(impl, w, h, ow, oh); es != NVCV_SUCCESS) { + lastError_ = std::string("EnsureImages/Load failed: ") + NvCV_GetErrorStringFromCode(es); ready_ = false; return false; + } + NvCVImage src{}; + NvCVImage_Init(&src, w, h, w * 4, const_cast(bgra), NVCV_BGRA, NVCV_U8, NVCV_CHUNKY, NVCV_CPU); + if (NvCVImage_Transfer(&src, &impl->srcGpu, 1.0f, impl->stream, &impl->stage) != NVCV_SUCCESS) { lastError_ = "upload failed"; return false; } + if (NvVFX_Run(impl->effect, 0) != NVCV_SUCCESS) { lastError_ = "NvVFX_Run failed"; return false; } + if (NvCVImage_Transfer(&impl->dstGpu, &impl->dstCpu, 1.0f, impl->stream, nullptr) != NVCV_SUCCESS) { lastError_ = "download failed"; return false; } + if (NvVFX_CudaStreamSynchronize(impl->stream) != NVCV_SUCCESS) { lastError_ = "stream sync failed"; return false; } + out.assign(static_cast(ow) * oh * 4, 0xFF); + const uint8_t* d = static_cast(impl->dstCpu.pixels); + const int dpitch = impl->dstCpu.pitch; + for (int y = 0; y < oh; ++y) { + const uint8_t* drow = d + static_cast(dpitch) * y; + uint8_t* prow = out.data() + static_cast(ow) * 4 * y; + for (int x = 0; x < ow; ++x) { + prow[x * 4 + 0] = drow[x * 3 + 0]; + prow[x * 4 + 1] = drow[x * 3 + 1]; + prow[x * 4 + 2] = drow[x * 3 + 2]; + prow[x * 4 + 3] = 0xFF; + } + } + outW = ow; outH = oh; + return true; +} + +#else +SuperRes::SuperRes() = default; +SuperRes::~SuperRes() = default; +bool SuperRes::Probe(std::string& detail) { detail = "Maxine SDK not built in"; return false; } +bool SuperRes::Start(int) { lastError_ = "Maxine SDK not built in"; ready_ = false; return false; } +void SuperRes::Stop() { ready_ = false; } +bool SuperRes::ProcessFrame(const uint8_t*, int, int, std::vector&, int&, int&) { return false; } +#endif +``` + +- [ ] **Step 3: Add `superres.cpp` to `native/shim/shim.vcxproj`** + +```xml + +``` + +- [ ] **Step 4: Extend the smoke** — append to `native/shim/smoke/effects_smoke.cpp` a SuperRes check (inside `main`, before the final puts): + +```cpp + { + std::string d; + if (SuperRes::Probe(d)) { + SuperRes sr; assert(sr.Start(20)); + std::vector in(640 * 480 * 4, 64), out; int ow = 0, oh = 0; + bool ok = sr.ProcessFrame(in.data(), 640, 480, out, ow, oh); + std::printf("SR ProcessFrame: %s -> %dx%d\n", ok ? "ok" : "fail", ow, oh); + assert(ok && ow == 1280 && oh == 960); + sr.Stop(); + } else { std::printf("SR unavailable: %s\n", d.c_str()); } + } +``` +Add `#include "../superres.h"` at the top. + +- [ ] **Step 5: Build the shim (clean) + run the smoke (dev box)** + +Expected: 0 warnings; smoke prints `SR ProcessFrame: ok -> 1280x960`. + +- [ ] **Step 6: Commit** + +```bash +git add native/shim/superres.h native/shim/superres.cpp native/shim/smoke/effects_smoke.cpp native/shim/shim.vcxproj +git commit -m "feat(shim): Maxine VFX Super Resolution effect (1.5x/2x)" +``` + +--- + +## Task 5: C ABI — params + capability gates + +Grow the contract once for both effects. After this task the shim accepts the new params and reports the new gates, but the worker does not yet run the effects (Task 6). + +**Files:** +- Modify: `native/shim/shim.h`, `native/shim/shim.cpp` + +**Interfaces:** +- Produces (C): `CosParams` gains `int artifact_reduction_enabled; int super_res_enabled; int super_res_scale;`. `CosCaps` gains `int artifact_reduction_available; int super_res_available;`. + +- [ ] **Step 1: Update `CosParams` + `CosCaps` in `native/shim/shim.h`** + +Replace the `CosParams` struct with: + +```cpp +typedef struct { + const char* camera_id; // UTF-8, may be null + int green_screen_enabled; + double green_screen_expand; + double green_screen_feather; + int eye_contact_enabled; + double eye_contact_sensitivity; + double eye_contact_look_away_range; + int artifact_reduction_enabled; + int super_res_enabled; + int super_res_scale; // 0=off, 15=1.5x, 20=2x +} CosParams; +``` + +Replace the `CosCaps` struct with (total 528 bytes on x64): + +```cpp +typedef struct { + int green_screen_available; + char detail[256]; + int eye_contact_available; + char ec_detail[256]; + int artifact_reduction_available; // 1 if Maxine ArtifactReduction can run + int super_res_available; // 1 if Maxine SuperRes can run +} CosCaps; +``` + +- [ ] **Step 2: Wire `cos_set_params`** — `native/shim/shim.cpp` + +Add the includes near the top: + +```cpp +#include "artifactreduction.h" +#include "superres.h" +``` + +In `cos_set_params`, after the existing `g_capture.SetEyeContact(...)` line add: + +```cpp + g_capture.SetArtifactReduction(p->artifact_reduction_enabled != 0); + g_capture.SetSuperRes(p->super_res_enabled != 0, p->super_res_scale); +``` + +- [ ] **Step 3: Wire `cos_query_capabilities`** — `native/shim/shim.cpp` + +Before the final `return` of `cos_query_capabilities`, add: + +```cpp + std::string arDetail, srDetail; + out->artifact_reduction_available = ArtifactReduction::Probe(arDetail) ? 1 : 0; + out->super_res_available = SuperRes::Probe(srDetail) ? 1 : 0; +``` + +And change the final return to also surface the new gates: + +```cpp + return (gsOk || ecOk || out->artifact_reduction_available || out->super_res_available) ? 1 : 0; +``` + +- [ ] **Step 4: Build the shim and verify exports + struct sizes** + +Run the shim build. Expected: 0 warnings; `dumpbin /exports` still shows exactly 9 `cos_*`. (Capture `Set*` methods are added in Task 6 — if the build fails on missing `SetArtifactReduction`/`SetSuperRes`, do Task 6 Step 1–3 before re-building; the two tasks share the ABI boundary. Recommended: implement Task 6 immediately after this step, then build once.) + +- [ ] **Step 5: Commit** (after Task 6 builds clean, or together) + +```bash +git add native/shim/shim.h native/shim/shim.cpp +git commit -m "feat(shim): ABI — AR/SR params + capability gates (CosParams/CosCaps 528B)" +``` + +--- + +## Task 6: Capture worker — run AR first, SR last + +**Files:** +- Modify: `native/shim/capture.h`, `native/shim/capture.cpp` + +**Interfaces:** +- Consumes: `ArtifactReduction`, `SuperRes` (Tasks 3–4); `cos_set_params` (Task 5). +- Produces: `Capture::SetArtifactReduction(bool)`, `Capture::SetSuperRes(bool, int scaleX10)`, `Capture::ArtifactReductionActive()/SuperResActive()` + error getters. + +- [ ] **Step 1: Declare the new methods in `native/shim/capture.h`** + +After the Eye Contact block add: + +```cpp + // Toggles Artifact Reduction for subsequent frames. Thread-safe; worker owns the object. + void SetArtifactReduction(bool enabled); + bool ArtifactReductionActive() const; + std::string ArtifactReductionError() const; + + // Toggles Super Resolution + scale (15=1.5x, 20=2x). Thread-safe; worker owns the object. + void SetSuperRes(bool enabled, int scaleX10); + bool SuperResActive() const; + std::string SuperResError() const; +``` + +- [ ] **Step 2: Add shared state** — `native/shim/capture.cpp`, in `struct CaptureState` + +```cpp + std::atomic artifactReductionEnabled{false}; + std::atomic artifactReductionActive{false}; + std::mutex arErrMtx; // leaf lock + std::string arError; + + std::atomic superResEnabled{false}; + std::atomic superResScale{20}; // 15 or 20 + std::atomic superResActive{false}; + std::mutex srErrMtx; // leaf lock + std::string srError; +``` + +Add the include at the top: `#include "superres.h"` (alongside the existing `#include "aigs.h"` / `#include "eyecontact.h"`; `artifactreduction.h` is needed too): + +```cpp +#include "artifactreduction.h" +#include "superres.h" +``` + +- [ ] **Step 3: Implement the setters/getters** — `native/shim/capture.cpp` (near the existing `SetEyeContact` etc.) + +```cpp +void Capture::SetArtifactReduction(bool enabled) { + g_state.artifactReductionEnabled.store(enabled, std::memory_order_release); +} +bool Capture::ArtifactReductionActive() const { + return g_state.artifactReductionActive.load(std::memory_order_acquire); +} +std::string Capture::ArtifactReductionError() const { + std::lock_guard e(g_state.arErrMtx); return g_state.arError; +} +void Capture::SetSuperRes(bool enabled, int scaleX10) { + g_state.superResScale.store(scaleX10 == 15 ? 15 : 20, std::memory_order_release); + g_state.superResEnabled.store(enabled, std::memory_order_release); +} +bool Capture::SuperResActive() const { + return g_state.superResActive.load(std::memory_order_acquire); +} +std::string Capture::SuperResError() const { + std::lock_guard e(g_state.srErrMtx); return g_state.srError; +} +``` + +- [ ] **Step 4: Insert AR + SR into `WorkerLoop`** + +Declare the effect objects next to the existing `Aigs aigs;` / `EyeContact eyeContact;` (worker-thread-local): + +```cpp + ArtifactReduction artifactReduction; + SuperRes superRes; +``` + +**Artifact Reduction — before Eye Contact.** Inside `if (CopyFrame(...))`, immediately BEFORE the existing `const bool ecWant = ...` line, add: + +```cpp + // Artifact Reduction runs FIRST so the AI effects downstream get a clean frame. + const bool arWant = g_state.artifactReductionEnabled.load(std::memory_order_acquire); + if (arWant && !artifactReduction.IsReady()) { + if (!artifactReduction.Start()) { + std::lock_guard e(g_state.arErrMtx); + const std::string& ne = artifactReduction.LastError(); + if (g_state.arError != ne) g_state.arError = ne; + } + } else if (!arWant && artifactReduction.IsReady()) { + artifactReduction.Stop(); + std::lock_guard e(g_state.arErrMtx); + if (!g_state.arError.empty()) g_state.arError.clear(); + } + bool arApplied = false; + if (arWant && artifactReduction.IsReady()) { + arApplied = artifactReduction.ProcessFrame(scratch.data(), width, height); + std::lock_guard e(g_state.arErrMtx); + if (!arApplied) { + const std::string& ne = artifactReduction.LastError(); + if (g_state.arError != ne) g_state.arError = ne; + } else if (!g_state.arError.empty()) { g_state.arError.clear(); } + } + g_state.artifactReductionActive.store(arApplied, std::memory_order_release); +``` + +**Super Resolution — after Green Screen, before publish.** The publish block currently does `g_state.frame.swap(scratch); g_state.width = width; g_state.height = height;`. Replace that publish block (the four lines under `std::lock_guard lock(g_state.mtx);`) with SR-aware publish: + +```cpp + // Super Resolution runs LAST; it changes the frame dimensions. + const bool srWant = g_state.superResEnabled.load(std::memory_order_acquire); + const int srScale = g_state.superResScale.load(std::memory_order_acquire); + if (srWant && !superRes.IsReady()) { + if (!superRes.Start(srScale)) { + std::lock_guard e(g_state.srErrMtx); + const std::string& ne = superRes.LastError(); + if (g_state.srError != ne) g_state.srError = ne; + } + } else if (!srWant && superRes.IsReady()) { + superRes.Stop(); + std::lock_guard e(g_state.srErrMtx); + if (!g_state.srError.empty()) g_state.srError.clear(); + } + + int pubW = width, pubH = height; + std::vector srOut; + bool srApplied = false; + if (srWant && superRes.IsReady()) { + srApplied = superRes.ProcessFrame(scratch.data(), width, height, srOut, pubW, pubH); + std::lock_guard e(g_state.srErrMtx); + if (!srApplied) { + const std::string& ne = superRes.LastError(); + if (g_state.srError != ne) g_state.srError = ne; + } else if (!g_state.srError.empty()) { g_state.srError.clear(); } + } + g_state.superResActive.store(srApplied, std::memory_order_release); + + { + std::lock_guard lock(g_state.mtx); + if (srApplied) g_state.frame.swap(srOut); + else g_state.frame.swap(scratch); + g_state.width = pubW; + g_state.height = pubH; + g_state.hasNewFrame = true; + g_state.framesProduced.fetch_add(1, std::memory_order_release); + } +``` + +> Note: this replaces the Task 1 Step 6 increment location — the increment now lives inside the new publish block. Remove the standalone increment line added in Task 1 if it is now duplicated. + +Add teardown next to the existing `eyeContact.Stop(); aigs.Stop();` (after the loop): + +```cpp + artifactReduction.Stop(); + superRes.Stop(); +``` + +- [ ] **Step 5: Reset active flags + clear errors in `Capture::Stop`** + +After the eye-contact reset block add: + +```cpp + g_state.artifactReductionActive.store(false, std::memory_order_release); + { std::lock_guard e(g_state.arErrMtx); g_state.arError.clear(); } + g_state.superResActive.store(false, std::memory_order_release); + { std::lock_guard e(g_state.srErrMtx); g_state.srError.clear(); } +``` + +- [ ] **Step 6: Surface AR/SR errors in `cos_get_status`** — `native/shim/shim.cpp` + +In `cos_get_status`, extend the error fallback chain: + +```cpp + std::string err = g_capture.GreenScreenError(); + if (err.empty()) err = g_capture.EyeContactError(); + if (err.empty()) err = g_capture.ArtifactReductionError(); + if (err.empty()) err = g_capture.SuperResError(); +``` + +- [ ] **Step 7: Build the shim (SDK config) — clean, exports intact** + +Run the shim build. Expected: 0 warnings; 9 `cos_*` exports. Verify the deployed DLL still exports `GreenScreen` and `GazeRedirection` and lacks `not built in` (grep the DLL). + +- [ ] **Step 8: Commit** + +```bash +git add native/shim/capture.h native/shim/capture.cpp native/shim/shim.h native/shim/shim.cpp +git commit -m "feat(shim): run Artifact Reduction (first) + Super Resolution (last) on the worker" +``` + +--- + +## Task 7: Managed contracts + Core logic (xUnit) + +This is the automated-test task. All changes are in `CameraOnScreen.Core` + tests (run in CI), plus the `PInvokeShim` mirror in the App (build-verified). + +**Files:** +- Modify: `src/CameraOnScreen.Core/Native/Contracts.cs`, `FakeShim.cs`, `Config/Models.cs`, `Orchestration/Orchestrator.cs`, `ViewModels/MainViewModel.cs` +- Modify: `src/CameraOnScreen.App/Native/PInvokeShim.cs` +- Test: `tests/CameraOnScreen.Core.Tests/ViewModels/MainViewModelTests.cs`, `tests/CameraOnScreen.Core.Tests/Orchestration/OrchestratorTests.cs` + +**Interfaces:** +- Produces: `ShimParams` gains `bool ArtifactReductionEnabled = false, bool SuperResEnabled = false, int SuperResScale = 0` (defaulted — existing call sites keep compiling). `ShimCapabilities` gains `bool ArtifactReductionAvailable = false, bool SuperResAvailable = false`. `MainViewModel` gains `ArtifactReductionEnabled`, `SuperResEnabled`, `SuperResScaleIndex` (0=off,1=1.5x,2=2x), `ArtifactReductionAvailable`, `SuperResAvailable`. + +- [ ] **Step 1: Write the failing test** — append to `tests/CameraOnScreen.Core.Tests/ViewModels/MainViewModelTests.cs` + +```csharp +[Fact] +public void BuildParams_includes_artifact_reduction_and_superres() +{ + var shim = new FakeShim { GreenScreenAvailable = true, EyeContactAvailable = true, + ArtifactReductionAvailable = true, SuperResAvailable = true }; + var orch = new Orchestrator(shim, GpuTier.Rtx); + orch.ProbeCapabilities(); + var vm = new MainViewModel(orch, shim); + + vm.ArtifactReductionEnabled = true; + vm.SuperResEnabled = true; + vm.SuperResScaleIndex = 2; // 2x + + var p = vm.BuildParams(); + Assert.True(p.ArtifactReductionEnabled); + Assert.True(p.SuperResEnabled); + Assert.Equal(20, p.SuperResScale); +} + +[Fact] +public void ToAppConfig_roundtrips_new_effects() +{ + var shim = new FakeShim(); + var vm = new MainViewModel(new Orchestrator(shim, GpuTier.Rtx), shim) + { + ArtifactReductionEnabled = true, SuperResEnabled = true, SuperResScaleIndex = 1 + }; + var cfg = vm.ToAppConfig(0, 0, 320, 240); + Assert.True(cfg.Effects.ArtifactReductionEnabled); + Assert.True(cfg.Effects.SuperResEnabled); + Assert.Equal(15, cfg.Effects.SuperResScale); + + var vm2 = new MainViewModel(new Orchestrator(shim, GpuTier.Rtx), shim); + vm2.LoadFrom(cfg); + Assert.True(vm2.ArtifactReductionEnabled); + Assert.Equal(1, vm2.SuperResScaleIndex); +} +``` + +Append to `tests/CameraOnScreen.Core.Tests/Orchestration/OrchestratorTests.cs`: + +```csharp +[Fact] +public void ApplyParams_forces_effects_off_when_unavailable() +{ + var shim = new FakeShim { GreenScreenAvailable = false, ArtifactReductionAvailable = false, SuperResAvailable = false }; + var orch = new Orchestrator(shim, GpuTier.Rtx); + orch.ProbeCapabilities(); + orch.ApplyParams(new ShimParams("cam", true, 0, 0, false, 0.5, 0.5, + ArtifactReductionEnabled: true, SuperResEnabled: true, SuperResScale: 20)); + Assert.False(shim.LastParams!.ArtifactReductionEnabled); + Assert.False(shim.LastParams!.SuperResEnabled); +} +``` + +- [ ] **Step 2: Run the tests to verify they fail** + +Run: `dotnet test tests/CameraOnScreen.Core.Tests/CameraOnScreen.Core.Tests.csproj --filter "FullyQualifiedName~artifact OR FullyQualifiedName~SuperRes OR FullyQualifiedName~forces_effects_off"` +Expected: FAIL to compile — new members do not exist. + +- [ ] **Step 3: Extend `ShimParams` + `ShimCapabilities`** — `src/CameraOnScreen.Core/Native/Contracts.cs` + +```csharp +public sealed record ShimParams( + string? CameraId, + bool GreenScreenEnabled, + double GreenScreenExpand, + double GreenScreenFeather, + bool EyeContactEnabled, + double EyeContactSensitivity, + double EyeContactLookAwayRange, + bool ArtifactReductionEnabled = false, + bool SuperResEnabled = false, + int SuperResScale = 0); // 0=off, 15=1.5x, 20=2x +``` + +```csharp +public sealed record ShimCapabilities( + bool GreenScreenAvailable, string Detail, + bool EyeContactAvailable = false, string EyeContactDetail = "", + bool ArtifactReductionAvailable = false, bool SuperResAvailable = false); +``` + +- [ ] **Step 4: Extend `EffectSettings`** — `src/CameraOnScreen.Core/Config/Models.cs` + +Add to the record: + +```csharp + public bool ArtifactReductionEnabled { get; init; } + public bool SuperResEnabled { get; init; } + public int SuperResScale { get; init; } // 0=off, 15=1.5x, 20=2x +``` + +- [ ] **Step 5: Extend `FakeShim`** — `src/CameraOnScreen.Core/Native/FakeShim.cs` + +Add properties: + +```csharp + public bool ArtifactReductionAvailable { get; set; } + public bool SuperResAvailable { get; set; } +``` + +Update `QueryCapabilities`: + +```csharp + public ShimCapabilities QueryCapabilities() => + new(GreenScreenAvailable, + GreenScreenAvailable ? "fake: available" : "fake: unavailable", + EyeContactAvailable, + EyeContactAvailable ? "fake: ec available" : "fake: ec unavailable", + ArtifactReductionAvailable, SuperResAvailable); +``` + +- [ ] **Step 6: Extend `Orchestrator`** — `src/CameraOnScreen.Core/Orchestration/Orchestrator.cs` + +Add properties: + +```csharp + public bool ArtifactReductionAvailable { get; private set; } + public bool SuperResAvailable { get; private set; } +``` + +In `ProbeCapabilities` add: + +```csharp + ArtifactReductionAvailable = caps.ArtifactReductionAvailable; + SuperResAvailable = caps.SuperResAvailable; +``` + +In `ApplyParams`, extend the `with`: + +```csharp + var effective = requested with + { + GreenScreenEnabled = requested.GreenScreenEnabled && EffectsAvailable, + EyeContactEnabled = requested.EyeContactEnabled && EyeContactAvailable, + ArtifactReductionEnabled = requested.ArtifactReductionEnabled && ArtifactReductionAvailable, + SuperResEnabled = requested.SuperResEnabled && SuperResAvailable, + }; +``` + +- [ ] **Step 7: Extend `MainViewModel`** — `src/CameraOnScreen.Core/ViewModels/MainViewModel.cs` + +Add observable props (next to the green-screen ones): + +```csharp + [ObservableProperty] private bool artifactReductionEnabled; + [ObservableProperty] private bool superResEnabled; + [ObservableProperty] private int superResScaleIndex; // 0=off, 1=1.5x, 2=2x + [ObservableProperty] private bool artifactReductionAvailable; + [ObservableProperty] private bool superResAvailable; +``` + +Map index→scale once: + +```csharp + // SuperResScaleIndex 0/1/2 -> shim scale 0/15/20. + private static int ScaleFromIndex(int i) => i switch { 1 => 15, 2 => 20, _ => 0 }; + private static int IndexFromScale(int s) => s switch { 15 => 1, 20 => 2, _ => 0 }; +``` + +In the ctor (after the existing `EyeContact*` mirroring) add: + +```csharp + ArtifactReductionAvailable = orchestrator.ArtifactReductionAvailable; + SuperResAvailable = orchestrator.SuperResAvailable; +``` + +In `ProbeCapabilitiesAsync` (after the existing assignments) add: + +```csharp + ArtifactReductionAvailable = _orchestrator.ArtifactReductionAvailable; + SuperResAvailable = _orchestrator.SuperResAvailable; +``` + +Add live-push partials: + +```csharp + partial void OnArtifactReductionEnabledChanged(bool value) => ApplyLiveParams(); + partial void OnSuperResEnabledChanged(bool value) => ApplyLiveParams(); + partial void OnSuperResScaleIndexChanged(int value) => ApplyLiveParams(); +``` + +Extend `BuildParams`: + +```csharp + public ShimParams BuildParams() => new( + CameraId: SelectedCamera?.Id, + GreenScreenEnabled: GreenScreenEnabled, + GreenScreenExpand: GreenScreenExpand, + GreenScreenFeather: GreenScreenFeather, + EyeContactEnabled: EyeContactEnabled, + EyeContactSensitivity: EyeContactSensitivity, + EyeContactLookAwayRange: EyeContactLookAwayRange, + ArtifactReductionEnabled: ArtifactReductionEnabled, + SuperResEnabled: SuperResEnabled, + SuperResScale: ScaleFromIndex(SuperResScaleIndex)); +``` + +Extend `LoadFrom` (after the eye-contact lines): + +```csharp + ArtifactReductionEnabled = config.Effects.ArtifactReductionEnabled; + SuperResEnabled = config.Effects.SuperResEnabled; + SuperResScaleIndex = IndexFromScale(config.Effects.SuperResScale); +``` + +Extend `ToAppConfig`'s `Effects = new EffectSettings { ... }`: + +```csharp + ArtifactReductionEnabled = ArtifactReductionEnabled, + SuperResEnabled = SuperResEnabled, + SuperResScale = ScaleFromIndex(SuperResScaleIndex), +``` + +- [ ] **Step 8: Run the new tests to verify they pass** + +Run: `dotnet test tests/CameraOnScreen.Core.Tests/CameraOnScreen.Core.Tests.csproj --filter "FullyQualifiedName~artifact OR FullyQualifiedName~SuperRes OR FullyQualifiedName~forces_effects_off"` +Expected: PASS. + +- [ ] **Step 9: Mirror the ABI in `PInvokeShim`** — `src/CameraOnScreen.App/Native/PInvokeShim.cs` + +Extend `CosParams`: + +```csharp + [StructLayout(LayoutKind.Sequential)] + private struct CosParams + { + public IntPtr camera_id; + public int green_screen_enabled; public double green_screen_expand; + public double green_screen_feather; + public int eye_contact_enabled; public double eye_contact_sensitivity; + public double eye_contact_look_away_range; + public int artifact_reduction_enabled; + public int super_res_enabled; + public int super_res_scale; + } +``` + +Extend `CosCaps`: + +```csharp + [StructLayout(LayoutKind.Sequential)] + private struct CosCaps + { + public int GreenScreenAvailable; + [MarshalAs(UnmanagedType.ByValArray, SizeConst = 256)] public byte[] Detail; + public int EyeContactAvailable; + [MarshalAs(UnmanagedType.ByValArray, SizeConst = 256)] public byte[] EcDetail; + public int ArtifactReductionAvailable; + public int SuperResAvailable; + } +``` + +Extend `SetParams`' `new CosParams { ... }`: + +```csharp + artifact_reduction_enabled = p.ArtifactReductionEnabled ? 1 : 0, + super_res_enabled = p.SuperResEnabled ? 1 : 0, + super_res_scale = p.SuperResScale, +``` + +Extend `QueryCapabilities`' returned `ShimCapabilities`: + +```csharp + return new ShimCapabilities( + caps.GreenScreenAvailable != 0, ReadUtf8(caps.Detail, 0, 256), + caps.EyeContactAvailable != 0, ReadUtf8(caps.EcDetail, 0, 256), + caps.ArtifactReductionAvailable != 0, caps.SuperResAvailable != 0); +``` + +- [ ] **Step 10: Run the full Core test suite (no regressions)** + +Run: `dotnet test tests/CameraOnScreen.Core.Tests/CameraOnScreen.Core.Tests.csproj` +Expected: all PASS, 0 warnings. + +- [ ] **Step 11: Commit** + +```bash +git add src/CameraOnScreen.Core src/CameraOnScreen.App/Native/PInvokeShim.cs tests/CameraOnScreen.Core.Tests +git commit -m "feat(core): AR/SR params, capability gates, persistence, live push" +``` + +--- + +## Task 8: App UI — toggles, scale combo, 4K frame buffer + +**Files:** +- Modify: `src/CameraOnScreen.App/MainWindow.xaml`, `src/CameraOnScreen.App/MainWindow.xaml.cs` + +**Interfaces:** +- Consumes: `MainViewModel.ArtifactReductionEnabled/SuperResEnabled/SuperResScaleIndex/ArtifactReductionAvailable/SuperResAvailable` (Task 7). + +- [ ] **Step 1: Add controls** — `src/CameraOnScreen.App/MainWindow.xaml` + +After the Eye Contact toggle + its note (`` ending the eye-contact detail), add: + +```xml + + + + + + + +``` + +- [ ] **Step 2: Pre-size the frame buffer to 4K** — `src/CameraOnScreen.App/MainWindow.xaml.cs:25` + +Replace: + +```csharp + // Big enough for 1920x1080 BGRA; the test camera is 640x480. TryGetFrame writes the actual size. + private readonly byte[] _frameBuf = new byte[1920 * 1080 * 4]; +``` + +with: + +```csharp + // Pre-sized to 4K (3840x2160) BGRA so Super Resolution (up to 2x of 1080p) fits without a + // resize. TryGetFrame writes the actual size; cos_get_frame rejects frames larger than this. + private readonly byte[] _frameBuf = new byte[3840 * 2160 * 4]; +``` + +- [ ] **Step 3: Build the App (shim built SDK-config last) — clean** + +Run: +``` +& "C:/Program Files (x86)/Microsoft Visual Studio/2022/BuildTools/MSBuild/Current/Bin/MSBuild.exe" native/shim/shim.vcxproj /p:Configuration=Debug /p:Platform=x64 +dotnet build src/CameraOnScreen.App/CameraOnScreen.App.csproj -t:Rebuild +``` +Expected: 0 warnings, build succeeds. + +- [ ] **Step 4: Commit** + +```bash +git add src/CameraOnScreen.App/MainWindow.xaml src/CameraOnScreen.App/MainWindow.xaml.cs +git commit -m "feat(app): Artifact Reduction + Super Resolution UI, 4K frame buffer" +``` + +--- + +## Task 9: Bundle the new feature DLLs + models + +**Files:** +- Modify: `native/shim/bundle/maxine-manifest.psd1`, `scripts/assemble-maxine-stage.ps1` + +**Interfaces:** +- Consumes: the produced bundle is verified by `trace_closure` / `verify-bundle.ps1`. + +> **Verify at impl:** confirm the exact VFX feature-DLL filenames for Artifact Reduction and Super Resolution in the VFX 1.2.0.0 SDK `bin` tree (alongside `nvVFXGreenScreen.dll`), and the model file globs under `models` for each effect. + +- [ ] **Step 1: Add the feature DLLs to the manifest** — `native/shim/bundle/maxine-manifest.psd1` + +Add the AR + SR feature DLL names to the `Dlls` list (next to the green-screen feature DLL), e.g.: + +```powershell + 'nvVFXArtifactReduction.dll' + 'nvVFXSuperRes.dll' +``` + +- [ ] **Step 2: Ensure the model globs cover the new effects** — `scripts/assemble-maxine-stage.ps1` + +In the VFX model copy step (the one that stages `models\vfx`), confirm the glob includes the AR + SR model files (they live under the same VFX `models` tree as green screen; if the script copies the whole `models\vfx` dir, no change is needed — verify it is not filtering to green-screen-only model names). + +- [ ] **Step 3: Re-stage + re-run the load-closure trace (dev box)** + +Run `scripts/assemble-maxine-stage.ps1` then `scripts/bundle-maxine.ps1 -OutDir -MaxineStage `, then run `native/shim/smoke/trace_closure` (or `bundle_probe`) against the produced bundle with `COS_*` unset. +Expected: AR + SR both load (Probe success); the trace lists the new feature DLLs in the closure; no `not built in`. + +- [ ] **Step 4: Update the manifest `Dlls` list from the trace output** if the trace surfaced additional dependency DLLs for the new effects, then re-run the trace to confirm a stable closure. + +- [ ] **Step 5: Commit** + +```bash +git add native/shim/bundle/maxine-manifest.psd1 scripts/assemble-maxine-stage.ps1 +git commit -m "build(bundle): stage Artifact Reduction + Super Resolution DLLs + models" +``` + +--- + +## Final verification (human gate) + +- [ ] Build shim **SDK config last**, build App, run. Confirm: + - Real fps shows in the status line and drops as effects stack (Task 1). + - Artifact Reduction toggle visibly removes MJPG blockiness on the webcam. + - Super Resolution at 2× visibly adds detail when the overlay is shown large; the overlay re-pins to the larger frame without artifacts. + - Toggles grey out correctly when the probe reports an effect unavailable. +- [ ] Visual/recorder confirmation via Windows.Graphics.Capture / OBS (inherent human gate, `docs/superpowers/verification/`). + +> Reminder (spec §13): stacking AR + Eye Contact + Green Screen + Super Res will not sustain high fps on an RTX 3090 — the fps readout is the tuning signal; no automatic management by design. + +--- + +## Self-Review + +**Spec coverage:** Artifact Reduction (Tasks 3,5,6,7,8,9) ✓; Super Resolution incl. 2× cap + 4K buffer (Tasks 4,5,6,7,8,9) ✓; real fps counter (Task 1) ✓; independent toggles + capability gates + live push + JSON persistence (Task 7) ✓; worker order AR-first/SR-last (Task 6) ✓; ABI parity 528-byte CosCaps + 3 CosParams fields (Tasks 5,7) ✓; bundler (Task 9) ✓; no-auto-management / fps-as-tuning-signal (final gate note) ✓; minification fix correctly **absent** (deferred, spec §11) ✓. + +**Placeholder scan:** no TBD/TODO; "verify at impl" notes name the exact symbol to confirm (selector macros, feature-DLL filenames) — these are real verification steps against headers, not gaps. Every code step shows complete code. + +**Type consistency:** `ShimParams`/`CosParams` field names and order match across `shim.h`, `PInvokeShim`, `Contracts.cs`, `BuildParams`. `SuperResScale` is the shim-facing int (0/15/20); `SuperResScaleIndex` is the VM/combo int (0/1/2) with `ScaleFromIndex`/`IndexFromScale` as the single conversion point. `ArtifactReductionAvailable`/`SuperResAvailable` consistent across `CosCaps`, `ShimCapabilities`, `FakeShim`, `Orchestrator`, `MainViewModel`. Effect class APIs (`Probe`/`Start`/`Stop`/`ProcessFrame`/`IsReady`/`LastError`) consistent with the `capture.cpp` call sites in Task 6. From 00b4865d8201797ad7fd134445cd4bea54270af5 Mon Sep 17 00:00:00 2001 From: opariffazman Date: Wed, 24 Jun 2026 00:27:53 +0800 Subject: [PATCH 04/26] =?UTF-8?q?docs:=20plan=20Task=202=20=E2=80=94=20ful?= =?UTF-8?q?ly-DRY=20VFX=20path=20resolver=20(refactor=20aigs=20to=20share)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Per execution decision: vfx_paths owns g_nvVFXSDKPath; aigs.cpp drops its private resolver copies and calls vfx::. Adds a green-screen re-verify gate. Co-Authored-By: Claude Opus 4.8 (1M context) --- ...26-06-24-ai-sharpness-resolution-phase1.md | 65 ++++++++++++++----- 1 file changed, 48 insertions(+), 17 deletions(-) diff --git a/docs/superpowers/plans/2026-06-24-ai-sharpness-resolution-phase1.md b/docs/superpowers/plans/2026-06-24-ai-sharpness-resolution-phase1.md index 3dba841..547da3a 100644 --- a/docs/superpowers/plans/2026-06-24-ai-sharpness-resolution-phase1.md +++ b/docs/superpowers/plans/2026-06-24-ai-sharpness-resolution-phase1.md @@ -211,16 +211,17 @@ git commit -m "feat(shim): real measured fps in cos_get_status (replaces 30.0 st --- -## Task 2: Shared VFX path resolver +## Task 2: Shared VFX path resolver (single source of truth) -Extract the VFX SDK path + proxy-pointer logic so the two new effects reuse it. `aigs.cpp` keeps its own copy untouched (do not destabilize working green screen). `g_nvVFXSDKPath` stays **defined** in `aigs.cpp`; this module `extern`s and writes it. +Extract the VFX SDK path + proxy-pointer logic into one module that **green screen, Artifact Reduction, and Super Resolution all use**. `g_nvVFXSDKPath` ownership **moves into `vfx_paths.cpp`**; `aigs.cpp` drops its private copies and calls the shared helpers. This is the fully-DRY choice (user decision) — it edits the proven green-screen path, so it carries a **green-screen re-verify gate**. **Files:** - Create: `native/shim/vfx_paths.h`, `native/shim/vfx_paths.cpp` -- Modify: `native/shim/shim.vcxproj` +- Modify: `native/shim/aigs.cpp` (use shared helpers; remove its private `ResolveSdkPaths`/`PointProxiesAt` + its `g_nvVFXSDKPath` definition), `native/shim/shim.vcxproj` +- Verify (human gate): green screen still loads + mattes after the refactor. **Interfaces:** -- Produces: `namespace vfx { bool ResolveSdkPaths(std::string& binDir, std::string& modelDir, std::string& err); void PointProxiesAt(const std::string& binDir); }` +- Produces: `namespace vfx { bool ResolveSdkPaths(std::string& binDir, std::string& modelDir, std::string& err); void PointProxiesAt(const std::string& binDir); }`. `g_nvVFXSDKPath` is now **defined** in `vfx_paths.cpp` (was `aigs.cpp`). - [ ] **Step 1: Create `native/shim/vfx_paths.h`** @@ -228,19 +229,19 @@ Extract the VFX SDK path + proxy-pointer logic so the two new effects reuse it. #pragma once #include -// Shared VFX-SDK runtime discovery for the VFX-based effects (Artifact Reduction, -// Super Resolution). Mirrors the resolver in aigs.cpp; kept separate so the proven -// green-screen path is not touched. Model subdir is models\vfx in the bundled layout. +// Shared VFX-SDK runtime discovery for ALL VFX effects (Green Screen, Artifact +// Reduction, Super Resolution). Single source of truth — aigs.cpp uses these too. +// Model subdir is models\vfx in the bundled layout. namespace vfx { // Fills binDir (holds NVVideoEffects.dll) and modelDir. Order: COS_VFX_RUNTIME_DIR // (flat) -> COS_VFX_SDK_DIR (legacy \bin) -> \maxine (bundled). false if none. bool ResolveSdkPaths(std::string& binDir, std::string& modelDir, std::string& err); -// Points the VFX proxy global (g_nvVFXSDKPath, defined in aigs.cpp) at binDir. +// Points the VFX proxy global (g_nvVFXSDKPath, defined in vfx_paths.cpp) at binDir. void PointProxiesAt(const std::string& binDir); } ``` -- [ ] **Step 2: Create `native/shim/vfx_paths.cpp`** (logic copied verbatim from `aigs.cpp` ResolveSdkPaths/PointProxiesAt) +- [ ] **Step 2: Create `native/shim/vfx_paths.cpp`** (owns `g_nvVFXSDKPath`; logic moved from `aigs.cpp`) ```cpp #include "vfx_paths.h" @@ -248,8 +249,10 @@ void PointProxiesAt(const std::string& binDir); #include #include "paths.h" -// Defined in aigs.cpp; the VFX proxy reads it before LoadLibrary. -extern char* g_nvVFXSDKPath; +// The proxy stub (nvVideoEffectsProxy.cpp) declares this extern; we own the single +// definition here (moved out of aigs.cpp). Non-null => dir holding NVVideoEffects.dll, +// which the proxy passes to SetDllDirectory before LoadLibrary. +char* g_nvVFXSDKPath = nullptr; namespace vfx { bool ResolveSdkPaths(std::string& binDir, std::string& modelDir, std::string& err) { @@ -288,7 +291,33 @@ void PointProxiesAt(const std::string&) {} #endif ``` -- [ ] **Step 3: Add to `native/shim/shim.vcxproj`** +- [ ] **Step 3: Refactor `native/shim/aigs.cpp` to use the shared module** + +In the `COS_HAS_MAXINE` block: +1. Add `#include "vfx_paths.h"` next to the other includes. +2. **Delete** the `g_nvVFXSDKPath` definition (the `char* g_nvVFXSDKPath = nullptr;` line and its comment) — it now lives in `vfx_paths.cpp`. +3. **Delete** the anonymous-namespace `ResolveSdkPaths(...)` and `PointProxiesAt(...)` functions (the whole `namespace { ... }` block that defines them, lines ~26–76). +4. In `Aigs::Probe`, replace the resolver call: + +```cpp + std::string binDir, modelDir, err; + if (!vfx::ResolveSdkPaths(binDir, modelDir, err)) { detail = err; return false; } + vfx::PointProxiesAt(binDir); +``` + +5. In `Aigs::Start`, replace likewise: + +```cpp + std::string binDir, err; + if (!vfx::ResolveSdkPaths(binDir, impl->modelDir, err)) { + lastError_ = err; delete impl; return false; + } + vfx::PointProxiesAt(binDir); +``` + +(Both call sites already use these exact local variable names; only the function namespace changes from the file-local helpers to `vfx::`.) + +- [ ] **Step 4: Add to `native/shim/shim.vcxproj`** Inside the `` that holds the other `` entries (e.g. `aigs.cpp`), add: @@ -296,17 +325,19 @@ Inside the `` that holds the other `` entries (e.g. `aigs. ``` -- [ ] **Step 4: Build the shim (clean)** +- [ ] **Step 5: Build the shim (SDK config) — clean, single `g_nvVFXSDKPath` definition** -Run the shim build command. Expected: 0 warnings (file compiles; not yet referenced — that is fine). +Run the shim build with `COS_VFX_SDK_DIR` set. Expected: 0 warnings, links cleanly (exactly one definition of `g_nvVFXSDKPath` — a duplicate-symbol linker error here means the old definition was not removed from `aigs.cpp`). -- [ ] **Step 5: Commit** +- [ ] **Step 6: Commit** ```bash -git add native/shim/vfx_paths.h native/shim/vfx_paths.cpp native/shim/shim.vcxproj -git commit -m "feat(shim): shared VFX SDK path resolver for new effects" +git add native/shim/vfx_paths.h native/shim/vfx_paths.cpp native/shim/aigs.cpp native/shim/shim.vcxproj +git commit -m "refactor(shim): shared VFX SDK path resolver (green screen + new effects)" ``` +> **Human gate (after build):** run the app with green screen ON and confirm the matte still works — this task edited the proven green-screen load path. Tasks 3–4 reuse `vfx::` so a green-screen regression here would surface everywhere. + --- ## Task 3: Artifact Reduction effect From eed43c52b61dadd30406ab4c1db02adde3ffb06c Mon Sep 17 00:00:00 2001 From: opariffazman Date: Wed, 24 Jun 2026 00:33:31 +0800 Subject: [PATCH 05/26] feat(shim): real measured fps in cos_get_status (replaces 30.0 stub) Co-Authored-By: Claude Opus 4.8 (1M context) --- native/shim/capture.cpp | 16 ++++++++++++++++ native/shim/capture.h | 4 ++++ native/shim/fps_counter.h | 27 +++++++++++++++++++++++++++ native/shim/shim.cpp | 2 +- native/shim/smoke/fps_smoke.cpp | 19 +++++++++++++++++++ 5 files changed, 67 insertions(+), 1 deletion(-) create mode 100644 native/shim/fps_counter.h create mode 100644 native/shim/smoke/fps_smoke.cpp diff --git a/native/shim/capture.cpp b/native/shim/capture.cpp index fa14fee..94bedef 100644 --- a/native/shim/capture.cpp +++ b/native/shim/capture.cpp @@ -3,8 +3,10 @@ #include "eyecontact.h" #include +#include #include #include +#include "fps_counter.h" #include #include @@ -45,6 +47,9 @@ struct CaptureState { std::atomic eyeContactActive{false}; // set by worker std::mutex ecErrMtx; // leaf lock, never nested under mtx/lifecycle std::string ecError; // guarded by ecErrMtx + + std::atomic framesProduced{0}; // bumped by worker after each published frame + FpsCounter fpsCounter; // read only by MeasuredFps (status-poll thread) }; // Module-level singleton state. Capture is used as a single global instance by the @@ -360,6 +365,7 @@ void Capture::WorkerLoop() { g_state.width = width; g_state.height = height; g_state.hasNewFrame = true; + g_state.framesProduced.fetch_add(1, std::memory_order_release); } SafeRelease(sample); } else { @@ -390,6 +396,8 @@ bool Capture::Start(const std::string& symbolicLink) { g_state.width = 0; g_state.height = 0; g_state.hasNewFrame = false; + g_state.framesProduced.store(0, std::memory_order_release); + g_state.fpsCounter.Reset(); } g_state.stopRequested.store(false, std::memory_order_release); g_state.worker = std::thread(&Capture::WorkerLoop, this); @@ -511,3 +519,11 @@ std::vector Capture::Enumerate() { if (devices) CoTaskMemFree(devices); return result; } + +double Capture::MeasuredFps() const { + using clock = std::chrono::steady_clock; + const double nowSec = + std::chrono::duration(clock::now().time_since_epoch()).count(); + const uint64_t frames = g_state.framesProduced.load(std::memory_order_acquire); + return g_state.fpsCounter.Sample(nowSec, frames); +} diff --git a/native/shim/capture.h b/native/shim/capture.h index dec09d3..7d34de0 100644 --- a/native/shim/capture.h +++ b/native/shim/capture.h @@ -42,6 +42,10 @@ class Capture { bool EyeContactActive() const; // true only while gaze redirection is transforming frames std::string EyeContactError() const; // empty when none + // Measured frames-per-second over a rolling window (status polling). Thread-safe to + // call from the single status-poll thread; 0 until the first interval elapses. + double MeasuredFps() const; + private: void WorkerLoop(); }; diff --git a/native/shim/fps_counter.h b/native/shim/fps_counter.h new file mode 100644 index 0000000..332a123 --- /dev/null +++ b/native/shim/fps_counter.h @@ -0,0 +1,27 @@ +#pragma once +#include + +// Rolling FPS estimate. SINGLE-CONSUMER (the status poll thread) — not internally +// synchronized. Feed a monotonically increasing cumulative frame count plus a +// monotonic clock reading in seconds. Recomputes over the elapsed window, but only +// refreshes once at least kMinInterval has passed so a fast poll can't divide by ~0. +class FpsCounter { +public: + double Sample(double nowSec, uint64_t totalFrames) { + if (!started_) { started_ = true; lastSec_ = nowSec; lastFrames_ = totalFrames; return fps_; } + const double dt = nowSec - lastSec_; + const auto dframes = totalFrames - lastFrames_; + lastSec_ = nowSec; // always advance baseline so next window starts here + lastFrames_ = totalFrames; + if (dt < kMinInterval) return fps_; // too soon: keep the last estimate + fps_ = static_cast(dframes) / dt; + return fps_; + } + void Reset() { started_ = false; fps_ = 0.0; lastSec_ = 0.0; lastFrames_ = 0; } +private: + static constexpr double kMinInterval = 0.5; // refresh at most ~2x/sec + bool started_ = false; + double lastSec_ = 0.0; + double fps_ = 0.0; + uint64_t lastFrames_ = 0; +}; diff --git a/native/shim/shim.cpp b/native/shim/shim.cpp index d6c705b..a2459c2 100644 --- a/native/shim/shim.cpp +++ b/native/shim/shim.cpp @@ -73,7 +73,7 @@ COS_API void cos_get_status(CosStatus* out) { if (!out) return; std::memset(out, 0, sizeof(*out)); out->running = g_running ? 1 : 0; - out->fps = g_running ? 30.0 : 0.0; // still a stub count (documented) + out->fps = g_running ? g_capture.MeasuredFps() : 0.0; out->green_screen_active = g_capture.GreenScreenActive() ? 1 : 0; out->eye_contact_active = g_capture.EyeContactActive() ? 1 : 0; std::string err = g_capture.GreenScreenError(); diff --git a/native/shim/smoke/fps_smoke.cpp b/native/shim/smoke/fps_smoke.cpp new file mode 100644 index 0000000..73c7644 --- /dev/null +++ b/native/shim/smoke/fps_smoke.cpp @@ -0,0 +1,19 @@ +// Standalone smoke for FpsCounter. Build: cl /EHsc /I.. fps_smoke.cpp +#include +#include +#include +#include "../fps_counter.h" + +static bool near(double a, double b) { return std::fabs(a - b) < 1e-6; } + +int main() { + FpsCounter c; + assert(near(c.Sample(0.0, 0), 0.0)); // first sample primes, no estimate yet + assert(near(c.Sample(1.0, 30), 30.0)); // 30 frames in 1.0s -> 30 fps + assert(near(c.Sample(1.2, 45), 30.0)); // 0.2s < min interval -> keep last estimate + assert(near(c.Sample(2.0, 90), 56.25)); // 45 frames over 0.8s (since last accepted at t=1.0) + c.Reset(); + assert(near(c.Sample(5.0, 999), 0.0)); // reset re-primes + std::puts("fps_smoke OK"); + return 0; +} From d5e4e5204b0842305dd9697446967401f1769d84 Mon Sep 17 00:00:00 2001 From: opariffazman Date: Wed, 24 Jun 2026 00:38:06 +0800 Subject: [PATCH 06/26] fix(shim): correct FpsCounter window semantics + test expectation (review) Sub-interval Sample() calls must NOT advance the baseline, else the ~30 Hz status poll never accumulates a >=0.5s window and fps stays 0.0 forever. Restore the brief's original semantics; the bug was the test's expected value (56.25 -> 60.0: the t=1.2 sub-interval call leaves the baseline at 1.0/30, so the 1.0->2.0 window is 60 frames / 1.0s). Co-Authored-By: Claude Opus 4.8 (1M context) --- native/shim/fps_counter.h | 9 ++++----- native/shim/smoke/fps_smoke.cpp | 2 +- 2 files changed, 5 insertions(+), 6 deletions(-) diff --git a/native/shim/fps_counter.h b/native/shim/fps_counter.h index 332a123..394cdcc 100644 --- a/native/shim/fps_counter.h +++ b/native/shim/fps_counter.h @@ -9,12 +9,11 @@ class FpsCounter { public: double Sample(double nowSec, uint64_t totalFrames) { if (!started_) { started_ = true; lastSec_ = nowSec; lastFrames_ = totalFrames; return fps_; } - const double dt = nowSec - lastSec_; - const auto dframes = totalFrames - lastFrames_; - lastSec_ = nowSec; // always advance baseline so next window starts here + const double dt = nowSec - lastSec_; + if (dt < kMinInterval) return fps_; // too soon: keep last estimate, baseline unchanged + fps_ = static_cast(totalFrames - lastFrames_) / dt; + lastSec_ = nowSec; lastFrames_ = totalFrames; - if (dt < kMinInterval) return fps_; // too soon: keep the last estimate - fps_ = static_cast(dframes) / dt; return fps_; } void Reset() { started_ = false; fps_ = 0.0; lastSec_ = 0.0; lastFrames_ = 0; } diff --git a/native/shim/smoke/fps_smoke.cpp b/native/shim/smoke/fps_smoke.cpp index 73c7644..a088354 100644 --- a/native/shim/smoke/fps_smoke.cpp +++ b/native/shim/smoke/fps_smoke.cpp @@ -11,7 +11,7 @@ int main() { assert(near(c.Sample(0.0, 0), 0.0)); // first sample primes, no estimate yet assert(near(c.Sample(1.0, 30), 30.0)); // 30 frames in 1.0s -> 30 fps assert(near(c.Sample(1.2, 45), 30.0)); // 0.2s < min interval -> keep last estimate - assert(near(c.Sample(2.0, 90), 56.25)); // 45 frames over 0.8s (since last accepted at t=1.0) + assert(near(c.Sample(2.0, 90), 60.0)); // window 1.0->2.0: 60 frames / 1.0s (the 1.2 sub-interval call did not move the baseline) c.Reset(); assert(near(c.Sample(5.0, 999), 0.0)); // reset re-primes std::puts("fps_smoke OK"); From 292f630c31edda32b4755391376fbc9d14f978ba Mon Sep 17 00:00:00 2001 From: opariffazman Date: Wed, 24 Jun 2026 00:43:01 +0800 Subject: [PATCH 07/26] refactor(shim): shared VFX SDK path resolver (green screen + new effects) Co-Authored-By: Claude Opus 4.8 (1M context) --- native/shim/aigs.cpp | 69 +++------------------------------------ native/shim/shim.vcxproj | 2 ++ native/shim/vfx_paths.cpp | 45 +++++++++++++++++++++++++ native/shim/vfx_paths.h | 13 ++++++++ 4 files changed, 65 insertions(+), 64 deletions(-) create mode 100644 native/shim/vfx_paths.cpp create mode 100644 native/shim/vfx_paths.h diff --git a/native/shim/aigs.cpp b/native/shim/aigs.cpp index 27951b6..79cd156 100644 --- a/native/shim/aigs.cpp +++ b/native/shim/aigs.cpp @@ -14,66 +14,7 @@ #include "nvVideoEffects.h" #include "nvVFXGreenScreen.h" // defines NVVFX_FX_GREEN_SCREEN "GreenScreen" #include "paths.h" - -// The proxy stub (nvVideoEffectsProxy.cpp) declares this as extern; we define it here. -// When non-null it points at the dir that holds NVVideoEffects.dll, which the proxy -// passes to SetDllDirectory before LoadLibrary. -// (Defined in Task 1 — NOT redefined here.) -char* g_nvVFXSDKPath = nullptr; -// Note: NVCVImage.dll resolves via the search path the VFX proxy's SetDllDirectory sets; -// nvCVImageProxy.cpp has no g_nvCVImageSDKPath global, so none is defined here. - -namespace { -// Resolves "\bin" (DLLs) and "\bin\models" once. -// Returns false if the env var is unset/empty. Stores results in out-params so -// callers can keep them alive as needed. -bool ResolveSdkPaths(std::string& binDir, std::string& modelDir, std::string& err) { - char buf[1024] = {0}; - // Prefer COS_VFX_RUNTIME_DIR — a FLAT runtime layout (DLLs in , models in - // \models). This pins the green-screen VFX runtime to the AR-matched 0.7.6 SDK so - // both Maxine SDKs load the SAME TensorRT 10.4 / CUDA 12.1 runtime (co-versioned — a - // mismatch makes NvVFX_Load fail with "no kernel image"). Fall back to COS_VFX_SDK_DIR - // (legacy full-SDK tree: DLLs in \bin, models in \bin\models). - DWORD n = GetEnvironmentVariableA("COS_VFX_RUNTIME_DIR", buf, sizeof(buf)); - if (n > 0 && n < sizeof(buf)) { - std::string root(buf, n); - if (!root.empty() && (root.back() == '\\' || root.back() == '/')) root.pop_back(); - binDir = root; - modelDir = root + "\\models"; - return true; - } - n = GetEnvironmentVariableA("COS_VFX_SDK_DIR", buf, sizeof(buf)); - if (n > 0 && n < sizeof(buf)) { - std::string root(buf, n); - if (!root.empty() && (root.back() == '\\' || root.back() == '/')) root.pop_back(); - binDir = root + "\\bin"; - modelDir = root + "\\bin\\models"; - return true; - } - // Ship default: runtime bundled beside the app at \maxine (shared co-versioned - // root; green-screen models in models\vfx). - std::string appDir = ShimModuleDir(); - if (!appDir.empty()) { - std::string maxine = appDir + "\\maxine"; - if (DirExists(maxine)) { - binDir = maxine; - modelDir = maxine + "\\models\\vfx"; - return true; - } - } - err = "VFX runtime not found: set COS_VFX_RUNTIME_DIR or bundle maxine\\ beside the app"; - return false; -} - -// Points the VFX proxy global at the bin dir so NVVideoEffects.dll is found. -// NVCVImage.dll resolves from the same search path that VFX proxy sets. -// Idempotent; s_bin must outlive every Maxine call. -void PointProxiesAt(const std::string& binDir) { - static std::string s_bin; - s_bin = binDir; - g_nvVFXSDKPath = const_cast(s_bin.c_str()); -} -} // namespace +#include "vfx_paths.h" // Real per-effect state, hidden behind the opaque impl_ pointer. struct AigsImpl { @@ -95,8 +36,8 @@ Aigs::~Aigs() { Stop(); } bool Aigs::Probe(std::string& detail) { std::string binDir, modelDir, err; - if (!ResolveSdkPaths(binDir, modelDir, err)) { detail = err; return false; } - PointProxiesAt(binDir); + if (!vfx::ResolveSdkPaths(binDir, modelDir, err)) { detail = err; return false; } + vfx::PointProxiesAt(binDir); // Probe uses the SDK's default CUDA stream (no explicit CudaStreamCreate); sufficient for a load-only check. NvVFX_Handle eff = nullptr; @@ -128,10 +69,10 @@ bool Aigs::Start() { if (!impl) { lastError_ = "out of memory"; return false; } std::string binDir, err; - if (!ResolveSdkPaths(binDir, impl->modelDir, err)) { + if (!vfx::ResolveSdkPaths(binDir, impl->modelDir, err)) { lastError_ = err; delete impl; return false; } - PointProxiesAt(binDir); + vfx::PointProxiesAt(binDir); if (NvVFX_CudaStreamCreate(&impl->stream) != NVCV_SUCCESS) { lastError_ = "NvVFX_CudaStreamCreate failed"; delete impl; return false; diff --git a/native/shim/shim.vcxproj b/native/shim/shim.vcxproj index 43f9ded..0580280 100644 --- a/native/shim/shim.vcxproj +++ b/native/shim/shim.vcxproj @@ -93,6 +93,7 @@ + @@ -117,6 +118,7 @@ + diff --git a/native/shim/vfx_paths.cpp b/native/shim/vfx_paths.cpp new file mode 100644 index 0000000..1bcab1e --- /dev/null +++ b/native/shim/vfx_paths.cpp @@ -0,0 +1,45 @@ +#include "vfx_paths.h" +#ifdef COS_HAS_MAXINE +#include +#include "paths.h" + +// The proxy stub (nvVideoEffectsProxy.cpp) declares this extern; we own the single +// definition here (moved out of aigs.cpp). Non-null => dir holding NVVideoEffects.dll, +// which the proxy passes to SetDllDirectory before LoadLibrary. +char* g_nvVFXSDKPath = nullptr; + +namespace vfx { +bool ResolveSdkPaths(std::string& binDir, std::string& modelDir, std::string& err) { + char buf[1024] = {0}; + DWORD n = GetEnvironmentVariableA("COS_VFX_RUNTIME_DIR", buf, sizeof(buf)); + if (n > 0 && n < sizeof(buf)) { + std::string root(buf, n); + if (!root.empty() && (root.back() == '\\' || root.back() == '/')) root.pop_back(); + binDir = root; modelDir = root + "\\models"; return true; + } + n = GetEnvironmentVariableA("COS_VFX_SDK_DIR", buf, sizeof(buf)); + if (n > 0 && n < sizeof(buf)) { + std::string root(buf, n); + if (!root.empty() && (root.back() == '\\' || root.back() == '/')) root.pop_back(); + binDir = root + "\\bin"; modelDir = root + "\\bin\\models"; return true; + } + std::string appDir = ShimModuleDir(); + if (!appDir.empty()) { + std::string maxine = appDir + "\\maxine"; + if (DirExists(maxine)) { binDir = maxine; modelDir = maxine + "\\models\\vfx"; return true; } + } + err = "VFX runtime not found: set COS_VFX_RUNTIME_DIR or bundle maxine\\ beside the app"; + return false; +} +void PointProxiesAt(const std::string& binDir) { + static std::string s_bin; + s_bin = binDir; + g_nvVFXSDKPath = const_cast(s_bin.c_str()); +} +} +#else +namespace vfx { +bool ResolveSdkPaths(std::string&, std::string&, std::string& err) { err = "Maxine SDK not built in"; return false; } +void PointProxiesAt(const std::string&) {} +} +#endif diff --git a/native/shim/vfx_paths.h b/native/shim/vfx_paths.h new file mode 100644 index 0000000..266d07e --- /dev/null +++ b/native/shim/vfx_paths.h @@ -0,0 +1,13 @@ +#pragma once +#include + +// Shared VFX-SDK runtime discovery for ALL VFX effects (Green Screen, Artifact +// Reduction, Super Resolution). Single source of truth — aigs.cpp uses these too. +// Model subdir is models\vfx in the bundled layout. +namespace vfx { +// Fills binDir (holds NVVideoEffects.dll) and modelDir. Order: COS_VFX_RUNTIME_DIR +// (flat) -> COS_VFX_SDK_DIR (legacy \bin) -> \maxine (bundled). false if none. +bool ResolveSdkPaths(std::string& binDir, std::string& modelDir, std::string& err); +// Points the VFX proxy global (g_nvVFXSDKPath, defined in vfx_paths.cpp) at binDir. +void PointProxiesAt(const std::string& binDir); +} From e49135dbd6e3846ef08cbf48e8c512806c77618c Mon Sep 17 00:00:00 2001 From: opariffazman Date: Wed, 24 Jun 2026 00:53:38 +0800 Subject: [PATCH 08/26] feat(shim): Maxine VFX Artifact Reduction effect Add ArtifactReduction class (artifactreduction.h/.cpp) mirroring the aigs.cpp pattern: BGRA CPU frame -> GPU BGR upload -> NvVFX_Run -> BGR download -> write-back (RGB replaced, alpha untouched). Stub when COS_HAS_MAXINE undefined. Add effects_smoke.cpp smoke. Wire into shim.vcxproj (ClCompile + ClInclude + AR feature include dir). Selector: NVVFX_FX_ARTIFACT_REDUCTION "ArtifactReduction" from nvVFXArtifactReduction.h (stub header in SDK features tree; real header ships with nvvfxartifactreduction feature downloaded from NGC via install_feature.ps1). Build: 0 warnings, 0 errors. 9 cos_* exports unchanged. GreenScreen+GazeRedirection+ArtifactReduction present in DLL; "not built in" absent. Co-Authored-By: Claude Opus 4.8 (1M context) --- native/shim/artifactreduction.cpp | 136 ++++++++++++++++++++++++++++ native/shim/artifactreduction.h | 23 +++++ native/shim/shim.vcxproj | 4 +- native/shim/smoke/effects_smoke.cpp | 38 ++++++++ 4 files changed, 200 insertions(+), 1 deletion(-) create mode 100644 native/shim/artifactreduction.cpp create mode 100644 native/shim/artifactreduction.h create mode 100644 native/shim/smoke/effects_smoke.cpp diff --git a/native/shim/artifactreduction.cpp b/native/shim/artifactreduction.cpp new file mode 100644 index 0000000..d8959ef --- /dev/null +++ b/native/shim/artifactreduction.cpp @@ -0,0 +1,136 @@ +#include "artifactreduction.h" + +#ifdef COS_HAS_MAXINE +#define NOMINMAX +#include +#include +#include +#include +#include "nvCVStatus.h" +#include "nvCVImage.h" +#include "nvVideoEffects.h" +#include "nvVFXArtifactReduction.h" // defines NVVFX_FX_ARTIFACT_REDUCTION "ArtifactReduction" +#include "vfx_paths.h" + +struct ArImpl { + NvVFX_Handle effect = nullptr; + CUstream stream = nullptr; + std::string modelDir; + NvCVImage srcGpu{}; // BGR u8 chunky GPU (input) + NvCVImage dstGpu{}; // BGR u8 chunky GPU (output) + NvCVImage dstCpu{}; // BGR u8 chunky CPU (downloaded) + NvCVImage stage{}; // BGRA u8 chunky GPU (transfer staging) + int w = 0, h = 0; + bool loaded = false; +}; + +ArtifactReduction::ArtifactReduction() = default; +ArtifactReduction::~ArtifactReduction() { Stop(); } + +bool ArtifactReduction::Probe(std::string& detail) { + std::string binDir, modelDir, err; + if (!vfx::ResolveSdkPaths(binDir, modelDir, err)) { detail = err; return false; } + vfx::PointProxiesAt(binDir); + NvVFX_Handle eff = nullptr; + if (NvVFX_CreateEffect(NVVFX_FX_ARTIFACT_REDUCTION, &eff) != NVCV_SUCCESS || !eff) { + detail = "NvVFX_CreateEffect(ArtifactReduction) failed (DLL/SDK load?)"; return false; + } + NvVFX_SetString(eff, NVVFX_MODEL_DIRECTORY, modelDir.c_str()); + NvVFX_SetU32(eff, NVVFX_MODE, 0u); + NvVFX_SetU32(eff, NVVFX_MAX_INPUT_WIDTH, 1920u); + NvVFX_SetU32(eff, NVVFX_MAX_INPUT_HEIGHT, 1080u); + NvCV_Status load = NvVFX_Load(eff); + NvVFX_DestroyEffect(eff); + if (load != NVCV_SUCCESS) { detail = "NvVFX_Load(ArtifactReduction) failed (models missing?)"; return false; } + detail = "ArtifactReduction available"; + return true; +} + +bool ArtifactReduction::Start() { + Stop(); + ArImpl* impl = new (std::nothrow) ArImpl(); + if (!impl) { lastError_ = "out of memory"; return false; } + std::string binDir, err; + if (!vfx::ResolveSdkPaths(binDir, impl->modelDir, err)) { lastError_ = err; delete impl; return false; } + vfx::PointProxiesAt(binDir); + if (NvVFX_CudaStreamCreate(&impl->stream) != NVCV_SUCCESS) { lastError_ = "CudaStreamCreate failed"; delete impl; return false; } + if (NvVFX_CreateEffect(NVVFX_FX_ARTIFACT_REDUCTION, &impl->effect) != NVCV_SUCCESS || !impl->effect) { + lastError_ = "CreateEffect failed"; delete impl; return false; + } + NvVFX_SetString(impl->effect, NVVFX_MODEL_DIRECTORY, impl->modelDir.c_str()); + NvVFX_SetCudaStream(impl->effect, NVVFX_CUDA_STREAM, impl->stream); + NvVFX_SetU32(impl->effect, NVVFX_MODE, 1u); // mode 1 = stronger / for compressed input + impl_ = impl; ready_ = true; lastError_.clear(); + return true; +} + +void ArtifactReduction::Stop() { + auto* impl = static_cast(impl_); + if (!impl) { ready_ = false; return; } + NvCVImage_Dealloc(&impl->srcGpu); + NvCVImage_Dealloc(&impl->dstGpu); + NvCVImage_Dealloc(&impl->dstCpu); + NvCVImage_Dealloc(&impl->stage); + if (impl->effect) NvVFX_DestroyEffect(impl->effect); + if (impl->stream) NvVFX_CudaStreamDestroy(impl->stream); + delete impl; impl_ = nullptr; ready_ = false; +} + +namespace { +NvCV_Status EnsureImages(ArImpl* impl, int w, int h) { + if (impl->loaded && impl->w == w && impl->h == h) return NVCV_SUCCESS; + NvCVImage_Dealloc(&impl->srcGpu); + NvCVImage_Dealloc(&impl->dstGpu); + NvCVImage_Dealloc(&impl->dstCpu); + NvCVImage_Dealloc(&impl->stage); + NvCV_Status s; + s = NvCVImage_Alloc(&impl->srcGpu, w, h, NVCV_BGR, NVCV_U8, NVCV_CHUNKY, NVCV_GPU, 1); if (s != NVCV_SUCCESS) return s; + s = NvCVImage_Alloc(&impl->dstGpu, w, h, NVCV_BGR, NVCV_U8, NVCV_CHUNKY, NVCV_GPU, 1); if (s != NVCV_SUCCESS) return s; + s = NvCVImage_Alloc(&impl->dstCpu, w, h, NVCV_BGR, NVCV_U8, NVCV_CHUNKY, NVCV_CPU, 1); if (s != NVCV_SUCCESS) return s; + s = NvCVImage_Alloc(&impl->stage, w, h, NVCV_BGRA, NVCV_U8, NVCV_CHUNKY, NVCV_GPU, 1); if (s != NVCV_SUCCESS) return s; + s = NvVFX_SetImage(impl->effect, NVVFX_INPUT_IMAGE, &impl->srcGpu); if (s != NVCV_SUCCESS) return s; + s = NvVFX_SetImage(impl->effect, NVVFX_OUTPUT_IMAGE, &impl->dstGpu); if (s != NVCV_SUCCESS) return s; + s = NvVFX_SetU32(impl->effect, NVVFX_MAX_INPUT_WIDTH, static_cast(w)); if (s != NVCV_SUCCESS) return s; + s = NvVFX_SetU32(impl->effect, NVVFX_MAX_INPUT_HEIGHT, static_cast(h)); if (s != NVCV_SUCCESS) return s; + s = NvVFX_Load(impl->effect); if (s != NVCV_SUCCESS) return s; + impl->w = w; impl->h = h; impl->loaded = true; + return NVCV_SUCCESS; +} +} // namespace + +bool ArtifactReduction::ProcessFrame(uint8_t* bgra, int w, int h) { + auto* impl = static_cast(impl_); + if (!impl || !impl->effect || !bgra || w <= 0 || h <= 0) return false; + if (NvCV_Status es = EnsureImages(impl, w, h); es != NVCV_SUCCESS) { + lastError_ = std::string("EnsureImages/Load failed: ") + NvCV_GetErrorStringFromCode(es); ready_ = false; return false; + } + NvCVImage src{}; + NvCVImage_Init(&src, w, h, w * 4, bgra, NVCV_BGRA, NVCV_U8, NVCV_CHUNKY, NVCV_CPU); + if (NvCVImage_Transfer(&src, &impl->srcGpu, 1.0f, impl->stream, &impl->stage) != NVCV_SUCCESS) { lastError_ = "upload failed"; return false; } + if (NvVFX_Run(impl->effect, 0) != NVCV_SUCCESS) { lastError_ = "NvVFX_Run failed"; return false; } + if (NvCVImage_Transfer(&impl->dstGpu, &impl->dstCpu, 1.0f, impl->stream, nullptr) != NVCV_SUCCESS) { lastError_ = "download failed"; return false; } + if (NvVFX_CudaStreamSynchronize(impl->stream) != NVCV_SUCCESS) { lastError_ = "stream sync failed"; return false; } + // Write cleaned BGR back over the BGRA buffer; leave alpha untouched (forced 0xFF upstream). + const uint8_t* d = static_cast(impl->dstCpu.pixels); + const int dpitch = impl->dstCpu.pitch; + for (int y = 0; y < h; ++y) { + const uint8_t* drow = d + static_cast(dpitch) * y; + uint8_t* prow = bgra + static_cast(w) * 4 * y; + for (int x = 0; x < w; ++x) { + prow[x * 4 + 0] = drow[x * 3 + 0]; + prow[x * 4 + 1] = drow[x * 3 + 1]; + prow[x * 4 + 2] = drow[x * 3 + 2]; + } + } + return true; +} + +#else +// ---- Passthrough stub: built when no SDK is configured. ---- +ArtifactReduction::ArtifactReduction() = default; +ArtifactReduction::~ArtifactReduction() = default; +bool ArtifactReduction::Probe(std::string& detail) { detail = "Maxine SDK not built in"; return false; } +bool ArtifactReduction::Start() { lastError_ = "Maxine SDK not built in"; ready_ = false; return false; } +void ArtifactReduction::Stop() { ready_ = false; } +bool ArtifactReduction::ProcessFrame(uint8_t*, int, int) { return false; } +#endif diff --git a/native/shim/artifactreduction.h b/native/shim/artifactreduction.h new file mode 100644 index 0000000..7e15aac --- /dev/null +++ b/native/shim/artifactreduction.h @@ -0,0 +1,23 @@ +#pragma once +#include +#include + +// Wraps the Maxine VFX Artifact Reduction effect. CPU-copy: a BGRA frame is uploaded +// to the GPU as BGR, cleaned, downloaded, and written back over the same BGRA buffer +// (RGB replaced, alpha left untouched). No size change. All methods no-throw; failure +// via IsReady()/LastError(). Without COS_HAS_MAXINE this is a never-ready stub. +class ArtifactReduction { +public: + ArtifactReduction(); + ~ArtifactReduction(); + static bool Probe(std::string& detail); + bool Start(); + void Stop(); + bool ProcessFrame(uint8_t* bgra, int width, int height); + bool IsReady() const { return ready_; } + const std::string& LastError() const { return lastError_; } +private: + bool ready_ = false; + std::string lastError_; + void* impl_ = nullptr; +}; diff --git a/native/shim/shim.vcxproj b/native/shim/shim.vcxproj index 0580280..e81a0b7 100644 --- a/native/shim/shim.vcxproj +++ b/native/shim/shim.vcxproj @@ -76,7 +76,7 @@ COS_HAS_MAXINE;%(PreprocessorDefinitions) - $(CosVfxSdkDir)\nvvfx\include;$(CosVfxSdkDir)\features\nvvfxgreenscreen\include;%(AdditionalIncludeDirectories) + $(CosVfxSdkDir)\nvvfx\include;$(CosVfxSdkDir)\features\nvvfxgreenscreen\include;$(CosVfxSdkDir)\features\nvvfxartifactreduction\include;%(AdditionalIncludeDirectories) @@ -91,6 +91,7 @@ + @@ -116,6 +117,7 @@ + diff --git a/native/shim/smoke/effects_smoke.cpp b/native/shim/smoke/effects_smoke.cpp new file mode 100644 index 0000000..231560a --- /dev/null +++ b/native/shim/smoke/effects_smoke.cpp @@ -0,0 +1,38 @@ +// Dev-box smoke (needs VFX SDK + RTX GPU). Build with the shim's include/lib setup. +// Verifies ArtifactReduction can start and process a synthetic 1280x720 frame. +// +// Build (Developer PowerShell for VS 2022, from repo root): +// $env:COS_VFX_SDK_DIR = "C:\path\to\VideoFX" +// cl /EHsc /std:c++17 ^ +// /I "%COS_VFX_SDK_DIR%\nvvfx\include" ^ +// /I "%COS_VFX_SDK_DIR%\features\nvvfxartifactreduction\include" ^ +// /DCOS_HAS_MAXINE ^ +// native\shim\smoke\effects_smoke.cpp ^ +// native\shim\artifactreduction.cpp ^ +// native\shim\vfx_paths.cpp native\shim\paths.cpp ^ +// "%COS_VFX_SDK_DIR%\nvvfx\src\nvVideoEffectsProxy.cpp" ^ +// "%COS_VFX_SDK_DIR%\nvvfx\src\nvCVImageProxy.cpp" +// .\effects_smoke.exe +// +// Expected output (RTX GPU with AR models installed): +// AR ProcessFrame: ok () +// effects_smoke AR OK +// Without GPU/models, Probe prints the reason and exits 0 (deferred to human gate). +#include +#include +#include +#include "../artifactreduction.h" + +int main() { + std::string detail; + if (!ArtifactReduction::Probe(detail)) { std::printf("AR unavailable: %s\n", detail.c_str()); return 0; } + ArtifactReduction ar; + assert(ar.Start()); + std::vector frame(1280 * 720 * 4, 128); + bool ok = ar.ProcessFrame(frame.data(), 1280, 720); + std::printf("AR ProcessFrame: %s (%s)\n", ok ? "ok" : "fail", ar.LastError().c_str()); + assert(ok); + ar.Stop(); + std::puts("effects_smoke AR OK"); + return 0; +} From 8ec212cb9171b0674c4786e7223038c3ec2eb415 Mon Sep 17 00:00:00 2001 From: opariffazman Date: Wed, 24 Jun 2026 00:57:39 +0800 Subject: [PATCH 09/26] fix(shim): define ArtifactReduction selector in-source, drop external header MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The VFX 1.2.0.0 base SDK has no artifact-reduction selector macro (it ships only in the nvvfxartifactreduction per-feature header downloaded from NGC). Including that header made the build depend on an out-of-repo file. Instead define the literal in artifactreduction.cpp behind an #ifndef guard — mirrors eyecontact.cpp's gaze-selector guard. The official header's define wins if/when installed. Drop the now-unneeded nvvfxartifactreduction include dir from the vcxproj and the smoke build comment. Build: 0 warnings, 0 errors. 9 cos_* exports unchanged; ArtifactReduction present in DLL. Co-Authored-By: Claude Opus 4.8 (1M context) --- native/shim/artifactreduction.cpp | 9 ++++++++- native/shim/shim.vcxproj | 2 +- native/shim/smoke/effects_smoke.cpp | 1 - 3 files changed, 9 insertions(+), 3 deletions(-) diff --git a/native/shim/artifactreduction.cpp b/native/shim/artifactreduction.cpp index d8959ef..c0e9dc6 100644 --- a/native/shim/artifactreduction.cpp +++ b/native/shim/artifactreduction.cpp @@ -9,9 +9,16 @@ #include "nvCVStatus.h" #include "nvCVImage.h" #include "nvVideoEffects.h" -#include "nvVFXArtifactReduction.h" // defines NVVFX_FX_ARTIFACT_REDUCTION "ArtifactReduction" #include "vfx_paths.h" +// VFX 1.2.0.0 ships the artifact-reduction selector only in its per-feature header +// (installed from NGC). Define the documented literal if absent — mirrors how +// eyecontact.cpp guards the gaze selector. NVIDIA VideoFX selector "ArtifactReduction" +// matches the nvvfxartifactreduction NGC feature name. +#ifndef NVVFX_FX_ARTIFACT_REDUCTION +#define NVVFX_FX_ARTIFACT_REDUCTION "ArtifactReduction" +#endif + struct ArImpl { NvVFX_Handle effect = nullptr; CUstream stream = nullptr; diff --git a/native/shim/shim.vcxproj b/native/shim/shim.vcxproj index e81a0b7..4942bf5 100644 --- a/native/shim/shim.vcxproj +++ b/native/shim/shim.vcxproj @@ -76,7 +76,7 @@ COS_HAS_MAXINE;%(PreprocessorDefinitions) - $(CosVfxSdkDir)\nvvfx\include;$(CosVfxSdkDir)\features\nvvfxgreenscreen\include;$(CosVfxSdkDir)\features\nvvfxartifactreduction\include;%(AdditionalIncludeDirectories) + $(CosVfxSdkDir)\nvvfx\include;$(CosVfxSdkDir)\features\nvvfxgreenscreen\include;%(AdditionalIncludeDirectories) diff --git a/native/shim/smoke/effects_smoke.cpp b/native/shim/smoke/effects_smoke.cpp index 231560a..8aaefea 100644 --- a/native/shim/smoke/effects_smoke.cpp +++ b/native/shim/smoke/effects_smoke.cpp @@ -5,7 +5,6 @@ // $env:COS_VFX_SDK_DIR = "C:\path\to\VideoFX" // cl /EHsc /std:c++17 ^ // /I "%COS_VFX_SDK_DIR%\nvvfx\include" ^ -// /I "%COS_VFX_SDK_DIR%\features\nvvfxartifactreduction\include" ^ // /DCOS_HAS_MAXINE ^ // native\shim\smoke\effects_smoke.cpp ^ // native\shim\artifactreduction.cpp ^ From dad29bf927b84ffcb1b644ecf4c2cfb18c1941bf Mon Sep 17 00:00:00 2001 From: opariffazman Date: Wed, 24 Jun 2026 01:08:35 +0800 Subject: [PATCH 10/26] feat(shim): Maxine VFX Super Resolution effect (1.5x/2x) Co-Authored-By: Claude Opus 4.8 (1M context) --- native/shim/shim.vcxproj | 2 + native/shim/smoke/effects_smoke.cpp | 40 ++++--- native/shim/superres.cpp | 162 ++++++++++++++++++++++++++++ native/shim/superres.h | 26 +++++ 4 files changed, 218 insertions(+), 12 deletions(-) create mode 100644 native/shim/superres.cpp create mode 100644 native/shim/superres.h diff --git a/native/shim/shim.vcxproj b/native/shim/shim.vcxproj index 4942bf5..b33811b 100644 --- a/native/shim/shim.vcxproj +++ b/native/shim/shim.vcxproj @@ -93,6 +93,7 @@ + @@ -119,6 +120,7 @@ + diff --git a/native/shim/smoke/effects_smoke.cpp b/native/shim/smoke/effects_smoke.cpp index 8aaefea..2318efa 100644 --- a/native/shim/smoke/effects_smoke.cpp +++ b/native/shim/smoke/effects_smoke.cpp @@ -1,5 +1,5 @@ // Dev-box smoke (needs VFX SDK + RTX GPU). Build with the shim's include/lib setup. -// Verifies ArtifactReduction can start and process a synthetic 1280x720 frame. +// Verifies ArtifactReduction and SuperRes can start and process synthetic frames. // // Build (Developer PowerShell for VS 2022, from repo root): // $env:COS_VFX_SDK_DIR = "C:\path\to\VideoFX" @@ -8,30 +8,46 @@ // /DCOS_HAS_MAXINE ^ // native\shim\smoke\effects_smoke.cpp ^ // native\shim\artifactreduction.cpp ^ +// native\shim\superres.cpp ^ // native\shim\vfx_paths.cpp native\shim\paths.cpp ^ // "%COS_VFX_SDK_DIR%\nvvfx\src\nvVideoEffectsProxy.cpp" ^ // "%COS_VFX_SDK_DIR%\nvvfx\src\nvCVImageProxy.cpp" // .\effects_smoke.exe // -// Expected output (RTX GPU with AR models installed): +// Expected output (RTX GPU with models installed): // AR ProcessFrame: ok () // effects_smoke AR OK +// SR ProcessFrame: ok -> 1280x960 (requires nvvfxvideosuperres NGC feature + models) // Without GPU/models, Probe prints the reason and exits 0 (deferred to human gate). #include #include #include #include "../artifactreduction.h" +#include "../superres.h" int main() { - std::string detail; - if (!ArtifactReduction::Probe(detail)) { std::printf("AR unavailable: %s\n", detail.c_str()); return 0; } - ArtifactReduction ar; - assert(ar.Start()); - std::vector frame(1280 * 720 * 4, 128); - bool ok = ar.ProcessFrame(frame.data(), 1280, 720); - std::printf("AR ProcessFrame: %s (%s)\n", ok ? "ok" : "fail", ar.LastError().c_str()); - assert(ok); - ar.Stop(); - std::puts("effects_smoke AR OK"); + { + std::string detail; + if (!ArtifactReduction::Probe(detail)) { std::printf("AR unavailable: %s\n", detail.c_str()); return 0; } + ArtifactReduction ar; + assert(ar.Start()); + std::vector frame(1280 * 720 * 4, 128); + bool ok = ar.ProcessFrame(frame.data(), 1280, 720); + std::printf("AR ProcessFrame: %s (%s)\n", ok ? "ok" : "fail", ar.LastError().c_str()); + assert(ok); + ar.Stop(); + std::puts("effects_smoke AR OK"); + } + { + std::string d; + if (SuperRes::Probe(d)) { + SuperRes sr; assert(sr.Start(20)); + std::vector in(640 * 480 * 4, 64), out; int ow = 0, oh = 0; + bool ok = sr.ProcessFrame(in.data(), 640, 480, out, ow, oh); + std::printf("SR ProcessFrame: %s -> %dx%d\n", ok ? "ok" : "fail", ow, oh); + assert(ok && ow == 1280 && oh == 960); + sr.Stop(); + } else { std::printf("SR unavailable: %s\n", d.c_str()); } + } return 0; } diff --git a/native/shim/superres.cpp b/native/shim/superres.cpp new file mode 100644 index 0000000..53101c0 --- /dev/null +++ b/native/shim/superres.cpp @@ -0,0 +1,162 @@ +#include "superres.h" + +#ifdef COS_HAS_MAXINE +#define NOMINMAX +#include +#include +#include +#include +#include "nvCVStatus.h" +#include "nvCVImage.h" +#include "nvVideoEffects.h" +#include "vfx_paths.h" + +// VFX 1.2.0.0 ships the super-res selector only in its per-feature header (NGC feature +// nvvfxvideosuperres). Define the documented literal if absent. NVIDIA VideoFX selector +// "SuperRes" confirmed by grepping the VFX 1.2.0.0 SDK doc tree. +#ifndef NVVFX_FX_SUPER_RES +#define NVVFX_FX_SUPER_RES "SuperRes" +#endif + +struct SrImpl { + NvVFX_Handle effect = nullptr; + CUstream stream = nullptr; + std::string modelDir; + NvCVImage srcGpu{}; // BGR u8 chunky GPU (input, w x h) + NvCVImage dstGpu{}; // BGR u8 chunky GPU (output, outW x outH) + NvCVImage dstCpu{}; // BGR u8 chunky CPU (downloaded) + NvCVImage stage{}; // BGRA u8 chunky GPU (upload staging, w x h) + int w = 0, h = 0, ow = 0, oh = 0; + bool loaded = false; +}; + +static void ScaledDims(int w, int h, int scaleX10, int& ow, int& oh) { + ow = (w * scaleX10) / 10; + oh = (h * scaleX10) / 10; +} + +SuperRes::SuperRes() = default; +SuperRes::~SuperRes() { Stop(); } + +// Probe loads the effect using mode 1 (VSR_Low), matching the mode Start() uses. +// This avoids a false-positive probe where mode 0 (bicubic) succeeds but mode 1 fails. +bool SuperRes::Probe(std::string& detail) { + std::string binDir, modelDir, err; + if (!vfx::ResolveSdkPaths(binDir, modelDir, err)) { detail = err; return false; } + vfx::PointProxiesAt(binDir); + NvVFX_Handle eff = nullptr; + if (NvVFX_CreateEffect(NVVFX_FX_SUPER_RES, &eff) != NVCV_SUCCESS || !eff) { + detail = "NvVFX_CreateEffect(SuperRes) failed (DLL/SDK load?)"; return false; + } + NvVFX_SetString(eff, NVVFX_MODEL_DIRECTORY, modelDir.c_str()); + NvVFX_SetU32(eff, NVVFX_MODE, 1u); // VSR_Low; same mode as Start() to avoid false-positive + NvVFX_SetU32(eff, NVVFX_MAX_INPUT_WIDTH, 1920u); + NvVFX_SetU32(eff, NVVFX_MAX_INPUT_HEIGHT, 1080u); + NvCV_Status load = NvVFX_Load(eff); + NvVFX_DestroyEffect(eff); + if (load != NVCV_SUCCESS) { detail = "NvVFX_Load(SuperRes) failed (models missing?)"; return false; } + detail = "SuperRes available"; + return true; +} + +bool SuperRes::Start(int scaleX10) { + Stop(); + scaleX10_ = (scaleX10 == 15) ? 15 : 20; // only 1.5x / 2x supported here + SrImpl* impl = new (std::nothrow) SrImpl(); + if (!impl) { lastError_ = "out of memory"; return false; } + std::string binDir, err; + if (!vfx::ResolveSdkPaths(binDir, impl->modelDir, err)) { lastError_ = err; delete impl; return false; } + vfx::PointProxiesAt(binDir); + if (NvVFX_CudaStreamCreate(&impl->stream) != NVCV_SUCCESS) { lastError_ = "CudaStreamCreate failed"; delete impl; return false; } + if (NvVFX_CreateEffect(NVVFX_FX_SUPER_RES, &impl->effect) != NVCV_SUCCESS || !impl->effect) { + lastError_ = "CreateEffect failed"; delete impl; return false; + } + NvVFX_SetString(impl->effect, NVVFX_MODEL_DIRECTORY, impl->modelDir.c_str()); + NvVFX_SetCudaStream(impl->effect, NVVFX_CUDA_STREAM, impl->stream); + NvVFX_SetU32(impl->effect, NVVFX_MODE, 1u); // VSR_Low: fast, good quality for live webcam + // Scale is inferred by the effect from the ratio of output to input image dimensions; + // no explicit NvVFX_SetF32(NVVFX_SCALE, ...) call is required. + impl_ = impl; ready_ = true; lastError_.clear(); + return true; +} + +void SuperRes::Stop() { + auto* impl = static_cast(impl_); + if (!impl) { ready_ = false; return; } + NvCVImage_Dealloc(&impl->srcGpu); + NvCVImage_Dealloc(&impl->dstGpu); + NvCVImage_Dealloc(&impl->dstCpu); + NvCVImage_Dealloc(&impl->stage); + if (impl->effect) NvVFX_DestroyEffect(impl->effect); + if (impl->stream) NvVFX_CudaStreamDestroy(impl->stream); + delete impl; impl_ = nullptr; ready_ = false; +} + +namespace { +// (Re)allocates GPU/CPU images for input w*h and output ow*oh, then loads the model. +// Scale is inferred by the effect from the ratio dstGpu.width/srcGpu.width. +// Returns NVCV_SUCCESS on success; called on first frame and on resolution change. +NvCV_Status EnsureImages(SrImpl* impl, int w, int h, int ow, int oh) { + if (impl->loaded && impl->w == w && impl->h == h && impl->ow == ow && impl->oh == oh) return NVCV_SUCCESS; + NvCVImage_Dealloc(&impl->srcGpu); + NvCVImage_Dealloc(&impl->dstGpu); + NvCVImage_Dealloc(&impl->dstCpu); + NvCVImage_Dealloc(&impl->stage); + NvCV_Status s; + s = NvCVImage_Alloc(&impl->srcGpu, w, h, NVCV_BGR, NVCV_U8, NVCV_CHUNKY, NVCV_GPU, 1); if (s != NVCV_SUCCESS) return s; + s = NvCVImage_Alloc(&impl->dstGpu, ow, oh, NVCV_BGR, NVCV_U8, NVCV_CHUNKY, NVCV_GPU, 1); if (s != NVCV_SUCCESS) return s; + s = NvCVImage_Alloc(&impl->dstCpu, ow, oh, NVCV_BGR, NVCV_U8, NVCV_CHUNKY, NVCV_CPU, 1); if (s != NVCV_SUCCESS) return s; + s = NvCVImage_Alloc(&impl->stage, w, h, NVCV_BGRA, NVCV_U8, NVCV_CHUNKY, NVCV_GPU, 1); if (s != NVCV_SUCCESS) return s; + s = NvVFX_SetImage(impl->effect, NVVFX_INPUT_IMAGE, &impl->srcGpu); if (s != NVCV_SUCCESS) return s; + s = NvVFX_SetImage(impl->effect, NVVFX_OUTPUT_IMAGE, &impl->dstGpu); if (s != NVCV_SUCCESS) return s; + // Set max input dimensions to pre-allocate internal buffers and enable load. + s = NvVFX_SetU32(impl->effect, NVVFX_MAX_INPUT_WIDTH, static_cast(w)); if (s != NVCV_SUCCESS) return s; + s = NvVFX_SetU32(impl->effect, NVVFX_MAX_INPUT_HEIGHT, static_cast(h)); if (s != NVCV_SUCCESS) return s; + s = NvVFX_Load(impl->effect); if (s != NVCV_SUCCESS) return s; + impl->w = w; impl->h = h; impl->ow = ow; impl->oh = oh; impl->loaded = true; + return NVCV_SUCCESS; +} +} // namespace + +bool SuperRes::ProcessFrame(const uint8_t* bgra, int w, int h, std::vector& out, int& outW, int& outH) { + auto* impl = static_cast(impl_); + if (!impl || !impl->effect || !bgra || w <= 0 || h <= 0) return false; + int ow, oh; ScaledDims(w, h, scaleX10_, ow, oh); + if (NvCV_Status es = EnsureImages(impl, w, h, ow, oh); es != NVCV_SUCCESS) { + lastError_ = std::string("EnsureImages/Load failed: ") + NvCV_GetErrorStringFromCode(es); ready_ = false; return false; + } + // Upload: CPU BGRA -> GPU BGR via staging buffer (NvCVImage_Transfer handles format conversion). + NvCVImage src{}; + NvCVImage_Init(&src, w, h, w * 4, const_cast(bgra), NVCV_BGRA, NVCV_U8, NVCV_CHUNKY, NVCV_CPU); + if (NvCVImage_Transfer(&src, &impl->srcGpu, 1.0f, impl->stream, &impl->stage) != NVCV_SUCCESS) { lastError_ = "upload failed"; return false; } + if (NvVFX_Run(impl->effect, 0) != NVCV_SUCCESS) { lastError_ = "NvVFX_Run failed"; return false; } + // Download: GPU BGR (ow x oh) -> CPU BGR. + if (NvCVImage_Transfer(&impl->dstGpu, &impl->dstCpu, 1.0f, impl->stream, nullptr) != NVCV_SUCCESS) { lastError_ = "download failed"; return false; } + if (NvVFX_CudaStreamSynchronize(impl->stream) != NVCV_SUCCESS) { lastError_ = "stream sync failed"; return false; } + // Write-back: BGR CPU -> BGRA output (alpha = 0xFF; fresh upscaled buffer, opaque). + out.assign(static_cast(ow) * oh * 4, 0xFF); + const uint8_t* d = static_cast(impl->dstCpu.pixels); + const int dpitch = impl->dstCpu.pitch; + for (int y = 0; y < oh; ++y) { + const uint8_t* drow = d + static_cast(dpitch) * y; + uint8_t* prow = out.data() + static_cast(ow) * 4 * y; + for (int x = 0; x < ow; ++x) { + prow[x * 4 + 0] = drow[x * 3 + 0]; // B + prow[x * 4 + 1] = drow[x * 3 + 1]; // G + prow[x * 4 + 2] = drow[x * 3 + 2]; // R + prow[x * 4 + 3] = 0xFF; // A = opaque (fresh upscaled buffer) + } + } + outW = ow; outH = oh; + return true; +} + +#else +// ---- Passthrough stub: built when no SDK is configured. ---- +SuperRes::SuperRes() = default; +SuperRes::~SuperRes() = default; +bool SuperRes::Probe(std::string& detail) { detail = "Maxine SDK not built in"; return false; } +bool SuperRes::Start(int) { lastError_ = "Maxine SDK not built in"; ready_ = false; return false; } +void SuperRes::Stop() { ready_ = false; } +bool SuperRes::ProcessFrame(const uint8_t*, int, int, std::vector&, int&, int&) { return false; } +#endif diff --git a/native/shim/superres.h b/native/shim/superres.h new file mode 100644 index 0000000..37e2610 --- /dev/null +++ b/native/shim/superres.h @@ -0,0 +1,26 @@ +#pragma once +#include +#include +#include + +// Wraps the Maxine VFX Super Resolution effect. Upscales a BGRA frame by 1.5x or 2x +// into a freshly sized BGRA output buffer (alpha = 0xFF). Without COS_HAS_MAXINE this +// is a never-ready stub. scaleX10: 15 => 1.5x, 20 => 2x. +class SuperRes { +public: + SuperRes(); + ~SuperRes(); + static bool Probe(std::string& detail); + bool Start(int scaleX10); + void Stop(); + // Reads w*h BGRA from 'bgra', writes outW*outH BGRA into 'out'. Returns false on failure + // (out untouched). outW/outH are w*scale, h*scale. + bool ProcessFrame(const uint8_t* bgra, int w, int h, std::vector& out, int& outW, int& outH); + bool IsReady() const { return ready_; } + const std::string& LastError() const { return lastError_; } +private: + bool ready_ = false; + int scaleX10_ = 20; + std::string lastError_; + void* impl_ = nullptr; +}; From bbaca8a03a8edd8241dae76838d1196ec717c41d Mon Sep 17 00:00:00 2001 From: opariffazman Date: Wed, 24 Jun 2026 01:14:06 +0800 Subject: [PATCH 11/26] fix(shim): destroy CUDA stream on CreateEffect failure in SuperRes/ArtifactReduction The Start() CreateEffect-failure branch deleted the impl without destroying the already-created CUDA stream, and Stop() cannot reclaim it because impl_ was never assigned. Destroy the stream before delete in both effects. Also drop a redundant per-pixel alpha write in SuperRes::ProcessFrame (out.assign pre-fills 0xFF). Co-Authored-By: Claude Opus 4.8 (1M context) --- native/shim/artifactreduction.cpp | 2 +- native/shim/superres.cpp | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/native/shim/artifactreduction.cpp b/native/shim/artifactreduction.cpp index c0e9dc6..ce3616d 100644 --- a/native/shim/artifactreduction.cpp +++ b/native/shim/artifactreduction.cpp @@ -62,7 +62,7 @@ bool ArtifactReduction::Start() { vfx::PointProxiesAt(binDir); if (NvVFX_CudaStreamCreate(&impl->stream) != NVCV_SUCCESS) { lastError_ = "CudaStreamCreate failed"; delete impl; return false; } if (NvVFX_CreateEffect(NVVFX_FX_ARTIFACT_REDUCTION, &impl->effect) != NVCV_SUCCESS || !impl->effect) { - lastError_ = "CreateEffect failed"; delete impl; return false; + lastError_ = "CreateEffect failed"; NvVFX_CudaStreamDestroy(impl->stream); delete impl; return false; } NvVFX_SetString(impl->effect, NVVFX_MODEL_DIRECTORY, impl->modelDir.c_str()); NvVFX_SetCudaStream(impl->effect, NVVFX_CUDA_STREAM, impl->stream); diff --git a/native/shim/superres.cpp b/native/shim/superres.cpp index 53101c0..4f112cf 100644 --- a/native/shim/superres.cpp +++ b/native/shim/superres.cpp @@ -69,7 +69,7 @@ bool SuperRes::Start(int scaleX10) { vfx::PointProxiesAt(binDir); if (NvVFX_CudaStreamCreate(&impl->stream) != NVCV_SUCCESS) { lastError_ = "CudaStreamCreate failed"; delete impl; return false; } if (NvVFX_CreateEffect(NVVFX_FX_SUPER_RES, &impl->effect) != NVCV_SUCCESS || !impl->effect) { - lastError_ = "CreateEffect failed"; delete impl; return false; + lastError_ = "CreateEffect failed"; NvVFX_CudaStreamDestroy(impl->stream); delete impl; return false; } NvVFX_SetString(impl->effect, NVVFX_MODEL_DIRECTORY, impl->modelDir.c_str()); NvVFX_SetCudaStream(impl->effect, NVVFX_CUDA_STREAM, impl->stream); @@ -144,7 +144,7 @@ bool SuperRes::ProcessFrame(const uint8_t* bgra, int w, int h, std::vector Date: Wed, 24 Jun 2026 01:18:28 +0800 Subject: [PATCH 12/26] =?UTF-8?q?feat(shim):=20ABI=20=E2=80=94=20AR/SR=20p?= =?UTF-8?q?arams=20+=20capability=20gates=20(CosParams/CosCaps=20528B)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Opus 4.8 (1M context) --- native/shim/shim.cpp | 14 ++++++++++++-- native/shim/shim.h | 5 +++++ 2 files changed, 17 insertions(+), 2 deletions(-) diff --git a/native/shim/shim.cpp b/native/shim/shim.cpp index a2459c2..04c5220 100644 --- a/native/shim/shim.cpp +++ b/native/shim/shim.cpp @@ -3,6 +3,8 @@ #include "capture.h" #include "aigs.h" #include "eyecontact.h" +#include "artifactreduction.h" +#include "superres.h" #include #include @@ -64,6 +66,8 @@ COS_API void cos_set_params(const CosParams* p) { g_capture.SetGreenScreen(p->green_screen_enabled != 0); g_capture.SetMatteParams(p->green_screen_expand, p->green_screen_feather); g_capture.SetEyeContact(p->eye_contact_enabled != 0); + g_capture.SetArtifactReduction(p->artifact_reduction_enabled != 0); + g_capture.SetSuperRes(p->super_res_enabled != 0, p->super_res_scale); } COS_API void cos_start(void) { g_capture.Start(g_cameraId); g_running = true; } @@ -78,6 +82,8 @@ COS_API void cos_get_status(CosStatus* out) { out->eye_contact_active = g_capture.EyeContactActive() ? 1 : 0; std::string err = g_capture.GreenScreenError(); if (err.empty()) err = g_capture.EyeContactError(); + if (err.empty()) err = g_capture.ArtifactReductionError(); + if (err.empty()) err = g_capture.SuperResError(); if (!err.empty()) { size_t n = err.size() < 255 ? err.size() : 255; std::memcpy(out->error, err.data(), n); @@ -115,8 +121,12 @@ COS_API int cos_query_capabilities(CosCaps* out) { std::memcpy(out->ec_detail, ecDetail.data(), en); out->ec_detail[en] = '\0'; - // Return 1 if either effect is available (the managed side reads the per-gate ints). - return (gsOk || ecOk) ? 1 : 0; + std::string arDetail, srDetail; + out->artifact_reduction_available = ArtifactReduction::Probe(arDetail) ? 1 : 0; + out->super_res_available = SuperRes::Probe(srDetail) ? 1 : 0; + + // Return 1 if any effect is available (the managed side reads the per-gate ints). + return (gsOk || ecOk || out->artifact_reduction_available || out->super_res_available) ? 1 : 0; } COS_API void cos_shutdown(void) { diff --git a/native/shim/shim.h b/native/shim/shim.h index 2fcd942..05aca2d 100644 --- a/native/shim/shim.h +++ b/native/shim/shim.h @@ -23,6 +23,9 @@ typedef struct { int eye_contact_enabled; double eye_contact_sensitivity; double eye_contact_look_away_range; + int artifact_reduction_enabled; + int super_res_enabled; + int super_res_scale; // 0=off, 15=1.5x, 20=2x } CosParams; typedef struct { @@ -30,6 +33,8 @@ typedef struct { char detail[256]; // green-screen status/error (UTF-8, NUL-terminated) int eye_contact_available; // 1 if Maxine GazeRedirection can run, else 0 char ec_detail[256]; // eye-contact status/error (UTF-8, NUL-terminated) + int artifact_reduction_available; // 1 if Maxine ArtifactReduction can run + int super_res_available; // 1 if Maxine SuperRes can run } CosCaps; COS_API int cos_init(void* d3d11_device); From 76ffe9d931a8d789e0ed1e56ff3ed85823001343 Mon Sep 17 00:00:00 2001 From: opariffazman Date: Wed, 24 Jun 2026 01:18:35 +0800 Subject: [PATCH 13/26] feat(shim): run Artifact Reduction (first) + Super Resolution (last) on the worker Co-Authored-By: Claude Opus 4.8 (1M context) --- native/shim/capture.cpp | 112 +++++++++++++++++++++++++++++++++++++--- native/shim/capture.h | 10 ++++ 2 files changed, 114 insertions(+), 8 deletions(-) diff --git a/native/shim/capture.cpp b/native/shim/capture.cpp index 94bedef..e7dd2d8 100644 --- a/native/shim/capture.cpp +++ b/native/shim/capture.cpp @@ -1,6 +1,8 @@ #include "capture.h" #include "aigs.h" #include "eyecontact.h" +#include "artifactreduction.h" +#include "superres.h" #include #include @@ -48,6 +50,17 @@ struct CaptureState { std::mutex ecErrMtx; // leaf lock, never nested under mtx/lifecycle std::string ecError; // guarded by ecErrMtx + std::atomic artifactReductionEnabled{false}; + std::atomic artifactReductionActive{false}; + std::mutex arErrMtx; // leaf lock + std::string arError; + + std::atomic superResEnabled{false}; + std::atomic superResScale{20}; // 15 or 20 + std::atomic superResActive{false}; + std::mutex srErrMtx; // leaf lock + std::string srError; + std::atomic framesProduced{0}; // bumped by worker after each published frame FpsCounter fpsCounter; // read only by MeasuredFps (status-poll thread) }; @@ -261,6 +274,8 @@ void Capture::WorkerLoop() { // affinity. Declare here (after CoInitializeEx) so ctor/dtor run on this thread. Aigs aigs; EyeContact eyeContact; + ArtifactReduction artifactReduction; + SuperRes superRes; IMFSourceReader* reader = nullptr; int width = 0, height = 0; @@ -301,7 +316,31 @@ void Capture::WorkerLoop() { if (sample) { if (CopyFrame(sample, width, height, stride, scratch)) { - // Eye Contact runs first, on the raw frame (needs real eyes/landmarks). + // Artifact Reduction runs FIRST so the AI effects downstream get a clean frame. + const bool arWant = g_state.artifactReductionEnabled.load(std::memory_order_acquire); + if (arWant && !artifactReduction.IsReady()) { + if (!artifactReduction.Start()) { + std::lock_guard e(g_state.arErrMtx); + const std::string& ne = artifactReduction.LastError(); + if (g_state.arError != ne) g_state.arError = ne; + } + } else if (!arWant && artifactReduction.IsReady()) { + artifactReduction.Stop(); + std::lock_guard e(g_state.arErrMtx); + if (!g_state.arError.empty()) g_state.arError.clear(); + } + bool arApplied = false; + if (arWant && artifactReduction.IsReady()) { + arApplied = artifactReduction.ProcessFrame(scratch.data(), width, height); + std::lock_guard e(g_state.arErrMtx); + if (!arApplied) { + const std::string& ne = artifactReduction.LastError(); + if (g_state.arError != ne) g_state.arError = ne; + } else if (!g_state.arError.empty()) { g_state.arError.clear(); } + } + g_state.artifactReductionActive.store(arApplied, std::memory_order_release); + + // Eye Contact runs second, on the (AR-cleaned) raw frame (needs real eyes/landmarks). const bool ecWant = g_state.eyeContactEnabled.load(std::memory_order_acquire); if (ecWant && !eyeContact.IsReady()) { if (!eyeContact.Start()) { @@ -360,12 +399,43 @@ void Capture::WorkerLoop() { } g_state.greenScreenActive.store(applied, std::memory_order_release); - std::lock_guard lock(g_state.mtx); - g_state.frame.swap(scratch); - g_state.width = width; - g_state.height = height; - g_state.hasNewFrame = true; - g_state.framesProduced.fetch_add(1, std::memory_order_release); + // Super Resolution runs LAST; it changes the frame dimensions. + const bool srWant = g_state.superResEnabled.load(std::memory_order_acquire); + const int srScale = g_state.superResScale.load(std::memory_order_acquire); + if (srWant && !superRes.IsReady()) { + if (!superRes.Start(srScale)) { + std::lock_guard e(g_state.srErrMtx); + const std::string& ne = superRes.LastError(); + if (g_state.srError != ne) g_state.srError = ne; + } + } else if (!srWant && superRes.IsReady()) { + superRes.Stop(); + std::lock_guard e(g_state.srErrMtx); + if (!g_state.srError.empty()) g_state.srError.clear(); + } + + int pubW = width, pubH = height; + std::vector srOut; + bool srApplied = false; + if (srWant && superRes.IsReady()) { + srApplied = superRes.ProcessFrame(scratch.data(), width, height, srOut, pubW, pubH); + std::lock_guard e(g_state.srErrMtx); + if (!srApplied) { + const std::string& ne = superRes.LastError(); + if (g_state.srError != ne) g_state.srError = ne; + } else if (!g_state.srError.empty()) { g_state.srError.clear(); } + } + g_state.superResActive.store(srApplied, std::memory_order_release); + + { + std::lock_guard lock(g_state.mtx); + if (srApplied) g_state.frame.swap(srOut); + else g_state.frame.swap(scratch); + g_state.width = pubW; + g_state.height = pubH; + g_state.hasNewFrame = true; + g_state.framesProduced.fetch_add(1, std::memory_order_release); + } } SafeRelease(sample); } else { @@ -375,7 +445,9 @@ void Capture::WorkerLoop() { } } - // Tear down Eye Contact and AIGS before releasing the reader (thread-affinity requirement). + // Tear down all effects before releasing the reader (thread-affinity requirement). + superRes.Stop(); + artifactReduction.Stop(); eyeContact.Stop(); aigs.Stop(); SafeRelease(reader); @@ -422,6 +494,10 @@ void Capture::Stop() { std::lock_guard e(g_state.ecErrMtx); g_state.ecError.clear(); } + g_state.artifactReductionActive.store(false, std::memory_order_release); + { std::lock_guard e(g_state.arErrMtx); g_state.arError.clear(); } + g_state.superResActive.store(false, std::memory_order_release); + { std::lock_guard e(g_state.srErrMtx); g_state.srError.clear(); } } bool Capture::LatestFrame(std::vector& out, int& w, int& h) { @@ -465,6 +541,26 @@ std::string Capture::EyeContactError() const { return g_state.ecError; } +void Capture::SetArtifactReduction(bool enabled) { + g_state.artifactReductionEnabled.store(enabled, std::memory_order_release); +} +bool Capture::ArtifactReductionActive() const { + return g_state.artifactReductionActive.load(std::memory_order_acquire); +} +std::string Capture::ArtifactReductionError() const { + std::lock_guard e(g_state.arErrMtx); return g_state.arError; +} +void Capture::SetSuperRes(bool enabled, int scaleX10) { + g_state.superResScale.store(scaleX10 == 15 ? 15 : 20, std::memory_order_release); + g_state.superResEnabled.store(enabled, std::memory_order_release); +} +bool Capture::SuperResActive() const { + return g_state.superResActive.load(std::memory_order_acquire); +} +std::string Capture::SuperResError() const { + std::lock_guard e(g_state.srErrMtx); return g_state.srError; +} + Capture::~Capture() { Stop(); } diff --git a/native/shim/capture.h b/native/shim/capture.h index 7d34de0..9ae9aa5 100644 --- a/native/shim/capture.h +++ b/native/shim/capture.h @@ -42,6 +42,16 @@ class Capture { bool EyeContactActive() const; // true only while gaze redirection is transforming frames std::string EyeContactError() const; // empty when none + // Toggles Artifact Reduction for subsequent frames. Thread-safe; worker owns the object. + void SetArtifactReduction(bool enabled); + bool ArtifactReductionActive() const; + std::string ArtifactReductionError() const; + + // Toggles Super Resolution + scale (15=1.5x, 20=2x). Thread-safe; worker owns the object. + void SetSuperRes(bool enabled, int scaleX10); + bool SuperResActive() const; + std::string SuperResError() const; + // Measured frames-per-second over a rolling window (status polling). Thread-safe to // call from the single status-poll thread; 0 until the first interval elapses. double MeasuredFps() const; From 8c0a99997740d65c48fdaceefa78a0de44593242 Mon Sep 17 00:00:00 2001 From: opariffazman Date: Wed, 24 Jun 2026 01:26:44 +0800 Subject: [PATCH 14/26] feat(core): AR/SR params, capability gates, persistence, live push Co-Authored-By: Claude Opus 4.8 (1M context) --- src/CameraOnScreen.App/Native/PInvokeShim.cs | 17 ++++++--- src/CameraOnScreen.Core/Config/Models.cs | 3 ++ src/CameraOnScreen.Core/Native/Contracts.cs | 8 +++- src/CameraOnScreen.Core/Native/FakeShim.cs | 5 ++- .../Orchestration/Orchestrator.cs | 10 +++++ .../ViewModels/MainViewModel.cs | 29 +++++++++++++- .../Orchestration/OrchestratorTests.cs | 12 ++++++ .../ViewModels/MainViewModelTests.cs | 38 +++++++++++++++++++ 8 files changed, 112 insertions(+), 10 deletions(-) diff --git a/src/CameraOnScreen.App/Native/PInvokeShim.cs b/src/CameraOnScreen.App/Native/PInvokeShim.cs index c75f06b..686e099 100644 --- a/src/CameraOnScreen.App/Native/PInvokeShim.cs +++ b/src/CameraOnScreen.App/Native/PInvokeShim.cs @@ -24,17 +24,20 @@ private struct CosParams public double green_screen_feather; public int eye_contact_enabled; public double eye_contact_sensitivity; public double eye_contact_look_away_range; + public int artifact_reduction_enabled; + public int super_res_enabled; + public int super_res_scale; } [StructLayout(LayoutKind.Sequential)] private struct CosCaps { public int GreenScreenAvailable; - [MarshalAs(UnmanagedType.ByValArray, SizeConst = 256)] - public byte[] Detail; + [MarshalAs(UnmanagedType.ByValArray, SizeConst = 256)] public byte[] Detail; public int EyeContactAvailable; - [MarshalAs(UnmanagedType.ByValArray, SizeConst = 256)] - public byte[] EcDetail; + [MarshalAs(UnmanagedType.ByValArray, SizeConst = 256)] public byte[] EcDetail; + public int ArtifactReductionAvailable; + public int SuperResAvailable; } [DllImport(Dll, CallingConvention = CallingConvention.Cdecl)] private static extern int cos_init(IntPtr device); @@ -79,6 +82,9 @@ public void SetParams(ShimParams p) eye_contact_enabled = p.EyeContactEnabled ? 1 : 0, eye_contact_sensitivity = p.EyeContactSensitivity, eye_contact_look_away_range = p.EyeContactLookAwayRange, + artifact_reduction_enabled = p.ArtifactReductionEnabled ? 1 : 0, + super_res_enabled = p.SuperResEnabled ? 1 : 0, + super_res_scale = p.SuperResScale, }; cos_set_params(ref native); } @@ -107,7 +113,8 @@ public ShimCapabilities QueryCapabilities() cos_query_capabilities(ref caps); return new ShimCapabilities( caps.GreenScreenAvailable != 0, ReadUtf8(caps.Detail, 0, 256), - caps.EyeContactAvailable != 0, ReadUtf8(caps.EcDetail, 0, 256)); + caps.EyeContactAvailable != 0, ReadUtf8(caps.EcDetail, 0, 256), + caps.ArtifactReductionAvailable != 0, caps.SuperResAvailable != 0); } public void Dispose() => cos_shutdown(); diff --git a/src/CameraOnScreen.Core/Config/Models.cs b/src/CameraOnScreen.Core/Config/Models.cs index 50c864c..04c9d70 100644 --- a/src/CameraOnScreen.Core/Config/Models.cs +++ b/src/CameraOnScreen.Core/Config/Models.cs @@ -29,6 +29,9 @@ public sealed record EffectSettings public bool EyeContactEnabled { get; init; } public double EyeContactSensitivity { get; init; } = 0.5; public double EyeContactLookAwayRange { get; init; } = 0.5; + public bool ArtifactReductionEnabled { get; init; } + public bool SuperResEnabled { get; init; } + public int SuperResScale { get; init; } // 0=off, 15=1.5x, 20=2x } public sealed record HotkeyBinding diff --git a/src/CameraOnScreen.Core/Native/Contracts.cs b/src/CameraOnScreen.Core/Native/Contracts.cs index ed05813..92276b5 100644 --- a/src/CameraOnScreen.Core/Native/Contracts.cs +++ b/src/CameraOnScreen.Core/Native/Contracts.cs @@ -6,7 +6,8 @@ namespace CameraOnScreen.Core.Native; /// so existing 2-arg call sites keep compiling. public sealed record ShimCapabilities( bool GreenScreenAvailable, string Detail, - bool EyeContactAvailable = false, string EyeContactDetail = ""); + bool EyeContactAvailable = false, string EyeContactDetail = "", + bool ArtifactReductionAvailable = false, bool SuperResAvailable = false); public enum GazeState { Unknown, OnCamera, Redirected, RealEyes } @@ -25,7 +26,10 @@ public sealed record ShimParams( double GreenScreenFeather, bool EyeContactEnabled, double EyeContactSensitivity, - double EyeContactLookAwayRange); + double EyeContactLookAwayRange, + bool ArtifactReductionEnabled = false, + bool SuperResEnabled = false, + int SuperResScale = 0); // 0=off, 15=1.5x, 20=2x public interface INativeShim : IDisposable { diff --git a/src/CameraOnScreen.Core/Native/FakeShim.cs b/src/CameraOnScreen.Core/Native/FakeShim.cs index 4eed06e..276f4b3 100644 --- a/src/CameraOnScreen.Core/Native/FakeShim.cs +++ b/src/CameraOnScreen.Core/Native/FakeShim.cs @@ -6,6 +6,8 @@ public sealed class FakeShim : INativeShim public ShimParams? LastParams { get; private set; } public bool GreenScreenAvailable { get; set; } public bool EyeContactAvailable { get; set; } + public bool ArtifactReductionAvailable { get; set; } + public bool SuperResAvailable { get; set; } private bool _running; public bool Init(IntPtr d3dDevice) => true; @@ -31,7 +33,8 @@ public ShimCapabilities QueryCapabilities() => new(GreenScreenAvailable, GreenScreenAvailable ? "fake: available" : "fake: unavailable", EyeContactAvailable, - EyeContactAvailable ? "fake: ec available" : "fake: ec unavailable"); + EyeContactAvailable ? "fake: ec available" : "fake: ec unavailable", + ArtifactReductionAvailable, SuperResAvailable); public bool Disposed { get; private set; } public void Dispose() => Disposed = true; diff --git a/src/CameraOnScreen.Core/Orchestration/Orchestrator.cs b/src/CameraOnScreen.Core/Orchestration/Orchestrator.cs index edef8a2..1a6f9f5 100644 --- a/src/CameraOnScreen.Core/Orchestration/Orchestrator.cs +++ b/src/CameraOnScreen.Core/Orchestration/Orchestrator.cs @@ -36,6 +36,12 @@ public Orchestrator(INativeShim shim, GpuTier tier) /// Human-readable reason from the eye-contact probe. "Checking…" until probed. public string EyeContactDetail { get; private set; } = "Checking effect availability…"; + /// True when the shim reports Artifact Reduction can run. False until . + public bool ArtifactReductionAvailable { get; private set; } + + /// True when the shim reports Super Resolution can run. False until . + public bool SuperResAvailable { get; private set; } + /// Runs the (blocking) native capability probe and records the result. Run this OFF the /// UI thread — the real probe does a ~1s TensorRT model load. The tier (RTX heuristic) is kept /// only for display; this probe is the authoritative effect gate. @@ -46,6 +52,8 @@ public void ProbeCapabilities() CapabilityDetail = caps.Detail; EyeContactAvailable = caps.EyeContactAvailable; EyeContactDetail = caps.EyeContactDetail; + ArtifactReductionAvailable = caps.ArtifactReductionAvailable; + SuperResAvailable = caps.SuperResAvailable; } /// The detected GPU tier — used for display only, not for effect gating. @@ -69,6 +77,8 @@ public void ApplyParams(ShimParams requested) { GreenScreenEnabled = requested.GreenScreenEnabled && EffectsAvailable, EyeContactEnabled = requested.EyeContactEnabled && EyeContactAvailable, + ArtifactReductionEnabled = requested.ArtifactReductionEnabled && ArtifactReductionAvailable, + SuperResEnabled = requested.SuperResEnabled && SuperResAvailable, }; _shim.SetParams(effective); } diff --git a/src/CameraOnScreen.Core/ViewModels/MainViewModel.cs b/src/CameraOnScreen.Core/ViewModels/MainViewModel.cs index 08ddc8f..54ef09d 100644 --- a/src/CameraOnScreen.Core/ViewModels/MainViewModel.cs +++ b/src/CameraOnScreen.Core/ViewModels/MainViewModel.cs @@ -31,6 +31,8 @@ public MainViewModel(Orchestrator orchestrator, INativeShim shim) CapabilityDetail = orchestrator.CapabilityDetail; EyeContactAvailable = orchestrator.EyeContactAvailable; EyeContactDetail = orchestrator.EyeContactDetail; + ArtifactReductionAvailable = orchestrator.ArtifactReductionAvailable; + SuperResAvailable = orchestrator.SuperResAvailable; _statusHandler = (_, s) => OnStatus(s); _orchestrator.StatusChanged += _statusHandler; } @@ -49,6 +51,8 @@ public async Task ProbeCapabilitiesAsync() CapabilityDetail = _orchestrator.CapabilityDetail; EyeContactAvailable = _orchestrator.EyeContactAvailable; EyeContactDetail = _orchestrator.EyeContactDetail; + ArtifactReductionAvailable = _orchestrator.ArtifactReductionAvailable; + SuperResAvailable = _orchestrator.SuperResAvailable; } public ObservableCollection Cameras { get; } = new(); @@ -60,10 +64,15 @@ public async Task ProbeCapabilitiesAsync() [ObservableProperty] private bool eyeContactEnabled; [ObservableProperty] private double eyeContactSensitivity = 0.5; [ObservableProperty] private double eyeContactLookAwayRange = 0.5; + [ObservableProperty] private bool artifactReductionEnabled; + [ObservableProperty] private bool superResEnabled; + [ObservableProperty] private int superResScaleIndex; // 0=off, 1=1.5x, 2=2x [ObservableProperty] private bool effectsAvailable; [ObservableProperty] private string capabilityDetail = "Checking effect availability…"; [ObservableProperty] private bool eyeContactAvailable; [ObservableProperty] private string eyeContactDetail = "Checking effect availability…"; + [ObservableProperty] private bool artifactReductionAvailable; + [ObservableProperty] private bool superResAvailable; [ObservableProperty] private bool isRunning; [ObservableProperty] private bool locked; [ObservableProperty] private bool clickThrough; @@ -73,6 +82,10 @@ public async Task ProbeCapabilitiesAsync() [ObservableProperty] private string? statusError; [ObservableProperty] private GazeState gaze; + // SuperResScaleIndex 0/1/2 -> shim scale 0/15/20. + private static int ScaleFromIndex(int i) => i switch { 1 => 15, 2 => 20, _ => 0 }; + private static int IndexFromScale(int s) => s switch { 15 => 1, 20 => 2, _ => 0 }; + public void LoadFrom(AppConfig config) { GreenScreenEnabled = config.Effects.GreenScreenEnabled; @@ -81,6 +94,9 @@ public void LoadFrom(AppConfig config) EyeContactEnabled = config.Effects.EyeContactEnabled; EyeContactSensitivity = config.Effects.EyeContactSensitivity; EyeContactLookAwayRange = config.Effects.EyeContactLookAwayRange; + ArtifactReductionEnabled = config.Effects.ArtifactReductionEnabled; + SuperResEnabled = config.Effects.SuperResEnabled; + SuperResScaleIndex = IndexFromScale(config.Effects.SuperResScale); Locked = config.Overlay.Locked; ClickThrough = config.Overlay.ClickThrough; Mirror = config.Overlay.Mirror; @@ -107,7 +123,10 @@ public void LoadFrom(AppConfig config) GreenScreenEnabled = GreenScreenEnabled, GreenScreenExpand = GreenScreenExpand, GreenScreenFeather = GreenScreenFeather, EyeContactEnabled = EyeContactEnabled, EyeContactSensitivity = EyeContactSensitivity, - EyeContactLookAwayRange = EyeContactLookAwayRange + EyeContactLookAwayRange = EyeContactLookAwayRange, + ArtifactReductionEnabled = ArtifactReductionEnabled, + SuperResEnabled = SuperResEnabled, + SuperResScale = ScaleFromIndex(SuperResScaleIndex), }, Hotkeys = _hotkeys }; @@ -123,6 +142,9 @@ public void LoadFrom(AppConfig config) partial void OnEyeContactEnabledChanged(bool value) => ApplyLiveParams(); partial void OnEyeContactSensitivityChanged(double value) => ApplyLiveParams(); partial void OnEyeContactLookAwayRangeChanged(double value) => ApplyLiveParams(); + partial void OnArtifactReductionEnabledChanged(bool value) => ApplyLiveParams(); + partial void OnSuperResEnabledChanged(bool value) => ApplyLiveParams(); + partial void OnSuperResScaleIndexChanged(int value) => ApplyLiveParams(); private void ApplyLiveParams() { @@ -136,7 +158,10 @@ private void ApplyLiveParams() GreenScreenFeather: GreenScreenFeather, EyeContactEnabled: EyeContactEnabled, EyeContactSensitivity: EyeContactSensitivity, - EyeContactLookAwayRange: EyeContactLookAwayRange); + EyeContactLookAwayRange: EyeContactLookAwayRange, + ArtifactReductionEnabled: ArtifactReductionEnabled, + SuperResEnabled: SuperResEnabled, + SuperResScale: ScaleFromIndex(SuperResScaleIndex)); public void OnStatus(ShimStatus s) { diff --git a/tests/CameraOnScreen.Core.Tests/Orchestration/OrchestratorTests.cs b/tests/CameraOnScreen.Core.Tests/Orchestration/OrchestratorTests.cs index 9bd4f7f..e0bcfb8 100644 --- a/tests/CameraOnScreen.Core.Tests/Orchestration/OrchestratorTests.cs +++ b/tests/CameraOnScreen.Core.Tests/Orchestration/OrchestratorTests.cs @@ -187,4 +187,16 @@ public void ProbeCapabilities_records_eye_contact_detail() orch.ProbeCapabilities(); Assert.Equal("fake: ec unavailable", orch.EyeContactDetail); } + + [Fact] + public void ApplyParams_forces_effects_off_when_unavailable() + { + var shim = new FakeShim { GreenScreenAvailable = false, ArtifactReductionAvailable = false, SuperResAvailable = false }; + var orch = new Orchestrator(shim, GpuTier.Rtx); + orch.ProbeCapabilities(); + orch.ApplyParams(new ShimParams("cam", true, 0, 0, false, 0.5, 0.5, + ArtifactReductionEnabled: true, SuperResEnabled: true, SuperResScale: 20)); + Assert.False(shim.LastParams!.ArtifactReductionEnabled); + Assert.False(shim.LastParams!.SuperResEnabled); + } } diff --git a/tests/CameraOnScreen.Core.Tests/ViewModels/MainViewModelTests.cs b/tests/CameraOnScreen.Core.Tests/ViewModels/MainViewModelTests.cs index efd28e7..ad27659 100644 --- a/tests/CameraOnScreen.Core.Tests/ViewModels/MainViewModelTests.cs +++ b/tests/CameraOnScreen.Core.Tests/ViewModels/MainViewModelTests.cs @@ -251,6 +251,44 @@ public void Toggling_eye_contact_while_running_pushes_params() Assert.False(shim.LastParams!.EyeContactEnabled); } + [Fact] + public void BuildParams_includes_artifact_reduction_and_superres() + { + var shim = new FakeShim { GreenScreenAvailable = true, EyeContactAvailable = true, + ArtifactReductionAvailable = true, SuperResAvailable = true }; + var orch = new Orchestrator(shim, GpuTier.Rtx); + orch.ProbeCapabilities(); + var vm = new MainViewModel(orch, shim); + + vm.ArtifactReductionEnabled = true; + vm.SuperResEnabled = true; + vm.SuperResScaleIndex = 2; // 2x + + var p = vm.BuildParams(); + Assert.True(p.ArtifactReductionEnabled); + Assert.True(p.SuperResEnabled); + Assert.Equal(20, p.SuperResScale); + } + + [Fact] + public void ToAppConfig_roundtrips_new_effects() + { + var shim = new FakeShim(); + var vm = new MainViewModel(new Orchestrator(shim, GpuTier.Rtx), shim) + { + ArtifactReductionEnabled = true, SuperResEnabled = true, SuperResScaleIndex = 1 + }; + var cfg = vm.ToAppConfig(0, 0, 320, 240); + Assert.True(cfg.Effects.ArtifactReductionEnabled); + Assert.True(cfg.Effects.SuperResEnabled); + Assert.Equal(15, cfg.Effects.SuperResScale); + + var vm2 = new MainViewModel(new Orchestrator(shim, GpuTier.Rtx), shim); + vm2.LoadFrom(cfg); + Assert.True(vm2.ArtifactReductionEnabled); + Assert.Equal(1, vm2.SuperResScaleIndex); + } + private sealed class ControllableFpsShim : INativeShim { public double FpsValue { get; set; } From c1dddf1ff906d2ade633310dd298ffdbeb98469a Mon Sep 17 00:00:00 2001 From: opariffazman Date: Wed, 24 Jun 2026 01:30:38 +0800 Subject: [PATCH 15/26] feat(app): Artifact Reduction + Super Resolution UI, 4K frame buffer Co-Authored-By: Claude Opus 4.8 (1M context) --- src/CameraOnScreen.App/MainWindow.xaml | 13 +++++++++++++ src/CameraOnScreen.App/MainWindow.xaml.cs | 5 +++-- 2 files changed, 16 insertions(+), 2 deletions(-) diff --git a/src/CameraOnScreen.App/MainWindow.xaml b/src/CameraOnScreen.App/MainWindow.xaml index 73fcbaf..6fda199 100644 --- a/src/CameraOnScreen.App/MainWindow.xaml +++ b/src/CameraOnScreen.App/MainWindow.xaml @@ -30,6 +30,19 @@ + + + + + + + diff --git a/src/CameraOnScreen.App/MainWindow.xaml.cs b/src/CameraOnScreen.App/MainWindow.xaml.cs index 446ced7..76191f3 100644 --- a/src/CameraOnScreen.App/MainWindow.xaml.cs +++ b/src/CameraOnScreen.App/MainWindow.xaml.cs @@ -21,8 +21,9 @@ public sealed partial class MainWindow : Window, INotifyPropertyChanged private readonly Overlay.OverlayWindow _overlay; private readonly Hotkeys.GlobalHotkeyService _hotkeys; private readonly JsonSettingsStore _store = new(JsonSettingsStore.DefaultPath()); - // Big enough for 1920x1080 BGRA; the test camera is 640x480. TryGetFrame writes the actual size. - private readonly byte[] _frameBuf = new byte[1920 * 1080 * 4]; + // Pre-sized to 4K (3840x2160) BGRA so Super Resolution (up to 2x of 1080p) fits without a + // resize. TryGetFrame writes the actual size; cos_get_frame rejects frames larger than this. + private readonly byte[] _frameBuf = new byte[3840 * 2160 * 4]; private Microsoft.UI.Dispatching.DispatcherQueueTimer? _timer; private Microsoft.UI.Dispatching.DispatcherQueueTimer? _saveTimer; // debounces persist after wheel-resize private Overlay.OverlayMouseHook? _mouseHook; From 51e4e9a7552bd6b8e82e308b9c32366588f74dc1 Mon Sep 17 00:00:00 2001 From: opariffazman Date: Wed, 24 Jun 2026 01:39:27 +0800 Subject: [PATCH 16/26] fix: Super-res scale combo is factor-only (1.5x/2x); toggle owns on/off Removes the redundant "Off" combo item that, combined with the SuperRes toggle, let scale=0 coerce to 2x. Index now 0=1.5x,1=2x; tests updated. Co-Authored-By: Claude Opus 4.8 (1M context) --- src/CameraOnScreen.App/MainWindow.xaml | 1 - src/CameraOnScreen.Core/ViewModels/MainViewModel.cs | 8 ++++---- .../ViewModels/MainViewModelTests.cs | 7 ++++--- 3 files changed, 8 insertions(+), 8 deletions(-) diff --git a/src/CameraOnScreen.App/MainWindow.xaml b/src/CameraOnScreen.App/MainWindow.xaml index 6fda199..740f284 100644 --- a/src/CameraOnScreen.App/MainWindow.xaml +++ b/src/CameraOnScreen.App/MainWindow.xaml @@ -39,7 +39,6 @@ - diff --git a/src/CameraOnScreen.Core/ViewModels/MainViewModel.cs b/src/CameraOnScreen.Core/ViewModels/MainViewModel.cs index 54ef09d..4b36800 100644 --- a/src/CameraOnScreen.Core/ViewModels/MainViewModel.cs +++ b/src/CameraOnScreen.Core/ViewModels/MainViewModel.cs @@ -66,7 +66,7 @@ public async Task ProbeCapabilitiesAsync() [ObservableProperty] private double eyeContactLookAwayRange = 0.5; [ObservableProperty] private bool artifactReductionEnabled; [ObservableProperty] private bool superResEnabled; - [ObservableProperty] private int superResScaleIndex; // 0=off, 1=1.5x, 2=2x + [ObservableProperty] private int superResScaleIndex; // 0=1.5x, 1=2x [ObservableProperty] private bool effectsAvailable; [ObservableProperty] private string capabilityDetail = "Checking effect availability…"; [ObservableProperty] private bool eyeContactAvailable; @@ -82,9 +82,9 @@ public async Task ProbeCapabilitiesAsync() [ObservableProperty] private string? statusError; [ObservableProperty] private GazeState gaze; - // SuperResScaleIndex 0/1/2 -> shim scale 0/15/20. - private static int ScaleFromIndex(int i) => i switch { 1 => 15, 2 => 20, _ => 0 }; - private static int IndexFromScale(int s) => s switch { 15 => 1, 20 => 2, _ => 0 }; + // SuperResScaleIndex 0/1 -> shim scale 15/20 (1.5x / 2x). On/off is the SuperResEnabled toggle. + private static int ScaleFromIndex(int i) => i switch { 1 => 20, _ => 15 }; + private static int IndexFromScale(int s) => s switch { 20 => 1, _ => 0 }; public void LoadFrom(AppConfig config) { diff --git a/tests/CameraOnScreen.Core.Tests/ViewModels/MainViewModelTests.cs b/tests/CameraOnScreen.Core.Tests/ViewModels/MainViewModelTests.cs index ad27659..358f578 100644 --- a/tests/CameraOnScreen.Core.Tests/ViewModels/MainViewModelTests.cs +++ b/tests/CameraOnScreen.Core.Tests/ViewModels/MainViewModelTests.cs @@ -262,7 +262,7 @@ public void BuildParams_includes_artifact_reduction_and_superres() vm.ArtifactReductionEnabled = true; vm.SuperResEnabled = true; - vm.SuperResScaleIndex = 2; // 2x + vm.SuperResScaleIndex = 1; // 2x var p = vm.BuildParams(); Assert.True(p.ArtifactReductionEnabled); @@ -276,7 +276,7 @@ public void ToAppConfig_roundtrips_new_effects() var shim = new FakeShim(); var vm = new MainViewModel(new Orchestrator(shim, GpuTier.Rtx), shim) { - ArtifactReductionEnabled = true, SuperResEnabled = true, SuperResScaleIndex = 1 + ArtifactReductionEnabled = true, SuperResEnabled = true, SuperResScaleIndex = 0 }; var cfg = vm.ToAppConfig(0, 0, 320, 240); Assert.True(cfg.Effects.ArtifactReductionEnabled); @@ -286,7 +286,8 @@ public void ToAppConfig_roundtrips_new_effects() var vm2 = new MainViewModel(new Orchestrator(shim, GpuTier.Rtx), shim); vm2.LoadFrom(cfg); Assert.True(vm2.ArtifactReductionEnabled); - Assert.Equal(1, vm2.SuperResScaleIndex); + Assert.True(vm2.SuperResEnabled); + Assert.Equal(0, vm2.SuperResScaleIndex); } private sealed class ControllableFpsShim : INativeShim From 8807c7f5bdcb48141637946c6066c2204b6bc8e8 Mon Sep 17 00:00:00 2001 From: opariffazman Date: Wed, 24 Jun 2026 02:03:32 +0800 Subject: [PATCH 17/26] docs: record real VFX 1.2.0.0 NGC feature catalog + verify-before-coding rule No nvvfxartifactreduction in the catalog (dead effect); Super Resolution is NGX VSR (selector "VideoSuperRes", QualityLevel param, nvngx_vsr.dll, no TRT engine). Verify a feature exists + read its real per-feature header before coding an effect; get the NGC key up front when it gates verification. Co-Authored-By: Claude Opus 4.8 (1M context) --- CLAUDE.md | 1 + 1 file changed, 1 insertion(+) diff --git a/CLAUDE.md b/CLAUDE.md index 0404397..03716b8 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -81,6 +81,7 @@ Three projects: `src/CameraOnScreen.Core` (pure .NET 8 logic, no WinUI/Win32 typ ## Maxine SDKs (not in repo; redistribution governed by the 2025 NVIDIA Software License + Open/Community Model Licenses) - **VFX** green screen (`nvvfxgreenscreen`) and **AR** eye contact (gaze) are separate NVIDIA products. No import `.lib` — link via the SDKs' proxy stubs (`nvVideoEffectsProxy.cpp`, `nvCVImageProxy.cpp`, `nvARProxy.cpp`) compiled into the shim. Models are prebuilt per-arch TensorRT engines; the bundle ships **sm75/86/89/100** (Turing/Ampere/Ada/Blackwell) — fetched from NGC by arch. **sm86 is the only arch verified on real silicon (RTX 3090); the others ship best-effort and grey out gracefully if an engine fails to deserialize.** +- **VFX feature catalog — verify availability + the REAL API BEFORE coding an effect (cost ~6 wasted tasks once).** Authoritative installable set: `\features\install_feature.ps1 -list_features` (NGC key via `NGC_CLI_API_KEY`). VFX 1.2.0.0 = `aigsrelighting, backgroundblur, denoising, greenscreen, relighting, transfer, upscale, videosuperres` (8). **There is NO `nvvfxartifactreduction`** — it's in the script's help text but NOT the catalog (removed since older VFX SDKs); coding against it = a dead, always-unavailable effect. **Super Resolution = NGX VSR** (`nvvfxvideosuperres`; ships `nvngx_vsr.dll`, NOT a per-arch TRT engine — "no models" is normal): real header `features\nvvfxvideosuperres\include\nvVFXVideoSuperRes.h` → selector `NVVFX_FX_VIDEO_SUPER_RES "VideoSuperRes"` + param `NVVFX_QUALITY_LEVEL` (NOT `"SuperRes"`/`NVVFX_MODE`/scale-from-dims). A feature's real selector + params + image format live ONLY in its per-feature header (`features\\include\*.h`), installed on demand — never infer them; if the NGC key is the gate, get it up front. - **App-relative discovery** (`paths.{h,cpp}` `ShimModuleDir()` via `GetModuleHandleExW(FROM_ADDRESS)`, CWD-independent): both resolvers gain an `\maxine\` tier so a shipped app finds the runtimes beside the exe with no env vars. Single shared co-versioned `maxine\` root (one TRT/CUDA runtime, dispatcher + feature DLLs, `models\vfx` + `models\ar`). - **Stage + Bundler.** Two steps now (dispatcher + per-feature-DLL SDK layout): (1) `scripts/assemble-maxine-stage.ps1` curates a co-versioned flat **stage** from the VFX 1.2.0.0 + AR 1.1.1.0 SDK trees (shared DLLs from VFX; AR dispatcher + 3 gaze feature DLLs from AR; the 6 license files; multi-arch engines via `scripts/fetch-maxine-engines.ps1`, NGC key). (2) `scripts/bundle-maxine.ps1 -OutDir X -MaxineStage ` PRUNES the stage to the manifest's verified load-closure (`native/shim/bundle/maxine-manifest.psd1`; the 19-DLL `Dlls` list was produced by `native/shim/smoke/trace_closure.cpp`) + model globs + required `LicenseFiles`, into `\maxine\`. Co-version is enforced at stage assembly, not by the bundler. `trace_closure`/`bundle_probe` re-run against the produced bundle (`COS_*` unset → both effects load) is the verify gate. End-user need: an **RTX GPU + recent driver**; no NVIDIA account or SDK download. - **Installer** (`scripts/bundle-maxine.ps1` consumer): `scripts/build-installer.ps1 -MaxineStage ` From 76b28f8244652d1a84f6a339a67f164d124df4ff Mon Sep 17 00:00:00 2001 From: opariffazman Date: Wed, 24 Jun 2026 07:51:50 +0800 Subject: [PATCH 18/26] feat(capture): select highest-fps native camera mode (+ capture_probe diag) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit CreateReader enumerated nothing — MF picked the camera's default native type, which can be a low-fps mode. Now enumerate native types, pick highest fps then highest res (<=1080p), and pin the RGB32 request to that size+rate so MF takes the MJPG-30 path; fall back to unconstrained if rejected. capture_probe.cpp is a standalone diagnostic (lists modes, measures RGB32 vs raw-MJPG delivery fps). Co-Authored-By: Claude Opus 4.8 (1M context) --- native/shim/capture.cpp | 43 +++++++- native/shim/smoke/capture_probe.cpp | 157 ++++++++++++++++++++++++++++ 2 files changed, 199 insertions(+), 1 deletion(-) create mode 100644 native/shim/smoke/capture_probe.cpp diff --git a/native/shim/capture.cpp b/native/shim/capture.cpp index e7dd2d8..cd7d76c 100644 --- a/native/shim/capture.cpp +++ b/native/shim/capture.cpp @@ -233,13 +233,54 @@ bool CreateReader(const std::string& symbolicLink, IMFSourceReader*& outReader, SafeRelease(source); if (FAILED(hr) || !reader) return false; - // Request RGB32 output on the first video stream. + // Pick the camera's best native type up front: HIGHEST frame rate, then highest + // resolution, capped at 1080p (the overlay downscales; _frameBuf is sized for 4K). + // Without this MF picks the camera's *default* native type, which on common webcams + // (e.g. Logitech Brio 100) is a ~15fps mode rather than the MJPG 1080p30 path — so + // the live overlay is choppy at half the camera's real capability. fps dominates the + // score; resolution is only the tiebreaker. + UINT32 bestW = 0, bestH = 0, bestNum = 0, bestDen = 1; + double bestScore = -1.0; + for (DWORD i = 0; ; ++i) { + IMFMediaType* nt = nullptr; + if (FAILED(reader->GetNativeMediaType( + static_cast(MF_SOURCE_READER_FIRST_VIDEO_STREAM), i, &nt))) break; + UINT32 w = 0, h = 0, num = 0, den = 0; + if (SUCCEEDED(MFGetAttributeSize(nt, MF_MT_FRAME_SIZE, &w, &h)) && + SUCCEEDED(MFGetAttributeRatio(nt, MF_MT_FRAME_RATE, &num, &den)) && + w > 0 && h > 0 && den > 0 && w <= 1920 && h <= 1080) { + const double fps = static_cast(num) / den; + const double score = fps * 1e8 + static_cast(w) * h; // fps wins, res breaks ties + if (score > bestScore) { + bestScore = score; bestW = w; bestH = h; bestNum = num; bestDen = den; + } + } + SafeRelease(nt); + } + + // Request RGB32 output on the first video stream. When a best native type was found, + // pin the output to its frame size + rate so MF selects that (high-fps) native path + // and converts it; if MF rejects the constrained request, retry unconstrained + // (previous behavior — MF picks the default). IMFMediaType* outType = nullptr; if (FAILED(MFCreateMediaType(&outType))) { SafeRelease(reader); return false; } outType->SetGUID(MF_MT_MAJOR_TYPE, MFMediaType_Video); outType->SetGUID(MF_MT_SUBTYPE, MFVideoFormat_RGB32); + if (bestScore >= 0.0) { + MFSetAttributeSize(outType, MF_MT_FRAME_SIZE, bestW, bestH); + MFSetAttributeRatio(outType, MF_MT_FRAME_RATE, bestNum, bestDen); + } hr = reader->SetCurrentMediaType( static_cast(MF_SOURCE_READER_FIRST_VIDEO_STREAM), nullptr, outType); + if (FAILED(hr) && bestScore >= 0.0) { + // Constrained request rejected — fall back to letting MF choose the native type. + SafeRelease(outType); + if (FAILED(MFCreateMediaType(&outType))) { SafeRelease(reader); return false; } + outType->SetGUID(MF_MT_MAJOR_TYPE, MFMediaType_Video); + outType->SetGUID(MF_MT_SUBTYPE, MFVideoFormat_RGB32); + hr = reader->SetCurrentMediaType( + static_cast(MF_SOURCE_READER_FIRST_VIDEO_STREAM), nullptr, outType); + } SafeRelease(outType); if (FAILED(hr)) { SafeRelease(reader); return false; } diff --git a/native/shim/smoke/capture_probe.cpp b/native/shim/smoke/capture_probe.cpp new file mode 100644 index 0000000..75db6dc --- /dev/null +++ b/native/shim/smoke/capture_probe.cpp @@ -0,0 +1,157 @@ +// Standalone capture diagnostic — runs WITHOUT the GUI. +// Opens the first video device, lists native types, then measures actual delivery fps for: +// (A) RGB32 output with MF video processing (what the app uses), pinned to the best native rate. +// (B) the native MJPG type directly (no conversion) — the raw camera ceiling. +// Distinguishes "camera/MF negotiates low" from "RGB32 conversion is the CPU cap". +// +// Build (from a VS x64 Developer prompt): +// cl /EHsc /DUNICODE /D_UNICODE capture_probe.cpp ^ +// mfplat.lib mf.lib mfreadwrite.lib mfuuid.lib ole32.lib /Fe:capture_probe.exe +#include +#include +#include +#include +#include +#include +#include +#include + +#pragma comment(lib, "mfplat.lib") +#pragma comment(lib, "mf.lib") +#pragma comment(lib, "mfreadwrite.lib") +#pragma comment(lib, "mfuuid.lib") +#pragma comment(lib, "ole32.lib") + +template void Rel(T*& p) { if (p) { p->Release(); p = nullptr; } } + +static std::string FourCC(const GUID& g) { + // MF video subtypes embed a FOURCC in Data1. + char c[5] = {0}; + memcpy(c, &g.Data1, 4); + for (int i = 0; i < 4; ++i) if (c[i] < 32 || c[i] > 126) c[i] = '?'; + return std::string(c, 4); +} + +static IMFMediaSource* OpenFirstDevice() { + IMFAttributes* a = nullptr; + MFCreateAttributes(&a, 1); + a->SetGUID(MF_DEVSOURCE_ATTRIBUTE_SOURCE_TYPE, MF_DEVSOURCE_ATTRIBUTE_SOURCE_TYPE_VIDCAP_GUID); + IMFActivate** devs = nullptr; UINT32 n = 0; + MFEnumDeviceSources(a, &devs, &n); + Rel(a); + IMFMediaSource* src = nullptr; + if (n > 0) { + WCHAR name[256]; UINT32 len = 0; + devs[0]->GetString(MF_DEVSOURCE_ATTRIBUTE_FRIENDLY_NAME, name, 256, &len); + wprintf(L"Device: %s\n", name); + devs[0]->ActivateObject(IID_PPV_ARGS(&src)); + } + for (UINT32 i = 0; i < n; ++i) Rel(devs[i]); + if (devs) CoTaskMemFree(devs); + return src; +} + +static double MeasureFps(IMFSourceReader* r, int frames) { + using clk = std::chrono::steady_clock; + // warm up + for (int i = 0; i < 5; ++i) { + DWORD si = 0, fl = 0; LONGLONG ts = 0; IMFSample* s = nullptr; + if (FAILED(r->ReadSample((DWORD)MF_SOURCE_READER_FIRST_VIDEO_STREAM, 0, &si, &fl, &ts, &s))) return -1; + Rel(s); + } + auto t0 = clk::now(); + int got = 0; + for (int i = 0; i < frames; ++i) { + DWORD si = 0, fl = 0; LONGLONG ts = 0; IMFSample* s = nullptr; + if (FAILED(r->ReadSample((DWORD)MF_SOURCE_READER_FIRST_VIDEO_STREAM, 0, &si, &fl, &ts, &s))) break; + if (s) got++; + Rel(s); + if (fl & MF_SOURCE_READERF_ENDOFSTREAM) break; + } + double sec = std::chrono::duration(clk::now() - t0).count(); + return sec > 0 ? got / sec : -1; +} + +static IMFSourceReader* MakeReader(IMFMediaSource* src, bool videoProcessing) { + IMFAttributes* ra = nullptr; MFCreateAttributes(&ra, 1); + if (videoProcessing) ra->SetUINT32(MF_SOURCE_READER_ENABLE_VIDEO_PROCESSING, TRUE); + IMFSourceReader* r = nullptr; + MFCreateSourceReaderFromMediaSource(src, ra, &r); + Rel(ra); + return r; +} + +int main() { + if (FAILED(MFStartup(MF_VERSION))) { printf("MFStartup failed\n"); return 1; } + + // ---- list native types + pick best (fps, then res, cap 1080) ---- + IMFMediaSource* src = OpenFirstDevice(); + if (!src) { printf("no device\n"); MFShutdown(); return 1; } + IMFSourceReader* lister = MakeReader(src, false); + printf("\n-- native types --\n"); + UINT32 bestW=0,bestH=0,bestNum=0,bestDen=1; double bestScore=-1; + for (DWORD i = 0; ; ++i) { + IMFMediaType* t = nullptr; + if (FAILED(lister->GetNativeMediaType((DWORD)MF_SOURCE_READER_FIRST_VIDEO_STREAM, i, &t))) break; + UINT32 w=0,h=0,num=0,den=0; GUID sub = {}; + MFGetAttributeSize(t, MF_MT_FRAME_SIZE, &w, &h); + MFGetAttributeRatio(t, MF_MT_FRAME_RATE, &num, &den); + t->GetGUID(MF_MT_SUBTYPE, &sub); + double fps = den ? (double)num/den : 0; + if (w && h && den && fps >= 29.0 && w<=1920 && h<=1080) + printf(" [%2lu] %4ux%-4u %5.1ffps %s\n", i, w, h, fps, FourCC(sub).c_str()); + if (w && h && den && w<=1920 && h<=1080) { + double score = fps*1e8 + (double)w*h; + if (score > bestScore) { bestScore=score; bestW=w; bestH=h; bestNum=num; bestDen=den; } + } + Rel(t); + } + printf(" (only >=29fps shown; many lower-fps modes omitted)\n"); + printf("BEST pick: %ux%u @ %.1ffps\n", bestW, bestH, bestDen?(double)bestNum/bestDen:0); + Rel(lister); Rel(src); + + // ---- TEST A: RGB32 + video processing, pinned to best rate (the app's path) ---- + src = OpenFirstDevice(); + IMFSourceReader* rA = MakeReader(src, true); + IMFMediaType* o = nullptr; MFCreateMediaType(&o); + o->SetGUID(MF_MT_MAJOR_TYPE, MFMediaType_Video); + o->SetGUID(MF_MT_SUBTYPE, MFVideoFormat_RGB32); + MFSetAttributeSize(o, MF_MT_FRAME_SIZE, bestW, bestH); + MFSetAttributeRatio(o, MF_MT_FRAME_RATE, bestNum, bestDen); + HRESULT hrA = rA->SetCurrentMediaType((DWORD)MF_SOURCE_READER_FIRST_VIDEO_STREAM, nullptr, o); + bool pinned = SUCCEEDED(hrA); + if (!pinned) { // fallback unconstrained + Rel(o); MFCreateMediaType(&o); + o->SetGUID(MF_MT_MAJOR_TYPE, MFMediaType_Video); + o->SetGUID(MF_MT_SUBTYPE, MFVideoFormat_RGB32); + hrA = rA->SetCurrentMediaType((DWORD)MF_SOURCE_READER_FIRST_VIDEO_STREAM, nullptr, o); + } + Rel(o); + IMFMediaType* cur = nullptr; UINT32 nw=0,nh=0,nnum=0,nden=1; + rA->GetCurrentMediaType((DWORD)MF_SOURCE_READER_FIRST_VIDEO_STREAM, &cur); + if (cur) { MFGetAttributeSize(cur,MF_MT_FRAME_SIZE,&nw,&nh); MFGetAttributeRatio(cur,MF_MT_FRAME_RATE,&nnum,&nden); } + Rel(cur); + printf("\n-- TEST A: RGB32 + video processing (app path) --\n"); + printf(" constrained-pin %s; negotiated %ux%u @ %.1ffps\n", + pinned?"OK":"REJECTED(fell back)", nw, nh, nden?(double)nnum/nden:0); + printf(" MEASURED delivery: %.1f fps\n", MeasureFps(rA, 90)); + Rel(rA); Rel(src); + + // ---- TEST B: native MJPG direct (no RGB32 conversion) — raw camera ceiling ---- + src = OpenFirstDevice(); + IMFSourceReader* rB = MakeReader(src, false); + IMFMediaType* mj = nullptr; MFCreateMediaType(&mj); + mj->SetGUID(MF_MT_MAJOR_TYPE, MFMediaType_Video); + mj->SetGUID(MF_MT_SUBTYPE, MFVideoFormat_MJPG); + MFSetAttributeSize(mj, MF_MT_FRAME_SIZE, bestW, bestH); + MFSetAttributeRatio(mj, MF_MT_FRAME_RATE, bestNum, bestDen); + HRESULT hrB = rB->SetCurrentMediaType((DWORD)MF_SOURCE_READER_FIRST_VIDEO_STREAM, nullptr, mj); + Rel(mj); + printf("\n-- TEST B: native MJPG direct (raw camera, no conversion) --\n"); + if (SUCCEEDED(hrB)) printf(" MEASURED delivery: %.1f fps\n", MeasureFps(rB, 90)); + else printf(" MJPG set failed hr=0x%08lX (camera may not expose MJPG at %ux%u)\n", hrB, bestW, bestH); + Rel(rB); Rel(src); + + MFShutdown(); + return 0; +} From d3c4911956e3c45c4e0238ea3cd89a1a17ce5a75 Mon Sep 17 00:00:00 2001 From: opariffazman Date: Wed, 24 Jun 2026 08:59:45 +0800 Subject: [PATCH 19/26] docs(#15): VSR-only resume plan + gitignore .env Resume plan for issue #15 amending the parked Phase-1 plan: VSR-only + mode combo decision, grounded NGC/probe findings, task list (T1 probe done). Ignore .env (holds NGC_CLI_API_KEY). Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01SfcBU4WRAy7HMNgtm92ogq --- .gitignore | 1 + .../plans/2026-06-24-issue-15-vsr-resume.md | 103 ++++++++++++++++++ 2 files changed, 104 insertions(+) create mode 100644 docs/superpowers/plans/2026-06-24-issue-15-vsr-resume.md diff --git a/.gitignore b/.gitignore index fcf8144..effcd11 100644 --- a/.gitignore +++ b/.gitignore @@ -1,6 +1,7 @@ [Bb]in/ [Oo]bj/ dist/ +.env *.user .vs/ *.suo diff --git a/docs/superpowers/plans/2026-06-24-issue-15-vsr-resume.md b/docs/superpowers/plans/2026-06-24-issue-15-vsr-resume.md new file mode 100644 index 0000000..7803b4e --- /dev/null +++ b/docs/superpowers/plans/2026-06-24-issue-15-vsr-resume.md @@ -0,0 +1,103 @@ +# Issue #15 Resume — VSR-only AI sharpness (amends the parked Phase-1 plan) + +> **For agentic workers:** Use superpowers:subagent-driven-development or superpowers:executing-plans. +> **When implementing any App/XAML task (T7), invoke the WinUI skills first:** `winui-design` +> (XAML correctness, Fluent layout, theming, accessibility review) and `winui-dev-workflow` +> (build/run). `winui-app` for broader WinUI guidance if needed. + +**Issue:** #15 — Parked AI sharpness effects. +**Supersedes the effect tasks of:** `docs/superpowers/plans/2026-06-24-ai-sharpness-resolution-phase1.md` +(parked branch `feat/ai-sharpness-resolution`). **Spec:** `docs/superpowers/specs/2026-06-23-camera-on-screen-ai-sharpness-resolution-design.md` — needs a spec update to drop Artifact Reduction and re-scope Super Resolution to VSR-only (see T0). + +## Decision (user, 2026-06-24) +**VSR-only + mode combo.** One effect (Video Super Resolution), expose its `QualityLevel` as a +mode selector covering **Upscale** *and* **Denoise/Deblur** — so it also covers the image-cleanup +goal the (dead) Artifact Reduction was meant to serve. **No second effect.** (`nvvfxdenoising` is +real + 4-arch on NGC and is the only viable stack-on-top add, but only needed if denoise must run +*in the same frame as* upscale — not wanted now.) + +## Grounded facts (verified 2026-06-24, see memory `vfx-vsr-ngc-grounded`) +- **Real VFX 1.2.0.0 catalog** (live NGC, collection `maxine_vfx_sdk`, 8 models): aigsrelighting, + backgroundblur, denoising, greenscreen, relighting, transfer, upscale, **videosuperres**. +- **Artifact Reduction is DEAD** — absent on disk, only a help-text typo in `install_feature.ps1`, + and live NGC returns `PAYMENT_REQUIRED`/"key lacks permission" for `nvvfxartifactreduction` + while all 8 real features return data. +- **Real VSR API** (`features/nvvfxvideosuperres/include/nvVFXVideoSuperRes.h`): selector + `NVVFX_FX_VIDEO_SUPER_RES "VideoSuperRes"`, param `NVVFX_QUALITY_LEVEL "QualityLevel"`. Image + in+out = **BGRA/RGBA u8 GPU** (matches our pipeline — no BGR convert). QualityLevel values: + 0 Bicubic, 1-4 VSR Low/Med/High/Ultra (upscale), 8-11 Denoise, 12-15 Deblur, 16-19 HighBitrate. + **Denoise/Deblur do NOT upscale (out must == in).** +- **VSR ships NO per-arch engine** — model baked into `nvngx_vsr.dll` (NGX, 45MB) + dispatcher + `nvVFXVideoSuperRes.dll`. **Runs on every RTX arch with zero per-arch deserialize risk** — + sidesteps the non-Ampere verify gate that greys out green screen / gaze. +- **T1 PROBE PASSED on the 3090** — `CreateEffect("VideoSuperRes")` → `SetU32(QualityLevel)` → + `Load` → `Run` all `0 No error` for quality=1 (720→1440 2x upscale) and quality=8 (denoise, + out==in). Confirms: QualityLevel takes the values directly (single param, no separate Mode); + scale inferred from out/in dims; `nvngx_vsr.dll` resolves from a flat dir via + `g_nvVFXSDKPath`→SetDllDirectory. Probe sources in session scratchpad + (`vsr_probe.cpp` + `build_run_vsr_probe.bat`). + +## Reuse from parked branch (do NOT rebuild) +`vfx_paths.{h,cpp}` (green screen adopted it); C-ABI effect slots + capture-worker wiring +(lazy start/stop, leaf-lock errors, dynamic-resolution publish); managed Orchestrator gating, +MVVM scale combo, persistence, 4K frame buffer, App UI scaffold, 57 xUnit tests. + +## Branch +After PR #14 (`feat/fps-and-capture-mode` = fps counter + highest-fps capture) merges, branch +`feat/ai-sharpness-vsr` off `main`; rebase the parked branch's effect commits on (the fps/capture +patches dedupe to empty since #14 carries them). The parked branch is the substrate. + +## Tasks + +- [x] **T1 — Probe VSR (DONE, 2026-06-24).** Verified the real API on the 3090 (above). + +- [ ] **T0 — Spec update.** Edit the design spec: drop Artifact Reduction entirely; re-scope SR to + VSR-only with a mode selector. Keep the generic-plumbing sections. + +- [ ] **T2 — Rewrite `superres.{h,cpp}`** against the real API (simpler than parked): + - selector `"VideoSuperRes"`, include the real header. + - **BGRA u8 GPU in+out** — delete the BGR conversion and the per-pixel alpha-repack loop. + - `Start(int qualityLevel, int scaleX10)`; set `NVVFX_QUALITY_LEVEL` (not `NVVFX_MODE`). + - Upscale modes 1-4 → out = in×scale; Denoise 8-11 / Deblur 12-15 → **out = in** (enforce). + - NGX path via `vfx_paths` (dir holding `nvngx_vsr.dll`). + +- [ ] **T3 — Delete Artifact Reduction** everywhere: `artifactreduction.{h,cpp}` + vcxproj entry, + CosParams flag, CosCaps gate, capture-worker call, MVVM toggle/gate/persistence, AR xUnit tests, + AR UI, AR bundle/manifest entry. + +- [ ] **T4 — C ABI.** SR enable + **mode (QualityLevel)** + scale. Update `CosParams`/`CosCaps` + (byte-parity x64) + `[StructLayout]` mirrors in `PInvokeShim` + `FakeShim` + `Contracts`. + Verify `dumpbin /exports` still lists exactly the **9** `cos_*` functions. + +- [ ] **T5 — Capture worker.** SR is now the only effect (AR gone), runs last; reuse the existing + dynamic output-resolution publish. + +- [ ] **T6 — Managed + Core + tests.** Orchestrator gate from the VSR cap-probe; MVVM mode + quality + + scale props with live push + persistence. Update the 57 xUnit tests (remove AR, add mode). + +- [ ] **T7 — App UI (INVOKE `winui-design` + `winui-dev-workflow` FIRST).** + 1. **Add VSR controls:** mode combo (Off / Upscale / Denoise / Deblur), quality (Low/Med/High/ + Ultra → QualityLevel), scale combo (1.5x / 2x — upscale modes only, disable otherwise). + 2. **Restructure the window for overflow + a pinned footer** (`MainWindow.xaml`): wrap the + controls in a `ScrollViewer` and pin the **"Powered by Maxine" attribution footer OUTSIDE** + it (Grid: row 0 `*` = ScrollViewer of controls, row 1 `Auto` = footer). + **Fixes a current bug:** the footer sits in a `*` spacer row pinned `Bottom`; once the stacked + toggles fill the 400×720 window the spacer collapses and the footer drops below the fold (only + visible after manually growing the window). Adding the VSR controls makes it worse. The + attribution is license-required (SDK Supplement §3.1) so it must always be visible. + Keep all existing `x:Bind` bindings. 4K frame buffer already present. + +- [ ] **T8 — Bundler.** Stage `nvVFXVideoSuperRes.dll` + `nvngx_vsr.dll` into the flat `maxine\`; + **no per-arch engines** for VSR. **Re-run `trace_closure`** — the NGX runtime may pull DLLs + outside the current 19-DLL closure (bundle size unknown until traced). Re-check the VSR model + license (`NVIDIA-Open-Model-License-Agreements-...pdf` in the feature dir) covers redistribution + like green screen. + +- [ ] **T9 — Verify (human gate, 3090).** Build the SDK shim config **last**, export-verify, run: + upscale 1080→2160 visibly sharper; denoise mode cleans; toggles gate correctly off when probe + unavailable. + +## Open risks (both grounded) +1. **NGX DLL closure unknown** until `trace_closure` re-runs against the VSR bundle (T8). +2. **VSR model license** needs a one-line redistribution re-check — it's a different model than + green screen (T8). From 1e111aa4a8ee4d735f99efcdf58fdfcae687c939 Mon Sep 17 00:00:00 2001 From: opariffazman Date: Wed, 24 Jun 2026 09:07:21 +0800 Subject: [PATCH 20/26] refactor(#15): remove dead Artifact Reduction effect Artifact Reduction does not exist in the VFX 1.2.0.0 SDK / NGC catalog (confirmed: absent on disk, NGC returns PAYMENT_REQUIRED, only a help-text typo in install_feature.ps1). Shipping VSR-only, whose own Denoise/Deblur modes cover the cleanup goal. Removes all AR code across the shim (effect, worker block, ABI field) and C# (ABI mirror, MVVM, Orchestrator, config, tests). Super Resolution / Green Screen / Eye Contact untouched; CosParams/CosCaps stay byte-parallel (one int dropped from both sides). Verified: shim stub 0 warnings, App 0 warnings, Core 57/57. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01SfcBU4WRAy7HMNgtm92ogq --- native/shim/artifactreduction.cpp | 143 ------------------ native/shim/artifactreduction.h | 23 --- native/shim/capture.cpp | 45 +----- native/shim/capture.h | 5 - native/shim/shim.cpp | 10 +- native/shim/shim.h | 2 - native/shim/shim.vcxproj | 2 - native/shim/smoke/effects_smoke.cpp | 18 +-- native/shim/vfx_paths.h | 4 +- src/CameraOnScreen.App/MainWindow.xaml | 3 - src/CameraOnScreen.App/Native/PInvokeShim.cs | 5 +- src/CameraOnScreen.Core/Config/Models.cs | 1 - src/CameraOnScreen.Core/Native/Contracts.cs | 3 +- src/CameraOnScreen.Core/Native/FakeShim.cs | 3 +- .../Orchestration/Orchestrator.cs | 5 - .../ViewModels/MainViewModel.cs | 8 - .../Orchestration/OrchestratorTests.cs | 5 +- .../ViewModels/MainViewModelTests.cs | 10 +- 18 files changed, 15 insertions(+), 280 deletions(-) delete mode 100644 native/shim/artifactreduction.cpp delete mode 100644 native/shim/artifactreduction.h diff --git a/native/shim/artifactreduction.cpp b/native/shim/artifactreduction.cpp deleted file mode 100644 index ce3616d..0000000 --- a/native/shim/artifactreduction.cpp +++ /dev/null @@ -1,143 +0,0 @@ -#include "artifactreduction.h" - -#ifdef COS_HAS_MAXINE -#define NOMINMAX -#include -#include -#include -#include -#include "nvCVStatus.h" -#include "nvCVImage.h" -#include "nvVideoEffects.h" -#include "vfx_paths.h" - -// VFX 1.2.0.0 ships the artifact-reduction selector only in its per-feature header -// (installed from NGC). Define the documented literal if absent — mirrors how -// eyecontact.cpp guards the gaze selector. NVIDIA VideoFX selector "ArtifactReduction" -// matches the nvvfxartifactreduction NGC feature name. -#ifndef NVVFX_FX_ARTIFACT_REDUCTION -#define NVVFX_FX_ARTIFACT_REDUCTION "ArtifactReduction" -#endif - -struct ArImpl { - NvVFX_Handle effect = nullptr; - CUstream stream = nullptr; - std::string modelDir; - NvCVImage srcGpu{}; // BGR u8 chunky GPU (input) - NvCVImage dstGpu{}; // BGR u8 chunky GPU (output) - NvCVImage dstCpu{}; // BGR u8 chunky CPU (downloaded) - NvCVImage stage{}; // BGRA u8 chunky GPU (transfer staging) - int w = 0, h = 0; - bool loaded = false; -}; - -ArtifactReduction::ArtifactReduction() = default; -ArtifactReduction::~ArtifactReduction() { Stop(); } - -bool ArtifactReduction::Probe(std::string& detail) { - std::string binDir, modelDir, err; - if (!vfx::ResolveSdkPaths(binDir, modelDir, err)) { detail = err; return false; } - vfx::PointProxiesAt(binDir); - NvVFX_Handle eff = nullptr; - if (NvVFX_CreateEffect(NVVFX_FX_ARTIFACT_REDUCTION, &eff) != NVCV_SUCCESS || !eff) { - detail = "NvVFX_CreateEffect(ArtifactReduction) failed (DLL/SDK load?)"; return false; - } - NvVFX_SetString(eff, NVVFX_MODEL_DIRECTORY, modelDir.c_str()); - NvVFX_SetU32(eff, NVVFX_MODE, 0u); - NvVFX_SetU32(eff, NVVFX_MAX_INPUT_WIDTH, 1920u); - NvVFX_SetU32(eff, NVVFX_MAX_INPUT_HEIGHT, 1080u); - NvCV_Status load = NvVFX_Load(eff); - NvVFX_DestroyEffect(eff); - if (load != NVCV_SUCCESS) { detail = "NvVFX_Load(ArtifactReduction) failed (models missing?)"; return false; } - detail = "ArtifactReduction available"; - return true; -} - -bool ArtifactReduction::Start() { - Stop(); - ArImpl* impl = new (std::nothrow) ArImpl(); - if (!impl) { lastError_ = "out of memory"; return false; } - std::string binDir, err; - if (!vfx::ResolveSdkPaths(binDir, impl->modelDir, err)) { lastError_ = err; delete impl; return false; } - vfx::PointProxiesAt(binDir); - if (NvVFX_CudaStreamCreate(&impl->stream) != NVCV_SUCCESS) { lastError_ = "CudaStreamCreate failed"; delete impl; return false; } - if (NvVFX_CreateEffect(NVVFX_FX_ARTIFACT_REDUCTION, &impl->effect) != NVCV_SUCCESS || !impl->effect) { - lastError_ = "CreateEffect failed"; NvVFX_CudaStreamDestroy(impl->stream); delete impl; return false; - } - NvVFX_SetString(impl->effect, NVVFX_MODEL_DIRECTORY, impl->modelDir.c_str()); - NvVFX_SetCudaStream(impl->effect, NVVFX_CUDA_STREAM, impl->stream); - NvVFX_SetU32(impl->effect, NVVFX_MODE, 1u); // mode 1 = stronger / for compressed input - impl_ = impl; ready_ = true; lastError_.clear(); - return true; -} - -void ArtifactReduction::Stop() { - auto* impl = static_cast(impl_); - if (!impl) { ready_ = false; return; } - NvCVImage_Dealloc(&impl->srcGpu); - NvCVImage_Dealloc(&impl->dstGpu); - NvCVImage_Dealloc(&impl->dstCpu); - NvCVImage_Dealloc(&impl->stage); - if (impl->effect) NvVFX_DestroyEffect(impl->effect); - if (impl->stream) NvVFX_CudaStreamDestroy(impl->stream); - delete impl; impl_ = nullptr; ready_ = false; -} - -namespace { -NvCV_Status EnsureImages(ArImpl* impl, int w, int h) { - if (impl->loaded && impl->w == w && impl->h == h) return NVCV_SUCCESS; - NvCVImage_Dealloc(&impl->srcGpu); - NvCVImage_Dealloc(&impl->dstGpu); - NvCVImage_Dealloc(&impl->dstCpu); - NvCVImage_Dealloc(&impl->stage); - NvCV_Status s; - s = NvCVImage_Alloc(&impl->srcGpu, w, h, NVCV_BGR, NVCV_U8, NVCV_CHUNKY, NVCV_GPU, 1); if (s != NVCV_SUCCESS) return s; - s = NvCVImage_Alloc(&impl->dstGpu, w, h, NVCV_BGR, NVCV_U8, NVCV_CHUNKY, NVCV_GPU, 1); if (s != NVCV_SUCCESS) return s; - s = NvCVImage_Alloc(&impl->dstCpu, w, h, NVCV_BGR, NVCV_U8, NVCV_CHUNKY, NVCV_CPU, 1); if (s != NVCV_SUCCESS) return s; - s = NvCVImage_Alloc(&impl->stage, w, h, NVCV_BGRA, NVCV_U8, NVCV_CHUNKY, NVCV_GPU, 1); if (s != NVCV_SUCCESS) return s; - s = NvVFX_SetImage(impl->effect, NVVFX_INPUT_IMAGE, &impl->srcGpu); if (s != NVCV_SUCCESS) return s; - s = NvVFX_SetImage(impl->effect, NVVFX_OUTPUT_IMAGE, &impl->dstGpu); if (s != NVCV_SUCCESS) return s; - s = NvVFX_SetU32(impl->effect, NVVFX_MAX_INPUT_WIDTH, static_cast(w)); if (s != NVCV_SUCCESS) return s; - s = NvVFX_SetU32(impl->effect, NVVFX_MAX_INPUT_HEIGHT, static_cast(h)); if (s != NVCV_SUCCESS) return s; - s = NvVFX_Load(impl->effect); if (s != NVCV_SUCCESS) return s; - impl->w = w; impl->h = h; impl->loaded = true; - return NVCV_SUCCESS; -} -} // namespace - -bool ArtifactReduction::ProcessFrame(uint8_t* bgra, int w, int h) { - auto* impl = static_cast(impl_); - if (!impl || !impl->effect || !bgra || w <= 0 || h <= 0) return false; - if (NvCV_Status es = EnsureImages(impl, w, h); es != NVCV_SUCCESS) { - lastError_ = std::string("EnsureImages/Load failed: ") + NvCV_GetErrorStringFromCode(es); ready_ = false; return false; - } - NvCVImage src{}; - NvCVImage_Init(&src, w, h, w * 4, bgra, NVCV_BGRA, NVCV_U8, NVCV_CHUNKY, NVCV_CPU); - if (NvCVImage_Transfer(&src, &impl->srcGpu, 1.0f, impl->stream, &impl->stage) != NVCV_SUCCESS) { lastError_ = "upload failed"; return false; } - if (NvVFX_Run(impl->effect, 0) != NVCV_SUCCESS) { lastError_ = "NvVFX_Run failed"; return false; } - if (NvCVImage_Transfer(&impl->dstGpu, &impl->dstCpu, 1.0f, impl->stream, nullptr) != NVCV_SUCCESS) { lastError_ = "download failed"; return false; } - if (NvVFX_CudaStreamSynchronize(impl->stream) != NVCV_SUCCESS) { lastError_ = "stream sync failed"; return false; } - // Write cleaned BGR back over the BGRA buffer; leave alpha untouched (forced 0xFF upstream). - const uint8_t* d = static_cast(impl->dstCpu.pixels); - const int dpitch = impl->dstCpu.pitch; - for (int y = 0; y < h; ++y) { - const uint8_t* drow = d + static_cast(dpitch) * y; - uint8_t* prow = bgra + static_cast(w) * 4 * y; - for (int x = 0; x < w; ++x) { - prow[x * 4 + 0] = drow[x * 3 + 0]; - prow[x * 4 + 1] = drow[x * 3 + 1]; - prow[x * 4 + 2] = drow[x * 3 + 2]; - } - } - return true; -} - -#else -// ---- Passthrough stub: built when no SDK is configured. ---- -ArtifactReduction::ArtifactReduction() = default; -ArtifactReduction::~ArtifactReduction() = default; -bool ArtifactReduction::Probe(std::string& detail) { detail = "Maxine SDK not built in"; return false; } -bool ArtifactReduction::Start() { lastError_ = "Maxine SDK not built in"; ready_ = false; return false; } -void ArtifactReduction::Stop() { ready_ = false; } -bool ArtifactReduction::ProcessFrame(uint8_t*, int, int) { return false; } -#endif diff --git a/native/shim/artifactreduction.h b/native/shim/artifactreduction.h deleted file mode 100644 index 7e15aac..0000000 --- a/native/shim/artifactreduction.h +++ /dev/null @@ -1,23 +0,0 @@ -#pragma once -#include -#include - -// Wraps the Maxine VFX Artifact Reduction effect. CPU-copy: a BGRA frame is uploaded -// to the GPU as BGR, cleaned, downloaded, and written back over the same BGRA buffer -// (RGB replaced, alpha left untouched). No size change. All methods no-throw; failure -// via IsReady()/LastError(). Without COS_HAS_MAXINE this is a never-ready stub. -class ArtifactReduction { -public: - ArtifactReduction(); - ~ArtifactReduction(); - static bool Probe(std::string& detail); - bool Start(); - void Stop(); - bool ProcessFrame(uint8_t* bgra, int width, int height); - bool IsReady() const { return ready_; } - const std::string& LastError() const { return lastError_; } -private: - bool ready_ = false; - std::string lastError_; - void* impl_ = nullptr; -}; diff --git a/native/shim/capture.cpp b/native/shim/capture.cpp index cd7d76c..0de117f 100644 --- a/native/shim/capture.cpp +++ b/native/shim/capture.cpp @@ -1,7 +1,6 @@ #include "capture.h" #include "aigs.h" #include "eyecontact.h" -#include "artifactreduction.h" #include "superres.h" #include @@ -50,11 +49,6 @@ struct CaptureState { std::mutex ecErrMtx; // leaf lock, never nested under mtx/lifecycle std::string ecError; // guarded by ecErrMtx - std::atomic artifactReductionEnabled{false}; - std::atomic artifactReductionActive{false}; - std::mutex arErrMtx; // leaf lock - std::string arError; - std::atomic superResEnabled{false}; std::atomic superResScale{20}; // 15 or 20 std::atomic superResActive{false}; @@ -315,7 +309,6 @@ void Capture::WorkerLoop() { // affinity. Declare here (after CoInitializeEx) so ctor/dtor run on this thread. Aigs aigs; EyeContact eyeContact; - ArtifactReduction artifactReduction; SuperRes superRes; IMFSourceReader* reader = nullptr; @@ -357,31 +350,7 @@ void Capture::WorkerLoop() { if (sample) { if (CopyFrame(sample, width, height, stride, scratch)) { - // Artifact Reduction runs FIRST so the AI effects downstream get a clean frame. - const bool arWant = g_state.artifactReductionEnabled.load(std::memory_order_acquire); - if (arWant && !artifactReduction.IsReady()) { - if (!artifactReduction.Start()) { - std::lock_guard e(g_state.arErrMtx); - const std::string& ne = artifactReduction.LastError(); - if (g_state.arError != ne) g_state.arError = ne; - } - } else if (!arWant && artifactReduction.IsReady()) { - artifactReduction.Stop(); - std::lock_guard e(g_state.arErrMtx); - if (!g_state.arError.empty()) g_state.arError.clear(); - } - bool arApplied = false; - if (arWant && artifactReduction.IsReady()) { - arApplied = artifactReduction.ProcessFrame(scratch.data(), width, height); - std::lock_guard e(g_state.arErrMtx); - if (!arApplied) { - const std::string& ne = artifactReduction.LastError(); - if (g_state.arError != ne) g_state.arError = ne; - } else if (!g_state.arError.empty()) { g_state.arError.clear(); } - } - g_state.artifactReductionActive.store(arApplied, std::memory_order_release); - - // Eye Contact runs second, on the (AR-cleaned) raw frame (needs real eyes/landmarks). + // Eye Contact runs first, on the raw frame (needs real eyes/landmarks). const bool ecWant = g_state.eyeContactEnabled.load(std::memory_order_acquire); if (ecWant && !eyeContact.IsReady()) { if (!eyeContact.Start()) { @@ -488,7 +457,6 @@ void Capture::WorkerLoop() { // Tear down all effects before releasing the reader (thread-affinity requirement). superRes.Stop(); - artifactReduction.Stop(); eyeContact.Stop(); aigs.Stop(); SafeRelease(reader); @@ -535,8 +503,6 @@ void Capture::Stop() { std::lock_guard e(g_state.ecErrMtx); g_state.ecError.clear(); } - g_state.artifactReductionActive.store(false, std::memory_order_release); - { std::lock_guard e(g_state.arErrMtx); g_state.arError.clear(); } g_state.superResActive.store(false, std::memory_order_release); { std::lock_guard e(g_state.srErrMtx); g_state.srError.clear(); } } @@ -582,15 +548,6 @@ std::string Capture::EyeContactError() const { return g_state.ecError; } -void Capture::SetArtifactReduction(bool enabled) { - g_state.artifactReductionEnabled.store(enabled, std::memory_order_release); -} -bool Capture::ArtifactReductionActive() const { - return g_state.artifactReductionActive.load(std::memory_order_acquire); -} -std::string Capture::ArtifactReductionError() const { - std::lock_guard e(g_state.arErrMtx); return g_state.arError; -} void Capture::SetSuperRes(bool enabled, int scaleX10) { g_state.superResScale.store(scaleX10 == 15 ? 15 : 20, std::memory_order_release); g_state.superResEnabled.store(enabled, std::memory_order_release); diff --git a/native/shim/capture.h b/native/shim/capture.h index 9ae9aa5..4815bfc 100644 --- a/native/shim/capture.h +++ b/native/shim/capture.h @@ -42,11 +42,6 @@ class Capture { bool EyeContactActive() const; // true only while gaze redirection is transforming frames std::string EyeContactError() const; // empty when none - // Toggles Artifact Reduction for subsequent frames. Thread-safe; worker owns the object. - void SetArtifactReduction(bool enabled); - bool ArtifactReductionActive() const; - std::string ArtifactReductionError() const; - // Toggles Super Resolution + scale (15=1.5x, 20=2x). Thread-safe; worker owns the object. void SetSuperRes(bool enabled, int scaleX10); bool SuperResActive() const; diff --git a/native/shim/shim.cpp b/native/shim/shim.cpp index 04c5220..a560af1 100644 --- a/native/shim/shim.cpp +++ b/native/shim/shim.cpp @@ -3,7 +3,6 @@ #include "capture.h" #include "aigs.h" #include "eyecontact.h" -#include "artifactreduction.h" #include "superres.h" #include @@ -66,7 +65,6 @@ COS_API void cos_set_params(const CosParams* p) { g_capture.SetGreenScreen(p->green_screen_enabled != 0); g_capture.SetMatteParams(p->green_screen_expand, p->green_screen_feather); g_capture.SetEyeContact(p->eye_contact_enabled != 0); - g_capture.SetArtifactReduction(p->artifact_reduction_enabled != 0); g_capture.SetSuperRes(p->super_res_enabled != 0, p->super_res_scale); } @@ -82,7 +80,6 @@ COS_API void cos_get_status(CosStatus* out) { out->eye_contact_active = g_capture.EyeContactActive() ? 1 : 0; std::string err = g_capture.GreenScreenError(); if (err.empty()) err = g_capture.EyeContactError(); - if (err.empty()) err = g_capture.ArtifactReductionError(); if (err.empty()) err = g_capture.SuperResError(); if (!err.empty()) { size_t n = err.size() < 255 ? err.size() : 255; @@ -121,12 +118,11 @@ COS_API int cos_query_capabilities(CosCaps* out) { std::memcpy(out->ec_detail, ecDetail.data(), en); out->ec_detail[en] = '\0'; - std::string arDetail, srDetail; - out->artifact_reduction_available = ArtifactReduction::Probe(arDetail) ? 1 : 0; - out->super_res_available = SuperRes::Probe(srDetail) ? 1 : 0; + std::string srDetail; + out->super_res_available = SuperRes::Probe(srDetail) ? 1 : 0; // Return 1 if any effect is available (the managed side reads the per-gate ints). - return (gsOk || ecOk || out->artifact_reduction_available || out->super_res_available) ? 1 : 0; + return (gsOk || ecOk || out->super_res_available) ? 1 : 0; } COS_API void cos_shutdown(void) { diff --git a/native/shim/shim.h b/native/shim/shim.h index 05aca2d..fc28c39 100644 --- a/native/shim/shim.h +++ b/native/shim/shim.h @@ -23,7 +23,6 @@ typedef struct { int eye_contact_enabled; double eye_contact_sensitivity; double eye_contact_look_away_range; - int artifact_reduction_enabled; int super_res_enabled; int super_res_scale; // 0=off, 15=1.5x, 20=2x } CosParams; @@ -33,7 +32,6 @@ typedef struct { char detail[256]; // green-screen status/error (UTF-8, NUL-terminated) int eye_contact_available; // 1 if Maxine GazeRedirection can run, else 0 char ec_detail[256]; // eye-contact status/error (UTF-8, NUL-terminated) - int artifact_reduction_available; // 1 if Maxine ArtifactReduction can run int super_res_available; // 1 if Maxine SuperRes can run } CosCaps; diff --git a/native/shim/shim.vcxproj b/native/shim/shim.vcxproj index b33811b..173ef29 100644 --- a/native/shim/shim.vcxproj +++ b/native/shim/shim.vcxproj @@ -91,7 +91,6 @@ - @@ -118,7 +117,6 @@ - diff --git a/native/shim/smoke/effects_smoke.cpp b/native/shim/smoke/effects_smoke.cpp index 2318efa..311ab97 100644 --- a/native/shim/smoke/effects_smoke.cpp +++ b/native/shim/smoke/effects_smoke.cpp @@ -1,5 +1,5 @@ // Dev-box smoke (needs VFX SDK + RTX GPU). Build with the shim's include/lib setup. -// Verifies ArtifactReduction and SuperRes can start and process synthetic frames. +// Verifies SuperRes can start and process synthetic frames. // // Build (Developer PowerShell for VS 2022, from repo root): // $env:COS_VFX_SDK_DIR = "C:\path\to\VideoFX" @@ -7,7 +7,6 @@ // /I "%COS_VFX_SDK_DIR%\nvvfx\include" ^ // /DCOS_HAS_MAXINE ^ // native\shim\smoke\effects_smoke.cpp ^ -// native\shim\artifactreduction.cpp ^ // native\shim\superres.cpp ^ // native\shim\vfx_paths.cpp native\shim\paths.cpp ^ // "%COS_VFX_SDK_DIR%\nvvfx\src\nvVideoEffectsProxy.cpp" ^ @@ -15,29 +14,14 @@ // .\effects_smoke.exe // // Expected output (RTX GPU with models installed): -// AR ProcessFrame: ok () -// effects_smoke AR OK // SR ProcessFrame: ok -> 1280x960 (requires nvvfxvideosuperres NGC feature + models) // Without GPU/models, Probe prints the reason and exits 0 (deferred to human gate). #include #include #include -#include "../artifactreduction.h" #include "../superres.h" int main() { - { - std::string detail; - if (!ArtifactReduction::Probe(detail)) { std::printf("AR unavailable: %s\n", detail.c_str()); return 0; } - ArtifactReduction ar; - assert(ar.Start()); - std::vector frame(1280 * 720 * 4, 128); - bool ok = ar.ProcessFrame(frame.data(), 1280, 720); - std::printf("AR ProcessFrame: %s (%s)\n", ok ? "ok" : "fail", ar.LastError().c_str()); - assert(ok); - ar.Stop(); - std::puts("effects_smoke AR OK"); - } { std::string d; if (SuperRes::Probe(d)) { diff --git a/native/shim/vfx_paths.h b/native/shim/vfx_paths.h index 266d07e..2f556aa 100644 --- a/native/shim/vfx_paths.h +++ b/native/shim/vfx_paths.h @@ -1,8 +1,8 @@ #pragma once #include -// Shared VFX-SDK runtime discovery for ALL VFX effects (Green Screen, Artifact -// Reduction, Super Resolution). Single source of truth — aigs.cpp uses these too. +// Shared VFX-SDK runtime discovery for ALL VFX effects (Green Screen, +// Super Resolution). Single source of truth — aigs.cpp uses these too. // Model subdir is models\vfx in the bundled layout. namespace vfx { // Fills binDir (holds NVVideoEffects.dll) and modelDir. Order: COS_VFX_RUNTIME_DIR diff --git a/src/CameraOnScreen.App/MainWindow.xaml b/src/CameraOnScreen.App/MainWindow.xaml index 740f284..5494473 100644 --- a/src/CameraOnScreen.App/MainWindow.xaml +++ b/src/CameraOnScreen.App/MainWindow.xaml @@ -30,9 +30,6 @@ - diff --git a/src/CameraOnScreen.App/Native/PInvokeShim.cs b/src/CameraOnScreen.App/Native/PInvokeShim.cs index 686e099..83ed470 100644 --- a/src/CameraOnScreen.App/Native/PInvokeShim.cs +++ b/src/CameraOnScreen.App/Native/PInvokeShim.cs @@ -24,7 +24,6 @@ private struct CosParams public double green_screen_feather; public int eye_contact_enabled; public double eye_contact_sensitivity; public double eye_contact_look_away_range; - public int artifact_reduction_enabled; public int super_res_enabled; public int super_res_scale; } @@ -36,7 +35,6 @@ private struct CosCaps [MarshalAs(UnmanagedType.ByValArray, SizeConst = 256)] public byte[] Detail; public int EyeContactAvailable; [MarshalAs(UnmanagedType.ByValArray, SizeConst = 256)] public byte[] EcDetail; - public int ArtifactReductionAvailable; public int SuperResAvailable; } @@ -82,7 +80,6 @@ public void SetParams(ShimParams p) eye_contact_enabled = p.EyeContactEnabled ? 1 : 0, eye_contact_sensitivity = p.EyeContactSensitivity, eye_contact_look_away_range = p.EyeContactLookAwayRange, - artifact_reduction_enabled = p.ArtifactReductionEnabled ? 1 : 0, super_res_enabled = p.SuperResEnabled ? 1 : 0, super_res_scale = p.SuperResScale, }; @@ -114,7 +111,7 @@ public ShimCapabilities QueryCapabilities() return new ShimCapabilities( caps.GreenScreenAvailable != 0, ReadUtf8(caps.Detail, 0, 256), caps.EyeContactAvailable != 0, ReadUtf8(caps.EcDetail, 0, 256), - caps.ArtifactReductionAvailable != 0, caps.SuperResAvailable != 0); + caps.SuperResAvailable != 0); } public void Dispose() => cos_shutdown(); diff --git a/src/CameraOnScreen.Core/Config/Models.cs b/src/CameraOnScreen.Core/Config/Models.cs index 04c9d70..dcfb7aa 100644 --- a/src/CameraOnScreen.Core/Config/Models.cs +++ b/src/CameraOnScreen.Core/Config/Models.cs @@ -29,7 +29,6 @@ public sealed record EffectSettings public bool EyeContactEnabled { get; init; } public double EyeContactSensitivity { get; init; } = 0.5; public double EyeContactLookAwayRange { get; init; } = 0.5; - public bool ArtifactReductionEnabled { get; init; } public bool SuperResEnabled { get; init; } public int SuperResScale { get; init; } // 0=off, 15=1.5x, 20=2x } diff --git a/src/CameraOnScreen.Core/Native/Contracts.cs b/src/CameraOnScreen.Core/Native/Contracts.cs index 92276b5..9b1b432 100644 --- a/src/CameraOnScreen.Core/Native/Contracts.cs +++ b/src/CameraOnScreen.Core/Native/Contracts.cs @@ -7,7 +7,7 @@ namespace CameraOnScreen.Core.Native; public sealed record ShimCapabilities( bool GreenScreenAvailable, string Detail, bool EyeContactAvailable = false, string EyeContactDetail = "", - bool ArtifactReductionAvailable = false, bool SuperResAvailable = false); + bool SuperResAvailable = false); public enum GazeState { Unknown, OnCamera, Redirected, RealEyes } @@ -27,7 +27,6 @@ public sealed record ShimParams( bool EyeContactEnabled, double EyeContactSensitivity, double EyeContactLookAwayRange, - bool ArtifactReductionEnabled = false, bool SuperResEnabled = false, int SuperResScale = 0); // 0=off, 15=1.5x, 20=2x diff --git a/src/CameraOnScreen.Core/Native/FakeShim.cs b/src/CameraOnScreen.Core/Native/FakeShim.cs index 276f4b3..ec13973 100644 --- a/src/CameraOnScreen.Core/Native/FakeShim.cs +++ b/src/CameraOnScreen.Core/Native/FakeShim.cs @@ -6,7 +6,6 @@ public sealed class FakeShim : INativeShim public ShimParams? LastParams { get; private set; } public bool GreenScreenAvailable { get; set; } public bool EyeContactAvailable { get; set; } - public bool ArtifactReductionAvailable { get; set; } public bool SuperResAvailable { get; set; } private bool _running; @@ -34,7 +33,7 @@ public ShimCapabilities QueryCapabilities() => GreenScreenAvailable ? "fake: available" : "fake: unavailable", EyeContactAvailable, EyeContactAvailable ? "fake: ec available" : "fake: ec unavailable", - ArtifactReductionAvailable, SuperResAvailable); + SuperResAvailable); public bool Disposed { get; private set; } public void Dispose() => Disposed = true; diff --git a/src/CameraOnScreen.Core/Orchestration/Orchestrator.cs b/src/CameraOnScreen.Core/Orchestration/Orchestrator.cs index 1a6f9f5..4a3103d 100644 --- a/src/CameraOnScreen.Core/Orchestration/Orchestrator.cs +++ b/src/CameraOnScreen.Core/Orchestration/Orchestrator.cs @@ -36,9 +36,6 @@ public Orchestrator(INativeShim shim, GpuTier tier) /// Human-readable reason from the eye-contact probe. "Checking…" until probed. public string EyeContactDetail { get; private set; } = "Checking effect availability…"; - /// True when the shim reports Artifact Reduction can run. False until . - public bool ArtifactReductionAvailable { get; private set; } - /// True when the shim reports Super Resolution can run. False until . public bool SuperResAvailable { get; private set; } @@ -52,7 +49,6 @@ public void ProbeCapabilities() CapabilityDetail = caps.Detail; EyeContactAvailable = caps.EyeContactAvailable; EyeContactDetail = caps.EyeContactDetail; - ArtifactReductionAvailable = caps.ArtifactReductionAvailable; SuperResAvailable = caps.SuperResAvailable; } @@ -77,7 +73,6 @@ public void ApplyParams(ShimParams requested) { GreenScreenEnabled = requested.GreenScreenEnabled && EffectsAvailable, EyeContactEnabled = requested.EyeContactEnabled && EyeContactAvailable, - ArtifactReductionEnabled = requested.ArtifactReductionEnabled && ArtifactReductionAvailable, SuperResEnabled = requested.SuperResEnabled && SuperResAvailable, }; _shim.SetParams(effective); diff --git a/src/CameraOnScreen.Core/ViewModels/MainViewModel.cs b/src/CameraOnScreen.Core/ViewModels/MainViewModel.cs index 4b36800..461da56 100644 --- a/src/CameraOnScreen.Core/ViewModels/MainViewModel.cs +++ b/src/CameraOnScreen.Core/ViewModels/MainViewModel.cs @@ -31,7 +31,6 @@ public MainViewModel(Orchestrator orchestrator, INativeShim shim) CapabilityDetail = orchestrator.CapabilityDetail; EyeContactAvailable = orchestrator.EyeContactAvailable; EyeContactDetail = orchestrator.EyeContactDetail; - ArtifactReductionAvailable = orchestrator.ArtifactReductionAvailable; SuperResAvailable = orchestrator.SuperResAvailable; _statusHandler = (_, s) => OnStatus(s); _orchestrator.StatusChanged += _statusHandler; @@ -51,7 +50,6 @@ public async Task ProbeCapabilitiesAsync() CapabilityDetail = _orchestrator.CapabilityDetail; EyeContactAvailable = _orchestrator.EyeContactAvailable; EyeContactDetail = _orchestrator.EyeContactDetail; - ArtifactReductionAvailable = _orchestrator.ArtifactReductionAvailable; SuperResAvailable = _orchestrator.SuperResAvailable; } @@ -64,14 +62,12 @@ public async Task ProbeCapabilitiesAsync() [ObservableProperty] private bool eyeContactEnabled; [ObservableProperty] private double eyeContactSensitivity = 0.5; [ObservableProperty] private double eyeContactLookAwayRange = 0.5; - [ObservableProperty] private bool artifactReductionEnabled; [ObservableProperty] private bool superResEnabled; [ObservableProperty] private int superResScaleIndex; // 0=1.5x, 1=2x [ObservableProperty] private bool effectsAvailable; [ObservableProperty] private string capabilityDetail = "Checking effect availability…"; [ObservableProperty] private bool eyeContactAvailable; [ObservableProperty] private string eyeContactDetail = "Checking effect availability…"; - [ObservableProperty] private bool artifactReductionAvailable; [ObservableProperty] private bool superResAvailable; [ObservableProperty] private bool isRunning; [ObservableProperty] private bool locked; @@ -94,7 +90,6 @@ public void LoadFrom(AppConfig config) EyeContactEnabled = config.Effects.EyeContactEnabled; EyeContactSensitivity = config.Effects.EyeContactSensitivity; EyeContactLookAwayRange = config.Effects.EyeContactLookAwayRange; - ArtifactReductionEnabled = config.Effects.ArtifactReductionEnabled; SuperResEnabled = config.Effects.SuperResEnabled; SuperResScaleIndex = IndexFromScale(config.Effects.SuperResScale); Locked = config.Overlay.Locked; @@ -124,7 +119,6 @@ public void LoadFrom(AppConfig config) GreenScreenFeather = GreenScreenFeather, EyeContactEnabled = EyeContactEnabled, EyeContactSensitivity = EyeContactSensitivity, EyeContactLookAwayRange = EyeContactLookAwayRange, - ArtifactReductionEnabled = ArtifactReductionEnabled, SuperResEnabled = SuperResEnabled, SuperResScale = ScaleFromIndex(SuperResScaleIndex), }, @@ -142,7 +136,6 @@ public void LoadFrom(AppConfig config) partial void OnEyeContactEnabledChanged(bool value) => ApplyLiveParams(); partial void OnEyeContactSensitivityChanged(double value) => ApplyLiveParams(); partial void OnEyeContactLookAwayRangeChanged(double value) => ApplyLiveParams(); - partial void OnArtifactReductionEnabledChanged(bool value) => ApplyLiveParams(); partial void OnSuperResEnabledChanged(bool value) => ApplyLiveParams(); partial void OnSuperResScaleIndexChanged(int value) => ApplyLiveParams(); @@ -159,7 +152,6 @@ private void ApplyLiveParams() EyeContactEnabled: EyeContactEnabled, EyeContactSensitivity: EyeContactSensitivity, EyeContactLookAwayRange: EyeContactLookAwayRange, - ArtifactReductionEnabled: ArtifactReductionEnabled, SuperResEnabled: SuperResEnabled, SuperResScale: ScaleFromIndex(SuperResScaleIndex)); diff --git a/tests/CameraOnScreen.Core.Tests/Orchestration/OrchestratorTests.cs b/tests/CameraOnScreen.Core.Tests/Orchestration/OrchestratorTests.cs index e0bcfb8..1c7b1e5 100644 --- a/tests/CameraOnScreen.Core.Tests/Orchestration/OrchestratorTests.cs +++ b/tests/CameraOnScreen.Core.Tests/Orchestration/OrchestratorTests.cs @@ -191,12 +191,11 @@ public void ProbeCapabilities_records_eye_contact_detail() [Fact] public void ApplyParams_forces_effects_off_when_unavailable() { - var shim = new FakeShim { GreenScreenAvailable = false, ArtifactReductionAvailable = false, SuperResAvailable = false }; + var shim = new FakeShim { GreenScreenAvailable = false, SuperResAvailable = false }; var orch = new Orchestrator(shim, GpuTier.Rtx); orch.ProbeCapabilities(); orch.ApplyParams(new ShimParams("cam", true, 0, 0, false, 0.5, 0.5, - ArtifactReductionEnabled: true, SuperResEnabled: true, SuperResScale: 20)); - Assert.False(shim.LastParams!.ArtifactReductionEnabled); + SuperResEnabled: true, SuperResScale: 20)); Assert.False(shim.LastParams!.SuperResEnabled); } } diff --git a/tests/CameraOnScreen.Core.Tests/ViewModels/MainViewModelTests.cs b/tests/CameraOnScreen.Core.Tests/ViewModels/MainViewModelTests.cs index 358f578..e2918e0 100644 --- a/tests/CameraOnScreen.Core.Tests/ViewModels/MainViewModelTests.cs +++ b/tests/CameraOnScreen.Core.Tests/ViewModels/MainViewModelTests.cs @@ -252,20 +252,18 @@ public void Toggling_eye_contact_while_running_pushes_params() } [Fact] - public void BuildParams_includes_artifact_reduction_and_superres() + public void BuildParams_includes_superres() { var shim = new FakeShim { GreenScreenAvailable = true, EyeContactAvailable = true, - ArtifactReductionAvailable = true, SuperResAvailable = true }; + SuperResAvailable = true }; var orch = new Orchestrator(shim, GpuTier.Rtx); orch.ProbeCapabilities(); var vm = new MainViewModel(orch, shim); - vm.ArtifactReductionEnabled = true; vm.SuperResEnabled = true; vm.SuperResScaleIndex = 1; // 2x var p = vm.BuildParams(); - Assert.True(p.ArtifactReductionEnabled); Assert.True(p.SuperResEnabled); Assert.Equal(20, p.SuperResScale); } @@ -276,16 +274,14 @@ public void ToAppConfig_roundtrips_new_effects() var shim = new FakeShim(); var vm = new MainViewModel(new Orchestrator(shim, GpuTier.Rtx), shim) { - ArtifactReductionEnabled = true, SuperResEnabled = true, SuperResScaleIndex = 0 + SuperResEnabled = true, SuperResScaleIndex = 0 }; var cfg = vm.ToAppConfig(0, 0, 320, 240); - Assert.True(cfg.Effects.ArtifactReductionEnabled); Assert.True(cfg.Effects.SuperResEnabled); Assert.Equal(15, cfg.Effects.SuperResScale); var vm2 = new MainViewModel(new Orchestrator(shim, GpuTier.Rtx), shim); vm2.LoadFrom(cfg); - Assert.True(vm2.ArtifactReductionEnabled); Assert.True(vm2.SuperResEnabled); Assert.Equal(0, vm2.SuperResScaleIndex); } From 01733797817cf43834dee3bf9dac1b39d434e52e Mon Sep 17 00:00:00 2001 From: opariffazman Date: Wed, 24 Jun 2026 15:17:37 +0800 Subject: [PATCH 21/26] feat(#15): rewrite SuperRes against real VSR API (T2) Rewrite native/shim/superres.{h,cpp} against the verified VFX 1.2.0.0 Video Super Resolution API (NGX), replacing the guessed "SuperRes" literal: - selector NVVFX_FX_VIDEO_SUPER_RES ("VideoSuperRes") via the real per-feature header nvVFXVideoSuperRes.h (added the feature include dir to shim.vcxproj). - BGRA u8 GPU in+out end to end; deleted the BGR conversion and the per-pixel alpha-repack loop in favor of a per-row memcpy honoring source pitch. Alpha now flows through VSR (opaque in passthrough; the green-screen matte when AIGS ran upstream) instead of being force-filled 0xFF. - Start(int qualityLevel, int scaleX10) sets NVVFX_QUALITY_LEVEL (the mode selector): 1-4 upscale -> out = in*scale; 8-15 denoise/deblur -> out = in. - Probe mirrors the verified Start sequence (create + quality + images + load) so the capability gate matches the runtime path. Verified by building the shim SDK config against the runner SDK (C:\actions-runner\_sdk\VideoFX, after installing the nvvfxvideosuperres feature): 0 warnings, both effects + VideoSuperRes exported, "not built in" absent. Compile-only; runtime Load/Run remains the T9 human gate on the 3090. Call sites stubbed to keep the tree green until the ABI carries the mode: capture.cpp passes quality 1 (ponytail comment; T5/T6 thread the real mode), effects_smoke.cpp updated. Plan doc: T2 ticked + a "Compile-testing the real SDK shim on this host" runbook section. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01SfcBU4WRAy7HMNgtm92ogq --- .../plans/2026-06-24-issue-15-vsr-resume.md | 31 ++++- native/shim/capture.cpp | 4 +- native/shim/shim.vcxproj | 2 +- native/shim/smoke/effects_smoke.cpp | 2 +- native/shim/superres.cpp | 107 +++++++++--------- native/shim/superres.h | 14 ++- 6 files changed, 96 insertions(+), 64 deletions(-) diff --git a/docs/superpowers/plans/2026-06-24-issue-15-vsr-resume.md b/docs/superpowers/plans/2026-06-24-issue-15-vsr-resume.md index 7803b4e..96be1b8 100644 --- a/docs/superpowers/plans/2026-06-24-issue-15-vsr-resume.md +++ b/docs/superpowers/plans/2026-06-24-issue-15-vsr-resume.md @@ -47,6 +47,28 @@ After PR #14 (`feat/fps-and-capture-mode` = fps counter + highest-fps capture) m `feat/ai-sharpness-vsr` off `main`; rebase the parked branch's effect commits on (the fps/capture patches dedupe to empty since #14 carries them). The parked branch is the substrate. +## Compile-testing the real SDK shim on THIS host (no dev SDK at `C:\dev`) + +The interactive user's `C:\dev` VFX/AR trees are absent on this box, **but the self-hosted runner +SDK is present and usable** for real-headers compile tests of any shim task (T2–T9): + +- **SDK roots:** `C:\actions-runner\_sdk\VideoFX` (VFX 1.2.0.0), `C:\actions-runner\_sdk\Maxine-AR-SDK-1.1.1.0`. +- **NGC key** (for `install_feature.ps1`) lives in the gitignored repo `.env` as + `$env:NGC_CLI_API_KEY="nvapi-…"`. **Dot-sourcing `.env` is flaky** (no trailing newline) — extract + it: `$raw = Get-Content .env -Raw; if ($raw -match 'nvapi-[A-Za-z0-9_\-]+') { $env:NGC_CLI_API_KEY = $matches[0] }`. +- **VSR feature must be installed once** (was missing — only greenscreen shipped): + `cd C:\actions-runner\_sdk\VideoFX\features; .\install_feature.ps1 -features nvvfxvideosuperres`. + → installs `features\nvvfxvideosuperres\{include\nvVFXVideoSuperRes.h, bin\nvVFXVideoSuperRes.dll, bin\nvngx_vsr.dll}`. + **"No models found for SM86 (normal)" is expected** — VSR is NGX, baked into `nvngx_vsr.dll`, no per-arch engine. +- **Build the real path:** MSBuild `native/shim/shim.vcxproj /p:Configuration=Debug /p:Platform=x64` + `/p:CosVfxSdkDir=C:\actions-runner\_sdk\VideoFX /p:CosArSdkDir=C:\actions-runner\_sdk\Maxine-AR-SDK-1.1.1.0`. +- **Export-verify the deployed DLL** (`native/shim/x64/Debug/CameraOnScreen.Shim.dll`): `grep -ac` + finds `GreenScreen` + `GazeRedirection` + `VideoSuperRes`, and `not built in` is **absent**. +- **Confirmed 2026-06-24:** T2 superres.cpp compiles clean against the real header (0 warnings, + both effects). The header's `NVVFX_FX_VIDEO_SUPER_RES "VideoSuperRes"` + `NVVFX_QUALITY_LEVEL + "QualityLevel"` match the rewrite. This only proves **compilation**; runtime VSR (Load/Run on real + frames) is still the T9 human gate on the 3090. + ## Tasks - [x] **T1 — Probe VSR (DONE, 2026-06-24).** Verified the real API on the 3090 (above). @@ -54,7 +76,14 @@ patches dedupe to empty since #14 carries them). The parked branch is the substr - [ ] **T0 — Spec update.** Edit the design spec: drop Artifact Reduction entirely; re-scope SR to VSR-only with a mode selector. Keep the generic-plumbing sections. -- [ ] **T2 — Rewrite `superres.{h,cpp}`** against the real API (simpler than parked): +- [x] **T2 — Rewrite `superres.{h,cpp}` (DONE, 2026-06-24).** Real `"VideoSuperRes"` selector + + `nvVFXVideoSuperRes.h` (added the feature include dir to `shim.vcxproj`); BGRA u8 GPU in+out + (BGR convert + per-pixel repack deleted → per-row `memcpy` honoring pitch, alpha flows through); + `Start(int qualityLevel, int scaleX10)` sets `NVVFX_QUALITY_LEVEL`; upscale 1-4 → out=in×scale, + clean 8-15 → out=in. Probe mirrors the verified Start sequence. **Built clean against the REAL + header on the runner SDK** (0 warnings, both effects + VSR — see "Compile-testing" above), plus a + stub build. Call sites stubbed to keep the tree green: `capture.cpp` passes quality `1` (ponytail + comment, T5/T6 thread real mode), `effects_smoke.cpp` updated. Original spec below: - selector `"VideoSuperRes"`, include the real header. - **BGRA u8 GPU in+out** — delete the BGR conversion and the per-pixel alpha-repack loop. - `Start(int qualityLevel, int scaleX10)`; set `NVVFX_QUALITY_LEVEL` (not `NVVFX_MODE`). diff --git a/native/shim/capture.cpp b/native/shim/capture.cpp index 0de117f..6bf92f6 100644 --- a/native/shim/capture.cpp +++ b/native/shim/capture.cpp @@ -413,7 +413,9 @@ void Capture::WorkerLoop() { const bool srWant = g_state.superResEnabled.load(std::memory_order_acquire); const int srScale = g_state.superResScale.load(std::memory_order_acquire); if (srWant && !superRes.IsReady()) { - if (!superRes.Start(srScale)) { + // ponytail: hardcoded quality 1 (VSR_Low upscale); T5/T6 thread the real + // QualityLevel (mode combo) once the ABI carries it. + if (!superRes.Start(1, srScale)) { std::lock_guard e(g_state.srErrMtx); const std::string& ne = superRes.LastError(); if (g_state.srError != ne) g_state.srError = ne; diff --git a/native/shim/shim.vcxproj b/native/shim/shim.vcxproj index 173ef29..abe4136 100644 --- a/native/shim/shim.vcxproj +++ b/native/shim/shim.vcxproj @@ -76,7 +76,7 @@ COS_HAS_MAXINE;%(PreprocessorDefinitions) - $(CosVfxSdkDir)\nvvfx\include;$(CosVfxSdkDir)\features\nvvfxgreenscreen\include;%(AdditionalIncludeDirectories) + $(CosVfxSdkDir)\nvvfx\include;$(CosVfxSdkDir)\features\nvvfxgreenscreen\include;$(CosVfxSdkDir)\features\nvvfxvideosuperres\include;%(AdditionalIncludeDirectories) diff --git a/native/shim/smoke/effects_smoke.cpp b/native/shim/smoke/effects_smoke.cpp index 311ab97..6b6c748 100644 --- a/native/shim/smoke/effects_smoke.cpp +++ b/native/shim/smoke/effects_smoke.cpp @@ -25,7 +25,7 @@ int main() { { std::string d; if (SuperRes::Probe(d)) { - SuperRes sr; assert(sr.Start(20)); + SuperRes sr; assert(sr.Start(1, 20)); // quality 1 = VSR_Low upscale, scale 2x std::vector in(640 * 480 * 4, 64), out; int ow = 0, oh = 0; bool ok = sr.ProcessFrame(in.data(), 640, 480, out, ow, oh); std::printf("SR ProcessFrame: %s -> %dx%d\n", ok ? "ok" : "fail", ow, oh); diff --git a/native/shim/superres.cpp b/native/shim/superres.cpp index 4f112cf..45b3ba4 100644 --- a/native/shim/superres.cpp +++ b/native/shim/superres.cpp @@ -9,73 +9,80 @@ #include "nvCVStatus.h" #include "nvCVImage.h" #include "nvVideoEffects.h" +#include "nvVFXVideoSuperRes.h" // real header: NVVFX_FX_VIDEO_SUPER_RES + NVVFX_QUALITY_LEVEL #include "vfx_paths.h" -// VFX 1.2.0.0 ships the super-res selector only in its per-feature header (NGC feature -// nvvfxvideosuperres). Define the documented literal if absent. NVIDIA VideoFX selector -// "SuperRes" confirmed by grepping the VFX 1.2.0.0 SDK doc tree. -#ifndef NVVFX_FX_SUPER_RES -#define NVVFX_FX_SUPER_RES "SuperRes" -#endif - struct SrImpl { NvVFX_Handle effect = nullptr; CUstream stream = nullptr; std::string modelDir; - NvCVImage srcGpu{}; // BGR u8 chunky GPU (input, w x h) - NvCVImage dstGpu{}; // BGR u8 chunky GPU (output, outW x outH) - NvCVImage dstCpu{}; // BGR u8 chunky CPU (downloaded) + NvCVImage srcGpu{}; // BGRA u8 chunky GPU (input, w x h) + NvCVImage dstGpu{}; // BGRA u8 chunky GPU (output, ow x oh) + NvCVImage dstCpu{}; // BGRA u8 chunky CPU (downloaded) NvCVImage stage{}; // BGRA u8 chunky GPU (upload staging, w x h) int w = 0, h = 0, ow = 0, oh = 0; bool loaded = false; }; -static void ScaledDims(int w, int h, int scaleX10, int& ow, int& oh) { - ow = (w * scaleX10) / 10; - oh = (h * scaleX10) / 10; +// QualityLevel 1-4 are the upscale modes; 8-15 (denoise/deblur) keep input size. +static bool IsUpscale(int q) { return q >= 1 && q <= 4; } + +static void OutDims(int w, int h, int q, int scaleX10, int& ow, int& oh) { + if (IsUpscale(q)) { ow = (w * scaleX10) / 10; oh = (h * scaleX10) / 10; } + else { ow = w; oh = h; } // denoise/deblur do not upscale (out == in) } SuperRes::SuperRes() = default; SuperRes::~SuperRes() { Stop(); } -// Probe loads the effect using mode 1 (VSR_Low), matching the mode Start() uses. -// This avoids a false-positive probe where mode 0 (bicubic) succeeds but mode 1 fails. +// Probe mirrors the verified Start sequence (create + quality + images + load) so a load-only +// capability check matches the runtime path. False here greys the effect off for all users. bool SuperRes::Probe(std::string& detail) { std::string binDir, modelDir, err; if (!vfx::ResolveSdkPaths(binDir, modelDir, err)) { detail = err; return false; } vfx::PointProxiesAt(binDir); NvVFX_Handle eff = nullptr; - if (NvVFX_CreateEffect(NVVFX_FX_SUPER_RES, &eff) != NVCV_SUCCESS || !eff) { - detail = "NvVFX_CreateEffect(SuperRes) failed (DLL/SDK load?)"; return false; + if (NvVFX_CreateEffect(NVVFX_FX_VIDEO_SUPER_RES, &eff) != NVCV_SUCCESS || !eff) { + detail = "NvVFX_CreateEffect(VideoSuperRes) failed (DLL/SDK load?)"; return false; } - NvVFX_SetString(eff, NVVFX_MODEL_DIRECTORY, modelDir.c_str()); - NvVFX_SetU32(eff, NVVFX_MODE, 1u); // VSR_Low; same mode as Start() to avoid false-positive - NvVFX_SetU32(eff, NVVFX_MAX_INPUT_WIDTH, 1920u); - NvVFX_SetU32(eff, NVVFX_MAX_INPUT_HEIGHT, 1080u); - NvCV_Status load = NvVFX_Load(eff); + CUstream stream = nullptr; + NvVFX_CudaStreamCreate(&stream); + NvVFX_SetString(eff, NVVFX_MODEL_DIRECTORY, modelDir.c_str()); // harmless for NGX + NvVFX_SetCudaStream(eff, NVVFX_CUDA_STREAM, stream); + NvVFX_SetU32(eff, NVVFX_QUALITY_LEVEL, 1u); // VSR_Low (upscale); same family Start uses + NvCVImage src{}, dst{}; + NvCV_Status load = NVCV_ERR_GENERAL; + if (NvCVImage_Alloc(&src, 640, 360, NVCV_BGRA, NVCV_U8, NVCV_CHUNKY, NVCV_GPU, 1) == NVCV_SUCCESS && + NvCVImage_Alloc(&dst, 1280, 720, NVCV_BGRA, NVCV_U8, NVCV_CHUNKY, NVCV_GPU, 1) == NVCV_SUCCESS) { + NvVFX_SetImage(eff, NVVFX_INPUT_IMAGE, &src); + NvVFX_SetImage(eff, NVVFX_OUTPUT_IMAGE, &dst); + load = NvVFX_Load(eff); + } + NvCVImage_Dealloc(&src); NvCVImage_Dealloc(&dst); NvVFX_DestroyEffect(eff); - if (load != NVCV_SUCCESS) { detail = "NvVFX_Load(SuperRes) failed (models missing?)"; return false; } - detail = "SuperRes available"; + if (stream) NvVFX_CudaStreamDestroy(stream); + if (load != NVCV_SUCCESS) { detail = "NvVFX_Load(VideoSuperRes) failed (models missing or GPU incompatible?)"; return false; } + detail = "VideoSuperRes available"; return true; } -bool SuperRes::Start(int scaleX10) { +bool SuperRes::Start(int qualityLevel, int scaleX10) { Stop(); - scaleX10_ = (scaleX10 == 15) ? 15 : 20; // only 1.5x / 2x supported here + qualityLevel_ = qualityLevel; + scaleX10_ = (scaleX10 == 15) ? 15 : 20; // only 1.5x / 2x supported (upscale modes) SrImpl* impl = new (std::nothrow) SrImpl(); if (!impl) { lastError_ = "out of memory"; return false; } std::string binDir, err; if (!vfx::ResolveSdkPaths(binDir, impl->modelDir, err)) { lastError_ = err; delete impl; return false; } vfx::PointProxiesAt(binDir); if (NvVFX_CudaStreamCreate(&impl->stream) != NVCV_SUCCESS) { lastError_ = "CudaStreamCreate failed"; delete impl; return false; } - if (NvVFX_CreateEffect(NVVFX_FX_SUPER_RES, &impl->effect) != NVCV_SUCCESS || !impl->effect) { + if (NvVFX_CreateEffect(NVVFX_FX_VIDEO_SUPER_RES, &impl->effect) != NVCV_SUCCESS || !impl->effect) { lastError_ = "CreateEffect failed"; NvVFX_CudaStreamDestroy(impl->stream); delete impl; return false; } NvVFX_SetString(impl->effect, NVVFX_MODEL_DIRECTORY, impl->modelDir.c_str()); NvVFX_SetCudaStream(impl->effect, NVVFX_CUDA_STREAM, impl->stream); - NvVFX_SetU32(impl->effect, NVVFX_MODE, 1u); // VSR_Low: fast, good quality for live webcam - // Scale is inferred by the effect from the ratio of output to input image dimensions; - // no explicit NvVFX_SetF32(NVVFX_SCALE, ...) call is required. + NvVFX_SetU32(impl->effect, NVVFX_QUALITY_LEVEL, static_cast(qualityLevel_)); + // Scale is inferred by the effect from the output/input dimension ratio; no scale param. impl_ = impl; ready_ = true; lastError_.clear(); return true; } @@ -93,9 +100,8 @@ void SuperRes::Stop() { } namespace { -// (Re)allocates GPU/CPU images for input w*h and output ow*oh, then loads the model. -// Scale is inferred by the effect from the ratio dstGpu.width/srcGpu.width. -// Returns NVCV_SUCCESS on success; called on first frame and on resolution change. +// (Re)allocates GPU/CPU images for input w*h and output ow*oh, then loads the model. Scale is +// inferred by the effect from dstGpu.width/srcGpu.width. Called on first frame and on size change. NvCV_Status EnsureImages(SrImpl* impl, int w, int h, int ow, int oh) { if (impl->loaded && impl->w == w && impl->h == h && impl->ow == ow && impl->oh == oh) return NVCV_SUCCESS; NvCVImage_Dealloc(&impl->srcGpu); @@ -103,15 +109,12 @@ NvCV_Status EnsureImages(SrImpl* impl, int w, int h, int ow, int oh) { NvCVImage_Dealloc(&impl->dstCpu); NvCVImage_Dealloc(&impl->stage); NvCV_Status s; - s = NvCVImage_Alloc(&impl->srcGpu, w, h, NVCV_BGR, NVCV_U8, NVCV_CHUNKY, NVCV_GPU, 1); if (s != NVCV_SUCCESS) return s; - s = NvCVImage_Alloc(&impl->dstGpu, ow, oh, NVCV_BGR, NVCV_U8, NVCV_CHUNKY, NVCV_GPU, 1); if (s != NVCV_SUCCESS) return s; - s = NvCVImage_Alloc(&impl->dstCpu, ow, oh, NVCV_BGR, NVCV_U8, NVCV_CHUNKY, NVCV_CPU, 1); if (s != NVCV_SUCCESS) return s; + s = NvCVImage_Alloc(&impl->srcGpu, w, h, NVCV_BGRA, NVCV_U8, NVCV_CHUNKY, NVCV_GPU, 1); if (s != NVCV_SUCCESS) return s; + s = NvCVImage_Alloc(&impl->dstGpu, ow, oh, NVCV_BGRA, NVCV_U8, NVCV_CHUNKY, NVCV_GPU, 1); if (s != NVCV_SUCCESS) return s; + s = NvCVImage_Alloc(&impl->dstCpu, ow, oh, NVCV_BGRA, NVCV_U8, NVCV_CHUNKY, NVCV_CPU, 1); if (s != NVCV_SUCCESS) return s; s = NvCVImage_Alloc(&impl->stage, w, h, NVCV_BGRA, NVCV_U8, NVCV_CHUNKY, NVCV_GPU, 1); if (s != NVCV_SUCCESS) return s; s = NvVFX_SetImage(impl->effect, NVVFX_INPUT_IMAGE, &impl->srcGpu); if (s != NVCV_SUCCESS) return s; s = NvVFX_SetImage(impl->effect, NVVFX_OUTPUT_IMAGE, &impl->dstGpu); if (s != NVCV_SUCCESS) return s; - // Set max input dimensions to pre-allocate internal buffers and enable load. - s = NvVFX_SetU32(impl->effect, NVVFX_MAX_INPUT_WIDTH, static_cast(w)); if (s != NVCV_SUCCESS) return s; - s = NvVFX_SetU32(impl->effect, NVVFX_MAX_INPUT_HEIGHT, static_cast(h)); if (s != NVCV_SUCCESS) return s; s = NvVFX_Load(impl->effect); if (s != NVCV_SUCCESS) return s; impl->w = w; impl->h = h; impl->ow = ow; impl->oh = oh; impl->loaded = true; return NVCV_SUCCESS; @@ -121,32 +124,26 @@ NvCV_Status EnsureImages(SrImpl* impl, int w, int h, int ow, int oh) { bool SuperRes::ProcessFrame(const uint8_t* bgra, int w, int h, std::vector& out, int& outW, int& outH) { auto* impl = static_cast(impl_); if (!impl || !impl->effect || !bgra || w <= 0 || h <= 0) return false; - int ow, oh; ScaledDims(w, h, scaleX10_, ow, oh); + int ow, oh; OutDims(w, h, qualityLevel_, scaleX10_, ow, oh); if (NvCV_Status es = EnsureImages(impl, w, h, ow, oh); es != NVCV_SUCCESS) { lastError_ = std::string("EnsureImages/Load failed: ") + NvCV_GetErrorStringFromCode(es); ready_ = false; return false; } - // Upload: CPU BGRA -> GPU BGR via staging buffer (NvCVImage_Transfer handles format conversion). + // Upload: CPU BGRA -> GPU BGRA via staging buffer (same format, no conversion). NvCVImage src{}; NvCVImage_Init(&src, w, h, w * 4, const_cast(bgra), NVCV_BGRA, NVCV_U8, NVCV_CHUNKY, NVCV_CPU); if (NvCVImage_Transfer(&src, &impl->srcGpu, 1.0f, impl->stream, &impl->stage) != NVCV_SUCCESS) { lastError_ = "upload failed"; return false; } if (NvVFX_Run(impl->effect, 0) != NVCV_SUCCESS) { lastError_ = "NvVFX_Run failed"; return false; } - // Download: GPU BGR (ow x oh) -> CPU BGR. + // Download: GPU BGRA (ow x oh) -> CPU BGRA. if (NvCVImage_Transfer(&impl->dstGpu, &impl->dstCpu, 1.0f, impl->stream, nullptr) != NVCV_SUCCESS) { lastError_ = "download failed"; return false; } if (NvVFX_CudaStreamSynchronize(impl->stream) != NVCV_SUCCESS) { lastError_ = "stream sync failed"; return false; } - // Write-back: BGR CPU -> BGRA output (alpha = 0xFF; fresh upscaled buffer, opaque). - out.assign(static_cast(ow) * oh * 4, 0xFF); + // Write-back: BGRA CPU -> packed BGRA out, honoring source pitch. Alpha passes through VSR + // (opaque in passthrough; the green-screen matte if AIGS ran upstream). + out.resize(static_cast(ow) * oh * 4); const uint8_t* d = static_cast(impl->dstCpu.pixels); const int dpitch = impl->dstCpu.pitch; - for (int y = 0; y < oh; ++y) { - const uint8_t* drow = d + static_cast(dpitch) * y; - uint8_t* prow = out.data() + static_cast(ow) * 4 * y; - for (int x = 0; x < ow; ++x) { - prow[x * 4 + 0] = drow[x * 3 + 0]; // B - prow[x * 4 + 1] = drow[x * 3 + 1]; // G - prow[x * 4 + 2] = drow[x * 3 + 2]; // R - // Alpha already 0xFF from the out.assign() pre-fill (opaque upscaled buffer). - } - } + for (int y = 0; y < oh; ++y) + std::memcpy(out.data() + static_cast(ow) * 4 * y, + d + static_cast(dpitch) * y, static_cast(ow) * 4); outW = ow; outH = oh; return true; } @@ -156,7 +153,7 @@ bool SuperRes::ProcessFrame(const uint8_t* bgra, int w, int h, std::vector&, int&, int&) { return false; } #endif diff --git a/native/shim/superres.h b/native/shim/superres.h index 37e2610..583175f 100644 --- a/native/shim/superres.h +++ b/native/shim/superres.h @@ -3,23 +3,27 @@ #include #include -// Wraps the Maxine VFX Super Resolution effect. Upscales a BGRA frame by 1.5x or 2x -// into a freshly sized BGRA output buffer (alpha = 0xFF). Without COS_HAS_MAXINE this -// is a never-ready stub. scaleX10: 15 => 1.5x, 20 => 2x. +// Wraps the Maxine VFX Video Super Resolution effect (NGX, selector "VideoSuperRes"). +// QualityLevel doubles as the mode selector: +// 1-4 = upscale (VSR Low/Med/High/Ultra) -> output = input * scale (scaleX10: 15=1.5x, 20=2x) +// 8-11 = denoise, 12-15 = deblur -> output = input size (no upscale) +// In and out are BGRA u8 on the GPU (matches the capture pipeline; no BGR conversion). Alpha +// flows through VSR untouched. Without COS_HAS_MAXINE this is a never-ready passthrough stub. class SuperRes { public: SuperRes(); ~SuperRes(); static bool Probe(std::string& detail); - bool Start(int scaleX10); + bool Start(int qualityLevel, int scaleX10); void Stop(); // Reads w*h BGRA from 'bgra', writes outW*outH BGRA into 'out'. Returns false on failure - // (out untouched). outW/outH are w*scale, h*scale. + // (out untouched). For upscale modes outW/outH are w*scale,h*scale; otherwise w,h. bool ProcessFrame(const uint8_t* bgra, int w, int h, std::vector& out, int& outW, int& outH); bool IsReady() const { return ready_; } const std::string& LastError() const { return lastError_; } private: bool ready_ = false; + int qualityLevel_ = 1; int scaleX10_ = 20; std::string lastError_; void* impl_ = nullptr; From 554ff64aa22a27ba561f70cb0aa64ba911b8646c Mon Sep 17 00:00:00 2001 From: opariffazman Date: Wed, 24 Jun 2026 15:26:37 +0800 Subject: [PATCH 22/26] =?UTF-8?q?feat(#15):=20VSR=20mode/quality=20through?= =?UTF-8?q?=20the=20stack=20=E2=80=94=20ABI,=20capture,=20MVVM,=20UI=20(T4?= =?UTF-8?q?-T7)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Wire the VSR QualityLevel (mode selector) end to end on top of the rewritten SuperRes effect. T4 (C ABI): add super_res_quality_level to CosParams (C shim.h + the [StructLayout] mirror in PInvokeShim) and the ShimParams record. Exactly the 9 cos_* exports unchanged. T5 (capture worker): SetSuperRes(enabled, qualityLevel, scale); a superResQuality atomic; the worker restarts the effect when quality or scale changes live (both are baked at Start). Replaces the T2 hardcoded quality 1. T6 (managed + Core + tests): EffectSettings persists SuperResMode + SuperResQuality + SuperResScale (dropping the bool). MainViewModel exposes SuperResModeIndex (Off/Upscale/Denoise/Deblur) + SuperResQualityIndex (Low/Med/High/Ultra) + SuperResScaleIndex, mapping mode+quality to a single QualityLevel (Upscale 1-4, Denoise 8-11, Deblur 12-15) and forcing scale to 0 for the clean modes. Live push + persistence round-trip. 57 xUnit tests green. T7 (App UI): replace the SR toggle + scale combo with mode / quality / upscale factor combos (quality enabled when mode != Off, factor only for Upscale). Restructure MainWindow into a ScrollViewer (row 0) with the license-required Maxine attribution footer pinned outside it (row 1, Auto) so it stays visible once the stacked controls overflow the 400x720 window. Verified: shim SDK config builds clean against the runner SDK (export-verify GreenScreen + GazeRedirection + VideoSuperRes, no "not built in"); App builds 0 warnings; Core tests 57/57. Runtime VSR remains the T9 human gate (3090). Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01SfcBU4WRAy7HMNgtm92ogq --- native/shim/capture.cpp | 15 ++- native/shim/capture.h | 5 +- native/shim/shim.cpp | 2 +- native/shim/shim.h | 3 +- src/CameraOnScreen.App/MainWindow.xaml | 126 ++++++++++-------- src/CameraOnScreen.App/MainWindow.xaml.cs | 5 + src/CameraOnScreen.App/Native/PInvokeShim.cs | 2 + src/CameraOnScreen.Core/Config/Models.cs | 5 +- src/CameraOnScreen.Core/Native/Contracts.cs | 3 +- .../ViewModels/MainViewModel.cs | 26 ++-- .../ViewModels/MainViewModelTests.cs | 28 +++- 11 files changed, 143 insertions(+), 77 deletions(-) diff --git a/native/shim/capture.cpp b/native/shim/capture.cpp index 6bf92f6..2f68561 100644 --- a/native/shim/capture.cpp +++ b/native/shim/capture.cpp @@ -51,6 +51,7 @@ struct CaptureState { std::atomic superResEnabled{false}; std::atomic superResScale{20}; // 15 or 20 + std::atomic superResQuality{1}; // VSR QualityLevel (mode): 1-4 / 8-11 / 12-15 std::atomic superResActive{false}; std::mutex srErrMtx; // leaf lock std::string srError; @@ -310,6 +311,7 @@ void Capture::WorkerLoop() { Aigs aigs; EyeContact eyeContact; SuperRes superRes; + int srAppliedQuality = 0, srAppliedScale = 0; // last (quality,scale) Start ran with; for live restart IMFSourceReader* reader = nullptr; int width = 0, height = 0; @@ -412,13 +414,17 @@ void Capture::WorkerLoop() { // Super Resolution runs LAST; it changes the frame dimensions. const bool srWant = g_state.superResEnabled.load(std::memory_order_acquire); const int srScale = g_state.superResScale.load(std::memory_order_acquire); + const int srQuality = g_state.superResQuality.load(std::memory_order_acquire); + // QualityLevel + scale are baked at Start; restart the effect if either changed live. + if (srWant && superRes.IsReady() && (srQuality != srAppliedQuality || srScale != srAppliedScale)) + superRes.Stop(); if (srWant && !superRes.IsReady()) { - // ponytail: hardcoded quality 1 (VSR_Low upscale); T5/T6 thread the real - // QualityLevel (mode combo) once the ABI carries it. - if (!superRes.Start(1, srScale)) { + if (!superRes.Start(srQuality, srScale)) { std::lock_guard e(g_state.srErrMtx); const std::string& ne = superRes.LastError(); if (g_state.srError != ne) g_state.srError = ne; + } else { + srAppliedQuality = srQuality; srAppliedScale = srScale; } } else if (!srWant && superRes.IsReady()) { superRes.Stop(); @@ -550,8 +556,9 @@ std::string Capture::EyeContactError() const { return g_state.ecError; } -void Capture::SetSuperRes(bool enabled, int scaleX10) { +void Capture::SetSuperRes(bool enabled, int qualityLevel, int scaleX10) { g_state.superResScale.store(scaleX10 == 15 ? 15 : 20, std::memory_order_release); + g_state.superResQuality.store(qualityLevel, std::memory_order_release); g_state.superResEnabled.store(enabled, std::memory_order_release); } bool Capture::SuperResActive() const { diff --git a/native/shim/capture.h b/native/shim/capture.h index 4815bfc..f0c8613 100644 --- a/native/shim/capture.h +++ b/native/shim/capture.h @@ -42,8 +42,9 @@ class Capture { bool EyeContactActive() const; // true only while gaze redirection is transforming frames std::string EyeContactError() const; // empty when none - // Toggles Super Resolution + scale (15=1.5x, 20=2x). Thread-safe; worker owns the object. - void SetSuperRes(bool enabled, int scaleX10); + // Toggles Super Resolution + VSR QualityLevel (mode) + scale (15=1.5x, 20=2x; upscale modes + // only). Thread-safe; worker owns the object. + void SetSuperRes(bool enabled, int qualityLevel, int scaleX10); bool SuperResActive() const; std::string SuperResError() const; diff --git a/native/shim/shim.cpp b/native/shim/shim.cpp index a560af1..104130e 100644 --- a/native/shim/shim.cpp +++ b/native/shim/shim.cpp @@ -65,7 +65,7 @@ COS_API void cos_set_params(const CosParams* p) { g_capture.SetGreenScreen(p->green_screen_enabled != 0); g_capture.SetMatteParams(p->green_screen_expand, p->green_screen_feather); g_capture.SetEyeContact(p->eye_contact_enabled != 0); - g_capture.SetSuperRes(p->super_res_enabled != 0, p->super_res_scale); + g_capture.SetSuperRes(p->super_res_enabled != 0, p->super_res_quality_level, p->super_res_scale); } COS_API void cos_start(void) { g_capture.Start(g_cameraId); g_running = true; } diff --git a/native/shim/shim.h b/native/shim/shim.h index fc28c39..83cad75 100644 --- a/native/shim/shim.h +++ b/native/shim/shim.h @@ -24,7 +24,8 @@ typedef struct { double eye_contact_sensitivity; double eye_contact_look_away_range; int super_res_enabled; - int super_res_scale; // 0=off, 15=1.5x, 20=2x + int super_res_scale; // 0=off, 15=1.5x, 20=2x (upscale modes only) + int super_res_quality_level; // VSR QualityLevel: 1-4 upscale, 8-11 denoise, 12-15 deblur } CosParams; typedef struct { diff --git a/src/CameraOnScreen.App/MainWindow.xaml b/src/CameraOnScreen.App/MainWindow.xaml index 5494473..aaecb50 100644 --- a/src/CameraOnScreen.App/MainWindow.xaml +++ b/src/CameraOnScreen.App/MainWindow.xaml @@ -1,67 +1,87 @@ - + xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml" + xmlns:local="using:CameraOnScreen.App"> + - - - - + - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - - - - - - - - - - - - - - - - + + + + + - -