diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 23a01ba..1366f7e 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -16,6 +16,16 @@ jobs: - name: Checkout uses: actions/checkout@v4 + # The runner host has an x86 .NET (no SDK) at "C:\Program Files (x86)\dotnet" + # that can shadow the x64 SDK on PATH -> `dotnet build` fails "No .NET SDKs were + # found". Prepend the x64 install so the SDK wins for all later steps. + - name: Prefer x64 .NET on PATH + shell: pwsh + run: | + if (Test-Path "C:\Program Files\dotnet\dotnet.exe") { + "C:\Program Files\dotnet" | Out-File -FilePath $env:GITHUB_PATH -Append -Encoding utf8 + } + - name: Build native shim (SDK config, x64 Release) shell: pwsh run: | diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index d3c4393..5716dba 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -25,6 +25,16 @@ jobs: - name: Checkout uses: actions/checkout@v4 + # The runner host has an x86 .NET (no SDK) at "C:\Program Files (x86)\dotnet" + # that can shadow the x64 SDK on PATH -> `dotnet build` (inside build-installer.ps1) + # fails "No .NET SDKs were found". Prepend the x64 install for all later steps. + - name: Prefer x64 .NET on PATH + shell: pwsh + run: | + if (Test-Path "C:\Program Files\dotnet\dotnet.exe") { + "C:\Program Files\dotnet" | Out-File -FilePath $env:GITHUB_PATH -Append -Encoding utf8 + } + # Self-hosted runner runs as NETWORK SERVICE, but _work is owned by Administrators, # so git's dubious-ownership guard blocks `gh release create --verify-tag` (which # shells out to git). Trust the workspace under THIS account's gitconfig — an diff --git a/.gitignore b/.gitignore index fcf8144..effcd11 100644 --- a/.gitignore +++ b/.gitignore @@ -1,6 +1,7 @@ [Bb]in/ [Oo]bj/ dist/ +.env *.user .vs/ *.suo diff --git a/CLAUDE.md b/CLAUDE.md index 0404397..03716b8 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -81,6 +81,7 @@ Three projects: `src/CameraOnScreen.Core` (pure .NET 8 logic, no WinUI/Win32 typ ## Maxine SDKs (not in repo; redistribution governed by the 2025 NVIDIA Software License + Open/Community Model Licenses) - **VFX** green screen (`nvvfxgreenscreen`) and **AR** eye contact (gaze) are separate NVIDIA products. No import `.lib` — link via the SDKs' proxy stubs (`nvVideoEffectsProxy.cpp`, `nvCVImageProxy.cpp`, `nvARProxy.cpp`) compiled into the shim. Models are prebuilt per-arch TensorRT engines; the bundle ships **sm75/86/89/100** (Turing/Ampere/Ada/Blackwell) — fetched from NGC by arch. **sm86 is the only arch verified on real silicon (RTX 3090); the others ship best-effort and grey out gracefully if an engine fails to deserialize.** +- **VFX feature catalog — verify availability + the REAL API BEFORE coding an effect (cost ~6 wasted tasks once).** Authoritative installable set: `\features\install_feature.ps1 -list_features` (NGC key via `NGC_CLI_API_KEY`). VFX 1.2.0.0 = `aigsrelighting, backgroundblur, denoising, greenscreen, relighting, transfer, upscale, videosuperres` (8). **There is NO `nvvfxartifactreduction`** — it's in the script's help text but NOT the catalog (removed since older VFX SDKs); coding against it = a dead, always-unavailable effect. **Super Resolution = NGX VSR** (`nvvfxvideosuperres`; ships `nvngx_vsr.dll`, NOT a per-arch TRT engine — "no models" is normal): real header `features\nvvfxvideosuperres\include\nvVFXVideoSuperRes.h` → selector `NVVFX_FX_VIDEO_SUPER_RES "VideoSuperRes"` + param `NVVFX_QUALITY_LEVEL` (NOT `"SuperRes"`/`NVVFX_MODE`/scale-from-dims). A feature's real selector + params + image format live ONLY in its per-feature header (`features\\include\*.h`), installed on demand — never infer them; if the NGC key is the gate, get it up front. - **App-relative discovery** (`paths.{h,cpp}` `ShimModuleDir()` via `GetModuleHandleExW(FROM_ADDRESS)`, CWD-independent): both resolvers gain an `\maxine\` tier so a shipped app finds the runtimes beside the exe with no env vars. Single shared co-versioned `maxine\` root (one TRT/CUDA runtime, dispatcher + feature DLLs, `models\vfx` + `models\ar`). - **Stage + Bundler.** Two steps now (dispatcher + per-feature-DLL SDK layout): (1) `scripts/assemble-maxine-stage.ps1` curates a co-versioned flat **stage** from the VFX 1.2.0.0 + AR 1.1.1.0 SDK trees (shared DLLs from VFX; AR dispatcher + 3 gaze feature DLLs from AR; the 6 license files; multi-arch engines via `scripts/fetch-maxine-engines.ps1`, NGC key). (2) `scripts/bundle-maxine.ps1 -OutDir X -MaxineStage ` PRUNES the stage to the manifest's verified load-closure (`native/shim/bundle/maxine-manifest.psd1`; the 19-DLL `Dlls` list was produced by `native/shim/smoke/trace_closure.cpp`) + model globs + required `LicenseFiles`, into `\maxine\`. Co-version is enforced at stage assembly, not by the bundler. `trace_closure`/`bundle_probe` re-run against the produced bundle (`COS_*` unset → both effects load) is the verify gate. End-user need: an **RTX GPU + recent driver**; no NVIDIA account or SDK download. - **Installer** (`scripts/bundle-maxine.ps1` consumer): `scripts/build-installer.ps1 -MaxineStage ` diff --git a/docs/superpowers/plans/2026-06-24-ai-sharpness-resolution-phase1.md b/docs/superpowers/plans/2026-06-24-ai-sharpness-resolution-phase1.md new file mode 100644 index 0000000..547da3a --- /dev/null +++ b/docs/superpowers/plans/2026-06-24-ai-sharpness-resolution-phase1.md @@ -0,0 +1,1498 @@ +# AI Sharpness & Resolution (Phase 1) Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Add Maxine VFX **Artifact Reduction** + **Super Resolution** effects and a **real fps counter** to the overlay pipeline, giving a sharper/higher-resolution image than the 1080p30 webcam delivers. + +**Architecture:** Two new VFX effects run on the existing capture worker thread, mirroring `aigs.cpp` (green screen): Artifact Reduction first (cleans MJPG artifacts, no size change), Super Resolution last (upscales, larger output buffer). Both live in the same VFX 1.2.0.0 runtime already used for green screen — **no new co-version**. The C ABI grows three `CosParams` fields + two `CosCaps` gates; the managed MVVM chain gains toggles, capability gates, persistence, and live-param push, exactly like the green-screen props. + +**Tech Stack:** C++ (Media Foundation + Maxine VFX, `COS_HAS_MAXINE`), C# .NET 8 (Core + WinUI 3 App), CommunityToolkit.Mvvm, xUnit. + +**Spec:** `docs/superpowers/specs/2026-06-23-camera-on-screen-ai-sharpness-resolution-design.md`. Phase 2 (fps interpolation) = GitHub issue #13. Present-path minification fix is **deferred** (spec §11) — not in this plan. + +## Global Constraints + +- **Builds & tests must be pristine — 0 warnings** (CI enforces `/warnaserror` + `TreatWarningsAsErrors`). +- **Build the native shim BEFORE the App** (App copies `native/shim/x64/$(Configuration)/CameraOnScreen.Shim.dll`). Build the **SDK config last** before running (deploy-the-right-shim). +- **C ABI struct parity is byte-for-byte on x64.** `CosStatus`/`CosParams`/`CosCaps` in `shim.h` and their `[StructLayout(Sequential)]` mirrors in `PInvokeShim` must match. After ABI change verify exports: `dumpbin /exports` must still list exactly the **9** `cos_*` functions. +- **Maxine code is behind `COS_HAS_MAXINE`**; without the SDK the effect compiles as a passthrough stub (never ready). New VFX effects use the **same VFX runtime** as green screen — do NOT introduce a new TensorRT/CUDA runtime. +- **Effects run on the capture worker thread only.** The UI flips atomic enable flags; status crosses threads via atomics + **leaf locks** (never nested under `g_state.mtx` / `g_lifecycleMtx`). +- **Super Res output is capped at 2×** → max frame 3840×2160. The managed `_frameBuf` is pre-sized to that. +- Shim build (PowerShell): `& "C:/Program Files (x86)/Microsoft Visual Studio/2022/BuildTools/MSBuild/Current/Bin/MSBuild.exe" native/shim/shim.vcxproj /p:Configuration=Debug /p:Platform=x64` +- App build: `dotnet build src/CameraOnScreen.App/CameraOnScreen.App.csproj -t:Rebuild` +- Core tests: `dotnet test tests/CameraOnScreen.Core.Tests/CameraOnScreen.Core.Tests.csproj` (single: append `--filter "FullyQualifiedName~Name"`). + +--- + +## File Structure + +**New files:** +- `native/shim/fps_counter.h` — header-only rolling FPS estimator (no Win32/Maxine deps; unit-testable). +- `native/shim/vfx_paths.h` / `vfx_paths.cpp` — shared VFX SDK path resolver + proxy pointer, reused by the two new effects (green-screen `aigs.cpp` keeps its own copy untouched). +- `native/shim/artifactreduction.h` / `artifactreduction.cpp` — `ArtifactReduction` effect (VFX `NVVFX_FX_ARTIFACT_REDUCTION`), in-place, no size change. +- `native/shim/superres.h` / `superres.cpp` — `SuperRes` effect (VFX `NVVFX_FX_SUPER_RES`), larger output buffer. +- `native/shim/smoke/fps_smoke.cpp` — asserts on the FPS estimator (no GPU). +- `native/shim/smoke/effects_smoke.cpp` — dev-box smoke: AR + SR `Start`+`ProcessFrame` on a synthetic frame (needs SDK+GPU). + +**Modified files:** +- `native/shim/capture.h` / `capture.cpp` — fps counter, AR/SR enable atomics + Set methods + active/error state, worker-chain insertion. +- `native/shim/shim.h` — `CosParams` +3 fields, `CosCaps` +2 gates. +- `native/shim/shim.cpp` — real fps in `cos_get_status`; wire AR/SR in `cos_set_params` + `cos_query_capabilities`. +- `native/shim/shim.vcxproj` — compile the four new `.cpp` files. +- `src/CameraOnScreen.Core/Native/Contracts.cs` — `ShimParams` +3, `ShimCapabilities` +2. +- `src/CameraOnScreen.Core/Native/FakeShim.cs` — new capability gates. +- `src/CameraOnScreen.Core/Config/Models.cs` — `EffectSettings` +3. +- `src/CameraOnScreen.Core/Orchestration/Orchestrator.cs` — AR/SR availability + gating in `ApplyParams`. +- `src/CameraOnScreen.Core/ViewModels/MainViewModel.cs` — props, `BuildParams`, `LoadFrom`, `ToAppConfig`, live-push partials. +- `src/CameraOnScreen.App/Native/PInvokeShim.cs` — struct mirrors + `SetParams`/`QueryCapabilities` mapping. +- `src/CameraOnScreen.App/MainWindow.xaml` — two toggles + scale combo. +- `src/CameraOnScreen.App/MainWindow.xaml.cs` — `_frameBuf` pre-size to 4K. +- `scripts/assemble-maxine-stage.ps1` + `native/shim/bundle/maxine-manifest.psd1` — bundle AR/SR feature DLLs + models. + +--- + +## Task 1: Real fps counter (native) + +Replaces the hardcoded `30.0` stub so the user can see effect cost. Fully isolated: no ABI change, no managed change (the managed side already plumbs `Fps`). + +**Files:** +- Create: `native/shim/fps_counter.h` +- Create: `native/shim/smoke/fps_smoke.cpp` +- Modify: `native/shim/capture.h`, `native/shim/capture.cpp`, `native/shim/shim.cpp` + +**Interfaces:** +- Produces: `class FpsCounter { double Sample(double nowSec, uint64_t totalFrames); void Reset(); }` and `double Capture::MeasuredFps() const;` + +- [ ] **Step 1: Write the failing test** — `native/shim/smoke/fps_smoke.cpp` + +```cpp +// Standalone smoke for FpsCounter. Build: cl /EHsc /I.. fps_smoke.cpp +#include +#include +#include +#include "../fps_counter.h" + +static bool near(double a, double b) { return std::fabs(a - b) < 1e-6; } + +int main() { + FpsCounter c; + assert(near(c.Sample(0.0, 0), 0.0)); // first sample primes, no estimate yet + assert(near(c.Sample(1.0, 30), 30.0)); // 30 frames in 1.0s -> 30 fps + assert(near(c.Sample(1.2, 45), 30.0)); // 0.2s < min interval -> keep last estimate + assert(near(c.Sample(2.0, 90), 56.25)); // 45 frames over 0.8s (since last accepted at t=1.0) + c.Reset(); + assert(near(c.Sample(5.0, 999), 0.0)); // reset re-primes + std::puts("fps_smoke OK"); + return 0; +} +``` + +- [ ] **Step 2: Run it to verify it fails** + +Run: `cl /EHsc /I native/shim native/shim/smoke/fps_smoke.cpp /Fe:build/fps_smoke.exe` (from a VS Developer prompt) +Expected: FAIL to compile — `fps_counter.h` does not exist. + +- [ ] **Step 3: Implement `native/shim/fps_counter.h`** + +```cpp +#pragma once +#include + +// Rolling FPS estimate. SINGLE-CONSUMER (the status poll thread) — not internally +// synchronized. Feed a monotonically increasing cumulative frame count plus a +// monotonic clock reading in seconds. Recomputes over the elapsed window, but only +// refreshes once at least kMinInterval has passed so a fast poll can't divide by ~0. +class FpsCounter { +public: + double Sample(double nowSec, uint64_t totalFrames) { + if (!started_) { started_ = true; lastSec_ = nowSec; lastFrames_ = totalFrames; return fps_; } + const double dt = nowSec - lastSec_; + if (dt < kMinInterval) return fps_; // too soon: keep the last estimate + fps_ = static_cast(totalFrames - lastFrames_) / dt; + lastSec_ = nowSec; + lastFrames_ = totalFrames; + return fps_; + } + void Reset() { started_ = false; fps_ = 0.0; lastSec_ = 0.0; lastFrames_ = 0; } +private: + static constexpr double kMinInterval = 0.5; // refresh at most ~2x/sec + bool started_ = false; + double lastSec_ = 0.0; + double fps_ = 0.0; + uint64_t lastFrames_ = 0; +}; +``` + +- [ ] **Step 4: Run the smoke to verify it passes** + +Run: `cl /EHsc /I native/shim native/shim/smoke/fps_smoke.cpp /Fe:build/fps_smoke.exe && build\fps_smoke.exe` +Expected: prints `fps_smoke OK`, exit 0. + +- [ ] **Step 5: Wire the counter into capture** — `native/shim/capture.h` + +Add to the public section of `class Capture`: + +```cpp + // Measured frames-per-second over a rolling window (status polling). Thread-safe to + // call from the single status-poll thread; 0 until the first interval elapses. + double MeasuredFps() const; +``` + +- [ ] **Step 6: Implement counter state + increment** — `native/shim/capture.cpp` + +At the top includes add: + +```cpp +#include +#include "fps_counter.h" +``` + +In `struct CaptureState` add: + +```cpp + std::atomic framesProduced{0}; // bumped by worker after each published frame + FpsCounter fpsCounter; // read only by MeasuredFps (status-poll thread) +``` + +In `WorkerLoop`, immediately after `g_state.hasNewFrame = true;` (the line at the end of the publish block, ~line 362) add: + +```cpp + g_state.framesProduced.fetch_add(1, std::memory_order_release); +``` + +In `Capture::Start`, inside the `g_state.mtx` block that resets frame state (next to `g_state.hasNewFrame = false;`) add: + +```cpp + g_state.framesProduced.store(0, std::memory_order_release); + g_state.fpsCounter.Reset(); +``` + +At the end of the file add the method: + +```cpp +double Capture::MeasuredFps() const { + using clock = std::chrono::steady_clock; + const double nowSec = + std::chrono::duration(clock::now().time_since_epoch()).count(); + const uint64_t frames = g_state.framesProduced.load(std::memory_order_acquire); + return g_state.fpsCounter.Sample(nowSec, frames); +} +``` + +- [ ] **Step 7: Use it in `cos_get_status`** — `native/shim/shim.cpp` + +Replace: + +```cpp + out->fps = g_running ? 30.0 : 0.0; // still a stub count (documented) +``` + +with: + +```cpp + out->fps = g_running ? g_capture.MeasuredFps() : 0.0; +``` + +- [ ] **Step 8: Build the shim and verify clean + exports unchanged** + +Run (PowerShell): +``` +& "C:/Program Files (x86)/Microsoft Visual Studio/2022/BuildTools/MSBuild/Current/Bin/MSBuild.exe" native/shim/shim.vcxproj /p:Configuration=Debug /p:Platform=x64 +dumpbin /exports native/shim/x64/Debug/CameraOnScreen.Shim.dll | Select-String cos_ +``` +Expected: 0 warnings; exactly 9 `cos_*` exports. + +- [ ] **Step 9: Commit** + +```bash +git add native/shim/fps_counter.h native/shim/smoke/fps_smoke.cpp native/shim/capture.h native/shim/capture.cpp native/shim/shim.cpp +git commit -m "feat(shim): real measured fps in cos_get_status (replaces 30.0 stub)" +``` + +--- + +## Task 2: Shared VFX path resolver (single source of truth) + +Extract the VFX SDK path + proxy-pointer logic into one module that **green screen, Artifact Reduction, and Super Resolution all use**. `g_nvVFXSDKPath` ownership **moves into `vfx_paths.cpp`**; `aigs.cpp` drops its private copies and calls the shared helpers. This is the fully-DRY choice (user decision) — it edits the proven green-screen path, so it carries a **green-screen re-verify gate**. + +**Files:** +- Create: `native/shim/vfx_paths.h`, `native/shim/vfx_paths.cpp` +- Modify: `native/shim/aigs.cpp` (use shared helpers; remove its private `ResolveSdkPaths`/`PointProxiesAt` + its `g_nvVFXSDKPath` definition), `native/shim/shim.vcxproj` +- Verify (human gate): green screen still loads + mattes after the refactor. + +**Interfaces:** +- Produces: `namespace vfx { bool ResolveSdkPaths(std::string& binDir, std::string& modelDir, std::string& err); void PointProxiesAt(const std::string& binDir); }`. `g_nvVFXSDKPath` is now **defined** in `vfx_paths.cpp` (was `aigs.cpp`). + +- [ ] **Step 1: Create `native/shim/vfx_paths.h`** + +```cpp +#pragma once +#include + +// Shared VFX-SDK runtime discovery for ALL VFX effects (Green Screen, Artifact +// Reduction, Super Resolution). Single source of truth — aigs.cpp uses these too. +// Model subdir is models\vfx in the bundled layout. +namespace vfx { +// Fills binDir (holds NVVideoEffects.dll) and modelDir. Order: COS_VFX_RUNTIME_DIR +// (flat) -> COS_VFX_SDK_DIR (legacy \bin) -> \maxine (bundled). false if none. +bool ResolveSdkPaths(std::string& binDir, std::string& modelDir, std::string& err); +// Points the VFX proxy global (g_nvVFXSDKPath, defined in vfx_paths.cpp) at binDir. +void PointProxiesAt(const std::string& binDir); +} +``` + +- [ ] **Step 2: Create `native/shim/vfx_paths.cpp`** (owns `g_nvVFXSDKPath`; logic moved from `aigs.cpp`) + +```cpp +#include "vfx_paths.h" +#ifdef COS_HAS_MAXINE +#include +#include "paths.h" + +// The proxy stub (nvVideoEffectsProxy.cpp) declares this extern; we own the single +// definition here (moved out of aigs.cpp). Non-null => dir holding NVVideoEffects.dll, +// which the proxy passes to SetDllDirectory before LoadLibrary. +char* g_nvVFXSDKPath = nullptr; + +namespace vfx { +bool ResolveSdkPaths(std::string& binDir, std::string& modelDir, std::string& err) { + char buf[1024] = {0}; + DWORD n = GetEnvironmentVariableA("COS_VFX_RUNTIME_DIR", buf, sizeof(buf)); + if (n > 0 && n < sizeof(buf)) { + std::string root(buf, n); + if (!root.empty() && (root.back() == '\\' || root.back() == '/')) root.pop_back(); + binDir = root; modelDir = root + "\\models"; return true; + } + n = GetEnvironmentVariableA("COS_VFX_SDK_DIR", buf, sizeof(buf)); + if (n > 0 && n < sizeof(buf)) { + std::string root(buf, n); + if (!root.empty() && (root.back() == '\\' || root.back() == '/')) root.pop_back(); + binDir = root + "\\bin"; modelDir = root + "\\bin\\models"; return true; + } + std::string appDir = ShimModuleDir(); + if (!appDir.empty()) { + std::string maxine = appDir + "\\maxine"; + if (DirExists(maxine)) { binDir = maxine; modelDir = maxine + "\\models\\vfx"; return true; } + } + err = "VFX runtime not found: set COS_VFX_RUNTIME_DIR or bundle maxine\\ beside the app"; + return false; +} +void PointProxiesAt(const std::string& binDir) { + static std::string s_bin; + s_bin = binDir; + g_nvVFXSDKPath = const_cast(s_bin.c_str()); +} +} +#else +namespace vfx { +bool ResolveSdkPaths(std::string&, std::string&, std::string& err) { err = "Maxine SDK not built in"; return false; } +void PointProxiesAt(const std::string&) {} +} +#endif +``` + +- [ ] **Step 3: Refactor `native/shim/aigs.cpp` to use the shared module** + +In the `COS_HAS_MAXINE` block: +1. Add `#include "vfx_paths.h"` next to the other includes. +2. **Delete** the `g_nvVFXSDKPath` definition (the `char* g_nvVFXSDKPath = nullptr;` line and its comment) — it now lives in `vfx_paths.cpp`. +3. **Delete** the anonymous-namespace `ResolveSdkPaths(...)` and `PointProxiesAt(...)` functions (the whole `namespace { ... }` block that defines them, lines ~26–76). +4. In `Aigs::Probe`, replace the resolver call: + +```cpp + std::string binDir, modelDir, err; + if (!vfx::ResolveSdkPaths(binDir, modelDir, err)) { detail = err; return false; } + vfx::PointProxiesAt(binDir); +``` + +5. In `Aigs::Start`, replace likewise: + +```cpp + std::string binDir, err; + if (!vfx::ResolveSdkPaths(binDir, impl->modelDir, err)) { + lastError_ = err; delete impl; return false; + } + vfx::PointProxiesAt(binDir); +``` + +(Both call sites already use these exact local variable names; only the function namespace changes from the file-local helpers to `vfx::`.) + +- [ ] **Step 4: Add to `native/shim/shim.vcxproj`** + +Inside the `` that holds the other `` entries (e.g. `aigs.cpp`), add: + +```xml + +``` + +- [ ] **Step 5: Build the shim (SDK config) — clean, single `g_nvVFXSDKPath` definition** + +Run the shim build with `COS_VFX_SDK_DIR` set. Expected: 0 warnings, links cleanly (exactly one definition of `g_nvVFXSDKPath` — a duplicate-symbol linker error here means the old definition was not removed from `aigs.cpp`). + +- [ ] **Step 6: Commit** + +```bash +git add native/shim/vfx_paths.h native/shim/vfx_paths.cpp native/shim/aigs.cpp native/shim/shim.vcxproj +git commit -m "refactor(shim): shared VFX SDK path resolver (green screen + new effects)" +``` + +> **Human gate (after build):** run the app with green screen ON and confirm the matte still works — this task edited the proven green-screen load path. Tasks 3–4 reuse `vfx::` so a green-screen regression here would surface everywhere. + +--- + +## Task 3: Artifact Reduction effect + +VFX effect that removes compression artifacts in place at source resolution. Mirrors `aigs.cpp` but the output is a full BGR image, not a matte. No dimension change. + +**Files:** +- Create: `native/shim/artifactreduction.h`, `native/shim/artifactreduction.cpp` +- Create/extend: `native/shim/smoke/effects_smoke.cpp` +- Modify: `native/shim/shim.vcxproj` + +**Interfaces:** +- Consumes: `vfx::ResolveSdkPaths`, `vfx::PointProxiesAt` (Task 2). +- Produces: `class ArtifactReduction { static bool Probe(std::string&); bool Start(); void Stop(); bool ProcessFrame(uint8_t* bgra, int w, int h); bool IsReady() const; const std::string& LastError() const; }` + +> **Verify at impl:** confirm the effect selector macro `NVVFX_FX_ARTIFACT_REDUCTION` and that AR input/output images are `NVCV_BGR`/`NVCV_U8`/`NVCV_CHUNKY` against the VFX 1.2.0.0 headers (`nvVideoEffects.h`). Mode key is `NVVFX_MODE` (0 or 1). These match the green-screen pattern; only the selector, the output format (full BGR, not `NVCV_A`), and the write-back differ. + +- [ ] **Step 1: Create `native/shim/artifactreduction.h`** + +```cpp +#pragma once +#include +#include + +// Wraps the Maxine VFX Artifact Reduction effect. CPU-copy: a BGRA frame is uploaded +// to the GPU as BGR, cleaned, downloaded, and written back over the same BGRA buffer +// (RGB replaced, alpha left untouched). No size change. All methods no-throw; failure +// via IsReady()/LastError(). Without COS_HAS_MAXINE this is a never-ready stub. +class ArtifactReduction { +public: + ArtifactReduction(); + ~ArtifactReduction(); + static bool Probe(std::string& detail); + bool Start(); + void Stop(); + bool ProcessFrame(uint8_t* bgra, int width, int height); + bool IsReady() const { return ready_; } + const std::string& LastError() const { return lastError_; } +private: + bool ready_ = false; + std::string lastError_; + void* impl_ = nullptr; +}; +``` + +- [ ] **Step 2: Create `native/shim/artifactreduction.cpp`** + +```cpp +#include "artifactreduction.h" + +#ifdef COS_HAS_MAXINE +#define NOMINMAX +#include +#include +#include +#include +#include "nvCVStatus.h" +#include "nvCVImage.h" +#include "nvVideoEffects.h" // NVVFX_FX_ARTIFACT_REDUCTION (verify symbol) +#include "vfx_paths.h" + +struct ArImpl { + NvVFX_Handle effect = nullptr; + CUstream stream = nullptr; + std::string modelDir; + NvCVImage srcGpu{}; // BGR u8 GPU (input) + NvCVImage dstGpu{}; // BGR u8 GPU (output) + NvCVImage dstCpu{}; // BGR u8 CPU (downloaded) + NvCVImage stage{}; // BGRA u8 GPU (transfer staging) + int w = 0, h = 0; + bool loaded = false; +}; + +ArtifactReduction::ArtifactReduction() = default; +ArtifactReduction::~ArtifactReduction() { Stop(); } + +bool ArtifactReduction::Probe(std::string& detail) { + std::string binDir, modelDir, err; + if (!vfx::ResolveSdkPaths(binDir, modelDir, err)) { detail = err; return false; } + vfx::PointProxiesAt(binDir); + NvVFX_Handle eff = nullptr; + if (NvVFX_CreateEffect(NVVFX_FX_ARTIFACT_REDUCTION, &eff) != NVCV_SUCCESS || !eff) { + detail = "NvVFX_CreateEffect(ArtifactReduction) failed (DLL/SDK load?)"; return false; + } + NvVFX_SetString(eff, NVVFX_MODEL_DIRECTORY, modelDir.c_str()); + NvVFX_SetU32(eff, NVVFX_MODE, 0u); + NvVFX_SetU32(eff, NVVFX_MAX_INPUT_WIDTH, 1920u); + NvVFX_SetU32(eff, NVVFX_MAX_INPUT_HEIGHT, 1080u); + NvCV_Status load = NvVFX_Load(eff); + NvVFX_DestroyEffect(eff); + if (load != NVCV_SUCCESS) { detail = "NvVFX_Load(ArtifactReduction) failed (models missing?)"; return false; } + detail = "ArtifactReduction available"; + return true; +} + +bool ArtifactReduction::Start() { + Stop(); + ArImpl* impl = new (std::nothrow) ArImpl(); + if (!impl) { lastError_ = "out of memory"; return false; } + std::string binDir, err; + if (!vfx::ResolveSdkPaths(binDir, impl->modelDir, err)) { lastError_ = err; delete impl; return false; } + vfx::PointProxiesAt(binDir); + if (NvVFX_CudaStreamCreate(&impl->stream) != NVCV_SUCCESS) { lastError_ = "CudaStreamCreate failed"; delete impl; return false; } + if (NvVFX_CreateEffect(NVVFX_FX_ARTIFACT_REDUCTION, &impl->effect) != NVCV_SUCCESS || !impl->effect) { + lastError_ = "CreateEffect failed"; delete impl; return false; + } + NvVFX_SetString(impl->effect, NVVFX_MODEL_DIRECTORY, impl->modelDir.c_str()); + NvVFX_SetCudaStream(impl->effect, NVVFX_CUDA_STREAM, impl->stream); + NvVFX_SetU32(impl->effect, NVVFX_MODE, 1u); // mode 1 = stronger / for compressed input + impl_ = impl; ready_ = true; lastError_.clear(); + return true; +} + +void ArtifactReduction::Stop() { + auto* impl = static_cast(impl_); + if (!impl) { ready_ = false; return; } + NvCVImage_Dealloc(&impl->srcGpu); + NvCVImage_Dealloc(&impl->dstGpu); + NvCVImage_Dealloc(&impl->dstCpu); + NvCVImage_Dealloc(&impl->stage); + if (impl->effect) NvVFX_DestroyEffect(impl->effect); + if (impl->stream) NvVFX_CudaStreamDestroy(impl->stream); + delete impl; impl_ = nullptr; ready_ = false; +} + +namespace { +NvCV_Status EnsureImages(ArImpl* impl, int w, int h) { + if (impl->loaded && impl->w == w && impl->h == h) return NVCV_SUCCESS; + NvCVImage_Dealloc(&impl->srcGpu); + NvCVImage_Dealloc(&impl->dstGpu); + NvCVImage_Dealloc(&impl->dstCpu); + NvCVImage_Dealloc(&impl->stage); + NvCV_Status s; + s = NvCVImage_Alloc(&impl->srcGpu, w, h, NVCV_BGR, NVCV_U8, NVCV_CHUNKY, NVCV_GPU, 1); if (s != NVCV_SUCCESS) return s; + s = NvCVImage_Alloc(&impl->dstGpu, w, h, NVCV_BGR, NVCV_U8, NVCV_CHUNKY, NVCV_GPU, 1); if (s != NVCV_SUCCESS) return s; + s = NvCVImage_Alloc(&impl->dstCpu, w, h, NVCV_BGR, NVCV_U8, NVCV_CHUNKY, NVCV_CPU, 1); if (s != NVCV_SUCCESS) return s; + s = NvCVImage_Alloc(&impl->stage, w, h, NVCV_BGRA,NVCV_U8, NVCV_CHUNKY, NVCV_GPU, 1); if (s != NVCV_SUCCESS) return s; + s = NvVFX_SetImage(impl->effect, NVVFX_INPUT_IMAGE, &impl->srcGpu); if (s != NVCV_SUCCESS) return s; + s = NvVFX_SetImage(impl->effect, NVVFX_OUTPUT_IMAGE, &impl->dstGpu); if (s != NVCV_SUCCESS) return s; + s = NvVFX_SetU32(impl->effect, NVVFX_MAX_INPUT_WIDTH, static_cast(w)); if (s != NVCV_SUCCESS) return s; + s = NvVFX_SetU32(impl->effect, NVVFX_MAX_INPUT_HEIGHT, static_cast(h)); if (s != NVCV_SUCCESS) return s; + s = NvVFX_Load(impl->effect); if (s != NVCV_SUCCESS) return s; + impl->w = w; impl->h = h; impl->loaded = true; + return NVCV_SUCCESS; +} +} // namespace + +bool ArtifactReduction::ProcessFrame(uint8_t* bgra, int w, int h) { + auto* impl = static_cast(impl_); + if (!impl || !impl->effect || !bgra || w <= 0 || h <= 0) return false; + if (NvCV_Status es = EnsureImages(impl, w, h); es != NVCV_SUCCESS) { + lastError_ = std::string("EnsureImages/Load failed: ") + NvCV_GetErrorStringFromCode(es); ready_ = false; return false; + } + NvCVImage src{}; + NvCVImage_Init(&src, w, h, w * 4, bgra, NVCV_BGRA, NVCV_U8, NVCV_CHUNKY, NVCV_CPU); + if (NvCVImage_Transfer(&src, &impl->srcGpu, 1.0f, impl->stream, &impl->stage) != NVCV_SUCCESS) { lastError_ = "upload failed"; return false; } + if (NvVFX_Run(impl->effect, 0) != NVCV_SUCCESS) { lastError_ = "NvVFX_Run failed"; return false; } + if (NvCVImage_Transfer(&impl->dstGpu, &impl->dstCpu, 1.0f, impl->stream, nullptr) != NVCV_SUCCESS) { lastError_ = "download failed"; return false; } + if (NvVFX_CudaStreamSynchronize(impl->stream) != NVCV_SUCCESS) { lastError_ = "stream sync failed"; return false; } + // Write cleaned BGR back over the BGRA buffer; leave alpha untouched (forced 0xFF upstream). + const uint8_t* d = static_cast(impl->dstCpu.pixels); + const int dpitch = impl->dstCpu.pitch; + for (int y = 0; y < h; ++y) { + const uint8_t* drow = d + static_cast(dpitch) * y; + uint8_t* prow = bgra + static_cast(w) * 4 * y; + for (int x = 0; x < w; ++x) { + prow[x * 4 + 0] = drow[x * 3 + 0]; + prow[x * 4 + 1] = drow[x * 3 + 1]; + prow[x * 4 + 2] = drow[x * 3 + 2]; + } + } + return true; +} + +#else +ArtifactReduction::ArtifactReduction() = default; +ArtifactReduction::~ArtifactReduction() = default; +bool ArtifactReduction::Probe(std::string& detail) { detail = "Maxine SDK not built in"; return false; } +bool ArtifactReduction::Start() { lastError_ = "Maxine SDK not built in"; ready_ = false; return false; } +void ArtifactReduction::Stop() { ready_ = false; } +bool ArtifactReduction::ProcessFrame(uint8_t*, int, int) { return false; } +#endif +``` + +- [ ] **Step 3: Add `artifactreduction.cpp` to `native/shim/shim.vcxproj`** + +```xml + +``` + +- [ ] **Step 4: Add an AR smoke** — append to `native/shim/smoke/effects_smoke.cpp` (create it) + +```cpp +// Dev-box smoke (needs VFX SDK + RTX GPU). Build with the shim's include/lib setup. +// Verifies AR can start and process a synthetic 1280x720 frame. +#include +#include +#include +#include "../artifactreduction.h" + +int main() { + std::string detail; + if (!ArtifactReduction::Probe(detail)) { std::printf("AR unavailable: %s\n", detail.c_str()); return 0; } + ArtifactReduction ar; + assert(ar.Start()); + std::vector frame(1280 * 720 * 4, 128); + bool ok = ar.ProcessFrame(frame.data(), 1280, 720); + std::printf("AR ProcessFrame: %s (%s)\n", ok ? "ok" : "fail", ar.LastError().c_str()); + assert(ok); + ar.Stop(); + std::puts("effects_smoke AR OK"); + return 0; +} +``` + +- [ ] **Step 5: Build the shim (clean) and run the AR smoke on the dev box (SDK config)** + +Run the shim build with `COS_VFX_SDK_DIR` set (SDK config). Expected: 0 warnings. Then build+run `effects_smoke.cpp` (dev box): prints `effects_smoke AR OK`. (If no SDK/GPU, `Probe` returns the stub message and the smoke exits 0 — acceptable on CI-style hosts.) + +- [ ] **Step 6: Commit** + +```bash +git add native/shim/artifactreduction.h native/shim/artifactreduction.cpp native/shim/smoke/effects_smoke.cpp native/shim/shim.vcxproj +git commit -m "feat(shim): Maxine VFX Artifact Reduction effect" +``` + +--- + +## Task 4: Super Resolution effect + +VFX effect that upscales the frame (1.5× or 2×) into a **larger output buffer**. Runs last in the chain. + +**Files:** +- Create: `native/shim/superres.h`, `native/shim/superres.cpp` +- Modify: `native/shim/smoke/effects_smoke.cpp`, `native/shim/shim.vcxproj` + +**Interfaces:** +- Consumes: `vfx::ResolveSdkPaths`, `vfx::PointProxiesAt`. +- Produces: `class SuperRes { static bool Probe(std::string&); bool Start(int scaleX10); void Stop(); bool ProcessFrame(const uint8_t* bgra, int w, int h, std::vector& out, int& outW, int& outH); bool IsReady() const; const std::string& LastError() const; }` where `scaleX10` ∈ {15, 20}. + +> **Verify at impl:** confirm `NVVFX_FX_SUPER_RES`, and how the upscale factor is selected — the VFX SuperRes effect infers scale from the output image size relative to the input (set both images), plus `NVVFX_MODE` (0/1). If a dedicated scale property (`NVVFX_SCALE`) is required by 1.2.0.0, set it. Output dims must be one of the SDK-supported factors of the input. + +- [ ] **Step 1: Create `native/shim/superres.h`** + +```cpp +#pragma once +#include +#include +#include + +// Wraps the Maxine VFX Super Resolution effect. Upscales a BGRA frame by 1.5x or 2x +// into a freshly sized BGRA output buffer (alpha = 0xFF). Without COS_HAS_MAXINE this +// is a never-ready stub. scaleX10: 15 => 1.5x, 20 => 2x. +class SuperRes { +public: + SuperRes(); + ~SuperRes(); + static bool Probe(std::string& detail); + bool Start(int scaleX10); + void Stop(); + // Reads w*h BGRA from 'bgra', writes outW*outH BGRA into 'out'. Returns false on failure + // (out untouched). outW/outH are w*scale, h*scale. + bool ProcessFrame(const uint8_t* bgra, int w, int h, std::vector& out, int& outW, int& outH); + bool IsReady() const { return ready_; } + const std::string& LastError() const { return lastError_; } +private: + bool ready_ = false; + int scaleX10_ = 20; + std::string lastError_; + void* impl_ = nullptr; +}; +``` + +- [ ] **Step 2: Create `native/shim/superres.cpp`** + +```cpp +#include "superres.h" + +#ifdef COS_HAS_MAXINE +#define NOMINMAX +#include +#include +#include +#include +#include "nvCVStatus.h" +#include "nvCVImage.h" +#include "nvVideoEffects.h" // NVVFX_FX_SUPER_RES (verify symbol) +#include "vfx_paths.h" + +struct SrImpl { + NvVFX_Handle effect = nullptr; + CUstream stream = nullptr; + std::string modelDir; + NvCVImage srcGpu{}; // BGR u8 GPU (input, w x h) + NvCVImage dstGpu{}; // BGR u8 GPU (output, outW x outH) + NvCVImage dstCpu{}; // BGR u8 CPU (downloaded) + NvCVImage stage{}; // BGRA u8 GPU (upload staging, w x h) + int w = 0, h = 0, ow = 0, oh = 0; + bool loaded = false; +}; + +static void ScaledDims(int w, int h, int scaleX10, int& ow, int& oh) { + ow = (w * scaleX10) / 10; + oh = (h * scaleX10) / 10; +} + +SuperRes::SuperRes() = default; +SuperRes::~SuperRes() { Stop(); } + +bool SuperRes::Probe(std::string& detail) { + std::string binDir, modelDir, err; + if (!vfx::ResolveSdkPaths(binDir, modelDir, err)) { detail = err; return false; } + vfx::PointProxiesAt(binDir); + NvVFX_Handle eff = nullptr; + if (NvVFX_CreateEffect(NVVFX_FX_SUPER_RES, &eff) != NVCV_SUCCESS || !eff) { + detail = "NvVFX_CreateEffect(SuperRes) failed (DLL/SDK load?)"; return false; + } + NvVFX_SetString(eff, NVVFX_MODEL_DIRECTORY, modelDir.c_str()); + NvVFX_SetU32(eff, NVVFX_MODE, 1u); + NvVFX_SetU32(eff, NVVFX_MAX_INPUT_WIDTH, 1920u); + NvVFX_SetU32(eff, NVVFX_MAX_INPUT_HEIGHT, 1080u); + NvCV_Status load = NvVFX_Load(eff); + NvVFX_DestroyEffect(eff); + if (load != NVCV_SUCCESS) { detail = "NvVFX_Load(SuperRes) failed (models missing?)"; return false; } + detail = "SuperRes available"; + return true; +} + +bool SuperRes::Start(int scaleX10) { + Stop(); + scaleX10_ = (scaleX10 == 15) ? 15 : 20; // only 1.5x / 2x supported here + SrImpl* impl = new (std::nothrow) SrImpl(); + if (!impl) { lastError_ = "out of memory"; return false; } + std::string binDir, err; + if (!vfx::ResolveSdkPaths(binDir, impl->modelDir, err)) { lastError_ = err; delete impl; return false; } + vfx::PointProxiesAt(binDir); + if (NvVFX_CudaStreamCreate(&impl->stream) != NVCV_SUCCESS) { lastError_ = "CudaStreamCreate failed"; delete impl; return false; } + if (NvVFX_CreateEffect(NVVFX_FX_SUPER_RES, &impl->effect) != NVCV_SUCCESS || !impl->effect) { + lastError_ = "CreateEffect failed"; delete impl; return false; + } + NvVFX_SetString(impl->effect, NVVFX_MODEL_DIRECTORY, impl->modelDir.c_str()); + NvVFX_SetCudaStream(impl->effect, NVVFX_CUDA_STREAM, impl->stream); + NvVFX_SetU32(impl->effect, NVVFX_MODE, 1u); + impl_ = impl; ready_ = true; lastError_.clear(); + return true; +} + +void SuperRes::Stop() { + auto* impl = static_cast(impl_); + if (!impl) { ready_ = false; return; } + NvCVImage_Dealloc(&impl->srcGpu); + NvCVImage_Dealloc(&impl->dstGpu); + NvCVImage_Dealloc(&impl->dstCpu); + NvCVImage_Dealloc(&impl->stage); + if (impl->effect) NvVFX_DestroyEffect(impl->effect); + if (impl->stream) NvVFX_CudaStreamDestroy(impl->stream); + delete impl; impl_ = nullptr; ready_ = false; +} + +namespace { +NvCV_Status EnsureImages(SrImpl* impl, int w, int h, int ow, int oh) { + if (impl->loaded && impl->w == w && impl->h == h && impl->ow == ow && impl->oh == oh) return NVCV_SUCCESS; + NvCVImage_Dealloc(&impl->srcGpu); + NvCVImage_Dealloc(&impl->dstGpu); + NvCVImage_Dealloc(&impl->dstCpu); + NvCVImage_Dealloc(&impl->stage); + NvCV_Status s; + s = NvCVImage_Alloc(&impl->srcGpu, w, h, NVCV_BGR, NVCV_U8, NVCV_CHUNKY, NVCV_GPU, 1); if (s != NVCV_SUCCESS) return s; + s = NvCVImage_Alloc(&impl->dstGpu, ow, oh, NVCV_BGR, NVCV_U8, NVCV_CHUNKY, NVCV_GPU, 1); if (s != NVCV_SUCCESS) return s; + s = NvCVImage_Alloc(&impl->dstCpu, ow, oh, NVCV_BGR, NVCV_U8, NVCV_CHUNKY, NVCV_CPU, 1); if (s != NVCV_SUCCESS) return s; + s = NvCVImage_Alloc(&impl->stage, w, h, NVCV_BGRA,NVCV_U8, NVCV_CHUNKY, NVCV_GPU, 1); if (s != NVCV_SUCCESS) return s; + s = NvVFX_SetImage(impl->effect, NVVFX_INPUT_IMAGE, &impl->srcGpu); if (s != NVCV_SUCCESS) return s; + s = NvVFX_SetImage(impl->effect, NVVFX_OUTPUT_IMAGE, &impl->dstGpu); if (s != NVCV_SUCCESS) return s; + s = NvVFX_Load(impl->effect); if (s != NVCV_SUCCESS) return s; + impl->w = w; impl->h = h; impl->ow = ow; impl->oh = oh; impl->loaded = true; + return NVCV_SUCCESS; +} +} // namespace + +bool SuperRes::ProcessFrame(const uint8_t* bgra, int w, int h, std::vector& out, int& outW, int& outH) { + auto* impl = static_cast(impl_); + if (!impl || !impl->effect || !bgra || w <= 0 || h <= 0) return false; + int ow, oh; ScaledDims(w, h, scaleX10_, ow, oh); + if (NvCV_Status es = EnsureImages(impl, w, h, ow, oh); es != NVCV_SUCCESS) { + lastError_ = std::string("EnsureImages/Load failed: ") + NvCV_GetErrorStringFromCode(es); ready_ = false; return false; + } + NvCVImage src{}; + NvCVImage_Init(&src, w, h, w * 4, const_cast(bgra), NVCV_BGRA, NVCV_U8, NVCV_CHUNKY, NVCV_CPU); + if (NvCVImage_Transfer(&src, &impl->srcGpu, 1.0f, impl->stream, &impl->stage) != NVCV_SUCCESS) { lastError_ = "upload failed"; return false; } + if (NvVFX_Run(impl->effect, 0) != NVCV_SUCCESS) { lastError_ = "NvVFX_Run failed"; return false; } + if (NvCVImage_Transfer(&impl->dstGpu, &impl->dstCpu, 1.0f, impl->stream, nullptr) != NVCV_SUCCESS) { lastError_ = "download failed"; return false; } + if (NvVFX_CudaStreamSynchronize(impl->stream) != NVCV_SUCCESS) { lastError_ = "stream sync failed"; return false; } + out.assign(static_cast(ow) * oh * 4, 0xFF); + const uint8_t* d = static_cast(impl->dstCpu.pixels); + const int dpitch = impl->dstCpu.pitch; + for (int y = 0; y < oh; ++y) { + const uint8_t* drow = d + static_cast(dpitch) * y; + uint8_t* prow = out.data() + static_cast(ow) * 4 * y; + for (int x = 0; x < ow; ++x) { + prow[x * 4 + 0] = drow[x * 3 + 0]; + prow[x * 4 + 1] = drow[x * 3 + 1]; + prow[x * 4 + 2] = drow[x * 3 + 2]; + prow[x * 4 + 3] = 0xFF; + } + } + outW = ow; outH = oh; + return true; +} + +#else +SuperRes::SuperRes() = default; +SuperRes::~SuperRes() = default; +bool SuperRes::Probe(std::string& detail) { detail = "Maxine SDK not built in"; return false; } +bool SuperRes::Start(int) { lastError_ = "Maxine SDK not built in"; ready_ = false; return false; } +void SuperRes::Stop() { ready_ = false; } +bool SuperRes::ProcessFrame(const uint8_t*, int, int, std::vector&, int&, int&) { return false; } +#endif +``` + +- [ ] **Step 3: Add `superres.cpp` to `native/shim/shim.vcxproj`** + +```xml + +``` + +- [ ] **Step 4: Extend the smoke** — append to `native/shim/smoke/effects_smoke.cpp` a SuperRes check (inside `main`, before the final puts): + +```cpp + { + std::string d; + if (SuperRes::Probe(d)) { + SuperRes sr; assert(sr.Start(20)); + std::vector in(640 * 480 * 4, 64), out; int ow = 0, oh = 0; + bool ok = sr.ProcessFrame(in.data(), 640, 480, out, ow, oh); + std::printf("SR ProcessFrame: %s -> %dx%d\n", ok ? "ok" : "fail", ow, oh); + assert(ok && ow == 1280 && oh == 960); + sr.Stop(); + } else { std::printf("SR unavailable: %s\n", d.c_str()); } + } +``` +Add `#include "../superres.h"` at the top. + +- [ ] **Step 5: Build the shim (clean) + run the smoke (dev box)** + +Expected: 0 warnings; smoke prints `SR ProcessFrame: ok -> 1280x960`. + +- [ ] **Step 6: Commit** + +```bash +git add native/shim/superres.h native/shim/superres.cpp native/shim/smoke/effects_smoke.cpp native/shim/shim.vcxproj +git commit -m "feat(shim): Maxine VFX Super Resolution effect (1.5x/2x)" +``` + +--- + +## Task 5: C ABI — params + capability gates + +Grow the contract once for both effects. After this task the shim accepts the new params and reports the new gates, but the worker does not yet run the effects (Task 6). + +**Files:** +- Modify: `native/shim/shim.h`, `native/shim/shim.cpp` + +**Interfaces:** +- Produces (C): `CosParams` gains `int artifact_reduction_enabled; int super_res_enabled; int super_res_scale;`. `CosCaps` gains `int artifact_reduction_available; int super_res_available;`. + +- [ ] **Step 1: Update `CosParams` + `CosCaps` in `native/shim/shim.h`** + +Replace the `CosParams` struct with: + +```cpp +typedef struct { + const char* camera_id; // UTF-8, may be null + int green_screen_enabled; + double green_screen_expand; + double green_screen_feather; + int eye_contact_enabled; + double eye_contact_sensitivity; + double eye_contact_look_away_range; + int artifact_reduction_enabled; + int super_res_enabled; + int super_res_scale; // 0=off, 15=1.5x, 20=2x +} CosParams; +``` + +Replace the `CosCaps` struct with (total 528 bytes on x64): + +```cpp +typedef struct { + int green_screen_available; + char detail[256]; + int eye_contact_available; + char ec_detail[256]; + int artifact_reduction_available; // 1 if Maxine ArtifactReduction can run + int super_res_available; // 1 if Maxine SuperRes can run +} CosCaps; +``` + +- [ ] **Step 2: Wire `cos_set_params`** — `native/shim/shim.cpp` + +Add the includes near the top: + +```cpp +#include "artifactreduction.h" +#include "superres.h" +``` + +In `cos_set_params`, after the existing `g_capture.SetEyeContact(...)` line add: + +```cpp + g_capture.SetArtifactReduction(p->artifact_reduction_enabled != 0); + g_capture.SetSuperRes(p->super_res_enabled != 0, p->super_res_scale); +``` + +- [ ] **Step 3: Wire `cos_query_capabilities`** — `native/shim/shim.cpp` + +Before the final `return` of `cos_query_capabilities`, add: + +```cpp + std::string arDetail, srDetail; + out->artifact_reduction_available = ArtifactReduction::Probe(arDetail) ? 1 : 0; + out->super_res_available = SuperRes::Probe(srDetail) ? 1 : 0; +``` + +And change the final return to also surface the new gates: + +```cpp + return (gsOk || ecOk || out->artifact_reduction_available || out->super_res_available) ? 1 : 0; +``` + +- [ ] **Step 4: Build the shim and verify exports + struct sizes** + +Run the shim build. Expected: 0 warnings; `dumpbin /exports` still shows exactly 9 `cos_*`. (Capture `Set*` methods are added in Task 6 — if the build fails on missing `SetArtifactReduction`/`SetSuperRes`, do Task 6 Step 1–3 before re-building; the two tasks share the ABI boundary. Recommended: implement Task 6 immediately after this step, then build once.) + +- [ ] **Step 5: Commit** (after Task 6 builds clean, or together) + +```bash +git add native/shim/shim.h native/shim/shim.cpp +git commit -m "feat(shim): ABI — AR/SR params + capability gates (CosParams/CosCaps 528B)" +``` + +--- + +## Task 6: Capture worker — run AR first, SR last + +**Files:** +- Modify: `native/shim/capture.h`, `native/shim/capture.cpp` + +**Interfaces:** +- Consumes: `ArtifactReduction`, `SuperRes` (Tasks 3–4); `cos_set_params` (Task 5). +- Produces: `Capture::SetArtifactReduction(bool)`, `Capture::SetSuperRes(bool, int scaleX10)`, `Capture::ArtifactReductionActive()/SuperResActive()` + error getters. + +- [ ] **Step 1: Declare the new methods in `native/shim/capture.h`** + +After the Eye Contact block add: + +```cpp + // Toggles Artifact Reduction for subsequent frames. Thread-safe; worker owns the object. + void SetArtifactReduction(bool enabled); + bool ArtifactReductionActive() const; + std::string ArtifactReductionError() const; + + // Toggles Super Resolution + scale (15=1.5x, 20=2x). Thread-safe; worker owns the object. + void SetSuperRes(bool enabled, int scaleX10); + bool SuperResActive() const; + std::string SuperResError() const; +``` + +- [ ] **Step 2: Add shared state** — `native/shim/capture.cpp`, in `struct CaptureState` + +```cpp + std::atomic artifactReductionEnabled{false}; + std::atomic artifactReductionActive{false}; + std::mutex arErrMtx; // leaf lock + std::string arError; + + std::atomic superResEnabled{false}; + std::atomic superResScale{20}; // 15 or 20 + std::atomic superResActive{false}; + std::mutex srErrMtx; // leaf lock + std::string srError; +``` + +Add the include at the top: `#include "superres.h"` (alongside the existing `#include "aigs.h"` / `#include "eyecontact.h"`; `artifactreduction.h` is needed too): + +```cpp +#include "artifactreduction.h" +#include "superres.h" +``` + +- [ ] **Step 3: Implement the setters/getters** — `native/shim/capture.cpp` (near the existing `SetEyeContact` etc.) + +```cpp +void Capture::SetArtifactReduction(bool enabled) { + g_state.artifactReductionEnabled.store(enabled, std::memory_order_release); +} +bool Capture::ArtifactReductionActive() const { + return g_state.artifactReductionActive.load(std::memory_order_acquire); +} +std::string Capture::ArtifactReductionError() const { + std::lock_guard e(g_state.arErrMtx); return g_state.arError; +} +void Capture::SetSuperRes(bool enabled, int scaleX10) { + g_state.superResScale.store(scaleX10 == 15 ? 15 : 20, std::memory_order_release); + g_state.superResEnabled.store(enabled, std::memory_order_release); +} +bool Capture::SuperResActive() const { + return g_state.superResActive.load(std::memory_order_acquire); +} +std::string Capture::SuperResError() const { + std::lock_guard e(g_state.srErrMtx); return g_state.srError; +} +``` + +- [ ] **Step 4: Insert AR + SR into `WorkerLoop`** + +Declare the effect objects next to the existing `Aigs aigs;` / `EyeContact eyeContact;` (worker-thread-local): + +```cpp + ArtifactReduction artifactReduction; + SuperRes superRes; +``` + +**Artifact Reduction — before Eye Contact.** Inside `if (CopyFrame(...))`, immediately BEFORE the existing `const bool ecWant = ...` line, add: + +```cpp + // Artifact Reduction runs FIRST so the AI effects downstream get a clean frame. + const bool arWant = g_state.artifactReductionEnabled.load(std::memory_order_acquire); + if (arWant && !artifactReduction.IsReady()) { + if (!artifactReduction.Start()) { + std::lock_guard e(g_state.arErrMtx); + const std::string& ne = artifactReduction.LastError(); + if (g_state.arError != ne) g_state.arError = ne; + } + } else if (!arWant && artifactReduction.IsReady()) { + artifactReduction.Stop(); + std::lock_guard e(g_state.arErrMtx); + if (!g_state.arError.empty()) g_state.arError.clear(); + } + bool arApplied = false; + if (arWant && artifactReduction.IsReady()) { + arApplied = artifactReduction.ProcessFrame(scratch.data(), width, height); + std::lock_guard e(g_state.arErrMtx); + if (!arApplied) { + const std::string& ne = artifactReduction.LastError(); + if (g_state.arError != ne) g_state.arError = ne; + } else if (!g_state.arError.empty()) { g_state.arError.clear(); } + } + g_state.artifactReductionActive.store(arApplied, std::memory_order_release); +``` + +**Super Resolution — after Green Screen, before publish.** The publish block currently does `g_state.frame.swap(scratch); g_state.width = width; g_state.height = height;`. Replace that publish block (the four lines under `std::lock_guard lock(g_state.mtx);`) with SR-aware publish: + +```cpp + // Super Resolution runs LAST; it changes the frame dimensions. + const bool srWant = g_state.superResEnabled.load(std::memory_order_acquire); + const int srScale = g_state.superResScale.load(std::memory_order_acquire); + if (srWant && !superRes.IsReady()) { + if (!superRes.Start(srScale)) { + std::lock_guard e(g_state.srErrMtx); + const std::string& ne = superRes.LastError(); + if (g_state.srError != ne) g_state.srError = ne; + } + } else if (!srWant && superRes.IsReady()) { + superRes.Stop(); + std::lock_guard e(g_state.srErrMtx); + if (!g_state.srError.empty()) g_state.srError.clear(); + } + + int pubW = width, pubH = height; + std::vector srOut; + bool srApplied = false; + if (srWant && superRes.IsReady()) { + srApplied = superRes.ProcessFrame(scratch.data(), width, height, srOut, pubW, pubH); + std::lock_guard e(g_state.srErrMtx); + if (!srApplied) { + const std::string& ne = superRes.LastError(); + if (g_state.srError != ne) g_state.srError = ne; + } else if (!g_state.srError.empty()) { g_state.srError.clear(); } + } + g_state.superResActive.store(srApplied, std::memory_order_release); + + { + std::lock_guard lock(g_state.mtx); + if (srApplied) g_state.frame.swap(srOut); + else g_state.frame.swap(scratch); + g_state.width = pubW; + g_state.height = pubH; + g_state.hasNewFrame = true; + g_state.framesProduced.fetch_add(1, std::memory_order_release); + } +``` + +> Note: this replaces the Task 1 Step 6 increment location — the increment now lives inside the new publish block. Remove the standalone increment line added in Task 1 if it is now duplicated. + +Add teardown next to the existing `eyeContact.Stop(); aigs.Stop();` (after the loop): + +```cpp + artifactReduction.Stop(); + superRes.Stop(); +``` + +- [ ] **Step 5: Reset active flags + clear errors in `Capture::Stop`** + +After the eye-contact reset block add: + +```cpp + g_state.artifactReductionActive.store(false, std::memory_order_release); + { std::lock_guard e(g_state.arErrMtx); g_state.arError.clear(); } + g_state.superResActive.store(false, std::memory_order_release); + { std::lock_guard e(g_state.srErrMtx); g_state.srError.clear(); } +``` + +- [ ] **Step 6: Surface AR/SR errors in `cos_get_status`** — `native/shim/shim.cpp` + +In `cos_get_status`, extend the error fallback chain: + +```cpp + std::string err = g_capture.GreenScreenError(); + if (err.empty()) err = g_capture.EyeContactError(); + if (err.empty()) err = g_capture.ArtifactReductionError(); + if (err.empty()) err = g_capture.SuperResError(); +``` + +- [ ] **Step 7: Build the shim (SDK config) — clean, exports intact** + +Run the shim build. Expected: 0 warnings; 9 `cos_*` exports. Verify the deployed DLL still exports `GreenScreen` and `GazeRedirection` and lacks `not built in` (grep the DLL). + +- [ ] **Step 8: Commit** + +```bash +git add native/shim/capture.h native/shim/capture.cpp native/shim/shim.h native/shim/shim.cpp +git commit -m "feat(shim): run Artifact Reduction (first) + Super Resolution (last) on the worker" +``` + +--- + +## Task 7: Managed contracts + Core logic (xUnit) + +This is the automated-test task. All changes are in `CameraOnScreen.Core` + tests (run in CI), plus the `PInvokeShim` mirror in the App (build-verified). + +**Files:** +- Modify: `src/CameraOnScreen.Core/Native/Contracts.cs`, `FakeShim.cs`, `Config/Models.cs`, `Orchestration/Orchestrator.cs`, `ViewModels/MainViewModel.cs` +- Modify: `src/CameraOnScreen.App/Native/PInvokeShim.cs` +- Test: `tests/CameraOnScreen.Core.Tests/ViewModels/MainViewModelTests.cs`, `tests/CameraOnScreen.Core.Tests/Orchestration/OrchestratorTests.cs` + +**Interfaces:** +- Produces: `ShimParams` gains `bool ArtifactReductionEnabled = false, bool SuperResEnabled = false, int SuperResScale = 0` (defaulted — existing call sites keep compiling). `ShimCapabilities` gains `bool ArtifactReductionAvailable = false, bool SuperResAvailable = false`. `MainViewModel` gains `ArtifactReductionEnabled`, `SuperResEnabled`, `SuperResScaleIndex` (0=off,1=1.5x,2=2x), `ArtifactReductionAvailable`, `SuperResAvailable`. + +- [ ] **Step 1: Write the failing test** — append to `tests/CameraOnScreen.Core.Tests/ViewModels/MainViewModelTests.cs` + +```csharp +[Fact] +public void BuildParams_includes_artifact_reduction_and_superres() +{ + var shim = new FakeShim { GreenScreenAvailable = true, EyeContactAvailable = true, + ArtifactReductionAvailable = true, SuperResAvailable = true }; + var orch = new Orchestrator(shim, GpuTier.Rtx); + orch.ProbeCapabilities(); + var vm = new MainViewModel(orch, shim); + + vm.ArtifactReductionEnabled = true; + vm.SuperResEnabled = true; + vm.SuperResScaleIndex = 2; // 2x + + var p = vm.BuildParams(); + Assert.True(p.ArtifactReductionEnabled); + Assert.True(p.SuperResEnabled); + Assert.Equal(20, p.SuperResScale); +} + +[Fact] +public void ToAppConfig_roundtrips_new_effects() +{ + var shim = new FakeShim(); + var vm = new MainViewModel(new Orchestrator(shim, GpuTier.Rtx), shim) + { + ArtifactReductionEnabled = true, SuperResEnabled = true, SuperResScaleIndex = 1 + }; + var cfg = vm.ToAppConfig(0, 0, 320, 240); + Assert.True(cfg.Effects.ArtifactReductionEnabled); + Assert.True(cfg.Effects.SuperResEnabled); + Assert.Equal(15, cfg.Effects.SuperResScale); + + var vm2 = new MainViewModel(new Orchestrator(shim, GpuTier.Rtx), shim); + vm2.LoadFrom(cfg); + Assert.True(vm2.ArtifactReductionEnabled); + Assert.Equal(1, vm2.SuperResScaleIndex); +} +``` + +Append to `tests/CameraOnScreen.Core.Tests/Orchestration/OrchestratorTests.cs`: + +```csharp +[Fact] +public void ApplyParams_forces_effects_off_when_unavailable() +{ + var shim = new FakeShim { GreenScreenAvailable = false, ArtifactReductionAvailable = false, SuperResAvailable = false }; + var orch = new Orchestrator(shim, GpuTier.Rtx); + orch.ProbeCapabilities(); + orch.ApplyParams(new ShimParams("cam", true, 0, 0, false, 0.5, 0.5, + ArtifactReductionEnabled: true, SuperResEnabled: true, SuperResScale: 20)); + Assert.False(shim.LastParams!.ArtifactReductionEnabled); + Assert.False(shim.LastParams!.SuperResEnabled); +} +``` + +- [ ] **Step 2: Run the tests to verify they fail** + +Run: `dotnet test tests/CameraOnScreen.Core.Tests/CameraOnScreen.Core.Tests.csproj --filter "FullyQualifiedName~artifact OR FullyQualifiedName~SuperRes OR FullyQualifiedName~forces_effects_off"` +Expected: FAIL to compile — new members do not exist. + +- [ ] **Step 3: Extend `ShimParams` + `ShimCapabilities`** — `src/CameraOnScreen.Core/Native/Contracts.cs` + +```csharp +public sealed record ShimParams( + string? CameraId, + bool GreenScreenEnabled, + double GreenScreenExpand, + double GreenScreenFeather, + bool EyeContactEnabled, + double EyeContactSensitivity, + double EyeContactLookAwayRange, + bool ArtifactReductionEnabled = false, + bool SuperResEnabled = false, + int SuperResScale = 0); // 0=off, 15=1.5x, 20=2x +``` + +```csharp +public sealed record ShimCapabilities( + bool GreenScreenAvailable, string Detail, + bool EyeContactAvailable = false, string EyeContactDetail = "", + bool ArtifactReductionAvailable = false, bool SuperResAvailable = false); +``` + +- [ ] **Step 4: Extend `EffectSettings`** — `src/CameraOnScreen.Core/Config/Models.cs` + +Add to the record: + +```csharp + public bool ArtifactReductionEnabled { get; init; } + public bool SuperResEnabled { get; init; } + public int SuperResScale { get; init; } // 0=off, 15=1.5x, 20=2x +``` + +- [ ] **Step 5: Extend `FakeShim`** — `src/CameraOnScreen.Core/Native/FakeShim.cs` + +Add properties: + +```csharp + public bool ArtifactReductionAvailable { get; set; } + public bool SuperResAvailable { get; set; } +``` + +Update `QueryCapabilities`: + +```csharp + public ShimCapabilities QueryCapabilities() => + new(GreenScreenAvailable, + GreenScreenAvailable ? "fake: available" : "fake: unavailable", + EyeContactAvailable, + EyeContactAvailable ? "fake: ec available" : "fake: ec unavailable", + ArtifactReductionAvailable, SuperResAvailable); +``` + +- [ ] **Step 6: Extend `Orchestrator`** — `src/CameraOnScreen.Core/Orchestration/Orchestrator.cs` + +Add properties: + +```csharp + public bool ArtifactReductionAvailable { get; private set; } + public bool SuperResAvailable { get; private set; } +``` + +In `ProbeCapabilities` add: + +```csharp + ArtifactReductionAvailable = caps.ArtifactReductionAvailable; + SuperResAvailable = caps.SuperResAvailable; +``` + +In `ApplyParams`, extend the `with`: + +```csharp + var effective = requested with + { + GreenScreenEnabled = requested.GreenScreenEnabled && EffectsAvailable, + EyeContactEnabled = requested.EyeContactEnabled && EyeContactAvailable, + ArtifactReductionEnabled = requested.ArtifactReductionEnabled && ArtifactReductionAvailable, + SuperResEnabled = requested.SuperResEnabled && SuperResAvailable, + }; +``` + +- [ ] **Step 7: Extend `MainViewModel`** — `src/CameraOnScreen.Core/ViewModels/MainViewModel.cs` + +Add observable props (next to the green-screen ones): + +```csharp + [ObservableProperty] private bool artifactReductionEnabled; + [ObservableProperty] private bool superResEnabled; + [ObservableProperty] private int superResScaleIndex; // 0=off, 1=1.5x, 2=2x + [ObservableProperty] private bool artifactReductionAvailable; + [ObservableProperty] private bool superResAvailable; +``` + +Map index→scale once: + +```csharp + // SuperResScaleIndex 0/1/2 -> shim scale 0/15/20. + private static int ScaleFromIndex(int i) => i switch { 1 => 15, 2 => 20, _ => 0 }; + private static int IndexFromScale(int s) => s switch { 15 => 1, 20 => 2, _ => 0 }; +``` + +In the ctor (after the existing `EyeContact*` mirroring) add: + +```csharp + ArtifactReductionAvailable = orchestrator.ArtifactReductionAvailable; + SuperResAvailable = orchestrator.SuperResAvailable; +``` + +In `ProbeCapabilitiesAsync` (after the existing assignments) add: + +```csharp + ArtifactReductionAvailable = _orchestrator.ArtifactReductionAvailable; + SuperResAvailable = _orchestrator.SuperResAvailable; +``` + +Add live-push partials: + +```csharp + partial void OnArtifactReductionEnabledChanged(bool value) => ApplyLiveParams(); + partial void OnSuperResEnabledChanged(bool value) => ApplyLiveParams(); + partial void OnSuperResScaleIndexChanged(int value) => ApplyLiveParams(); +``` + +Extend `BuildParams`: + +```csharp + public ShimParams BuildParams() => new( + CameraId: SelectedCamera?.Id, + GreenScreenEnabled: GreenScreenEnabled, + GreenScreenExpand: GreenScreenExpand, + GreenScreenFeather: GreenScreenFeather, + EyeContactEnabled: EyeContactEnabled, + EyeContactSensitivity: EyeContactSensitivity, + EyeContactLookAwayRange: EyeContactLookAwayRange, + ArtifactReductionEnabled: ArtifactReductionEnabled, + SuperResEnabled: SuperResEnabled, + SuperResScale: ScaleFromIndex(SuperResScaleIndex)); +``` + +Extend `LoadFrom` (after the eye-contact lines): + +```csharp + ArtifactReductionEnabled = config.Effects.ArtifactReductionEnabled; + SuperResEnabled = config.Effects.SuperResEnabled; + SuperResScaleIndex = IndexFromScale(config.Effects.SuperResScale); +``` + +Extend `ToAppConfig`'s `Effects = new EffectSettings { ... }`: + +```csharp + ArtifactReductionEnabled = ArtifactReductionEnabled, + SuperResEnabled = SuperResEnabled, + SuperResScale = ScaleFromIndex(SuperResScaleIndex), +``` + +- [ ] **Step 8: Run the new tests to verify they pass** + +Run: `dotnet test tests/CameraOnScreen.Core.Tests/CameraOnScreen.Core.Tests.csproj --filter "FullyQualifiedName~artifact OR FullyQualifiedName~SuperRes OR FullyQualifiedName~forces_effects_off"` +Expected: PASS. + +- [ ] **Step 9: Mirror the ABI in `PInvokeShim`** — `src/CameraOnScreen.App/Native/PInvokeShim.cs` + +Extend `CosParams`: + +```csharp + [StructLayout(LayoutKind.Sequential)] + private struct CosParams + { + public IntPtr camera_id; + public int green_screen_enabled; public double green_screen_expand; + public double green_screen_feather; + public int eye_contact_enabled; public double eye_contact_sensitivity; + public double eye_contact_look_away_range; + public int artifact_reduction_enabled; + public int super_res_enabled; + public int super_res_scale; + } +``` + +Extend `CosCaps`: + +```csharp + [StructLayout(LayoutKind.Sequential)] + private struct CosCaps + { + public int GreenScreenAvailable; + [MarshalAs(UnmanagedType.ByValArray, SizeConst = 256)] public byte[] Detail; + public int EyeContactAvailable; + [MarshalAs(UnmanagedType.ByValArray, SizeConst = 256)] public byte[] EcDetail; + public int ArtifactReductionAvailable; + public int SuperResAvailable; + } +``` + +Extend `SetParams`' `new CosParams { ... }`: + +```csharp + artifact_reduction_enabled = p.ArtifactReductionEnabled ? 1 : 0, + super_res_enabled = p.SuperResEnabled ? 1 : 0, + super_res_scale = p.SuperResScale, +``` + +Extend `QueryCapabilities`' returned `ShimCapabilities`: + +```csharp + return new ShimCapabilities( + caps.GreenScreenAvailable != 0, ReadUtf8(caps.Detail, 0, 256), + caps.EyeContactAvailable != 0, ReadUtf8(caps.EcDetail, 0, 256), + caps.ArtifactReductionAvailable != 0, caps.SuperResAvailable != 0); +``` + +- [ ] **Step 10: Run the full Core test suite (no regressions)** + +Run: `dotnet test tests/CameraOnScreen.Core.Tests/CameraOnScreen.Core.Tests.csproj` +Expected: all PASS, 0 warnings. + +- [ ] **Step 11: Commit** + +```bash +git add src/CameraOnScreen.Core src/CameraOnScreen.App/Native/PInvokeShim.cs tests/CameraOnScreen.Core.Tests +git commit -m "feat(core): AR/SR params, capability gates, persistence, live push" +``` + +--- + +## Task 8: App UI — toggles, scale combo, 4K frame buffer + +**Files:** +- Modify: `src/CameraOnScreen.App/MainWindow.xaml`, `src/CameraOnScreen.App/MainWindow.xaml.cs` + +**Interfaces:** +- Consumes: `MainViewModel.ArtifactReductionEnabled/SuperResEnabled/SuperResScaleIndex/ArtifactReductionAvailable/SuperResAvailable` (Task 7). + +- [ ] **Step 1: Add controls** — `src/CameraOnScreen.App/MainWindow.xaml` + +After the Eye Contact toggle + its note (`` ending the eye-contact detail), add: + +```xml + + + + + + + +``` + +- [ ] **Step 2: Pre-size the frame buffer to 4K** — `src/CameraOnScreen.App/MainWindow.xaml.cs:25` + +Replace: + +```csharp + // Big enough for 1920x1080 BGRA; the test camera is 640x480. TryGetFrame writes the actual size. + private readonly byte[] _frameBuf = new byte[1920 * 1080 * 4]; +``` + +with: + +```csharp + // Pre-sized to 4K (3840x2160) BGRA so Super Resolution (up to 2x of 1080p) fits without a + // resize. TryGetFrame writes the actual size; cos_get_frame rejects frames larger than this. + private readonly byte[] _frameBuf = new byte[3840 * 2160 * 4]; +``` + +- [ ] **Step 3: Build the App (shim built SDK-config last) — clean** + +Run: +``` +& "C:/Program Files (x86)/Microsoft Visual Studio/2022/BuildTools/MSBuild/Current/Bin/MSBuild.exe" native/shim/shim.vcxproj /p:Configuration=Debug /p:Platform=x64 +dotnet build src/CameraOnScreen.App/CameraOnScreen.App.csproj -t:Rebuild +``` +Expected: 0 warnings, build succeeds. + +- [ ] **Step 4: Commit** + +```bash +git add src/CameraOnScreen.App/MainWindow.xaml src/CameraOnScreen.App/MainWindow.xaml.cs +git commit -m "feat(app): Artifact Reduction + Super Resolution UI, 4K frame buffer" +``` + +--- + +## Task 9: Bundle the new feature DLLs + models + +**Files:** +- Modify: `native/shim/bundle/maxine-manifest.psd1`, `scripts/assemble-maxine-stage.ps1` + +**Interfaces:** +- Consumes: the produced bundle is verified by `trace_closure` / `verify-bundle.ps1`. + +> **Verify at impl:** confirm the exact VFX feature-DLL filenames for Artifact Reduction and Super Resolution in the VFX 1.2.0.0 SDK `bin` tree (alongside `nvVFXGreenScreen.dll`), and the model file globs under `models` for each effect. + +- [ ] **Step 1: Add the feature DLLs to the manifest** — `native/shim/bundle/maxine-manifest.psd1` + +Add the AR + SR feature DLL names to the `Dlls` list (next to the green-screen feature DLL), e.g.: + +```powershell + 'nvVFXArtifactReduction.dll' + 'nvVFXSuperRes.dll' +``` + +- [ ] **Step 2: Ensure the model globs cover the new effects** — `scripts/assemble-maxine-stage.ps1` + +In the VFX model copy step (the one that stages `models\vfx`), confirm the glob includes the AR + SR model files (they live under the same VFX `models` tree as green screen; if the script copies the whole `models\vfx` dir, no change is needed — verify it is not filtering to green-screen-only model names). + +- [ ] **Step 3: Re-stage + re-run the load-closure trace (dev box)** + +Run `scripts/assemble-maxine-stage.ps1` then `scripts/bundle-maxine.ps1 -OutDir -MaxineStage `, then run `native/shim/smoke/trace_closure` (or `bundle_probe`) against the produced bundle with `COS_*` unset. +Expected: AR + SR both load (Probe success); the trace lists the new feature DLLs in the closure; no `not built in`. + +- [ ] **Step 4: Update the manifest `Dlls` list from the trace output** if the trace surfaced additional dependency DLLs for the new effects, then re-run the trace to confirm a stable closure. + +- [ ] **Step 5: Commit** + +```bash +git add native/shim/bundle/maxine-manifest.psd1 scripts/assemble-maxine-stage.ps1 +git commit -m "build(bundle): stage Artifact Reduction + Super Resolution DLLs + models" +``` + +--- + +## Final verification (human gate) + +- [ ] Build shim **SDK config last**, build App, run. Confirm: + - Real fps shows in the status line and drops as effects stack (Task 1). + - Artifact Reduction toggle visibly removes MJPG blockiness on the webcam. + - Super Resolution at 2× visibly adds detail when the overlay is shown large; the overlay re-pins to the larger frame without artifacts. + - Toggles grey out correctly when the probe reports an effect unavailable. +- [ ] Visual/recorder confirmation via Windows.Graphics.Capture / OBS (inherent human gate, `docs/superpowers/verification/`). + +> Reminder (spec §13): stacking AR + Eye Contact + Green Screen + Super Res will not sustain high fps on an RTX 3090 — the fps readout is the tuning signal; no automatic management by design. + +--- + +## Self-Review + +**Spec coverage:** Artifact Reduction (Tasks 3,5,6,7,8,9) ✓; Super Resolution incl. 2× cap + 4K buffer (Tasks 4,5,6,7,8,9) ✓; real fps counter (Task 1) ✓; independent toggles + capability gates + live push + JSON persistence (Task 7) ✓; worker order AR-first/SR-last (Task 6) ✓; ABI parity 528-byte CosCaps + 3 CosParams fields (Tasks 5,7) ✓; bundler (Task 9) ✓; no-auto-management / fps-as-tuning-signal (final gate note) ✓; minification fix correctly **absent** (deferred, spec §11) ✓. + +**Placeholder scan:** no TBD/TODO; "verify at impl" notes name the exact symbol to confirm (selector macros, feature-DLL filenames) — these are real verification steps against headers, not gaps. Every code step shows complete code. + +**Type consistency:** `ShimParams`/`CosParams` field names and order match across `shim.h`, `PInvokeShim`, `Contracts.cs`, `BuildParams`. `SuperResScale` is the shim-facing int (0/15/20); `SuperResScaleIndex` is the VM/combo int (0/1/2) with `ScaleFromIndex`/`IndexFromScale` as the single conversion point. `ArtifactReductionAvailable`/`SuperResAvailable` consistent across `CosCaps`, `ShimCapabilities`, `FakeShim`, `Orchestrator`, `MainViewModel`. Effect class APIs (`Probe`/`Start`/`Stop`/`ProcessFrame`/`IsReady`/`LastError`) consistent with the `capture.cpp` call sites in Task 6. diff --git a/docs/superpowers/plans/2026-06-24-issue-15-vsr-resume.md b/docs/superpowers/plans/2026-06-24-issue-15-vsr-resume.md new file mode 100644 index 0000000..9f34c1b --- /dev/null +++ b/docs/superpowers/plans/2026-06-24-issue-15-vsr-resume.md @@ -0,0 +1,141 @@ +# Issue #15 Resume — VSR-only AI sharpness (amends the parked Phase-1 plan) + +> **For agentic workers:** Use superpowers:subagent-driven-development or superpowers:executing-plans. +> **When implementing any App/XAML task (T7), invoke the WinUI skills first:** `winui-design` +> (XAML correctness, Fluent layout, theming, accessibility review) and `winui-dev-workflow` +> (build/run). `winui-app` for broader WinUI guidance if needed. + +**Issue:** #15 — Parked AI sharpness effects. +**Supersedes the effect tasks of:** `docs/superpowers/plans/2026-06-24-ai-sharpness-resolution-phase1.md` +(parked branch `feat/ai-sharpness-resolution`). **Spec:** `docs/superpowers/specs/2026-06-23-camera-on-screen-ai-sharpness-resolution-design.md` — needs a spec update to drop Artifact Reduction and re-scope Super Resolution to VSR-only (see T0). + +## Decision (user, 2026-06-24) +**VSR-only + mode combo.** One effect (Video Super Resolution), expose its `QualityLevel` as a +mode selector covering **Upscale** *and* **Denoise/Deblur** — so it also covers the image-cleanup +goal the (dead) Artifact Reduction was meant to serve. **No second effect.** (`nvvfxdenoising` is +real + 4-arch on NGC and is the only viable stack-on-top add, but only needed if denoise must run +*in the same frame as* upscale — not wanted now.) + +## Grounded facts (verified 2026-06-24, see memory `vfx-vsr-ngc-grounded`) +- **Real VFX 1.2.0.0 catalog** (live NGC, collection `maxine_vfx_sdk`, 8 models): aigsrelighting, + backgroundblur, denoising, greenscreen, relighting, transfer, upscale, **videosuperres**. +- **Artifact Reduction is DEAD** — absent on disk, only a help-text typo in `install_feature.ps1`, + and live NGC returns `PAYMENT_REQUIRED`/"key lacks permission" for `nvvfxartifactreduction` + while all 8 real features return data. +- **Real VSR API** (`features/nvvfxvideosuperres/include/nvVFXVideoSuperRes.h`): selector + `NVVFX_FX_VIDEO_SUPER_RES "VideoSuperRes"`, param `NVVFX_QUALITY_LEVEL "QualityLevel"`. Image + in+out = **BGRA/RGBA u8 GPU** (matches our pipeline — no BGR convert). QualityLevel values: + 0 Bicubic, 1-4 VSR Low/Med/High/Ultra (upscale), 8-11 Denoise, 12-15 Deblur, 16-19 HighBitrate. + **Denoise/Deblur do NOT upscale (out must == in).** +- **VSR ships NO per-arch engine** — model baked into `nvngx_vsr.dll` (NGX, 45MB) + dispatcher + `nvVFXVideoSuperRes.dll`. **Runs on every RTX arch with zero per-arch deserialize risk** — + sidesteps the non-Ampere verify gate that greys out green screen / gaze. +- **T1 PROBE PASSED on the 3090** — `CreateEffect("VideoSuperRes")` → `SetU32(QualityLevel)` → + `Load` → `Run` all `0 No error` for quality=1 (720→1440 2x upscale) and quality=8 (denoise, + out==in). Confirms: QualityLevel takes the values directly (single param, no separate Mode); + scale inferred from out/in dims; `nvngx_vsr.dll` resolves from a flat dir via + `g_nvVFXSDKPath`→SetDllDirectory. Probe sources in session scratchpad + (`vsr_probe.cpp` + `build_run_vsr_probe.bat`). + +## Reuse from parked branch (do NOT rebuild) +`vfx_paths.{h,cpp}` (green screen adopted it); C-ABI effect slots + capture-worker wiring +(lazy start/stop, leaf-lock errors, dynamic-resolution publish); managed Orchestrator gating, +MVVM scale combo, persistence, 4K frame buffer, App UI scaffold, 57 xUnit tests. + +## Branch +After PR #14 (`feat/fps-and-capture-mode` = fps counter + highest-fps capture) merges, branch +`feat/ai-sharpness-vsr` off `main`; rebase the parked branch's effect commits on (the fps/capture +patches dedupe to empty since #14 carries them). The parked branch is the substrate. + +## Compile-testing the real SDK shim on THIS host (no dev SDK at `C:\dev`) + +The interactive user's `C:\dev` VFX/AR trees are absent on this box, **but the self-hosted runner +SDK is present and usable** for real-headers compile tests of any shim task (T2–T9): + +- **SDK roots:** `C:\actions-runner\_sdk\VideoFX` (VFX 1.2.0.0), `C:\actions-runner\_sdk\Maxine-AR-SDK-1.1.1.0`. +- **NGC key** (for `install_feature.ps1`) lives in the gitignored repo `.env` as + `$env:NGC_CLI_API_KEY="nvapi-…"`. **Dot-sourcing `.env` is flaky** (no trailing newline) — extract + it: `$raw = Get-Content .env -Raw; if ($raw -match 'nvapi-[A-Za-z0-9_\-]+') { $env:NGC_CLI_API_KEY = $matches[0] }`. +- **VSR feature must be installed once** (was missing — only greenscreen shipped): + `cd C:\actions-runner\_sdk\VideoFX\features; .\install_feature.ps1 -features nvvfxvideosuperres`. + → installs `features\nvvfxvideosuperres\{include\nvVFXVideoSuperRes.h, bin\nvVFXVideoSuperRes.dll, bin\nvngx_vsr.dll}`. + **"No models found for SM86 (normal)" is expected** — VSR is NGX, baked into `nvngx_vsr.dll`, no per-arch engine. +- **Build the real path:** MSBuild `native/shim/shim.vcxproj /p:Configuration=Debug /p:Platform=x64` + `/p:CosVfxSdkDir=C:\actions-runner\_sdk\VideoFX /p:CosArSdkDir=C:\actions-runner\_sdk\Maxine-AR-SDK-1.1.1.0`. +- **Export-verify the deployed DLL** (`native/shim/x64/Debug/CameraOnScreen.Shim.dll`): `grep -ac` + finds `GreenScreen` + `GazeRedirection` + `VideoSuperRes`, and `not built in` is **absent**. +- **Confirmed 2026-06-24:** T2 superres.cpp compiles clean against the real header (0 warnings, + both effects). The header's `NVVFX_FX_VIDEO_SUPER_RES "VideoSuperRes"` + `NVVFX_QUALITY_LEVEL + "QualityLevel"` match the rewrite. This only proves **compilation**; runtime VSR (Load/Run on real + frames) is still the T9 human gate on the 3090. + +## Tasks + +- [x] **T1 — Probe VSR (DONE, 2026-06-24).** Verified the real API on the 3090 (above). + +- [x] **T0 — Spec update (DONE, 2026-06-24).** Added a prominent AMENDED banner to the design spec + (`…ai-sharpness-resolution-design.md`): AR dropped (doesn't exist), SR re-scoped to NGX VSR with + the real API + QualityLevel mode selector, generic plumbing retained, points here. Banner over + line-by-line rewrite — the spec is implemented design-history; this plan + the code are authoritative. + +- [x] **T2 — Rewrite `superres.{h,cpp}` (DONE, 2026-06-24).** Real `"VideoSuperRes"` selector + + `nvVFXVideoSuperRes.h` (added the feature include dir to `shim.vcxproj`); BGRA u8 GPU in+out + (BGR convert + per-pixel repack deleted → per-row `memcpy` honoring pitch, alpha flows through); + `Start(int qualityLevel, int scaleX10)` sets `NVVFX_QUALITY_LEVEL`; upscale 1-4 → out=in×scale, + clean 8-15 → out=in. Probe mirrors the verified Start sequence. **Built clean against the REAL + header on the runner SDK** (0 warnings, both effects + VSR — see "Compile-testing" above), plus a + stub build. Call sites stubbed to keep the tree green: `capture.cpp` passes quality `1` (ponytail + comment, T5/T6 thread real mode), `effects_smoke.cpp` updated. Original spec below: + - selector `"VideoSuperRes"`, include the real header. + - **BGRA u8 GPU in+out** — delete the BGR conversion and the per-pixel alpha-repack loop. + - `Start(int qualityLevel, int scaleX10)`; set `NVVFX_QUALITY_LEVEL` (not `NVVFX_MODE`). + - Upscale modes 1-4 → out = in×scale; Denoise 8-11 / Deblur 12-15 → **out = in** (enforce). + - NGX path via `vfx_paths` (dir holding `nvngx_vsr.dll`). + +- [x] **T3 — Delete Artifact Reduction (DONE — prior commit `1e111aa`).** Verified no AR remnants + remain in code/tests/xaml/manifest (grep clean). + +- [x] **T4 — C ABI (DONE, 2026-06-24).** Added `super_res_quality_level` to `CosParams` (C `shim.h` + + `[StructLayout]` mirror in `PInvokeShim`) and the `ShimParams` record (`FakeShim` records it via + the record; `Contracts` updated). Managed computes the single QualityLevel from mode+quality, so + no CosCaps change needed. The 9 `cos_*` exports unchanged. + +- [x] **T5 — Capture worker (DONE, 2026-06-24).** `SetSuperRes(enabled, qualityLevel, scale)` + a + `superResQuality` atomic; worker restarts the effect on live quality/scale change (both baked at + Start); replaced the T2 hardcoded quality. SR runs last, reuses the dynamic-resolution publish. + +- [x] **T6 — Managed + Core + tests (DONE, 2026-06-24).** `EffectSettings` persists + `SuperResMode`+`SuperResQuality`+`SuperResScale`; MVVM `SuperResModeIndex`/`SuperResQualityIndex`/ + `SuperResScaleIndex` map to QualityLevel (Upscale 1-4 / Denoise 8-11 / Deblur 12-15) with scale=0 + for clean modes; live push + persistence round-trip. **57 xUnit tests green.** + +- [x] **T7 — App UI (DONE, 2026-06-24; `winui-design` invoked).** Mode / quality / upscale-factor + combos (quality enabled when mode≠Off, factor only for Upscale — code-behind `x:Bind` funcs). + Window restructured: `ScrollViewer` (row 0 `*`) + pinned Maxine attribution footer (row 1 `Auto`), + fixing the dropped-footer bug. App builds 0 warnings. (Build via CLAUDE.md's authoritative + unpackaged command, which overrides the generic `winui-dev-workflow` script on this host.) + +- [ ] **T8 — Bundler.** Stage `nvVFXVideoSuperRes.dll` + `nvngx_vsr.dll` into the flat `maxine\`; + **no per-arch engines** for VSR. **Re-run `trace_closure`** — the NGX runtime may pull DLLs + outside the current 19-DLL closure (bundle size unknown until traced). Re-check the VSR model + license (`NVIDIA-Open-Model-License-Agreements-...pdf` in the feature dir) covers redistribution + like green screen. + +- [x] **T8 — Bundler (DONE, 2026-06-24).** `assemble-maxine-stage.ps1` now stages + `nvVFXVideoSuperRes.dll` + `nvngx_vsr.dll` (NGX, no per-arch engine). Manifest `Dlls` += those 2 + plus `nvngxruntime.dll` (NGX loader, pulled from VFX\bin) — the closure delta confirmed by + `dumpbin` static imports AND by **`trace_closure` re-run on the 3090: all 3 effects Start=1, exit + 0, 22 modules** (19 prior + 3 VSR, no dynamic surprises). `trace_closure.cpp` + + `build_trace_closure.bat` updated to exercise VSR (and the stale bat fixed to compile + `vfx_paths.cpp`). VSR license = Open Model + Software + PST, **already in the manifest** — no new + file. Bundle-prune gate (against the produced `maxine\`) folds into T9/release verify. + +- [ ] **T9 — Verify (human gate, 3090).** Build the SDK shim config **last**, export-verify, run: + upscale 1080→2160 visibly sharper; denoise mode cleans; toggles gate correctly off when probe + unavailable. (Native Load/Run already proven by the T8 trace; remaining gate is **visual**.) + +## Open risks — both RESOLVED (2026-06-24) +1. ~~NGX DLL closure unknown~~ — **resolved**: `trace_closure` on the 3090 shows exactly 3 added + DLLs (`nvVFXVideoSuperRes.dll`, `nvngxruntime.dll`, `nvngx_vsr.dll`); 22 modules total. +2. ~~VSR model license re-check~~ — **resolved**: VSR's feature dir ships the same Open Model + + Software License + Product-Specific Terms already bundled; redistribution basis unchanged. diff --git a/docs/superpowers/specs/2026-06-23-camera-on-screen-ai-sharpness-resolution-design.md b/docs/superpowers/specs/2026-06-23-camera-on-screen-ai-sharpness-resolution-design.md new file mode 100644 index 0000000..79da8bd --- /dev/null +++ b/docs/superpowers/specs/2026-06-23-camera-on-screen-ai-sharpness-resolution-design.md @@ -0,0 +1,239 @@ +# Camera-on-Screen — Design Spec: AI Sharpness & Resolution (Phase 1) + +> **⚠️ AMENDED 2026-06-24 (issue #15) — re-scoped to VSR-only; implemented.** Two design +> assumptions below are **superseded** by what was verified against the live NGC catalog + the +> real VFX 1.2.0.0 headers (memory `vfx-vsr-ngc-grounded`): +> 1. **Artifact Reduction is DROPPED — it does not exist.** `NVVFX_FX_ARTIFACT_REDUCTION` is absent +> from the VFX 1.2.0.0 catalog (only a help-text typo in `install_feature.ps1`); NGC returns +> `PAYMENT_REQUIRED` for it. All `artifactreduction.{h,cpp}` / CosParams / CosCaps / UI / test +> references were removed. +> 2. **Super Resolution = NGX VSR, not the old `SuperRes` effect.** Real selector +> `NVVFX_FX_VIDEO_SUPER_RES "VideoSuperRes"`, param `NVVFX_QUALITY_LEVEL "QualityLevel"` (NOT +> `NVVFX_MODE`/`NVVFX_STRENGTH`/a scale param). BGRA u8 GPU in+out. **QualityLevel is the mode +> selector**: 1-4 upscale (Low/Med/High/Ultra, scaled by 1.5×/2×), 8-11 denoise, 12-15 deblur — +> so VSR's clean modes cover the image-cleanup goal the dead Artifact Reduction was meant to +> serve. VSR is NGX (model baked into `nvngx_vsr.dll`) → **no per-arch TRT engine**, runs on +> every RTX arch. Bundle delta = 3 DLLs (`nvVFXVideoSuperRes.dll`, `nvngxruntime.dll`, +> `nvngx_vsr.dll`), confirmed by `trace_closure` on the 3090 (22 modules). +> +> The **generic plumbing** below (worker-chain placement, dynamic output-resolution publish, 4K +> `_frameBuf`, factor cap, present-path ripple, single-effect MVVM/persistence pattern) still holds. +> Authoritative current scope + task state: `docs/superpowers/plans/2026-06-24-issue-15-vsr-resume.md`. +> Treat any `NVVFX_FX_SUPER_RES` / `NVVFX_MODE` / Artifact-Reduction wording below as historical. + +**Date:** 2026-06-23 +**Status:** AMENDED + implemented 2026-06-24 (issue #15, VSR-only) — see banner above +**Depends on:** M3 (AI Green Screen) + M4 (Eye Contact) — both merged to `main`. Reuses the +worker-thread effect chain (`aigs.{h,cpp}` pattern), the capability probe, and the live-param push. +**Parent spec:** `docs/superpowers/specs/2026-06-20-camera-on-screen-design.md`. +**Phase 2 (separate, deferred):** fps interpolation — GitHub issue #13. + +## 1. Goal + +Make the overlay image sharper and higher-resolution than the camera hardware delivers, using +NVIDIA Maxine AI. Two distinct user-visible wins: + +1. **Cleaner image** — remove the MJPG/H.264 compression artifacts the webcam emits at + 1080p30 (the source of the "softer than the native camera app" complaint). +2. **More effective resolution** — AI-upscale the source so the overlay stays sharp when shown + large, not just at widget size. + +The overlay display size **varies** (small widget ↔ near-fullscreen). Phase 1 cleans compression +artifacts (helps all sizes) and adds AI-upscaled detail (helps large sizes, where there is little +or no downscale). The remaining **small-size softness** — bilinear minification in the present path +— is a separate present-path redesign and is **deferred** (see §2 and §11), not solved by the +Maxine effects here. + +## 2. Scope + +**In scope (Phase 1):** +- **Artifact Reduction** (Maxine VFX `NVVFX_FX_ARTIFACT_REDUCTION`) — removes compression + artifacts, in-place at source resolution, no dimension change. +- **Super Resolution** (Maxine VFX `NVVFX_FX_SUPER_RES`) — AI upscale, off / 1.5× / 2×, writes a + larger output buffer. +- **Real fps counter** — replace the hardcoded `cos_get_status` `30.0` stub with a measured value + so the user can see each effect's cost. +- Independent on/off toggles per effect; capability gating; live-param push; JSON persistence. + +**Out of scope:** +- **Present-path minification fix (deferred — see §11).** The small-size bilinear-minification + softness needs a present-path redesign (window-sized back buffer + mipped shader scale), which + collides with load-bearing invariants (pinned-to-frame-res back buffer, mirror/zoom in the DComp + transform, the MPO hit-test gotcha). Separated from the Maxine effects to keep Phase 1 low-risk. + To be filed as its own issue if pursued. +- **fps interpolation (Phase 2)** — NVIDIA Optical Flow FRUC, separate SDK, co-version risk. + GitHub issue #13. Gated on a standalone co-version smoke test before any integration. +- **Automatic GPU-budget management** — no scheduler, no mutual exclusion, no auto-drop. The user + toggles effects and watches the fps readout (explicit decision). +- **Maxine Video Noise Removal / Upscale (non-AI) / relighting** — not needed for this goal. +- **3×/4× Super Res** — capped at 2× (see §7). + +## 3. Why this is low-risk: no new co-version + +Super Resolution and Artifact Reduction live in the **same Maxine VideoFX SDK** already integrated +for green screen (VFX 1.2.0.0). Same `NVVideoEffects.dll` dispatcher, same per-feature-DLL model, +**same TensorRT 10.9 / CUDA 12.x runtime**. Adding them is **not** a new co-version — unlike Phase 2's +Optical Flow SDK. They compile behind the existing `COS_HAS_MAXINE` flag; without the SDK they build +as passthrough stubs, exactly like green screen. + +## 4. Worker-thread effect chain + +`capture.cpp WorkerLoop`, per frame (each stage gated on its atomic enable flag, lazy Start/Stop +identical to the current green-screen block): + +``` +decode (RGB32 BGRA) + → [Artifact Reduction] in-place, source res + → Eye Contact (existing) + → Green Screen (existing) + → [Super Resolution] → LARGER output buffer, new w/h + → publish frame (w/h = SR output size) +``` + +**Order rationale:** +- Artifact Reduction runs **first** so the AI effects downstream consume a clean frame (better + matte and landmarks). +- Super Resolution runs **last** so the expensive upscale happens once on the final composite, and + the green-screen matte + gaze landmarks compute at native 1080p (cheaper, and the models are + tuned for that resolution). + +Maxine effects remain worker-thread-local (CUDA/NvVFX thread affinity); the UI only flips atomic +enable flags; status crosses threads via atomics + leaf-lock (never nested under `g_state.mtx` / +`g_lifecycleMtx`) — unchanged invariants. + +## 5. New shim effect modules + +Two files each, lifecycle shape copied from `aigs.{h,cpp}` +(`Start()/Stop()/IsReady()/ProcessFrame()/LastError()`, worker-thread-local): + +- `native/shim/artifactreduction.{h,cpp}` — `NVVFX_FX_ARTIFACT_REDUCTION`, mode 1 + (strong / for-compressed). Operates in place on the BGRA scratch buffer at source resolution. + **No dimension change.** +- `native/shim/superres.{h,cpp}` — `NVVFX_FX_SUPER_RES`, mode 1, upscale factor from params + (1.5× or 2×). Allocates/owns a **larger** output buffer; `ProcessFrame` returns the new width and + height to the worker, which publishes them. + +Both behind `COS_HAS_MAXINE` → CI-safe passthrough stub when the SDK is absent. The exact +feature-DLL names and Maxine property keys (`NVVFX_STRENGTH`, `NVVFX_MODE`, the upscale-factor key) +are verified against the VFX 1.2.0.0 headers at implementation time. + +## 6. Dimension-change plumbing (the only structurally new part) + +Super Res makes published frames larger than 1080p, which ripples through the consumer: + +- **`MainWindow.xaml.cs:25` `_frameBuf`** is a fixed `1920*1080*4` array — would overflow. + **Pre-size it to 4K** (`3840*2160*4` ≈ 33 MB). One constant change; no dynamic realloc logic. + *(ponytail: pre-size beats a resize-on-demand path; 33 MB is negligible.)* This is the hard cap + on SR output and is the reason SR is capped at 2× of 1080p. +- **Swap chain** — `OverlayWindow.PresentFrame` **already** re-pins the back buffer to the incoming + frame size (`if (_bufW != width || _bufH != height)`). SR's larger frame re-pins once on first + arrival. No change needed. +- **Frame C-ABI** already returns dynamic `w`/`h` from `get_frame`. No contract change for size. + +## 7. Super Res factor cap + +Capped at **2×** (1080p → 2160p). Rationale: 2× of 1080p hits exactly 4K, matching the pre-sized +`_frameBuf`; 3×/4× would need a bigger buffer and far more GPU for diminishing perceptual gain at +overlay sizes. Exposed as off / 1.5× / 2×. Higher factors are a later change if ever needed. + +## 8. C-ABI parity (load-bearing — x64 byte-for-byte) + +Both `shim.h` (C) and `PInvokeShim` (`[StructLayout(Sequential)]` mirror) change together: + +- **`CosParams`** — add `int artifactReductionEnabled`, `int superResEnabled`, `int superResScale` + (`0` = off, else factor ×10: `15` = 1.5×, `20` = 2×). +- **`CosCaps`** — add `int artifactReductionAvailable`, `int superResAvailable` gates. The struct + grows from two `int` gates + `char detail[512]` = **520 bytes** to **four** `int` gates + + `detail[512]` = **528 bytes**; update the size comment in `shim.h` and the `[StructLayout]` C# + mirror together. +- **`cos_query_capabilities`** — extend the probe to create + load each new effect, exactly as it + already does for green screen and gaze, and set the new gates from the result. + +## 9. Real fps counter + +Replace the hardcoded `30.0` in `cos_get_status`: +- The capture worker increments an atomic frame count and stamps `std::chrono::steady_clock` at + publish. +- `cos_get_status` computes fps over a rolling window (or simple EWMA) under the existing status + leaf-lock — never nested under `g_state.mtx`. +- The existing C# `Fps` property (already polled every pump tick → `PollStatus` → `GetStatus`) now + shows the real value. No new plumbing on the managed side. + +## 10. C# / MVVM (Core + App) + +- **`MainViewModel`** — new observable props `ArtifactReductionEnabled`, `SuperResEnabled`, + `SuperResScale`. Their `On…Changed` partials call `ApplyLiveParams()` → + `Orchestrator.ApplyParams(BuildParams())` → `shim.SetParams`, gated on `IsRunning` + + `EffectsAvailable`, identical to the green-screen props. `ApplyParams` already forces effects off + when `EffectsAvailable` is false — the new flags inherit that. +- **`BuildParams()`** — copy the three new fields into `ShimParams`/`CosParams`. +- **`ToAppConfig` / `AppConfig`** — persist the three fields in `config.json` (anything not copied + reverts to default — must add explicitly). +- **XAML** — two toggles + a Super Res factor combo, bound `Mode=OneWay` to the availability gates + (greyed until the deferred probe lands). The status line shows the now-real fps. + +## 11. Present-path minification softness (DEFERRED — not built in Phase 1) + +Recorded here so the analysis is not lost; this is **out of scope** for Phase 1 (§2). + +**Root cause (confirmed in code):** the back buffer is pinned to the **frame** resolution +(`PresentFrame` — `CopyResource(back, _frameTex)` is 1:1, `_frameTex` is `MipLevels = 1`), and the +frame-res → window scaling is done by the **DComp visual transform** (`UpdateScale` → `SetTransform`), +which samples **bilinear** with no mip control. A 1080p → ~480px widget downscale therefore +undersamples → softness/aliasing. + +**Why it is not a drop-in blit swap:** putting mips on `_frameTex` and drawing a quad into the +*current* frame-res back buffer changes nothing — DComp still does the final minification afterward +and ignores our mips. A correct fix requires moving the scale into our own D3D draw: +- back buffer becomes **window-sized** (drops the pinned-to-frame-res / `CopyResource` 1:1 invariant), +- draw a fullscreen quad sampling a **mipped** `_frameTex` (`GENERATE_MIPS` + `GenerateMips()`) with + a **trilinear** sampler, +- DComp transform becomes **identity**, and **mirror + center-zoom** move out of the DComp + `Matrix3x2` into the quad's transform, +- re-verify `WS_EX_NOREDIRECTIONBITMAP` clean capture and the size-dependent **MPO hit-test** gotcha + still hold with a window-sized back buffer. + +This touches load-bearing present-path invariants, so it is its own milestone. Note: Super Res does +**not** substitute for it — the upscaled detail is also thrown away by the DComp bilinear downscale +at small sizes. + +## 12. Bundler / installer + +- `scripts/assemble-maxine-stage.ps1` + `native/shim/bundle/maxine-manifest.psd1` — add the + SuperRes + ArtifactReduction feature DLLs and their model files to the curated stage. Same VFX + runtime, so **no new TRT/CUDA DLLs** — only the two feature DLLs + models. +- Re-run `native/shim/smoke/trace_closure.cpp` against the produced bundle to capture the new + load closure; update the manifest `Dlls` list from its output. +- `verify-bundle.ps1` runtime probe (`COS_*` unset) must still load all effects. + +## 13. Performance stance + +Independent toggles, real fps readout, **no auto-management** (explicit user decision). Stacking +four AI inferences (Artifact Reduction + Eye Contact + Green Screen + Super Res) will not sustain +high fps on an RTX 3090; the user watches the fps readout and disables to taste. Document this in +the UI note + CLAUDE.md. No scheduler, no mutual exclusion (YAGNI). + +## 14. Testing + +- **Core unit tests** — `BuildParams` / `ToAppConfig` round-trip the three new fields; + `ApplyParams` forces them off when `EffectsAvailable == false` (extend existing green-screen tests). +- **Native smoke** — extend `cos_query_capabilities` probe coverage; an AR/SR `Start` + + `ProcessFrame` smoke on a synthetic frame returns success and (for SR) a larger output size + (mirrors `aigs_smoke.cpp`). +- **fps counter self-check** — feed N frames over a known elapsed time, assert reported fps ≈ N/elapsed. +- **Human gate** — visual sharpness confirmation via a real recorder (Windows.Graphics.Capture / + OBS), the inherent verification gate (`docs/superpowers/verification/`). + +## 15. Incremental landing order + +Each step is independently shippable and independently valuable: + +1. **Real fps counter** — needed to tune everything after. +2. **Artifact Reduction** — clean source, no dimension change (low risk). +3. **Super Resolution** + 4K `_frameBuf` pre-size — the dimension-change step, landed last. + +## 16. Open questions / verify-at-impl + +- Exact VFX 1.2.0.0 feature-DLL filenames + property keys for SuperRes / ArtifactReduction. +- Whether Super Res mode 1 (compressed) or mode 0 looks better on this webcam's MJPG output — + decide visually during the human gate. diff --git a/native/shim/aigs.cpp b/native/shim/aigs.cpp index 27951b6..79cd156 100644 --- a/native/shim/aigs.cpp +++ b/native/shim/aigs.cpp @@ -14,66 +14,7 @@ #include "nvVideoEffects.h" #include "nvVFXGreenScreen.h" // defines NVVFX_FX_GREEN_SCREEN "GreenScreen" #include "paths.h" - -// The proxy stub (nvVideoEffectsProxy.cpp) declares this as extern; we define it here. -// When non-null it points at the dir that holds NVVideoEffects.dll, which the proxy -// passes to SetDllDirectory before LoadLibrary. -// (Defined in Task 1 — NOT redefined here.) -char* g_nvVFXSDKPath = nullptr; -// Note: NVCVImage.dll resolves via the search path the VFX proxy's SetDllDirectory sets; -// nvCVImageProxy.cpp has no g_nvCVImageSDKPath global, so none is defined here. - -namespace { -// Resolves "\bin" (DLLs) and "\bin\models" once. -// Returns false if the env var is unset/empty. Stores results in out-params so -// callers can keep them alive as needed. -bool ResolveSdkPaths(std::string& binDir, std::string& modelDir, std::string& err) { - char buf[1024] = {0}; - // Prefer COS_VFX_RUNTIME_DIR — a FLAT runtime layout (DLLs in , models in - // \models). This pins the green-screen VFX runtime to the AR-matched 0.7.6 SDK so - // both Maxine SDKs load the SAME TensorRT 10.4 / CUDA 12.1 runtime (co-versioned — a - // mismatch makes NvVFX_Load fail with "no kernel image"). Fall back to COS_VFX_SDK_DIR - // (legacy full-SDK tree: DLLs in \bin, models in \bin\models). - DWORD n = GetEnvironmentVariableA("COS_VFX_RUNTIME_DIR", buf, sizeof(buf)); - if (n > 0 && n < sizeof(buf)) { - std::string root(buf, n); - if (!root.empty() && (root.back() == '\\' || root.back() == '/')) root.pop_back(); - binDir = root; - modelDir = root + "\\models"; - return true; - } - n = GetEnvironmentVariableA("COS_VFX_SDK_DIR", buf, sizeof(buf)); - if (n > 0 && n < sizeof(buf)) { - std::string root(buf, n); - if (!root.empty() && (root.back() == '\\' || root.back() == '/')) root.pop_back(); - binDir = root + "\\bin"; - modelDir = root + "\\bin\\models"; - return true; - } - // Ship default: runtime bundled beside the app at \maxine (shared co-versioned - // root; green-screen models in models\vfx). - std::string appDir = ShimModuleDir(); - if (!appDir.empty()) { - std::string maxine = appDir + "\\maxine"; - if (DirExists(maxine)) { - binDir = maxine; - modelDir = maxine + "\\models\\vfx"; - return true; - } - } - err = "VFX runtime not found: set COS_VFX_RUNTIME_DIR or bundle maxine\\ beside the app"; - return false; -} - -// Points the VFX proxy global at the bin dir so NVVideoEffects.dll is found. -// NVCVImage.dll resolves from the same search path that VFX proxy sets. -// Idempotent; s_bin must outlive every Maxine call. -void PointProxiesAt(const std::string& binDir) { - static std::string s_bin; - s_bin = binDir; - g_nvVFXSDKPath = const_cast(s_bin.c_str()); -} -} // namespace +#include "vfx_paths.h" // Real per-effect state, hidden behind the opaque impl_ pointer. struct AigsImpl { @@ -95,8 +36,8 @@ Aigs::~Aigs() { Stop(); } bool Aigs::Probe(std::string& detail) { std::string binDir, modelDir, err; - if (!ResolveSdkPaths(binDir, modelDir, err)) { detail = err; return false; } - PointProxiesAt(binDir); + if (!vfx::ResolveSdkPaths(binDir, modelDir, err)) { detail = err; return false; } + vfx::PointProxiesAt(binDir); // Probe uses the SDK's default CUDA stream (no explicit CudaStreamCreate); sufficient for a load-only check. NvVFX_Handle eff = nullptr; @@ -128,10 +69,10 @@ bool Aigs::Start() { if (!impl) { lastError_ = "out of memory"; return false; } std::string binDir, err; - if (!ResolveSdkPaths(binDir, impl->modelDir, err)) { + if (!vfx::ResolveSdkPaths(binDir, impl->modelDir, err)) { lastError_ = err; delete impl; return false; } - PointProxiesAt(binDir); + vfx::PointProxiesAt(binDir); if (NvVFX_CudaStreamCreate(&impl->stream) != NVCV_SUCCESS) { lastError_ = "NvVFX_CudaStreamCreate failed"; delete impl; return false; diff --git a/native/shim/bundle/maxine-manifest.psd1 b/native/shim/bundle/maxine-manifest.psd1 index 4b30239..cf425d7 100644 --- a/native/shim/bundle/maxine-manifest.psd1 +++ b/native/shim/bundle/maxine-manifest.psd1 @@ -5,8 +5,13 @@ # # The Dlls list was PRODUCED by running native/shim/smoke/trace_closure against the assembled # co-versioned stage on an RTX 3090 (sm86) and enumerating the modules actually loaded from - # maxine\ (both effects Start=1, 19 modules). Re-run the trace and update this list on any SDK - # bump. See docs/superpowers/specs/2026-06-23-camera-on-screen-m5-multigpu-findings.md. + # maxine\ (all three effects Start=1, 22 modules). Re-run the trace and update this list on any + # SDK bump. See docs/superpowers/specs/2026-06-23-camera-on-screen-m5-multigpu-findings.md. + # + # VSR (issue #15) added 3 DLLs (static-import closure of nvVFXVideoSuperRes.dll via dumpbin): + # nvVFXVideoSuperRes.dll (feature) -> nvngxruntime.dll (NGX loader, from VFX\bin) + nvngx_vsr.dll + # (NGX model, 45 MB). VSR is NGX not TensorRT, so it pulls NO nvinfer/engine; nvcuda.dll is the + # driver DLL (System32) and is never bundled. NVCVImage.dll is shared (already listed). # # SOURCE MODEL (bundle-maxine.ps1): the bundler PRUNES a pre-assembled, co-versioned flat # "stage" dir (-MaxineStage) down to this allow-list + model globs + license set. The stage is @@ -36,6 +41,9 @@ 'nvonnxparser_10.dll' 'NVVideoEffects.dll' # VFX dispatcher 'nvVFXGreenScreen.dll' # VFX green-screen feature + 'nvVFXVideoSuperRes.dll' # VFX video-super-res feature (NGX) + 'nvngxruntime.dll' # NGX loader (pulled by nvVFXVideoSuperRes.dll) + 'nvngx_vsr.dll' # NGX VSR model (no per-arch engine) 'nvARPose.dll' # AR dispatcher 'nvARGazeRedirection.dll' # AR gaze feature 'nvARFaceBoxDetection.dll' # AR gaze dep diff --git a/native/shim/capture.cpp b/native/shim/capture.cpp index 4fcd1df..3bd034d 100644 --- a/native/shim/capture.cpp +++ b/native/shim/capture.cpp @@ -1,6 +1,7 @@ #include "capture.h" #include "aigs.h" #include "eyecontact.h" +#include "superres.h" #include #include @@ -48,6 +49,13 @@ struct CaptureState { std::mutex ecErrMtx; // leaf lock, never nested under mtx/lifecycle std::string ecError; // guarded by ecErrMtx + std::atomic superResEnabled{false}; + std::atomic superResScale{20}; // 15 or 20 + std::atomic superResQuality{1}; // VSR QualityLevel (mode): 1-4 / 8-11 / 12-15 + std::atomic superResActive{false}; + std::mutex srErrMtx; // leaf lock + std::string srError; + std::atomic framesProduced{0}; // bumped by worker after each published frame FpsCounter fpsCounter; // read only by MeasuredFps (status-poll thread) }; @@ -302,6 +310,8 @@ void Capture::WorkerLoop() { // affinity. Declare here (after CoInitializeEx) so ctor/dtor run on this thread. Aigs aigs; EyeContact eyeContact; + SuperRes superRes; + int srAppliedQuality = 0, srAppliedScale = 0; // last (quality,scale) Start ran with; for live restart IMFSourceReader* reader = nullptr; int width = 0, height = 0; @@ -369,6 +379,47 @@ void Capture::WorkerLoop() { } g_state.eyeContactActive.store(ecApplied, std::memory_order_release); + // Super Resolution runs BEFORE green screen: it sharpens/upscales the raw RGB, + // then green screen authors the final alpha matte on the result. (Running SR + // last clobbered the matte — NGX VSR does not preserve alpha — so a green- + // screened overlay went opaque/black.) SR may change the frame dimensions. + const bool srWant = g_state.superResEnabled.load(std::memory_order_acquire); + const int srScale = g_state.superResScale.load(std::memory_order_acquire); + const int srQuality = g_state.superResQuality.load(std::memory_order_acquire); + // QualityLevel + scale are baked at Start; restart the effect if either changed live. + if (srWant && superRes.IsReady() && (srQuality != srAppliedQuality || srScale != srAppliedScale)) + superRes.Stop(); + if (srWant && !superRes.IsReady()) { + if (!superRes.Start(srQuality, srScale)) { + std::lock_guard e(g_state.srErrMtx); + const std::string& ne = superRes.LastError(); + if (g_state.srError != ne) g_state.srError = ne; + } else { + srAppliedQuality = srQuality; srAppliedScale = srScale; + } + } else if (!srWant && superRes.IsReady()) { + superRes.Stop(); + std::lock_guard e(g_state.srErrMtx); + if (!g_state.srError.empty()) g_state.srError.clear(); + } + + int curW = width, curH = height; + std::vector srOut; + bool srApplied = false; + if (srWant && superRes.IsReady()) { + srApplied = superRes.ProcessFrame(scratch.data(), width, height, srOut, curW, curH); + std::lock_guard e(g_state.srErrMtx); + if (!srApplied) { + const std::string& ne = superRes.LastError(); + if (g_state.srError != ne) g_state.srError = ne; + } else if (!g_state.srError.empty()) { g_state.srError.clear(); } + } + g_state.superResActive.store(srApplied, std::memory_order_release); + + // The "current" frame is the SR output if it ran, else the raw capture. + // Green screen composites onto it and writes the final premultiplied alpha. + std::vector& cur = srApplied ? srOut : scratch; + // Lazily start/stop AIGS to match the enabled flag. const bool want = g_state.greenScreenEnabled.load(std::memory_order_acquire); if (want && !aigs.IsReady()) { @@ -388,7 +439,7 @@ void Capture::WorkerLoop() { if (want && aigs.IsReady()) { const double expand = g_state.matteExpand.load(std::memory_order_acquire); const double feather = g_state.matteFeather.load(std::memory_order_acquire); - applied = aigs.ProcessFrame(scratch.data(), width, height, expand, feather); + applied = aigs.ProcessFrame(cur.data(), curW, curH, expand, feather); if (!applied) { std::lock_guard e(g_state.gsErrMtx); const std::string& newErr = aigs.LastError(); @@ -401,12 +452,14 @@ void Capture::WorkerLoop() { } g_state.greenScreenActive.store(applied, std::memory_order_release); - std::lock_guard lock(g_state.mtx); - g_state.frame.swap(scratch); - g_state.width = width; - g_state.height = height; - g_state.hasNewFrame = true; - g_state.framesProduced.fetch_add(1, std::memory_order_release); + { + std::lock_guard lock(g_state.mtx); + g_state.frame.swap(cur); + g_state.width = curW; + g_state.height = curH; + g_state.hasNewFrame = true; + g_state.framesProduced.fetch_add(1, std::memory_order_release); + } } SafeRelease(sample); } else { @@ -416,7 +469,8 @@ void Capture::WorkerLoop() { } } - // Tear down Eye Contact and AIGS before releasing the reader (thread-affinity requirement). + // Tear down all effects before releasing the reader (thread-affinity requirement). + superRes.Stop(); eyeContact.Stop(); aigs.Stop(); SafeRelease(reader); @@ -463,6 +517,8 @@ void Capture::Stop() { std::lock_guard e(g_state.ecErrMtx); g_state.ecError.clear(); } + g_state.superResActive.store(false, std::memory_order_release); + { std::lock_guard e(g_state.srErrMtx); g_state.srError.clear(); } } bool Capture::LatestFrame(std::vector& out, int& w, int& h) { @@ -506,6 +562,18 @@ std::string Capture::EyeContactError() const { return g_state.ecError; } +void Capture::SetSuperRes(bool enabled, int qualityLevel, int scaleX10) { + g_state.superResScale.store(scaleX10 == 15 ? 15 : 20, std::memory_order_release); + g_state.superResQuality.store(qualityLevel, std::memory_order_release); + g_state.superResEnabled.store(enabled, std::memory_order_release); +} +bool Capture::SuperResActive() const { + return g_state.superResActive.load(std::memory_order_acquire); +} +std::string Capture::SuperResError() const { + std::lock_guard e(g_state.srErrMtx); return g_state.srError; +} + Capture::~Capture() { Stop(); } diff --git a/native/shim/capture.h b/native/shim/capture.h index 7d34de0..f0c8613 100644 --- a/native/shim/capture.h +++ b/native/shim/capture.h @@ -42,6 +42,12 @@ class Capture { bool EyeContactActive() const; // true only while gaze redirection is transforming frames std::string EyeContactError() const; // empty when none + // Toggles Super Resolution + VSR QualityLevel (mode) + scale (15=1.5x, 20=2x; upscale modes + // only). Thread-safe; worker owns the object. + void SetSuperRes(bool enabled, int qualityLevel, int scaleX10); + bool SuperResActive() const; + std::string SuperResError() const; + // Measured frames-per-second over a rolling window (status polling). Thread-safe to // call from the single status-poll thread; 0 until the first interval elapses. double MeasuredFps() const; diff --git a/native/shim/shim.cpp b/native/shim/shim.cpp index a2459c2..104130e 100644 --- a/native/shim/shim.cpp +++ b/native/shim/shim.cpp @@ -3,6 +3,7 @@ #include "capture.h" #include "aigs.h" #include "eyecontact.h" +#include "superres.h" #include #include @@ -64,6 +65,7 @@ COS_API void cos_set_params(const CosParams* p) { g_capture.SetGreenScreen(p->green_screen_enabled != 0); g_capture.SetMatteParams(p->green_screen_expand, p->green_screen_feather); g_capture.SetEyeContact(p->eye_contact_enabled != 0); + g_capture.SetSuperRes(p->super_res_enabled != 0, p->super_res_quality_level, p->super_res_scale); } COS_API void cos_start(void) { g_capture.Start(g_cameraId); g_running = true; } @@ -78,6 +80,7 @@ COS_API void cos_get_status(CosStatus* out) { out->eye_contact_active = g_capture.EyeContactActive() ? 1 : 0; std::string err = g_capture.GreenScreenError(); if (err.empty()) err = g_capture.EyeContactError(); + if (err.empty()) err = g_capture.SuperResError(); if (!err.empty()) { size_t n = err.size() < 255 ? err.size() : 255; std::memcpy(out->error, err.data(), n); @@ -115,8 +118,11 @@ COS_API int cos_query_capabilities(CosCaps* out) { std::memcpy(out->ec_detail, ecDetail.data(), en); out->ec_detail[en] = '\0'; - // Return 1 if either effect is available (the managed side reads the per-gate ints). - return (gsOk || ecOk) ? 1 : 0; + std::string srDetail; + out->super_res_available = SuperRes::Probe(srDetail) ? 1 : 0; + + // Return 1 if any effect is available (the managed side reads the per-gate ints). + return (gsOk || ecOk || out->super_res_available) ? 1 : 0; } COS_API void cos_shutdown(void) { diff --git a/native/shim/shim.h b/native/shim/shim.h index 2fcd942..83cad75 100644 --- a/native/shim/shim.h +++ b/native/shim/shim.h @@ -23,6 +23,9 @@ typedef struct { int eye_contact_enabled; double eye_contact_sensitivity; double eye_contact_look_away_range; + int super_res_enabled; + int super_res_scale; // 0=off, 15=1.5x, 20=2x (upscale modes only) + int super_res_quality_level; // VSR QualityLevel: 1-4 upscale, 8-11 denoise, 12-15 deblur } CosParams; typedef struct { @@ -30,6 +33,7 @@ typedef struct { char detail[256]; // green-screen status/error (UTF-8, NUL-terminated) int eye_contact_available; // 1 if Maxine GazeRedirection can run, else 0 char ec_detail[256]; // eye-contact status/error (UTF-8, NUL-terminated) + int super_res_available; // 1 if Maxine SuperRes can run } CosCaps; COS_API int cos_init(void* d3d11_device); diff --git a/native/shim/shim.vcxproj b/native/shim/shim.vcxproj index 43f9ded..abe4136 100644 --- a/native/shim/shim.vcxproj +++ b/native/shim/shim.vcxproj @@ -76,7 +76,7 @@ COS_HAS_MAXINE;%(PreprocessorDefinitions) - $(CosVfxSdkDir)\nvvfx\include;$(CosVfxSdkDir)\features\nvvfxgreenscreen\include;%(AdditionalIncludeDirectories) + $(CosVfxSdkDir)\nvvfx\include;$(CosVfxSdkDir)\features\nvvfxgreenscreen\include;$(CosVfxSdkDir)\features\nvvfxvideosuperres\include;%(AdditionalIncludeDirectories) @@ -92,7 +92,9 @@ + + @@ -116,7 +118,9 @@ + + diff --git a/native/shim/smoke/build_trace_closure.bat b/native/shim/smoke/build_trace_closure.bat index 6174f5f..370b31c 100644 --- a/native/shim/smoke/build_trace_closure.bat +++ b/native/shim/smoke/build_trace_closure.bat @@ -3,5 +3,5 @@ call "C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Auxiliar if not defined VFX set VFX=C:\dev\VideoFX if not defined AR set AR=C:\dev\Maxine-AR-SDK set OUT=%~1 -cl /nologo /EHsc /std:c++17 /DCOS_HAS_MAXINE /DCOS_HAS_MAXINE_AR /I "%VFX%\nvvfx\include" /I "%VFX%\features\nvvfxgreenscreen\include" /I "%AR%\nvar\include" native\shim\smoke\trace_closure.cpp native\shim\aigs.cpp native\shim\eyecontact.cpp native\shim\paths.cpp "%VFX%\nvvfx\src\nvVideoEffectsProxy.cpp" "%VFX%\nvvfx\src\nvCVImageProxy.cpp" "%AR%\nvar\src\nvARProxy.cpp" Psapi.lib /Fo"native\shim\smoke\\" /Fe"%OUT%" +cl /nologo /EHsc /std:c++17 /DCOS_HAS_MAXINE /DCOS_HAS_MAXINE_AR /I "%VFX%\nvvfx\include" /I "%VFX%\features\nvvfxgreenscreen\include" /I "%VFX%\features\nvvfxvideosuperres\include" /I "%AR%\nvar\include" native\shim\smoke\trace_closure.cpp native\shim\aigs.cpp native\shim\eyecontact.cpp native\shim\superres.cpp native\shim\paths.cpp native\shim\vfx_paths.cpp "%VFX%\nvvfx\src\nvVideoEffectsProxy.cpp" "%VFX%\nvvfx\src\nvCVImageProxy.cpp" "%AR%\nvar\src\nvARProxy.cpp" Psapi.lib /Fo"native\shim\smoke\\" /Fe"%OUT%" exit /b %ERRORLEVEL% diff --git a/native/shim/smoke/effects_smoke.cpp b/native/shim/smoke/effects_smoke.cpp new file mode 100644 index 0000000..6b6c748 --- /dev/null +++ b/native/shim/smoke/effects_smoke.cpp @@ -0,0 +1,37 @@ +// Dev-box smoke (needs VFX SDK + RTX GPU). Build with the shim's include/lib setup. +// Verifies SuperRes can start and process synthetic frames. +// +// Build (Developer PowerShell for VS 2022, from repo root): +// $env:COS_VFX_SDK_DIR = "C:\path\to\VideoFX" +// cl /EHsc /std:c++17 ^ +// /I "%COS_VFX_SDK_DIR%\nvvfx\include" ^ +// /DCOS_HAS_MAXINE ^ +// native\shim\smoke\effects_smoke.cpp ^ +// native\shim\superres.cpp ^ +// native\shim\vfx_paths.cpp native\shim\paths.cpp ^ +// "%COS_VFX_SDK_DIR%\nvvfx\src\nvVideoEffectsProxy.cpp" ^ +// "%COS_VFX_SDK_DIR%\nvvfx\src\nvCVImageProxy.cpp" +// .\effects_smoke.exe +// +// Expected output (RTX GPU with models installed): +// SR ProcessFrame: ok -> 1280x960 (requires nvvfxvideosuperres NGC feature + models) +// Without GPU/models, Probe prints the reason and exits 0 (deferred to human gate). +#include +#include +#include +#include "../superres.h" + +int main() { + { + std::string d; + if (SuperRes::Probe(d)) { + SuperRes sr; assert(sr.Start(1, 20)); // quality 1 = VSR_Low upscale, scale 2x + std::vector in(640 * 480 * 4, 64), out; int ow = 0, oh = 0; + bool ok = sr.ProcessFrame(in.data(), 640, 480, out, ow, oh); + std::printf("SR ProcessFrame: %s -> %dx%d\n", ok ? "ok" : "fail", ow, oh); + assert(ok && ow == 1280 && oh == 960); + sr.Stop(); + } else { std::printf("SR unavailable: %s\n", d.c_str()); } + } + return 0; +} diff --git a/native/shim/smoke/trace_closure.cpp b/native/shim/smoke/trace_closure.cpp index 55bfaaf..1f4dd74 100644 --- a/native/shim/smoke/trace_closure.cpp +++ b/native/shim/smoke/trace_closure.cpp @@ -20,6 +20,7 @@ #include #include "../aigs.h" #include "../eyecontact.h" +#include "../superres.h" #include "../paths.h" static std::string ToLower(std::string s) { @@ -31,15 +32,19 @@ int main() { // 1. Load + run both effects (same thread = CUDA affinity satisfied). Aigs gs; EyeContact ec; + SuperRes sr; const bool gsOk = gs.Start(); const bool ecOk = ec.Start(); + const bool srOk = sr.Start(1, 20); // quality 1 = VSR_Low upscale (pulls nvngx_vsr.dll closure) const int W = 1280, H = 720; std::vector frame((size_t)W * H * 4, (uint8_t)128); if (gsOk) gs.ProcessFrame(frame.data(), W, H, 0.0, 0.0); if (ecOk) ec.ProcessFrame(frame.data(), W, H); - std::printf("# green-screen Start=%d eye-contact Start=%d\n", gsOk ? 1 : 0, ecOk ? 1 : 0); + if (srOk) { std::vector out; int ow = 0, oh = 0; sr.ProcessFrame(frame.data(), W, H, out, ow, oh); } + std::printf("# green-screen Start=%d eye-contact Start=%d super-res Start=%d\n", gsOk ? 1 : 0, ecOk ? 1 : 0, srOk ? 1 : 0); if (!gsOk) std::printf("# GS error: %s\n", gs.LastError().c_str()); if (!ecOk) std::printf("# EC error: %s\n", ec.LastError().c_str()); + if (!srOk) std::printf("# SR error: %s\n", sr.LastError().c_str()); // 2. Enumerate modules loaded from \maxine\ . const std::string root = ToLower(ShimModuleDir()) + "\\maxine\\"; @@ -67,5 +72,6 @@ int main() { // 3. Clean up so worker objects release the CUDA stream before exit. gs.Stop(); ec.Stop(); - return (gsOk && ecOk) ? 0 : 1; + sr.Stop(); + return (gsOk && ecOk && srOk) ? 0 : 1; } diff --git a/native/shim/superres.cpp b/native/shim/superres.cpp new file mode 100644 index 0000000..c84be59 --- /dev/null +++ b/native/shim/superres.cpp @@ -0,0 +1,163 @@ +#include "superres.h" + +#ifdef COS_HAS_MAXINE +#define NOMINMAX +#include +#include +#include +#include +#include "nvCVStatus.h" +#include "nvCVImage.h" +#include "nvVideoEffects.h" +#include "nvVFXVideoSuperRes.h" // real header: NVVFX_FX_VIDEO_SUPER_RES + NVVFX_QUALITY_LEVEL +#include "vfx_paths.h" + +struct SrImpl { + NvVFX_Handle effect = nullptr; + CUstream stream = nullptr; + std::string modelDir; + NvCVImage srcGpu{}; // BGRA u8 chunky GPU (input, w x h) + NvCVImage dstGpu{}; // BGRA u8 chunky GPU (output, ow x oh) + NvCVImage dstCpu{}; // BGRA u8 chunky CPU (downloaded) + NvCVImage stage{}; // BGRA u8 chunky GPU (upload staging, w x h) + int w = 0, h = 0, ow = 0, oh = 0; + bool loaded = false; +}; + +// QualityLevel 1-4 are the upscale modes; 8-15 (denoise/deblur) keep input size. +static bool IsUpscale(int q) { return q >= 1 && q <= 4; } + +static void OutDims(int w, int h, int q, int scaleX10, int& ow, int& oh) { + if (IsUpscale(q)) { ow = (w * scaleX10) / 10; oh = (h * scaleX10) / 10; } + else { ow = w; oh = h; } // denoise/deblur do not upscale (out == in) +} + +SuperRes::SuperRes() = default; +SuperRes::~SuperRes() { Stop(); } + +// Probe mirrors the verified Start sequence (create + quality + images + load) so a load-only +// capability check matches the runtime path. False here greys the effect off for all users. +bool SuperRes::Probe(std::string& detail) { + std::string binDir, modelDir, err; + if (!vfx::ResolveSdkPaths(binDir, modelDir, err)) { detail = err; return false; } + vfx::PointProxiesAt(binDir); + NvVFX_Handle eff = nullptr; + if (NvVFX_CreateEffect(NVVFX_FX_VIDEO_SUPER_RES, &eff) != NVCV_SUCCESS || !eff) { + detail = "NvVFX_CreateEffect(VideoSuperRes) failed (DLL/SDK load?)"; return false; + } + CUstream stream = nullptr; + NvVFX_CudaStreamCreate(&stream); + NvVFX_SetString(eff, NVVFX_MODEL_DIRECTORY, modelDir.c_str()); // harmless for NGX + NvVFX_SetCudaStream(eff, NVVFX_CUDA_STREAM, stream); + NvVFX_SetU32(eff, NVVFX_QUALITY_LEVEL, 1u); // VSR_Low (upscale); same family Start uses + NvCVImage src{}, dst{}; + NvCV_Status load = NVCV_ERR_GENERAL; + if (NvCVImage_Alloc(&src, 640, 360, NVCV_BGRA, NVCV_U8, NVCV_CHUNKY, NVCV_GPU, 1) == NVCV_SUCCESS && + NvCVImage_Alloc(&dst, 1280, 720, NVCV_BGRA, NVCV_U8, NVCV_CHUNKY, NVCV_GPU, 1) == NVCV_SUCCESS) { + NvVFX_SetImage(eff, NVVFX_INPUT_IMAGE, &src); + NvVFX_SetImage(eff, NVVFX_OUTPUT_IMAGE, &dst); + load = NvVFX_Load(eff); + } + NvCVImage_Dealloc(&src); NvCVImage_Dealloc(&dst); + NvVFX_DestroyEffect(eff); + if (stream) NvVFX_CudaStreamDestroy(stream); + if (load != NVCV_SUCCESS) { detail = "NvVFX_Load(VideoSuperRes) failed (models missing or GPU incompatible?)"; return false; } + detail = "VideoSuperRes available"; + return true; +} + +bool SuperRes::Start(int qualityLevel, int scaleX10) { + Stop(); + qualityLevel_ = qualityLevel; + scaleX10_ = (scaleX10 == 15) ? 15 : 20; // only 1.5x / 2x supported (upscale modes) + SrImpl* impl = new (std::nothrow) SrImpl(); + if (!impl) { lastError_ = "out of memory"; return false; } + std::string binDir, err; + if (!vfx::ResolveSdkPaths(binDir, impl->modelDir, err)) { lastError_ = err; delete impl; return false; } + vfx::PointProxiesAt(binDir); + if (NvVFX_CudaStreamCreate(&impl->stream) != NVCV_SUCCESS) { lastError_ = "CudaStreamCreate failed"; delete impl; return false; } + if (NvVFX_CreateEffect(NVVFX_FX_VIDEO_SUPER_RES, &impl->effect) != NVCV_SUCCESS || !impl->effect) { + lastError_ = "CreateEffect failed"; NvVFX_CudaStreamDestroy(impl->stream); delete impl; return false; + } + NvVFX_SetString(impl->effect, NVVFX_MODEL_DIRECTORY, impl->modelDir.c_str()); + NvVFX_SetCudaStream(impl->effect, NVVFX_CUDA_STREAM, impl->stream); + NvVFX_SetU32(impl->effect, NVVFX_QUALITY_LEVEL, static_cast(qualityLevel_)); + // Scale is inferred by the effect from the output/input dimension ratio; no scale param. + impl_ = impl; ready_ = true; lastError_.clear(); + return true; +} + +void SuperRes::Stop() { + auto* impl = static_cast(impl_); + if (!impl) { ready_ = false; return; } + NvCVImage_Dealloc(&impl->srcGpu); + NvCVImage_Dealloc(&impl->dstGpu); + NvCVImage_Dealloc(&impl->dstCpu); + NvCVImage_Dealloc(&impl->stage); + if (impl->effect) NvVFX_DestroyEffect(impl->effect); + if (impl->stream) NvVFX_CudaStreamDestroy(impl->stream); + delete impl; impl_ = nullptr; ready_ = false; +} + +namespace { +// (Re)allocates GPU/CPU images for input w*h and output ow*oh, then loads the model. Scale is +// inferred by the effect from dstGpu.width/srcGpu.width. Called on first frame and on size change. +NvCV_Status EnsureImages(SrImpl* impl, int w, int h, int ow, int oh) { + if (impl->loaded && impl->w == w && impl->h == h && impl->ow == ow && impl->oh == oh) return NVCV_SUCCESS; + NvCVImage_Dealloc(&impl->srcGpu); + NvCVImage_Dealloc(&impl->dstGpu); + NvCVImage_Dealloc(&impl->dstCpu); + NvCVImage_Dealloc(&impl->stage); + NvCV_Status s; + s = NvCVImage_Alloc(&impl->srcGpu, w, h, NVCV_BGRA, NVCV_U8, NVCV_CHUNKY, NVCV_GPU, 1); if (s != NVCV_SUCCESS) return s; + s = NvCVImage_Alloc(&impl->dstGpu, ow, oh, NVCV_BGRA, NVCV_U8, NVCV_CHUNKY, NVCV_GPU, 1); if (s != NVCV_SUCCESS) return s; + s = NvCVImage_Alloc(&impl->dstCpu, ow, oh, NVCV_BGRA, NVCV_U8, NVCV_CHUNKY, NVCV_CPU, 1); if (s != NVCV_SUCCESS) return s; + s = NvCVImage_Alloc(&impl->stage, w, h, NVCV_BGRA, NVCV_U8, NVCV_CHUNKY, NVCV_GPU, 1); if (s != NVCV_SUCCESS) return s; + s = NvVFX_SetImage(impl->effect, NVVFX_INPUT_IMAGE, &impl->srcGpu); if (s != NVCV_SUCCESS) return s; + s = NvVFX_SetImage(impl->effect, NVVFX_OUTPUT_IMAGE, &impl->dstGpu); if (s != NVCV_SUCCESS) return s; + s = NvVFX_Load(impl->effect); if (s != NVCV_SUCCESS) return s; + impl->w = w; impl->h = h; impl->ow = ow; impl->oh = oh; impl->loaded = true; + return NVCV_SUCCESS; +} +} // namespace + +bool SuperRes::ProcessFrame(const uint8_t* bgra, int w, int h, std::vector& out, int& outW, int& outH) { + auto* impl = static_cast(impl_); + if (!impl || !impl->effect || !bgra || w <= 0 || h <= 0) return false; + int ow, oh; OutDims(w, h, qualityLevel_, scaleX10_, ow, oh); + if (NvCV_Status es = EnsureImages(impl, w, h, ow, oh); es != NVCV_SUCCESS) { + lastError_ = std::string("EnsureImages/Load failed: ") + NvCV_GetErrorStringFromCode(es); ready_ = false; return false; + } + // Upload: CPU BGRA -> GPU BGRA via staging buffer (same format, no conversion). + NvCVImage src{}; + NvCVImage_Init(&src, w, h, w * 4, const_cast(bgra), NVCV_BGRA, NVCV_U8, NVCV_CHUNKY, NVCV_CPU); + if (NvCVImage_Transfer(&src, &impl->srcGpu, 1.0f, impl->stream, &impl->stage) != NVCV_SUCCESS) { lastError_ = "upload failed"; return false; } + if (NvVFX_Run(impl->effect, 0) != NVCV_SUCCESS) { lastError_ = "NvVFX_Run failed"; return false; } + // Download: GPU BGRA (ow x oh) -> CPU BGRA. + if (NvCVImage_Transfer(&impl->dstGpu, &impl->dstCpu, 1.0f, impl->stream, nullptr) != NVCV_SUCCESS) { lastError_ = "download failed"; return false; } + if (NvVFX_CudaStreamSynchronize(impl->stream) != NVCV_SUCCESS) { lastError_ = "stream sync failed"; return false; } + // Write-back: BGRA CPU -> packed BGRA out, honoring source pitch. Force alpha opaque: VSR + // does not preserve the alpha channel, and SR now runs on a pre-matte frame (green screen + // authors the real matte AFTER SR). Opaque matches CopyFrame's passthrough invariant. + out.resize(static_cast(ow) * oh * 4); + const uint8_t* d = static_cast(impl->dstCpu.pixels); + const int dpitch = impl->dstCpu.pitch; + const int rowBytes = ow * 4; + for (int y = 0; y < oh; ++y) { + uint8_t* orow = out.data() + static_cast(rowBytes) * y; + std::memcpy(orow, d + static_cast(dpitch) * y, static_cast(rowBytes)); + for (int x = 3; x < rowBytes; x += 4) orow[x] = 0xFF; + } + outW = ow; outH = oh; + return true; +} + +#else +// ---- Passthrough stub: built when no SDK is configured. ---- +SuperRes::SuperRes() = default; +SuperRes::~SuperRes() = default; +bool SuperRes::Probe(std::string& detail) { detail = "Maxine SDK not built in"; return false; } +bool SuperRes::Start(int, int) { lastError_ = "Maxine SDK not built in"; ready_ = false; return false; } +void SuperRes::Stop() { ready_ = false; } +bool SuperRes::ProcessFrame(const uint8_t*, int, int, std::vector&, int&, int&) { return false; } +#endif diff --git a/native/shim/superres.h b/native/shim/superres.h new file mode 100644 index 0000000..583175f --- /dev/null +++ b/native/shim/superres.h @@ -0,0 +1,30 @@ +#pragma once +#include +#include +#include + +// Wraps the Maxine VFX Video Super Resolution effect (NGX, selector "VideoSuperRes"). +// QualityLevel doubles as the mode selector: +// 1-4 = upscale (VSR Low/Med/High/Ultra) -> output = input * scale (scaleX10: 15=1.5x, 20=2x) +// 8-11 = denoise, 12-15 = deblur -> output = input size (no upscale) +// In and out are BGRA u8 on the GPU (matches the capture pipeline; no BGR conversion). Alpha +// flows through VSR untouched. Without COS_HAS_MAXINE this is a never-ready passthrough stub. +class SuperRes { +public: + SuperRes(); + ~SuperRes(); + static bool Probe(std::string& detail); + bool Start(int qualityLevel, int scaleX10); + void Stop(); + // Reads w*h BGRA from 'bgra', writes outW*outH BGRA into 'out'. Returns false on failure + // (out untouched). For upscale modes outW/outH are w*scale,h*scale; otherwise w,h. + bool ProcessFrame(const uint8_t* bgra, int w, int h, std::vector& out, int& outW, int& outH); + bool IsReady() const { return ready_; } + const std::string& LastError() const { return lastError_; } +private: + bool ready_ = false; + int qualityLevel_ = 1; + int scaleX10_ = 20; + std::string lastError_; + void* impl_ = nullptr; +}; diff --git a/native/shim/vfx_paths.cpp b/native/shim/vfx_paths.cpp new file mode 100644 index 0000000..1bcab1e --- /dev/null +++ b/native/shim/vfx_paths.cpp @@ -0,0 +1,45 @@ +#include "vfx_paths.h" +#ifdef COS_HAS_MAXINE +#include +#include "paths.h" + +// The proxy stub (nvVideoEffectsProxy.cpp) declares this extern; we own the single +// definition here (moved out of aigs.cpp). Non-null => dir holding NVVideoEffects.dll, +// which the proxy passes to SetDllDirectory before LoadLibrary. +char* g_nvVFXSDKPath = nullptr; + +namespace vfx { +bool ResolveSdkPaths(std::string& binDir, std::string& modelDir, std::string& err) { + char buf[1024] = {0}; + DWORD n = GetEnvironmentVariableA("COS_VFX_RUNTIME_DIR", buf, sizeof(buf)); + if (n > 0 && n < sizeof(buf)) { + std::string root(buf, n); + if (!root.empty() && (root.back() == '\\' || root.back() == '/')) root.pop_back(); + binDir = root; modelDir = root + "\\models"; return true; + } + n = GetEnvironmentVariableA("COS_VFX_SDK_DIR", buf, sizeof(buf)); + if (n > 0 && n < sizeof(buf)) { + std::string root(buf, n); + if (!root.empty() && (root.back() == '\\' || root.back() == '/')) root.pop_back(); + binDir = root + "\\bin"; modelDir = root + "\\bin\\models"; return true; + } + std::string appDir = ShimModuleDir(); + if (!appDir.empty()) { + std::string maxine = appDir + "\\maxine"; + if (DirExists(maxine)) { binDir = maxine; modelDir = maxine + "\\models\\vfx"; return true; } + } + err = "VFX runtime not found: set COS_VFX_RUNTIME_DIR or bundle maxine\\ beside the app"; + return false; +} +void PointProxiesAt(const std::string& binDir) { + static std::string s_bin; + s_bin = binDir; + g_nvVFXSDKPath = const_cast(s_bin.c_str()); +} +} +#else +namespace vfx { +bool ResolveSdkPaths(std::string&, std::string&, std::string& err) { err = "Maxine SDK not built in"; return false; } +void PointProxiesAt(const std::string&) {} +} +#endif diff --git a/native/shim/vfx_paths.h b/native/shim/vfx_paths.h new file mode 100644 index 0000000..2f556aa --- /dev/null +++ b/native/shim/vfx_paths.h @@ -0,0 +1,13 @@ +#pragma once +#include + +// Shared VFX-SDK runtime discovery for ALL VFX effects (Green Screen, +// Super Resolution). Single source of truth — aigs.cpp uses these too. +// Model subdir is models\vfx in the bundled layout. +namespace vfx { +// Fills binDir (holds NVVideoEffects.dll) and modelDir. Order: COS_VFX_RUNTIME_DIR +// (flat) -> COS_VFX_SDK_DIR (legacy \bin) -> \maxine (bundled). false if none. +bool ResolveSdkPaths(std::string& binDir, std::string& modelDir, std::string& err); +// Points the VFX proxy global (g_nvVFXSDKPath, defined in vfx_paths.cpp) at binDir. +void PointProxiesAt(const std::string& binDir); +} diff --git a/scripts/assemble-maxine-stage.ps1 b/scripts/assemble-maxine-stage.ps1 index 85cef04..f6b7132 100644 --- a/scripts/assemble-maxine-stage.ps1 +++ b/scripts/assemble-maxine-stage.ps1 @@ -58,6 +58,7 @@ if ($DryRun) { Write-Host "DRY RUN — assemble stage at $OutStage" Write-Host " VFX bin DLLs (shared + NVVideoEffects dispatcher) <- $VfxSdk\bin" Write-Host " nvVFXGreenScreen.dll <- $VfxSdk\features\nvvfxgreenscreen\bin" + Write-Host " nvVFXVideoSuperRes.dll + nvngx_vsr.dll <- $VfxSdk\features\nvvfxvideosuperres\bin" Write-Host " nvARPose.dll <- $ArSdk\bin" foreach ($f in $arFeatures) { Write-Host " $($f.dll) <- $(Feature-Dir $f.name)\bin" } foreach ($l in $licenses) { Write-Host " license: $($l.dst) <- $($l.src)" } @@ -78,6 +79,10 @@ function Copy-One([string]$src, [string]$dst) { Get-ChildItem (Join-Path $VfxSdk 'bin') -Filter *.dll -File | ForEach-Object { Copy-Item -LiteralPath $_.FullName -Destination $OutStage -Force } # 2. VFX green-screen feature DLL. Copy-One (Join-Path $VfxSdk 'features\nvvfxgreenscreen\bin\nvVFXGreenScreen.dll') $OutStage +# 2b. VFX video-super-resolution feature DLL + its NGX runtime. VSR is NGX (model baked into +# nvngx_vsr.dll) -- no per-arch engine to fetch, runs on every RTX arch. +Copy-One (Join-Path $VfxSdk 'features\nvvfxvideosuperres\bin\nvVFXVideoSuperRes.dll') $OutStage +Copy-One (Join-Path $VfxSdk 'features\nvvfxvideosuperres\bin\nvngx_vsr.dll') $OutStage # 3. AR dispatcher + the three gaze feature DLLs (AR-unique only -- shared DLLs already from VFX). Copy-One (Join-Path $ArSdk 'bin\nvARPose.dll') $OutStage foreach ($f in $arFeatures) { Copy-One (Join-Path (Feature-Dir $f.name) (Join-Path 'bin' $f.dll)) $OutStage } diff --git a/src/CameraOnScreen.App/MainWindow.xaml b/src/CameraOnScreen.App/MainWindow.xaml index 73fcbaf..cdd8f27 100644 --- a/src/CameraOnScreen.App/MainWindow.xaml +++ b/src/CameraOnScreen.App/MainWindow.xaml @@ -1,58 +1,80 @@ - + xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml" + xmlns:local="using:CameraOnScreen.App"> + - - - - + - + + + + + + + + + + + + + + + + + + + + + + + + + - - - - - - - - - - - - + + + + + - -