feat(#15): AI sharpness via Maxine VSR (Denoise/Deblur)#17
Merged
Conversation
Artifact Reduction + Super Resolution (Maxine VFX, no new co-version) + mipmapped downscale + real fps counter. Independent toggles, no auto GPU management. Phase 2 (fps interpolation via Optical Flow FRUC) tracked as #13. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Found inconsistency: the mip-downscale "swap CopyResource for a quad" cannot fix minification while the back buffer stays pinned to frame res — DComp does the final bilinear minification and ignores the mips. The real fix needs a window-sized back buffer + shader scale (mirror/zoom/MPO invariants), a present-path redesign. Descoped to deferred §11; Phase 1 = Artifact Reduction + Super Res + real fps counter. Tighten CosCaps size 520->528. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
9 tasks: real fps counter, shared VFX path resolver, Artifact Reduction + Super Resolution effects, ABI growth (CosParams +3 / CosCaps 528B), worker chain wiring, Core+managed plumbing with xUnit coverage, App UI + 4K buffer, bundler. TDD steps with full code; minification fix deferred per spec §11. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Per execution decision: vfx_paths owns g_nvVFXSDKPath; aigs.cpp drops its private resolver copies and calls vfx::. Adds a green-screen re-verify gate. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…view) Sub-interval Sample() calls must NOT advance the baseline, else the ~30 Hz status poll never accumulates a >=0.5s window and fps stays 0.0 forever. Restore the brief's original semantics; the bug was the test's expected value (56.25 -> 60.0: the t=1.2 sub-interval call leaves the baseline at 1.0/30, so the 1.0->2.0 window is 60 frames / 1.0s). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…cts) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add ArtifactReduction class (artifactreduction.h/.cpp) mirroring the aigs.cpp pattern: BGRA CPU frame -> GPU BGR upload -> NvVFX_Run -> BGR download -> write-back (RGB replaced, alpha untouched). Stub when COS_HAS_MAXINE undefined. Add effects_smoke.cpp smoke. Wire into shim.vcxproj (ClCompile + ClInclude + AR feature include dir). Selector: NVVFX_FX_ARTIFACT_REDUCTION "ArtifactReduction" from nvVFXArtifactReduction.h (stub header in SDK features tree; real header ships with nvvfxartifactreduction feature downloaded from NGC via install_feature.ps1). Build: 0 warnings, 0 errors. 9 cos_* exports unchanged. GreenScreen+GazeRedirection+ArtifactReduction present in DLL; "not built in" absent. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… header The VFX 1.2.0.0 base SDK has no artifact-reduction selector macro (it ships only in the nvvfxartifactreduction per-feature header downloaded from NGC). Including that header made the build depend on an out-of-repo file. Instead define the literal in artifactreduction.cpp behind an #ifndef guard — mirrors eyecontact.cpp's gaze-selector guard. The official header's define wins if/when installed. Drop the now-unneeded nvvfxartifactreduction include dir from the vcxproj and the smoke build comment. Build: 0 warnings, 0 errors. 9 cos_* exports unchanged; ArtifactReduction present in DLL. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…tifactReduction The Start() CreateEffect-failure branch deleted the impl without destroying the already-created CUDA stream, and Stop() cannot reclaim it because impl_ was never assigned. Destroy the stream before delete in both effects. Also drop a redundant per-pixel alpha write in SuperRes::ProcessFrame (out.assign pre-fills 0xFF). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…528B) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…on the worker Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Removes the redundant "Off" combo item that, combined with the SuperRes toggle, let scale=0 coerce to 2x. Index now 0=1.5x,1=2x; tests updated. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ing rule No nvvfxartifactreduction in the catalog (dead effect); Super Resolution is NGX VSR (selector "VideoSuperRes", QualityLevel param, nvngx_vsr.dll, no TRT engine). Verify a feature exists + read its real per-feature header before coding an effect; get the NGC key up front when it gates verification. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… diag) CreateReader enumerated nothing — MF picked the camera's default native type, which can be a low-fps mode. Now enumerate native types, pick highest fps then highest res (<=1080p), and pin the RGB32 request to that size+rate so MF takes the MJPG-30 path; fall back to unconstrained if rejected. capture_probe.cpp is a standalone diagnostic (lists modes, measures RGB32 vs raw-MJPG delivery fps). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
# Conflicts: # native/shim/capture.cpp # native/shim/capture.h
Resume plan for issue #15 amending the parked Phase-1 plan: VSR-only + mode combo decision, grounded NGC/probe findings, task list (T1 probe done). Ignore .env (holds NGC_CLI_API_KEY). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SfcBU4WRAy7HMNgtm92ogq
Artifact Reduction does not exist in the VFX 1.2.0.0 SDK / NGC catalog (confirmed: absent on disk, NGC returns PAYMENT_REQUIRED, only a help-text typo in install_feature.ps1). Shipping VSR-only, whose own Denoise/Deblur modes cover the cleanup goal. Removes all AR code across the shim (effect, worker block, ABI field) and C# (ABI mirror, MVVM, Orchestrator, config, tests). Super Resolution / Green Screen / Eye Contact untouched; CosParams/CosCaps stay byte-parallel (one int dropped from both sides). Verified: shim stub 0 warnings, App 0 warnings, Core 57/57. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SfcBU4WRAy7HMNgtm92ogq
Rewrite native/shim/superres.{h,cpp} against the verified VFX 1.2.0.0
Video Super Resolution API (NGX), replacing the guessed "SuperRes" literal:
- selector NVVFX_FX_VIDEO_SUPER_RES ("VideoSuperRes") via the real per-feature
header nvVFXVideoSuperRes.h (added the feature include dir to shim.vcxproj).
- BGRA u8 GPU in+out end to end; deleted the BGR conversion and the per-pixel
alpha-repack loop in favor of a per-row memcpy honoring source pitch. Alpha
now flows through VSR (opaque in passthrough; the green-screen matte when
AIGS ran upstream) instead of being force-filled 0xFF.
- Start(int qualityLevel, int scaleX10) sets NVVFX_QUALITY_LEVEL (the mode
selector): 1-4 upscale -> out = in*scale; 8-15 denoise/deblur -> out = in.
- Probe mirrors the verified Start sequence (create + quality + images + load)
so the capability gate matches the runtime path.
Verified by building the shim SDK config against the runner SDK
(C:\actions-runner\_sdk\VideoFX, after installing the nvvfxvideosuperres
feature): 0 warnings, both effects + VideoSuperRes exported, "not built in"
absent. Compile-only; runtime Load/Run remains the T9 human gate on the 3090.
Call sites stubbed to keep the tree green until the ABI carries the mode:
capture.cpp passes quality 1 (ponytail comment; T5/T6 thread the real mode),
effects_smoke.cpp updated. Plan doc: T2 ticked + a "Compile-testing the real
SDK shim on this host" runbook section.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SfcBU4WRAy7HMNgtm92ogq
…I (T4-T7) Wire the VSR QualityLevel (mode selector) end to end on top of the rewritten SuperRes effect. T4 (C ABI): add super_res_quality_level to CosParams (C shim.h + the [StructLayout] mirror in PInvokeShim) and the ShimParams record. Exactly the 9 cos_* exports unchanged. T5 (capture worker): SetSuperRes(enabled, qualityLevel, scale); a superResQuality atomic; the worker restarts the effect when quality or scale changes live (both are baked at Start). Replaces the T2 hardcoded quality 1. T6 (managed + Core + tests): EffectSettings persists SuperResMode + SuperResQuality + SuperResScale (dropping the bool). MainViewModel exposes SuperResModeIndex (Off/Upscale/Denoise/Deblur) + SuperResQualityIndex (Low/Med/High/Ultra) + SuperResScaleIndex, mapping mode+quality to a single QualityLevel (Upscale 1-4, Denoise 8-11, Deblur 12-15) and forcing scale to 0 for the clean modes. Live push + persistence round-trip. 57 xUnit tests green. T7 (App UI): replace the SR toggle + scale combo with mode / quality / upscale factor combos (quality enabled when mode != Off, factor only for Upscale). Restructure MainWindow into a ScrollViewer (row 0) with the license-required Maxine attribution footer pinned outside it (row 1, Auto) so it stays visible once the stacked controls overflow the 400x720 window. Verified: shim SDK config builds clean against the runner SDK (export-verify GreenScreen + GazeRedirection + VideoSuperRes, no "not built in"); App builds 0 warnings; Core tests 57/57. Runtime VSR remains the T9 human gate (3090). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SfcBU4WRAy7HMNgtm92ogq
T8 (bundler): assemble-maxine-stage.ps1 stages the VSR feature DLLs (nvVFXVideoSuperRes.dll + nvngx_vsr.dll); the manifest Dlls allow-list gains those plus nvngxruntime.dll (the NGX loader, already staged from VFX\bin). The closure delta was grounded by dumpbin static imports and CONFIRMED by re-running trace_closure on the RTX 3090: all three effects Start=1, exit 0, 22 modules loaded from maxine\ (the prior 19 + the 3 VSR DLLs, no dynamic surprises). trace_closure.cpp now exercises SuperRes too, and build_trace_closure.bat was fixed (it was missing vfx_paths.cpp, so it could not link since the vfx_paths split) and given the VSR include + superres.cpp. VSR is NGX, so no per-arch engine and no nvinfer dependency. Its license set (Open Model + Software License + Product-Specific Terms) is already in the manifest — no new license file. Resolves both open risks from the plan. T0 (spec): added an AMENDED banner to the AI-sharpness design spec — Artifact Reduction dropped (does not exist in VFX 1.2.0.0), Super Resolution re-scoped to NGX VSR with the real API (VideoSuperRes / QualityLevel) and the QualityLevel mode selector. Generic plumbing retained; points to the resume plan as the authoritative current scope. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SfcBU4WRAy7HMNgtm92ogq
Two visual bugs and a scope trim for the VSR effect: - Green-screen-black: SR ran last, after green screen wrote the matte into alpha and premultiplied RGB. NGX VSR does not preserve the alpha channel, so it clobbered the matte opaque -> transparent background rendered black. Reorder the capture pipeline to eyeContact -> SR -> green screen so green screen is the last alpha author; correct regardless of VSR's alpha behavior. AIGS already re-allocs per frame size and ignores input alpha, so running it on the (possibly upscaled) SR output is safe. SR write-back now forces alpha opaque since it only ever sees pre-matte frames (matches CopyFrame's passthrough invariant). - Drop upscale + scale combo: upscaling is wasted on this overlay (the DComp visual downsamples the back buffer to the on-screen window, so the extra pixels are discarded unless the window exceeds camera-native res). Keep the two native-res modes that survive downscale: Denoise (QualityLevel 8-11) and Deblur (12-15). UI is now one "AI Sharpness" combo (Off/Denoise/Deblur) + Quality. Native shim is unchanged (already generic over QualityLevel). - Startup crash on stale config: a config.json saved by the 4-mode build could hold SuperResMode = 3, past the new 3-item combo -> ArgumentException when the binding sets SelectedIndex. Clamp mode/quality to range on load. Tests: 58/58 (added the clamp regression). Shim + App build 0-warn. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The self-hosted runner host has an x86 .NET runtime (no SDK) that can sort ahead of the x64 SDK on PATH, so `dotnet build` fails with "No .NET SDKs were found". Prepend the x64 install dir to GITHUB_PATH so the SDK wins for the App build + Core test steps. Build commands unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Same x86-dotnet-shadows-SDK guard as ci.yml, for the installer build (build-installer.ps1 calls dotnet internally). GITHUB_PATH applies to all later steps including scripts they invoke. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #15.
Adds an AI Sharpness effect backed by Maxine VFX Video Super Resolution (NGX VSR), exposed as Off / Denoise / Deblur + a Quality tier (Low/Med/High/Ultra). VSR is NGX (model baked into
nvngx_vsr.dll), so it runs on every RTX arch with no per-arch TRT engine — sidesteps the non-Ampere deserialize gate that greys out green screen / gaze.What changed
superres.{h,cpp}against the actual header: selectorNVVFX_FX_VIDEO_SUPER_RES "VideoSuperRes", paramNVVFX_QUALITY_LEVEL, BGRA u8 GPU in+out. (The parked code guessed"SuperRes"/NVVFX_MODE, which never existed in VFX 1.2.0.0.) Dropped the dead Artifact Reduction effect (not in the catalog).super_res_quality_leveladded to the C ABI (CosParams) + P/Invoke mirror;SetSuperRes(enabled, qualityLevel, scaleX10)with live restart-on-change; MVVM mode/quality → QualityLevel (Denoise 8-11 / Deblur 12-15); one combo in the UI; config persistence.nvVFXVideoSuperRes.dll,nvngx_vsr.dll) +nvngxruntime.dll;trace_closurere-run on 3090 confirms the load closure (22 modules).Bug fixes in the final commit
eyeContact → SR → green screen(green screen is now the last alpha author). SR forces output alpha opaque since it only sees pre-matte frames.ArgumentException. Clamp mode/quality on load.Verification
GreenScreen+GazeRedirection+VideoSuperRes, no stub.trace_closureon RTX 3090: all three effects Load+Run.Related: filed #16 (optional exposure/framerate lock — webcam auto-exposure halves FPS in low light; out of scope here).
🤖 Generated with Claude Code