Skip to content

feat(#13): AI fps interpolation 30→60 via NVIDIA Optical Flow FRUC#22

Merged
opariffazman merged 21 commits into
mainfrom
spike/13-fruc-coversion-smoke
Jun 26, 2026
Merged

feat(#13): AI fps interpolation 30→60 via NVIDIA Optical Flow FRUC#22
opariffazman merged 21 commits into
mainfrom
spike/13-fruc-coversion-smoke

Conversation

@opariffazman

@opariffazman opariffazman commented Jun 26, 2026

Copy link
Copy Markdown
Contributor

Closes #13.

What

Optional AI frame interpolation (camera 30 fps → 60 fps overlay) via the NVIDIA Optical Flow SDK FRUC library (NvOFFRUC.dll) — a separate product from Maxine VFX/AR. Toggled + capability-gated like the other effects.

The hard gate (the original question)

#13's blocker was co-version: can FRUC coexist in-process with the pinned Maxine VFX 1.2.0.0 + AR 1.1.1.0 (TRT 10.9 / CUDA 12)? Yes — proven by a standalone smoke (spike commit). FRUC uses no TensorRT and ships its own cudart64_110.dll (CUDA 11, distinct name from cudart64_12.dll), so the same-DLL-name first-load-wins hazard cannot trigger; only nvcuda.dll (the driver) is shared. Both runtimes verified co-resident on RTX 3090.

How

  • Fruc wrapper (native/shim/fruc.{h,cpp}): streaming feed-once Submit(cur)→mid — FRUC is a stateful pipeline on a monotonic timeline (a prime-every-call model triggers its frame-repetition fallback; the streaming model is correct and ~halves latency to ~8 ms/call). Own CUDA-11 context, CPU round-trip BGRA, ping-pong render buffers.
  • Runs last on the final composite in the capture worker; the worker double-publishes (midpoint then real frame); C# pump runs 16 ms while active else 33 ms.
  • New CosParams.frame_interp_enabled + CosCaps.frame_interp_available/fi_detail (ABI parity verified byte-for-byte), Core contracts, Orchestrator gate, MVVM toggle, WinUI "Smooth 60 fps (AI)" switch, config persistence.
  • Bundler: FRUC DLLs ship as an optional tier (OptionalDlls) — best-effort, tolerated-absent — so release builds succeed FRUC-less and no legally-uncleared binary is coerced into shipping.

Verification

  • GPU smoke: genuine temporal-midpoint interpolation confirmed (meanAbsDiff mid↔each = exactly ½ the full motion, symmetric), correct color order, 8.35 ms/call.
  • Human runtime gate PASSED: smooth ~60 fps, no objectionable jitter/artifacts.
  • 63 Core tests pass, 0 warnings. Native shim builds clean (stub + full SDK); export-verify intact.
  • Built via subagent-driven development: per-task reviews + a final whole-branch review, all findings fixed.

Open (non-code) items — flagged in-tree, NOT done here

  • ⚖️ Legal gate: OF SDK redistribution of NvOFFRUC.dll + cudart64_110.dll is NOT yet human-verified (see THIRD-PARTY-NOTICES.md). Must clear before shipping them in an installer.
  • 🏃 Release host: add COS_FRUC_SDK_DIR to the runner .env; run the full assemble + bundle + trace_closure/bundle_probe (deferred — needs the runner).
  • 🧪 CI export-verify could add a FRUC presence marker once the runner stages it.

Notes

  • native/shim/smoke/* are dev tools/tests, not shipped code.
  • Non-RTX / FRUC-unavailable machines run unchanged (toggle greys out, passthrough).

🤖 Generated with Claude Code

opariffazman and others added 21 commits June 26, 2026 01:23
@opariffazman @claude
spike(#13): co-version smoke proves FRUC coexists with VFX+AR
05c4f41 
HARD GATE cleared. NvOFFRUC.dll (CUDA 11.0 / NVOFA, no TensorRT) loads, creates
and runs in the same process as the pinned Maxine VFX 1.2.0.0 + AR 1.1.1.0 stack
(CUDA 12.x / TRT 10.9). cudart64_110.dll and cudart64_12.dll co-resident with no
cudaErrorNoKernelImageForDevice -- the documented same-DLL-name conflict can't
trigger because FRUC shares no runtime DLL name with VFX/AR (only nvcuda.dll, the
driver). Verified on RTX 3090 (sm86), exit 0.

Adds native/shim/smoke/of_fruc_smoke.cpp + build script and a findings doc.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@opariffazman @claude
docs(#13): FRUC fps-interpolation implementation plan
7f08093 
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@opariffazman @claude
feat(#13): Fruc wrapper interface + passthrough stub
bd320d8 
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@opariffazman @claude
feat(#13): Fruc GPU implementation (NvOFFRUC, CUDA-11 ctx)
8d61a4e 
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@opariffazman @claude
fix(#13/task-2): fix 7 review findings in Fruc GPU wrapper
7245f5f 
- DLL handle leak: FreeLibrary on every error path after LoadFruc succeeds
  (exports-missing, CUDA init, NvOFFRUCCreate, cuMemAlloc, RegisterResource)
- Partial cuMemAlloc leak: free buf[0..i-1] before pDestroy on loop failure
- CUDA ctx leak: track ctxRetained separately; release on cuCtxSetCurrent fail
- Move=delete: add Fruc(Fruc&&)=delete + operator=(Fruc&&)=delete to fruc.h
- Stop order: Unregister -> cuMemFree buffers -> pDestroy -> CtxRelease -> FreeLibrary
- Interpolate: download into local tmp; swap into outMid only on success
- Remove unused FrucImpl::toggle member (YAGNI)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@opariffazman @claude
test(#13): FRUC interpolation color+latency smoke (Task 3)
9108ae3 
Drives Fruc::Interpolate on red-gradient frames (prev=0px, cur=+40px shift),
dumps prev/mid/cur BMPs to scratchpad, and measures avg latency (13.95 ms <16 ms budget).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@opariffazman @claude
refactor(fruc): replace broken Interpolate with streaming Submit
f278d33 
NvOFFRUC is a stateful pipeline on a monotonic timeline — re-priming
every call caused frame-repetition (mid == prev byte-for-byte, proven
in Task 3 smoke). New Submit feeds one frame per call: ping-pongs
buf[1]/buf[2] so FRUC's previous input stays valid, advances lastTs
by kInterval=1.0 each call, returns hasMid=false on the first call
(no previous frame), and produces a real midpoint on every subsequent
call. Probe unchanged (Start+Stop only). Stub updated to match signature.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@opariffazman @claude
test(fruc): rewrite smoke to streaming Submit API over 6-frame sequence
e5abdde 
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@opariffazman @claude
test(fruc): smooth sinusoidal content for genuine optical-flow interp…
b530f11 
…olation

Replace hard-edge red bar (80 px/frame jump, triggered frame-repetition
fallback) with a smooth 2D sinusoidal grayscale texture translated 6 px/frame.
Increase to 8 frames; sample mid between frame6 and frame7 (warmed up).
Replace color/bar-range analysis with % identical + meanAbsDiff stats.

Result: meanAbsDiff mid-f6=2.995, mid-f7=2.995, f6-f7=5.980 (exact halving),
% identical 31.46%/31.47% -- confirms genuine temporal interpolation.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@opariffazman @claude
feat(#13): wire FRUC interpolation through capture worker + shim C-ABI
c89d4cf 
- CosParams.frame_interp_enabled, CosCaps.frame_interp_available/fi_detail
- Capture: SetFrameInterp/FrameInterpActive/FrameInterpError accessors
- WorkerLoop: Fruc runs last after green-screen; publish midpoint then real frame
- shim.cpp: SetParams/GetStatus/QueryCapabilities all wire to FRUC
- vcxproj: CosFrucSdkDir/CosCudaDir gates COS_HAS_FRUC + cuda.lib

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@opariffazman @claude
feat(#13): frame-interp through Core contracts + PInvoke mirror
8dfc820 
Add FrameInterpEnabled/FrameInterpAvailable through ShimParams,
ShimCapabilities, FakeShim, Orchestrator (with gate), and PInvokeShim
struct mirrors (CosParams.frame_interp_enabled + CosCaps fields).
TDD: BuildParams_carries_FrameInterp_flag RED→GREEN; full 60-test suite green, 0 warnings.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@opariffazman @claude
feat(#13): frame-interp VM toggle + capability gate
d3b33b7 
Add FrameInterpEnabled observable property with live-push handler,
FrameInterpAvailable capability gate (set from probe), and include
FrameInterpEnabled in BuildParams(). Two new VM tests (RED→GREEN).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@opariffazman @claude
feat(#13): frame-interp UI toggle + 60Hz pump
a43600d 
Add "Smooth 60 fps (AI)" ToggleSwitch in MainWindow.xaml bound to
Vm.FrameInterpEnabled (TwoWay) / Vm.FrameInterpAvailable (OneWay).
Bump the frame-pump interval to 16 ms while FRUC is active, drop back
to 33 ms otherwise (single existing timer, no second timer).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@opariffazman @claude
feat(#13): bundle NvOFFRUC + cudart64_110 into maxine runtime
fb56298 
Add NvOFFRUC.dll and cudart64_110.dll to the manifest allow-list and
the stage assembler so shipped apps resolve FRUC from <exe>\maxine\ with
no env vars. FRUC is optional in assemble (warn+skip if COS_FRUC_SDK_DIR
unset) but required by the manifest for bundling (release host must have
the Optical Flow SDK). cudart64_110 (CUDA 11) coexists with cudart64_12
(CUDA 12) -- distinct DLL names, no conflict proven.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@opariffazman @claude
docs(#13): document FRUC effect + Optical Flow SDK notice
1111444 
- CLAUDE.md: woven-in FRUC guidance (COS_FRUC_SDK_DIR/COS_FRUC_RUNTIME_DIR build/runtime
  env vars, CosCudaDir prop, CO-VERSION safety note, worker-chain double-publish + 16 ms pump,
  CI runner .env gap, Maxine SDKs FRUC bullet with pending legal gate callout)
- THIRD-PARTY-NOTICES.md: NVIDIA Optical Flow SDK entry flagged as redistribution NOT YET
  human-verified; references LicenseAgreement.pdf path under runner SDK tree
- maxine-manifest.psd1: update header comment 22→24 modules, three→four effects

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@opariffazman @claude
docs(#13): fix FRUC section heading + remove CI runner path from notices
ec35c0c 
Move FRUC bullet out of Maxine SDKs section (whose heading falsely implied
Maxine license governs FRUC) into its own subsection with heading that names
it a separate product with an open legal gate.

Replace machine-specific C:\actions-runner\ path in the shipped
THIRD-PARTY-NOTICES.md with the public NVIDIA developer download URL.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@opariffazman @claude
fix(bundle): FRUC DLLs optional tier + persist FrameInterpEnabled
3b4243e 
- maxine-manifest.psd1: move NvOFFRUC.dll + cudart64_110.dll from
  mandatory Dlls to new OptionalDlls array; fix module-count comment
  (24 -> 22 mandatory; 2 optional noted separately)
- bundle-maxine.ps1: add Copy-Optional loop over OptionalDlls (warn,
  not throw, when absent); null-guard for older manifests
- assemble-maxine-stage.ps1: update FRUC warning to reflect optional
  tier (bundle proceeds without them; FRUC unavailable in shipped app)
- Models.cs + MainViewModel.cs: persist FrameInterpEnabled to config
- Add FrameInterp_config_round_trips_through_config test (RED->GREEN)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@opariffazman @claude
fix(ci): optional-FRUC stub must not trip the required-effect stub guard
2b4aa8c 
The export-verify greps the deployed shim for 'not built in' to detect a
passthrough stub of the REQUIRED Maxine effects (green screen / gaze). The
FRUC stub reused that exact phrase, so a build without COS_FRUC_SDK_DIR (FRUC
is optional — legal gate open, optional bundler tier) false-failed CI even
with green screen + gaze built real.

- FRUC stub strings now say 'FRUC unavailable (COS_HAS_FRUC unset)' (no 'not
  built in' sentinel) so an optional-FRUC stub never trips the guard.
- Verify reports FRUC presence (NvOFFRUC string) as a non-failing warning so
  the runner build still surfaces whether FRUC compiled.

Verified: Release build with VFX+AR real + FRUC stubbed -> stub=False, PASS.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@opariffazman @claude
docs(#13): clear FRUC legal gate — bundled-with-app redistribution
6233d6f 
Reviewed the actual NVIDIA DesignWorks SDK License (LicenseAgreement.pdf,
v.2022-05-10) governing the Optical Flow SDK. Redistribution of NvOFFRUC.dll +
cudart64_110.dll is permitted as part of the application bundle (never
standalone), per the license's distributable-portions terms — the app has
material additional functionality and the SDK is accessed only by our app.

- THIRD-PARTY-NOTICES.md: replace the open legal-gate block with the
  permitted-as-bundled notice + conditions met + end-user driver req (>=528.24;
  CUDA 11 runtime bundled, no toolkit needed).
- CLAUDE.md: FRUC redistribution = bundled-with-app; runner .env now sets
  COS_FRUC_SDK_DIR (service restart to take effect) so releases bundle FRUC.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@opariffazman @claude
@
bb79f1e 
docs(#13): revise plan to streaming feed-once FRUC API

Task-3 smoke proved prime-every-call Interpolate(prev,cur) re-primes the
stateful NvOFFRUC timeline and repeats frames; switch to Submit(cur) feed-once.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@
@opariffazman
Merge remote-tracking branch 'origin/main' into spike/13-fruc-coversi…
2222447 
…on-smoke

# Conflicts:
#	src/CameraOnScreen.App/MainWindow.xaml
#	src/CameraOnScreen.App/MainWindow.xaml.cs
#	src/CameraOnScreen.Core/ViewModels/MainViewModel.cs
@opariffazman
opariffazman merged commit ddd2f42 into main Jun 26, 2026
1 check passed
@opariffazman
opariffazman deleted the spike/13-fruc-coversion-smoke branch June 26, 2026 01:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Phase 2: AI fps interpolation (30->60) via NVIDIA Optical Flow FRUC — gated on co-version smoke test

1 participant

@opariffazman