Skip to content

fix(render): quality + fluidity defects across the local and cloud paths - #2

Closed
lBroth wants to merge 1 commit into
local-models-wan22-mlxfrom
claude/local-mode-pr-status-6bvg3d
Closed

fix(render): quality + fluidity defects across the local and cloud paths#2
lBroth wants to merge 1 commit into
local-models-wan22-mlxfrom
claude/local-mode-pr-status-6bvg3d

Conversation

@lBroth

@lBroth lBroth commented Aug 3, 2026

Copy link
Copy Markdown
Owner

Stacked on #1 (base is local-models-wan22-mlx, not main), so it merges into the local-mode branch.

Eight fixes to the render chain, found while auditing it for maximum quality and maximum fluidity. Every one was silent in production — nothing crashed, the video just came out worse than the code was already capable of. Each new behavior is env-gated back to the old one, so any single piece rolls back with one variable.

The one that matters most

setEnv only ever merged, so a one-shot env override leaked into every later render in the same process.

render:requality injects VB_LOCAL_WAN_STEPS=20, and the resolver emits it for the 5B. Either one then poisoned the next 14B render:

  • 20 steps forced through a 4-step Lightning distillation → flat, slow-motion output (the code's own comment warns about exactly this)
  • worse: isHd was defined as quality === 'hd' && !env('VB_LOCAL_WAN_STEPS'), so the leftover key made HD read as false — the HD path silently dropped to the fast tiny-VAE and the short deadline while also skipping the LoRA

Net: selecting Quality/HD produced a few-step, no-LoRA, lossy-VAE render — strictly worse than both Fast and HD. setEnv is now a per-op replace, and isHd no longer lets a step override redefine what HD means.

All eight

# Fix Where
1 setEnv is a per-op replace, not a merge — the root cause engine/config.ts
2 isHd no longer depends on the absence of a step override engine/localVideo.ts
3 render:requality bumps steps for the 5B only (native steps → 20 is a real gain; the 14B's HD path already runs the full 40-step schedule) main/index.ts
4 RIFE targets an exact multiple of the timeline rate localVideo.ts, local/interp.py
5 A failed interpolation is logged, not swallowed engine/localVideo.ts
6 Cloud retime is motion-compensated when the deviation is visible engine/ffmpeg.ts
7 Clamped anti-drift on chained start frames ffmpeg.ts, backends/local/video.ts
8 One x264 generation removed; the grade stops fighting the render ffmpeg.ts, pipeline.ts

Notes on the fluidity ones

#4 — a fixed 2x took the 14B's native 16fps to 32, and the 24fps conform then dropped 1 frame in 4 at uneven phase: a repeating cadence break on the default model. 3x lands on 48fps, an exact 2:1 decimation. interp.py now generalizes to an arbitrary factor using RIFE v4's timestep conditioning — one pass, not a recursive 2x-of-2x that would interpolate already-synthesized frames.

#6 — the provider only accepts whole-second durations, so a scene window is almost never an exact match and essentially every cloud clip gets retimed. setpts,fps= resamples by duplicating or dropping whole frames, and the cloud path had no RIFE equivalent. The smoothed result is accepted only at the exact frame-grid length, so motion smoothing can never cost frame accuracy (and therefore never desync the song).

#7 — chained i2v compounds exposure/saturation error per link with nothing bounding it: VB_LOCAL_CHAIN_MAX caps the chain across scenes, never within one, so a 12s scene is ~6 unbounded links at the 14B's ~2.3s native clip. The correction is clamped (±8% luma, ±6% saturation), which is what makes it safe to apply unconditionally — it rounds to a no-op when there is no drift and can never override deliberate lighting.

#8 — timeline concat now stream-copies when the inputs are compatible, verified against the summed input duration (the concat demuxer can exit 0 on a truncated file), falling back to the re-encode otherwise. On an upscaled master the temporal denoise comes off — it averages each pixel against neighbouring frames, smearing the very frames RIFE just synthesized — along with the second unsharp over an already-sharpened ESRGAN output.

Also: timeline resolution gets one definition in src/shared/videoRes.ts instead of two hand-copied literal pairs that nothing tied together (a mismatch there is silent — clips still render, conformClip just scales and pads them, quietly costing resolution). And autoconfig.ts's "not yet wired into sidecarEnv" comment is corrected: it has been wired all along.

Verification

  • npm test green: typecheck + 29 unit tests (12 new) + check-no-cloud
  • npm run build green
  • 15 checks executing the real ffmpeg helpers against synthetic clips — filter strings were the risk here, since a bad one degrades silently behind a best-effort fallback. Confirmed: signalstats parses, matchLevels moves luma 17.0/255 toward the anchor (inside the clamp) and no-ops at zero drift, stream-copy concat is frame-exact, fitToWindow lands exactly 67/67 frames, the minterpolate filtergraph is valid.

Not verified here — worth a look before merge

No Apple GPU or model weights in this environment, so two things are correct-by-construction but unproven at runtime:

  1. The RIFE 3x path at runtime. The frame math is proven (1+(n-1)·f+(f-1) == n·f across every tested n,f) and the file compiles, but Rife.process(timestep=1/3) was never actually executed.
  2. The visual judgment on #7 and #8. Both are clamped and reversible, but they are aesthetic changes — they deserve an A/B on a long scene (≥10s, so ≥5 chain links). If they don't convince, VB_LOCAL_CHAIN_MATCH=0 / VB_FINISH_TEMPORAL=1 / VB_FINISH_SHARPEN=1 restore the old behavior.

Escape hatches: VB_CONCAT_COPY, VB_SMOOTH_RETIME, VB_SMOOTH_RETIME_MIN, VB_LOCAL_CHAIN_MATCH, VB_FINISH_TEMPORAL, VB_FINISH_SHARPEN.

🤖 Generated with Claude Code

https://claude.ai/code/session_01TD7w2Asmu4Ga5HBGK99261


Generated by Claude Code

Eight fixes to the render chain. Every one of them was silent in production —
nothing crashed, the video just came out worse than the code was capable of.

1. setEnv is now a per-op REPLACE, not a merge (engine/config.ts).
   CFG only ever accumulated, so any key emitted for one model/op survived into
   the next. `render:requality` injects VB_LOCAL_WAN_STEPS=20 and the resolver
   emits it for the 5B — either one then poisoned every later 14B render in the
   same process: 20 steps forced through a 4-step Lightning distillation (flat,
   slow-motion output) and, worse, `isHd` read the leftover key and silently
   demoted an HD render to the fast VAE + short deadline while also skipping the
   LoRA — strictly worse than both Fast and HD.

2. isHd no longer depends on the absence of a step override (localVideo.ts).
   Quality is a settings question; a step override must not redefine it.

3. render:requality bumps steps for the 5B only (main/index.ts).
   The 5B runs native steps, so 20 is a real gain. Pushing 20 through the 14B's
   Lightning distillation over-denoises into flat motion, and its HD path already
   runs the full 40-step schedule.

4. RIFE targets an exact multiple of the timeline rate (localVideo.ts, interp.py).
   A fixed 2x took the 14B's native 16fps to 32, and the 24fps conform then
   dropped 1 frame in 4 at uneven phase — a repeating cadence break on the
   DEFAULT model. 3x lands on 48fps, an exact 2:1. interp.py generalizes to an
   arbitrary factor using RIFE v4's timestep conditioning (one pass, not a
   recursive 2x-of-2x that would interpolate synthesized frames).

5. A failed interpolation is logged instead of swallowed (localVideo.ts).
   The fallback conforms 16->24 by duplicating one frame in two — constant
   judder, previously with nothing in the logs to explain it.

6. The cloud retime is motion-compensated when the deviation is visible
   (ffmpeg.ts). The provider only accepts whole-second durations, so nearly
   every cloud clip is retimed, and setpts+fps resamples by duplicating or
   dropping whole frames with no local-style RIFE equivalent. Accepted only at
   the exact frame-grid length, so smoothing can never cost frame accuracy.

7. Clamped anti-drift on chained start frames (ffmpeg.ts, backends/local/video.ts).
   Chained i2v compounds exposure/saturation error per link with nothing bounding
   it — VB_LOCAL_CHAIN_MAX caps the chain across scenes, never within one, so a
   12s scene is 6 unbounded links. The clamp makes it a no-op when there is no
   drift and keeps it from ever fighting deliberate lighting.

8. One x264 generation removed, and the grade stops fighting the render
   (ffmpeg.ts, pipeline.ts). Timeline concat now stream-copies when the inputs
   are compatible, verified against the summed input duration. On an upscaled
   master the temporal denoise (which smears the frames RIFE just synthesized)
   and the second unsharp (over an already-sharpened ESRGAN output) come off.

Also: timeline resolution gets one definition in shared/videoRes.ts instead of
two hand-copied literal pairs, and autoconfig's stale "not yet wired" comment is
corrected — it has been wired into sidecarEnv all along.

Every new behavior is env-gated back to the old one. Verified: typecheck, 29 unit
tests (12 new), check-no-cloud, build, plus the ffmpeg helpers executed against
synthetic clips to prove the filtergraphs are valid and frame-exact rather than
silently falling back.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TD7w2Asmu4Ga5HBGK99261
@lBroth

lBroth commented Aug 3, 2026

Copy link
Copy Markdown
Owner Author

Merged into #3 rather than into this base. The commit was rebased on top of the newer 5B/taew2 work, which meant resolving two conflicts — renderClip/renderLocalScene had independently gained a new last argument on both branches, for two different purposes. Both are kept: endImg (the next scene's keyframe, sent to the model as end_image) and anchor (the still that extracted last frames are level-matched against to bound drift). See the "Merge note" section in #3. All eight fixes and the 12 tests in test/render-quality.test.ts are in; 33/33 pass on the merged tree. Closing this and deleting the branch.

@lBroth lBroth closed this Aug 3, 2026
@lBroth
lBroth deleted the claude/local-mode-pr-status-6bvg3d branch August 3, 2026 16:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants