fix(render): quality + fluidity defects across the local and cloud paths - #2
Closed
lBroth wants to merge 1 commit into
Closed
fix(render): quality + fluidity defects across the local and cloud paths#2lBroth wants to merge 1 commit into
lBroth wants to merge 1 commit into
Conversation
Eight fixes to the render chain. Every one of them was silent in production — nothing crashed, the video just came out worse than the code was capable of. 1. setEnv is now a per-op REPLACE, not a merge (engine/config.ts). CFG only ever accumulated, so any key emitted for one model/op survived into the next. `render:requality` injects VB_LOCAL_WAN_STEPS=20 and the resolver emits it for the 5B — either one then poisoned every later 14B render in the same process: 20 steps forced through a 4-step Lightning distillation (flat, slow-motion output) and, worse, `isHd` read the leftover key and silently demoted an HD render to the fast VAE + short deadline while also skipping the LoRA — strictly worse than both Fast and HD. 2. isHd no longer depends on the absence of a step override (localVideo.ts). Quality is a settings question; a step override must not redefine it. 3. render:requality bumps steps for the 5B only (main/index.ts). The 5B runs native steps, so 20 is a real gain. Pushing 20 through the 14B's Lightning distillation over-denoises into flat motion, and its HD path already runs the full 40-step schedule. 4. RIFE targets an exact multiple of the timeline rate (localVideo.ts, interp.py). A fixed 2x took the 14B's native 16fps to 32, and the 24fps conform then dropped 1 frame in 4 at uneven phase — a repeating cadence break on the DEFAULT model. 3x lands on 48fps, an exact 2:1. interp.py generalizes to an arbitrary factor using RIFE v4's timestep conditioning (one pass, not a recursive 2x-of-2x that would interpolate synthesized frames). 5. A failed interpolation is logged instead of swallowed (localVideo.ts). The fallback conforms 16->24 by duplicating one frame in two — constant judder, previously with nothing in the logs to explain it. 6. The cloud retime is motion-compensated when the deviation is visible (ffmpeg.ts). The provider only accepts whole-second durations, so nearly every cloud clip is retimed, and setpts+fps resamples by duplicating or dropping whole frames with no local-style RIFE equivalent. Accepted only at the exact frame-grid length, so smoothing can never cost frame accuracy. 7. Clamped anti-drift on chained start frames (ffmpeg.ts, backends/local/video.ts). Chained i2v compounds exposure/saturation error per link with nothing bounding it — VB_LOCAL_CHAIN_MAX caps the chain across scenes, never within one, so a 12s scene is 6 unbounded links. The clamp makes it a no-op when there is no drift and keeps it from ever fighting deliberate lighting. 8. One x264 generation removed, and the grade stops fighting the render (ffmpeg.ts, pipeline.ts). Timeline concat now stream-copies when the inputs are compatible, verified against the summed input duration. On an upscaled master the temporal denoise (which smears the frames RIFE just synthesized) and the second unsharp (over an already-sharpened ESRGAN output) come off. Also: timeline resolution gets one definition in shared/videoRes.ts instead of two hand-copied literal pairs, and autoconfig's stale "not yet wired" comment is corrected — it has been wired into sidecarEnv all along. Every new behavior is env-gated back to the old one. Verified: typecheck, 29 unit tests (12 new), check-no-cloud, build, plus the ffmpeg helpers executed against synthetic clips to prove the filtergraphs are valid and frame-exact rather than silently falling back. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TD7w2Asmu4Ga5HBGK99261
Owner
Author
|
Merged into #3 rather than into this base. The commit was rebased on top of the newer 5B/taew2 work, which meant resolving two conflicts — |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #1 (base is
local-models-wan22-mlx, notmain), so it merges into the local-mode branch.Eight fixes to the render chain, found while auditing it for maximum quality and maximum fluidity. Every one was silent in production — nothing crashed, the video just came out worse than the code was already capable of. Each new behavior is env-gated back to the old one, so any single piece rolls back with one variable.
The one that matters most
setEnvonly ever merged, so a one-shot env override leaked into every later render in the same process.render:requalityinjectsVB_LOCAL_WAN_STEPS=20, and the resolver emits it for the 5B. Either one then poisoned the next 14B render:isHdwas defined asquality === 'hd' && !env('VB_LOCAL_WAN_STEPS'), so the leftover key made HD read as false — the HD path silently dropped to the fast tiny-VAE and the short deadline while also skipping the LoRANet: selecting Quality/HD produced a few-step, no-LoRA, lossy-VAE render — strictly worse than both Fast and HD.
setEnvis now a per-op replace, andisHdno longer lets a step override redefine what HD means.All eight
setEnvis a per-op replace, not a merge — the root causeengine/config.tsisHdno longer depends on the absence of a step overrideengine/localVideo.tsrender:requalitybumps steps for the 5B only (native steps → 20 is a real gain; the 14B's HD path already runs the full 40-step schedule)main/index.tslocalVideo.ts,local/interp.pyengine/localVideo.tsengine/ffmpeg.tsffmpeg.ts,backends/local/video.tsffmpeg.ts,pipeline.tsNotes on the fluidity ones
#4 — a fixed 2x took the 14B's native 16fps to 32, and the 24fps conform then dropped 1 frame in 4 at uneven phase: a repeating cadence break on the default model. 3x lands on 48fps, an exact 2:1 decimation.
interp.pynow generalizes to an arbitrary factor using RIFE v4's timestep conditioning — one pass, not a recursive 2x-of-2x that would interpolate already-synthesized frames.#6 — the provider only accepts whole-second durations, so a scene window is almost never an exact match and essentially every cloud clip gets retimed.
setpts,fps=resamples by duplicating or dropping whole frames, and the cloud path had no RIFE equivalent. The smoothed result is accepted only at the exact frame-grid length, so motion smoothing can never cost frame accuracy (and therefore never desync the song).#7 — chained i2v compounds exposure/saturation error per link with nothing bounding it:
VB_LOCAL_CHAIN_MAXcaps the chain across scenes, never within one, so a 12s scene is ~6 unbounded links at the 14B's ~2.3s native clip. The correction is clamped (±8% luma, ±6% saturation), which is what makes it safe to apply unconditionally — it rounds to a no-op when there is no drift and can never override deliberate lighting.#8 — timeline concat now stream-copies when the inputs are compatible, verified against the summed input duration (the concat demuxer can exit 0 on a truncated file), falling back to the re-encode otherwise. On an upscaled master the temporal denoise comes off — it averages each pixel against neighbouring frames, smearing the very frames RIFE just synthesized — along with the second
unsharpover an already-sharpened ESRGAN output.Also: timeline resolution gets one definition in
src/shared/videoRes.tsinstead of two hand-copied literal pairs that nothing tied together (a mismatch there is silent — clips still render,conformClipjust scales and pads them, quietly costing resolution). Andautoconfig.ts's "not yet wired into sidecarEnv" comment is corrected: it has been wired all along.Verification
npm testgreen: typecheck + 29 unit tests (12 new) +check-no-cloudnpm run buildgreensignalstatsparses,matchLevelsmoves luma 17.0/255 toward the anchor (inside the clamp) and no-ops at zero drift, stream-copy concat is frame-exact,fitToWindowlands exactly 67/67 frames, theminterpolatefiltergraph is valid.Not verified here — worth a look before merge
No Apple GPU or model weights in this environment, so two things are correct-by-construction but unproven at runtime:
1+(n-1)·f+(f-1) == n·facross every tested n,f) and the file compiles, butRife.process(timestep=1/3)was never actually executed.VB_LOCAL_CHAIN_MATCH=0/VB_FINISH_TEMPORAL=1/VB_FINISH_SHARPEN=1restore the old behavior.Escape hatches:
VB_CONCAT_COPY,VB_SMOOTH_RETIME,VB_SMOOTH_RETIME_MIN,VB_LOCAL_CHAIN_MATCH,VB_FINISH_TEMPORAL,VB_FINISH_SHARPEN.🤖 Generated with Claude Code
https://claude.ai/code/session_01TD7w2Asmu4Ga5HBGK99261
Generated by Claude Code