From 8f1394c07fa7ecdf6dbe8347c1e655cc19efee3d Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Fri, 14 Aug 2026 14:10:54 +0000 Subject: [PATCH] feat(ltx-2.5): image conditioning at crf=0, the VAE encoder load path, and the keyframe refusal repair (#644) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Row `LTX25-IMAGE-COND`, reconstructed onto `main` (`043e56862`). Carries the whole row: the feature reviewed as #657 and the review repair reviewed as #666. Neither ever reached `main`. ## Why this is a new branch rather than a merge #666's base branch was `row/LTX25-IMAGE-COND`, not `main`, so merging it squashed it onto #657's head. That head was `a7154bd1e`, itself a *merge of main*, and **squashing on top of a merge flattens the merge away**. Git then no longer knew the row had ever seen `main`: `git merge-base 247eba52b origin/main` fell back to `b81a5e413`, 67 commits behind, and the row's apparent diff grew to 252 files and 42,426 insertions — git believing this row had authored everything `main` landed since, down to `tools/oracle/music3_oracle.py` and `website/README.md`. That is invisible until the next merge, when it presents as ~19 conflicts including **add/add on files the row never touched** (`gpt2.*`, `talker.*`, `w2vbert.*`, `.agents/specs/indextts-2-5.md`, `.agents/specs/minimax-music3.md`). Hand-resolving those is how `main`'s newest work gets silently reverted, so the merge was abandoned and the row rebuilt instead. **The transferable rule: never squash-merge into a branch that has itself merged `main`.** The squash commit inherits authorship of everything `main` did since, and nothing reports it at the time. Check `baseRefName` before merging, not just whether one branch is an ancestor of the other. ## How it was rebuilt, and how that is verified Branched from an IMMUTABLE SHA, never the ref. `origin/main` is shared with the other worktrees in this checkout and moved four times during this work; at one point that made `main`'s own files (the `dit_front`/`dit_stack` cluster) read as content this row had deleted, which is a fabricated regression waiting to be "fixed". The row's real delta is `git diff 32d82c64d 247eba52b`: `32d82c64d` is the `main` the row was last gated against, and `247eba52b` is the squash, whose tree is byte-identical (`d276e5194`) to the gated `56c2309ca`. That is **19 files, all owned by this row**, applied three-way onto `main` and then rebased forward twice. CORRECTION for anyone repeating this: `git diff a7154bd1e 247eba52b` is NOT the repair. It spans 246 files and 38,314 insertions, 11 of them `main`'s, because `a7154bd1e` carries `main` only to `b81a5e413` while `247eba52b` carries it to `32d82c64d`. Using it would reintroduce exactly the problem being fixed. The repair-only diff is `git show eb58fd58f` — 8 files, +266/-57. `docs/FEATURES.md` conflicted twice and was resolved as a keyed record both times: `main`'s file taken WHOLESALE and this row's key reapplied onto it, never the automatic three-way result. `.agents/roadmap_v1.md` applied cleanly and was verified the same way. Both are now byte-identical to `043e56862` apart from exactly one line each — the LTX-2.5 DiT row and the `#644` row. `main`'s known duplicate `Safetensors direct load` key (#769) is carried through untouched at its existing multiplicity of 2, neither fixed here nor multiplied. Verified by file list, which is the check that catches a bad reconstruction: `git diff 043e56862 --name-only` is exactly 19 LTX-2.5 files, with zero occurrences of `gpt2.*`, `talker.*`, `w2vbert.*`, `music3_oracle.py`, `bigvgan` or `website/`. Both CMake files are purely additive (zero removed lines), the registration delta is exactly +1 — 446 `vllm_cpp_add_test` against 445 — for `test_ltx2_image_cond`, and `git merge-base HEAD 043e56862` is `043e56862` itself, so the base is healthy again. ## Gate (CPU-only, Release, VLLM_CPP_CUDA=OFF) BUILD_EXIT=0, 0 hits for "No space left" / "BFD assertion" ctest -N 461 (main's 460 + test_ltx2_image_cond) test_ltx2_video 33 cases / 576 assertions, exit 0 test_ltx2_image_cond 15 cases / 223 assertions, exit 0 test_ltx2_vae 36 / 3039 test_ltx2 35 / 2435 test_ltx2_loader 26 / 4826 test_ltx2_pipeline 37 / 2382 test_ltx2_device 15 / 523 scripts/agent-preflight.sh --staged All gates green. `test_ltx2_video` reads 33/576 where the pre-reconstruction record says 32/565. That drift is `main`'s: `fc903b8dd` (#674) added `TEST_CASE("ltx2 VAE weights load from an ODD safetensors payload offset (#674)")` to that same file, worth +1 case and +11 assertions. `main` went 30 -> 31 cases over the window while this row's own contribution stayed +2 (one rename, two additions), which is what the diff shows. `LTX2_CHECKPOINT_ROOT` is unset here, so `test_ltx2_video`'s shipped-checkpoint subcase SKIPPED and is not in those counts. **The full `ctest -j 8` at this exact head is VOID, not passing and not failing.** The shared disk went to 100% mid-run with two other sessions compiling: the run aborted at 265/461, and `test_tool_choice_grammar` "failed" by reading back an EMPTY `/tmp/vllm_toolchoice_gbnf_0.json` it had just written, with the ctest log itself truncated mid-sentence. An ENOSPC run reports on a filesystem, not on a diff, so it is recorded as void rather than interpreted. The build that preceded it completed cleanly with zero ENOSPC hits, and the seven suites above ran after it and passed. The last VALID full run was `eae25c745` — the identical 19-file row content on a base four commits older — at 459/461, the two being `test_engine_core_proc` (the documented `ctest -j` starvation, which passes serially) and `test_op_parity` (#737, since fixed on `main` by `043e56862`, which this branch now sits on). A full re-run is owed once the box has headroom; CI covers it on a clean runner. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code] --- .agents/roadmap_v1.md | 2 +- .agents/specs/ltx25-image-conditioning.md | 438 +++++++ CMakeLists.txt | 7 + docs/FEATURES.md | 2 +- docs/USAGE.md | 58 +- examples/ltx2_gen/main.cpp | 44 +- include/vllm.h | 12 +- .../models/ltx2_image_preprocess.h | 103 ++ .../models/ltx2_video_vae_encoder.h | 45 +- include/vllm/multimodal/ltx2_video.h | 72 +- scripts/gen-ltx2-image-cond-goldens.py | 698 ++++++++++++ .../models/ltx2_image_preprocess.cpp | 219 ++++ .../models/ltx2_video_vae_encoder_load.cpp | 266 +++++ src/vllm/multimodal/ltx2_video.cpp | 298 ++++- tests/CMakeLists.txt | 7 + .../multimodal/ltx2_image_cond_goldens.inc | 1007 +++++++++++++++++ tests/vllm/multimodal/ltx2_video_fixture.h | 56 + .../vllm/multimodal/test_ltx2_image_cond.cpp | 765 +++++++++++++ tests/vllm/multimodal/test_ltx2_video.cpp | 372 +++++- 19 files changed, 4379 insertions(+), 92 deletions(-) create mode 100644 .agents/specs/ltx25-image-conditioning.md create mode 100644 include/vllm/model_executor/models/ltx2_image_preprocess.h create mode 100644 scripts/gen-ltx2-image-cond-goldens.py create mode 100644 src/vllm/model_executor/models/ltx2_image_preprocess.cpp create mode 100644 src/vllm/model_executor/models/ltx2_video_vae_encoder_load.cpp create mode 100644 tests/vllm/multimodal/ltx2_image_cond_goldens.inc create mode 100644 tests/vllm/multimodal/test_ltx2_image_cond.cpp diff --git a/.agents/roadmap_v1.md b/.agents/roadmap_v1.md index 7c2772ab4..7419704d5 100644 --- a/.agents/roadmap_v1.md +++ b/.agents/roadmap_v1.md @@ -153,7 +153,7 @@ issue is not yet placed. Keyed record: update in place, never append. | [#435](https://github.com/mudler/vllm.cpp/issues/435) | `ROAD-V1-LTX25` | LTX-2.5: port the 21B joint video+audio DiT and generalize the video seam | feature | | [#560](https://github.com/mudler/vllm.cpp/issues/560) | `ROAD-V1-LTX25` | Stabilizing constants unreachable by any gate: FIVE instances found by sweep, incl. the DiT's own `norm_eps` field default | bug | | [#567](https://github.com/mudler/vllm.cpp/issues/567) | `ROAD-V1-LTX25` | `OpNameImpl` makes a SECOND file every new `OpId` must edit; collapse the enum and its names into one derived list | bug | -| [#644](https://github.com/mudler/vllm.cpp/issues/644) | `ROAD-V1-LTX25` | LTX-2.5 FULL PORT campaign. Row 0 `LTX25-PROMPT-ADALN` (spec [`ltx25-prompt-adaln.md`](specs/ltx25-prompt-adaln.md)): `use_prompt_adaln_single` defaults TRUE in BOTH references and the shipped DiT carries its 18 tensors, but `ltx2_loader.cpp:988` cleared the flag unconditionally, so every render dropped the timestep-conditioned half of the prompt K/V modulation — invisible to every shape and finiteness check | feature | +| [#644](https://github.com/mudler/vllm.cpp/issues/644) | `ROAD-V1-LTX25` | LTX-2.5 FULL PORT campaign: close every refused conditioning arm. Row 0 `LTX25-PROMPT-ADALN` (spec [`ltx25-prompt-adaln.md`](specs/ltx25-prompt-adaln.md)) restored `use_prompt_adaln_single`, which the loader cleared unconditionally so every render dropped the timestep-conditioned half of the prompt K/V modulation. Row 1 `LTX25-IMAGE-COND` (spec [`ltx25-image-conditioning.md`](specs/ltx25-image-conditioning.md)) builds the video VAE ENCODER's load path — `Ltx2VideoVaeEncoderKeyRules` existed nowhere in the tree — and serves an image at latent frame 0 at `crf = 0`; keyframe / reference / non-zero-CRF stay refused by name | feature | | [#615](https://github.com/mudler/vllm.cpp/issues/615) | `GATE-PR-SIZE-BINARY` | `check-pr-size` fail-closes on every binary path with no exemption route, so no golden-bearing PR can merge: it blocks #431 and post-dates the golden precedent it rejects, spec [`gate-pr-size-binary.md`](specs/gate-pr-size-binary.md) | bug | | [#670](https://github.com/mudler/vllm.cpp/issues/670) | `BACKEND-TENSTORRENT-MISTRAL` | Tenstorrent: allowlist `MistralForCausalLM` and gate it on-device; goldens are `transformers`-teacher-forced because vLLM has no TT backend, spec [`tenstorrent-mistral.md`](specs/tenstorrent-mistral.md) | feature | | [#238](https://github.com/mudler/vllm.cpp/issues/238) | `SAMPLE-LOGPROB-TOKEN-IDS` | `logprobs_mode`: three of four modes are runtime-refused stubs | bug | diff --git a/.agents/specs/ltx25-image-conditioning.md b/.agents/specs/ltx25-image-conditioning.md new file mode 100644 index 000000000..1ccd5766a --- /dev/null +++ b/.agents/specs/ltx25-image-conditioning.md @@ -0,0 +1,438 @@ +# LTX-2.5 IMAGE CONDITIONING — the VAE encoder's load path, and the first arm off `ltx2_video.cpp:1122` + +**Row:** `LTX25-IMAGE-COND` (row 1 of the `#644` full-port campaign). +**Issue:** [#644](https://github.com/mudler/vllm.cpp/issues/644). +**Branch:** `row/LTX25-IMAGE-COND`. +**Parent spec:** [`ltx-2-5.md`](ltx-2-5.md) — operator-owned, NOT edited by this row. +**Upstream root (primary):** Lightricks/LTX-2 @ `fd4ded7f2d88d3da713abcdd4ad41ecc4a9314ca`, +`packages/ltx-core/src/ltx_core/` and `packages/ltx-pipelines/src/ltx_pipelines/`. +**Upstream (cross-check only):** `huggingface/diffusers` `pipeline_ltx2_condition.py`, +`ltx2/utils.py`. Where the two disagree, §3.3 records it and we follow `ltx_core`. + +--- + +## 0. What is claimed, and what is not + +This row closes exactly ONE conditioning kind — an **image at latent frame 0**, at +**CRF 0** — and it does so by building the thing that was actually absent: a +**weight-loading path for the video VAE encoder**. Keyframes, reference video and +reference audio stay refused, with a message that names what is now true rather +than repeating a reason that has gone stale. + +Three things are stated up front so they cannot be discovered later. + +1. **`crf = 0` is OUT OF DISTRIBUTION for LTX-2.5 and is labelled as such.** + Upstream resolves an unset CRF from the checkpoint generation + (`ImageConditioner.resolve_crf`, `blocks.py:977-983`), and a 2.5 checkpoint + resolves to **18** — `detect_params` maps `(2,5)` onto the newest row at or + below it, `((2, 4), LTX_2_4_PARAMS)` (`constants.py:130-133`), whose + `default_image_crf` is `LTX_2_4_IMAGE_CRF = 18` (`constants.py:37, 124`). So + the DEFAULT request refuses, and a caller must ask for `image_crf = 0` + explicitly. That is upstream-legal — `preprocess` short-circuits at + `if crf == 0: return image` (`decode.py:425-426`), and an explicit `0` is + documented as "skip re-compression entirely" (`args.py:58-59`) — but it + conditions on pixels the model was not trained to see. It is not silently + rendered. +2. **No render-quality claim.** This row's evidence is numeric parity against + executed upstream at reduced dimensions plus a mutation gate. A finite, + correctly-shaped clip is not a quality result; the parent spec §0 already + records why. +3. **No speed claim.** Nothing here is measured for throughput; the parent spec's + `PENDING` speed axis is untouched. + +## 1. The gap, re-verified against the current tree (not against the record) + +The refusal at `src/vllm/multimodal/ltx2_video.cpp:1122` is honestly written and +its anchors verify. Its internal claim was re-checked, and it is TRUE: + +| claim | check | result | +|---|---|---| +| the encoder MATH landed | `Ltx2ConvVideoEncode`, `src/vllm/model_executor/models/ltx2_video_vae.cpp:988` | present, gated (`test_ltx2_vae`) | +| the conditioning ITEMS landed | `Ltx2ConditionVideoByLatentIndex`, `src/vllm/model_executor/models/ltx2_conditioning.cpp:137` | present, gated | +| the engine loads the DECODER filter only | `src/vllm/multimodal/ltx2_video.cpp:752` | `Ltx2LoadVaeWeights(f, Ltx2VideoVaeDecoderKeyRules())` | +| an encoder key filter exists anywhere | `grep -rn Ltx2VideoVaeEncoderKeyRules src include tests` | **NO MATCH** | +| an encoder CONFIG parser exists anywhere | `grep -rn ParseConvVideoEncoder src include tests` | **NO MATCH** | + +So the encoder is a brick with no delivery route: nothing turns a checkpoint into +an `Ltx2ConvVideoEncoderConfig`, and nothing puts `encoder.*` tensors in a +`Ltx2VaeWeights`. That, not the math, is what this row builds. + +The CRF round trip is **not** on the critical path, contrary to what the refusal +text implies by listing it alongside the encoder. `preprocess` returns the image +unchanged at `crf == 0` (`decode.py:425-426`; note `:427-428` is a *different* +early return, for a degenerate < 2px side), and this environment has no `av` +module at all, which is itself the demonstration: a `crf = 0` path needs no +codec. + +## 2. Upstream anchors — every stage, both sides + +| stage | upstream `file:line` | ours | +|---|---|---| +| encoder key filter | `ltx-core/.../video_vae/model_configurator.py:267-276` (`VAE_ENCODER_COMFY_KEYS_FILTER`) | `Ltx2VideoVaeEncoderKeyRules` | +| encoder config | `model_configurator.py:37-69` (`_prepare_video_encoder_kwargs`) + `:72-78` (`VideoEncoderConfigurator`) | `Ltx2ParseConvVideoEncoderConfig` | +| encoder lifecycle | `ltx-pipelines/.../utils/blocks.py:936-991` (`ImageConditioner`; build `:985-986`, build-and-free `:988-991`) | engine load path, `ltx2_video.cpp` | +| CRF resolution | `blocks.py:966-983` + `constants.py:36-37, 124, 130-133` | `Ltx2ResolveDefaultImageCrf` | +| CRF round trip | `decode.py:413-435`, `encode_single_frame:386-400` | **REFUSED BY NAME** (§3.4) | +| image decode | `decode.py:139-170` (`decode_image`: EXIF rotate, ICC→sRGB, uint8 RGB) | `Ltx2DecodePpmRgb` (PPM only, §3.2) | +| aspect-fill resize | `media_io/resize.py:41-73` (`resize_and_center_crop`) | `Ltx2ResizeAndCenterCrop` | +| normalize | `media_io/range_map.py:8-9` (`normalize_images`) | in `Ltx2LoadImageAndPreprocess` | +| the whole preprocess | `decode.py:46-79` (`load_image_and_preprocess`) | `Ltx2LoadImageAndPreprocess` | +| VAE encode | `ltx-pipelines/.../utils/helpers.py:285-294` | `Ltx2ConvVideoEncode` (already landed) | +| latent composition | `ltx-core/.../conditioning/types/latent_cond.py:32-43` | `Ltx2ConditionVideoByLatentIndex` (already landed) | +| noise composition | `ltx-core/.../components/noisers.py:30-37` | `ApplyGaussianNoise` (already landed) | + +## 3. Design + +### 3.1 The encoder load path + +`Ltx2VideoVaeEncoderKeyRules()` mirrors `VAE_ENCODER_COMFY_KEYS_FILTER` +(`model_configurator.py:267-276`) in the first-match-wins prefix form the +already-gated `Ltx2VideoVaeDecoderKeyRules()` uses. It is a TRANSLATION, not a +rule-for-rule copy: upstream's `SDOps` (`loader/sd_ops.py:101-122`) admits a key +by `any()` over four matchings and then chains three substring replacements, +while this port matches and replaces in one pass — so the fourth rule below is an +identity that exists to carry upstream's fourth matching. Equivalent on every key +a shipped checkpoint carries; see the comment on the function for exactly when +they would part. + +``` +{"vae.encoder.", ""} +{"vae.per_channel_statistics.", "per_channel_statistics."} +{"encoder.", ""} +{"per_channel_statistics.", "per_channel_statistics."} +``` + +`per_channel_statistics` is in BOTH filters upstream and must be in both here: +`Ltx2ConvVideoEncode` reads `per_channel_statistics.{std,mean}-of-means` to +normalize its output (`video_vae.py:336`), so an encoder bag without them cannot +produce a latent in the DiT's space. + +`Ltx2ParseConvVideoEncoderConfig` mirrors `_prepare_video_encoder_kwargs` key for +key, including the two-layout split (`:46-53`): a nested `vae.encoder` object +takes its latent width from `encoder.out_channels`, a flat +`CausalVideoAutoencoder` from `vae.latent_channels` — and the top-level +`out_channels` is the DECODER's RGB count and must never be read as the latent +width (`:41-43`). The encoder's `spatial_padding_mode` default is `zeros` where +the decoder's is `reflect` (`:63-67` vs `:90`); both read the same checkpoint key +on a flat config, so they diverge only when it is ABSENT — silently, by a +half-pixel border, in opposite directions. That is why the default lives in the +parser and is pinned by a test. + +A checkpoint carrying no `encoder.*` tensors is a decoder-only (Comfy-split) +file. The engine then holds no encoder and an image request is refused BY NAME — +never served by falling back to something else. + +### 3.2 Image preprocessing at `crf = 0` + +`load_image_and_preprocess` (`decode.py:74-78`) is four steps, and the ORDER is +load-bearing: + +``` +image = decode_image(path) # uint8 [H, W, 3], sRGB +image = preprocess(image, crf) # identity at crf == 0 +image = torch.tensor(image, float32) # values still 0..255 +image = resize_and_center_crop(image, height, width) # <- IN 0..255 SPACE +image = normalize_images(image, ...) # /127.5 - 1.0 +``` + +The resize happens BEFORE the normalize. Bilinear interpolation is affine so the +two commute mathematically, but not bit-for-bit, and this project's existing PPM +reader (`minimax_h3_video.cpp:87-120`) normalizes at decode time. Reusing it +would put the affine first. So this row decodes to **uint8** and keeps upstream's +order; `Ltx2DecodePpmRgb` is a separate reader for that reason and the reason is +recorded here rather than left as an unexplained duplicate. + +`resize_and_center_crop` (`resize.py:41-73`) is aspect-FILL then centre crop: +`scale = max(h/src_h, w/src_w)`, `new = ceil(src * scale)` (upstream comments the +`ceil` as avoiding negative crop offsets), `interpolate(mode="bilinear", +align_corners=False)`, then `crop_top = (new_h - h) // 2`. + +The bilinear kernel mirrors PyTorch's `align_corners=False` index map in **f32**, +which is what `HelperInterpLinear::compute_indices_weights` uses +(`aten/src/ATen/native/cpu/UpSampleKernel.cpp`, dispatched over `scalar_t`): +`scale = src / dst`, `real = scale * (i + 0.5) - 0.5`, clamped to `>= 0`, +`idx = floor(real)`, `lambda = clamp(real - idx, 0, 1)`, and the right tap is +`idx + (idx < src - 1)`. Width and height passes are separated, height first — +matching the generic Nd kernel's dimension order. + +**Only PPM (P6) is read**, and only `maxval == 255`. No PNG/JPEG codec is +vendored — the same NAMED residual `minimax_h3_video.cpp:84-86` already carries — +and a `maxval != 255` PPM is REFUSED rather than rescaled, because upstream's +decode is PIL's and mirroring PIL's rescale semantics is a separate port. EXIF +rotation and ICC→sRGB conversion (`decode.py:143-168`) do not apply: PPM carries +neither tag, so there is nothing to honour and nothing is silently dropped. + +### 3.3 Composition — and where the two references DISAGREE + +`ltx_core` writes the encoded latent into `clean_latent` and sets +`denoise_mask = 1 - strength` (`latent_cond.py:40-41`), leaving the NOISY tensor +alone; the noiser then composes the two with a DOUBLE lerp +(`noisers.py:32-33`): + +``` +latent = lerp(latent, noise, noise_scale) +latent = lerp(clean_latent, latent, denoise_mask) +``` + +diffusers instead writes the clean tokens into the noisy tensor as well +(`pipeline_ltx2_condition.py:1002-1004`, `:1229-1231`). The two agree ONLY at +`noise_scale == 1`. The two-stage distilled recipe's second phase does not run at +`noise_scale == 1`, so this is a live divergence, not a theoretical one. + +**We follow `ltx_core`.** `Ltx2ConditionVideoByLatentIndex` already implements +exactly that and `ltx2_conditioning.h` already records the divergence; this row +adds no new composition, it only reaches the existing one. Do not silently switch +to the diffusers form. + +Placement in `Generate`: the conditioning is applied to the video `StreamState` +AFTER `clean` is seeded from the patchified initial volume and BEFORE +`ApplyGaussianNoise` — which is `create_noised_state` order (`helpers.py:428-445`: +initial state, then the conditioning items, then the noiser). NOT `blocks.py:576-580`, +which this spec cited until the review of #657: those lines are the TEARDOWN +(`clear_conditioning` + `unpatchify`) and say nothing about conditioning order. +It is applied on EVERY phase, because every phase rebuilds its state from the +recipe and a conditioning dropped on phase 2 would be re-noised away. + +`strength` comes from the seam's existing `VideoGenParams::noise_aug` +("keyframe pinning strength; <= 0 => 1.0", `include/vllm.h:762`), whose polarity +already matches upstream's `ImageConditioningInput.strength` (`args.py:64`): 1.0 +pins, mask 0. + +### 3.4 What is still refused, and in what words + +Four distinct refusals, each naming a different missing piece so a later reader +can re-check its cause rather than trust it. This campaign has had FIVE refusals +whose stated reason went stale; each message below names the exact symbol or +`file:line` that would have to change for it to become false. + +1. **Non-zero CRF.** Names `encode_single_frame` / `decode_single_frame` + (`decode.py:386-410`) as unported, states that `crf = 0` is the supported + value and that it is out of distribution for a 2.5 checkpoint whose resolved + default is 18, and says which extra to set. +2. **No encoder in the checkpoint.** Names `Ltx2VideoVaeEncoderKeyRules` and says + the file carried no `encoder.*` / `vae.encoder.*` tensors. +3. **Keyframes** (`last_frame_path`, or an image at a non-zero frame index). + `Ltx2ConditionVideoByKeyframe` EXISTS and is gated; what is missing is the + TOKEN-APPEND machinery. `VideoConditionByKeyframeIndex.apply_to` + (`keyframe_cond.py:36-90`) appends tokens — concatenating onto `latent`, + `denoise_mask`, `positions` and `clean_latent` (`:79-82`), giving them their + own coordinates offset to `frame_idx` (`:46-59`) and rebuilding the attention + mask via `update_attention_mask` (`:68-76`) — and `clear_conditioning` + (`ltx_core/tools.py:88-105`) trims them back before unpatchify. `Ltx2LatentState` + carries no attention mask, and the engine's phase loop is fixed at the target + grid's token count from one `Ltx2VideoTokenCount` through the sigma schedule, + the `Ltx2ModalityInput` and `Ltx2VideoUnpatchify`. The first-frame arm needs + none of it because `VideoConditionByLatentIndex` REPLACES existing tokens. + + **This spec and the shipped message previously named + `keyframes_abs_pos_embedding` as the blocker. That was FALSE at pin + `fd4ded7f`, and a test had been written to assert it by name.** A supplied + keyframe is appended with `marked=False` (`keyframe_cond.py:84-86`), and its + sole consumer adds `mask * embedding` with `mask = keyframes_mask > 0` + (`model/transformer/transformer_args.py:42-43`, called once at `:269`), so the + embedding contributes nothing to those tokens and porting it would not serve + this arm. The tokens that DO reach it are the target's own first latent frame, + marked unconditionally by `_first_frame_keyframes_mask` + (`ltx_core/tools.py:184-196`) — the frame the SERVED first-frame arm writes + into. That is a real gap on the served arm and is tracked as + [#658](https://github.com/mudler/vllm.cpp/issues/658); it is not what blocks a + last-frame keyframe. +4. **Reference video / reference image / reference audio.** + `Ltx2ConditionVideoByReference` and `Ltx2ConditionAudioByReference` also + EXIST; what is missing is that both need the IC-LoRA's `downscale_factor` / + `temporal_scale_factor`, which upstream stores in LoRA metadata this project + does not read (already recorded at `ltx2_conditioning.h:110-114`), and the + audio arm additionally needs the audio VAE ENCODER, whose key rules this row + does not add. + +## 4. The things that fail silently here + +* **Reading `vae.out_channels` as the encoder's latent width.** It is the + decoder's RGB count (3). An encoder built with `out_channels = 3` still runs + and still produces a latent. +* **The encoder's `zeros` vs the decoder's `reflect` padding default.** Same + checkpoint key, different defaults, and they only diverge when the key is + absent (`model_configurator.py:63-68` vs `:92`). +* **Defaulting a `res_x` block's `num_layers`.** Upstream subscripts + `block_config["num_layers"]` (`video_vae.py:55`) and raises `KeyError`; no + other block kind reads the field. `ParseEncoderBlocks` defaulted it to 1, which + builds a one-layer `UNetMidBlock3D` out of a config upstream refuses. Made + strict in the review of #657, matching how `multiplier`'s sentinel two lines + below already treats an absent value. +* **Normalizing before resizing.** Same answer to ~1e-7 — and §8.1 records the + outcome: NO golden here sees it either, because the two orders are + algebraically equal. Mirrored because it is upstream's, and written down + rather than assumed to be covered. +* **Applying the conditioning after the noiser.** Produces a pinned first frame + that is pinned to the NOISED latent. Shapes, masks and finiteness all pass. +* **Inverting the mask (`strength` instead of `1 - strength`).** Renders an + unconditioned clip that looks like the feature not working. +* **A conditioning that is loaded but never read.** The whole class this row's + mutation evidence exists to exclude: `last_conditioning()` scaled x1.5 and + row-reversed both passed every assertion in an earlier phase. + +## 5. Tests and evidence + +**Goldens.** `scripts/gen-ltx2-image-cond-goldens.py` imports and EXECUTES +upstream under the pinned SHA and emits +`tests/vllm/multimodal/ltx2_image_cond_goldens.inc`. Sections: + +1. `resize_and_center_crop` alone, over shapes covering upscale, downscale, + wider-than-target and taller-than-target, including one where `ceil` + disagrees with both `round` and `floor` (§8.1 corrects what that case + actually demonstrates). +2. `load_image_and_preprocess`'s full chain at `crf = 0` (resize then normalize). +3. `VideoEncoder(image)` at reduced dims over deterministic weights. +4. `VideoConditionByLatentIndex.apply_to` over that encoded latent — `clean` and + `denoise_mask`. +5. `GaussianNoiser` over the conditioned state at a NON-unit `noise_scale`, which + is the arm at which `ltx_core` and diffusers disagree (§3.3). + +**Harness adaptations, recorded:** (a) `ltx_pipelines.utils.media_io.__init__` +imports `av`, which is absent in this environment, so `resize.py` and +`range_map.py` are loaded by FILE PATH with `importlib` rather than as package +members. Both import only `torch`/`einops`/stdlib, so nothing about the math +changes, and the workaround is itself the evidence for §0.1. (b) Weights are +filled from the same deterministic stream `gen-ltx2-vae-goldens.py` uses, so no +weight byte is checked in. + +**Mutation evidence (required, not optional).** Each of these is applied to a +scratch copy, rebuilt, run, and the tree restored byte-for-byte: + +| mutation | must RED | +|---|---| +| conditioned latent scaled x1.5 | golden §4/§5 | +| mask set to `strength` instead of `1 - strength` | golden §4 | +| conditioning applied AFTER the noiser | golden §5 | +| resize/normalize order swapped | golden §2 | +| encoder key rules replaced by decoder key rules | the load test (weights absent) | +| the encoder config's latent width read from `vae.out_channels` | the config test | + +**Encoder weights are LOADED AND USED, proven separately from "loadable":** the +engine test perturbs ONE tensor of the encoder half of the fixture checkpoint and +asserts the conditioning trace's image digest moves, with every other byte of the +request identical. + +## 6. Gates + +* `ctest -R 'test_ltx2_image_cond|test_ltx2_video|test_ltx2_vae'` — focused, with + CASE and ASSERTION counts recorded on both sides of every mutation. +* Full `ctest` with the case count asserted against `ctest -N`. +* `scripts/agent-preflight.sh --staged` before commit. + +## 7. Stop conditions + +* If closing the image arm would require editing the prompt-AdaLN path in + `ltx2_loader.cpp` / `ltx2.cpp`, STOP and return `NEEDS_DECISION` — that is + `row/LTX25-PROMPT-ADALN`'s surface. +* If executed upstream disagrees with a ported stage by more than + `kLtx2GoldenTol`, the port is wrong; do not widen the band. + +## 8. Outcome — what was measured, and what was refuted + +Recorded here rather than in the code, because none of it is derivable from the +tree. All measurements: CPU Release, gcc, `-ffp-contract=off` (the tree's own +flag), `build-lic`, this box, at the head this spec landed on. + +### 8.1 Three claims in §3 and §5 were WRONG, and are corrected here + +* **The `ceil` reason.** §3.2 repeated upstream's own comment — that `ceil` + guards against `src * scale` landing just above an integer and producing a + negative crop offset. Swept every source/target pair in `3..40 -> {16, 24}`: + **no pair does that** in IEEE double. The `ceil` is still load-bearing, for + the ordinary reason that it disagrees with `round` and `floor` at a + non-integer scale (case 1, `32 * 16/24 = 21.333` → 22 rows and a 3-row crop, + against 21 and a 2-row crop), and the generator asserts THAT rather than the + claim upstream makes. +* **A golden can see the resize/normalize ORDER.** It cannot, and the reason is + structural: resize is a convex combination and normalize is affine, so the two + orders are equal in exact arithmetic and their f32 gap is pure rounding — + **1.94e-07 measured**, below the golden band and below this port's own distance + from torch. It cannot be amplified by choosing a different image. The order is + mirrored because it is upstream's; that is now written down in three places + rather than assumed to be covered, and `kLtx2ImgPreOrderGap` asserts the gap + stays below the band so a future change that makes it gateable is visible. +* **The port can match torch's bilinear bit for bit.** It cannot, portably. The + index map and the lambdas were probed with basis images and match EXACTLY + (`0.61111104`, `0.35185182`, `0.09259248` … reproduced to the bit). The + residual is in the ACCUMULATION and appears on an output element whose width + weights are `(1, 0)` — a pure two-term `a*h0 + c*h1` — which rules out + dimension order. Plain-f32, f64-accumulate and premultiplied-weight orderings + all land 1 ulp away on the same elements: FMA contraction inside torch's + kernel, which this tree compiles with `-ffp-contract=off` and so cannot + reproduce. Hence `kLtx2ImgPixelTol`. + +### 8.2 The bands, derived rather than picked + +Measured by setting both to `1e-12` and reading the reported `worst`: + +| section | space | worst | band | +|---|---|---|---| +| 1 resize | 0..255 | `6.10352e-05` (identity case: 0) | `kLtx2ImgPixelTol = 2e-4` | +| 2 preprocess | [-1, 1] | `4.76837e-07` | `kLtx2ImgGoldenTol = 2e-6` | +| 3 encoded latent | latent | `2.68221e-07` | same | +| 4 conditioned clean | latent | `2.68221e-07` | same | +| 5 noised latent | latent | `1.78814e-07` | same | + +### 8.3 Mutation evidence — RED, with counts + +Each applied to the tree, rebuilt, run, and restored byte-for-byte (md5 checked). +Green baseline: `test_ltx2_image_cond` **15 cases / 198 assertions**, +`test_ltx2_video` **32 cases / 550 assertions**, both exit 0. + +| mutation | file | result | +|---|---|---| +| `ceil` → `llround` in the resize | `ltx2_image_preprocess.cpp` | image_cond 14/15, 197/198, exit 1 | +| encoder key rules → the DECODER's | `ltx2_video_vae_encoder_load.cpp` | image_cond 14/15, 195/198; video 31/32 (THREW, assertions 517 — the COUNT itself moved) | +| mask `1 - strength` → `strength` | `ltx2_conditioning.cpp` | image_cond 13/15, 193/198, exit 1 | +| latent width from `vae.out_channels` | `ltx2_video_vae_encoder_load.cpp` | image_cond 14/15, 192/198; video 9/32, 36/53 | +| CRF refusal removed | `ltx2_image_preprocess.cpp` | image_cond 14/15, 193/198; video 31/32 | +| encode the image, never PLACE it | `ltx2_video.cpp` | video 31/32, 548/550, exit 1 | + +### 8.4 A mutation that SURVIVED, and what that says + +`video.latent = state.clean` inserted after the placement left `test_ltx2_video` +fully green (32/32, 550/550). Two findings, and the first is not a gap: + +1. **Phase 0 runs at `noise_scale = 1.0`** (`ltx2_pipeline.cpp:1068`), where + `lerp(latent, noise, 1)` discards `latent` entirely — so on that phase the + mutation is genuinely inert. AGENTS.md's rule applies: a mutation that moves + nothing is not evidence of unreachability. It IS live on phase 1 + (`noise_scale = 0.909375`). +2. **`test_ltx2_video` gates no VALUE of the composed latent.** It gates that the + conditioning is placed, that it depends on the image, and that it depends on + the ENCODER'S OWN WEIGHTS — not the arithmetic. That arithmetic is gated + against executed upstream in `test_ltx2_image_cond`, over the identical + functions. This is recorded as a named residual rather than closed with an + unanchored digest, which would detect change without pinning anything. + +That mutation is also what caused `Ltx2ConditioningTrace::image_digest` to be +taken over the TOKENS AS WRITTEN rather than over the encoder's output: the +first version digested `encoded.data`, which stays healthy for a build that +encodes an image and never places it. The "never place it" mutation above is RED +only because of that change. + +### 8.5 Why the defaults are what they are + +* **`image_crf` has no default that renders.** Absent resolves 18 and refuses. + A default of 0 would silently condition every request out of distribution. +* **The encoder stays RESIDENT**, where `ImageConditioner` builds and frees it + per call (built at `blocks.py:985-986`, built-and-freed around `fn` at + `:988-991`). A conditioning image arrives per request and + the encoder is small next to the DiT; the divergence is lifecycle only. +* **The conditioning is applied PER PHASE**, because the two-stage recipe renders + its stages at different resolutions and upstream passes each stage's own + height/width. One encode would either be re-noised away or placed at the wrong + scale. **This reason was gated by nothing until the review of #657.** MEASURED: + changing the guard to `wants_image && phase_index == 0` left `test_ltx2_video` + at 32 cases / 550 assertions / exit 0, because `image_tokens` and `image_digest` + are overwritten each phase and the suite only asked `image_tokens > 0`. The + trace now pins the LAST phase's per-latent-frame token count (4 in the fixture, + 1 at phase 0) and contrasts it with a `max_phase = 0` engine over the same + request, which REDs that mutant at `1 == 4`. It matters because hoisting the + per-phase decode+encode out of the loop is the obvious optimization. + +## 9. Now + +`ACTIVE` — spec committed before implementation; implementation landed with the +outcome above. diff --git a/CMakeLists.txt b/CMakeLists.txt index 6f53b5811..e52404d48 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -836,6 +836,13 @@ add_library(vllm STATIC # LTX-2.5 (ROW MODEL-DIFFUSION-LTX25, phase L11): the conditioning items that # place the VAE ENCODERS' output into the denoise state. src/vllm/model_executor/models/ltx2_conditioning.cpp + # LTX-2.5 (ROW LTX25-IMAGE-COND, issue #644): the video VAE ENCODER's LOAD + # path — key filter and config parser, which existed nowhere before — and the + # pixel front-end an image conditioning arrives through. Separate TUs rather + # than additions to ltx2_loader.cpp, which two concurrent rows of the same + # campaign need to edit. + src/vllm/model_executor/models/ltx2_video_vae_encoder_load.cpp + src/vllm/model_executor/models/ltx2_image_preprocess.cpp # LTX-2.5 (ROW MODEL-DIFFUSION-LTX25, phase L6): the quantized loaders — the # FP8 and torchao-NVFP4 DiT arms, the torchao-NVFP4 text encoder, and the # load-time device staging GB10's ATS penalty makes the default. diff --git a/docs/FEATURES.md b/docs/FEATURES.md index 9083978b7..585df8ede 100644 --- a/docs/FEATURES.md +++ b/docs/FEATURES.md @@ -163,7 +163,7 @@ in `ltx2_text_encoder.cpp` is the call that would have to change. | Voxtral audio (`VoxtralForConditionalGeneration`) | Voxtral-Mini-3B-2507 | near-tie-robust 16/16 vs vLLM 0.25.0 | decode 0.97x (beats vLLM); encoder FORWARD 15.90x of vLLM's whole TTFT (pin 46.02 ms), or 2.89x with opt-in `VT_WHISPER_ENC_FA2=1` (costs 3 near-tie divergences vs 0). Not a TTFT ratio. Pending | | Whisper audio encoder | openai/whisper-small; whisper-large-v3 (Voxtral cfg) | encoder tower 77/77; large-v3 tower 203/203 | pending | | MiniMax-H3 DiT (`MiniMaxH3DiTModel`, vllm-omni lane) | MiniMax-H3 (33.1B video+audio) | portable 79/79; all three modalities COHERENT on Q4_K_M (§8.20); PRUNED ckpts run, Q8_0 seam 0.9941 (§8.21); ref2va grid was NVFP4 quant error, §8.9 REFUTED; GGUF/NVFP4/bf16 shards stream | FP4/Marlin landed; speed pending; no bf16 render yet. Render from the Q4_K_M GGUF, not the NVFP4 arm. Krea 2 text-to-image (roadmap C11) is scoped to reuse these DiT seams | -| LTX-2.5 DiT (`LTX2VideoTransformer3DModel`, Lightricks lane) | LTX-2.5 (21.00B video+audio) | `SPIKE`. DiT, VAEs+ENCODERS, conditioning, pipeline, quant loaders gated at reduced dims. Prompt-side AdaLN ported, host+device. Typed prompt to Gemma-4 to cross-attn, FIXTURE-gated. A prompted render is OWED | Family `ltx-2.5`, `ltx2-gen`. ~29 GB NVFP4/GB10, FP8 ~44 GB, +~24 GB tower. FP8/torchao/1st-party NVFP4 load; `keyframes_abs_pos_embedding` alone needs `allow_unported`. DiffVAE, LoRA, image cond refused. Speed PENDING | +| LTX-2.5 DiT (`LTX2VideoTransformer3DModel`, Lightricks lane) | LTX-2.5 (21.00B video+audio) | `SPIKE`. DiT, VAEs+ENCODERS, conditioning, pipeline, quant loaders gated at reduced dims. Prompt AdaLN host+device; prompt->Gemma-4->cross-attn FIXTURE-gated. Image chain PPM->resize->encode->place->noise. Render OWED | `ltx-2.5`/`ltx2-gen`. ~29 GB NVFP4/GB10, FP8 ~44 GB, +24 GB tower. FP8/torchao/NVFP4 load; `keyframes_abs_pos_embedding` needs `allow_unported`. IMAGE cond SERVED `crf=0`; DiffVAE/LoRA/keyframe/ref refused. Speed PENDING | | MiniMax-Music3 (`MiniMaxMusic3ForConditionalGeneration`, diffusers lane) | MiniMax-Music3 (8.6B Qwen3 LLM + 0.646B RVQ decoder + 2.4B fp32 DiT + DAC Flow-VAE); diffusers arm, ~28.5 GB | `ACTIVE`. Loader 1413/1413 (#714); AR half (W2/W3) and acoustic half (W4/W5: DiT, scheduler, vocoder) gated vs the real weights, waveform 88,064 samples. No greedy path upstream, so no token gate. W2 LM fwd, W6, W7 owed | Not measured. The denominator will be SGLang-Omni in its production configuration (both CUDA graphs, compiled DIT and DAV, batched seeded sampling) | | MTP speculator | Qwen3.6-27B, Qwen3.6-35B-A3B | token-identical to vLLM `mtp` at c1 | ~4% faster c1; +16% output tput (MoE) | | DFlash block-diffusion | Qwen3 (DFlash draft) | near-tie e2e 27/27 vs vLLM | 2.9x over spec-off, 1.003x vs vLLM DFlash-on | diff --git a/docs/USAGE.md b/docs/USAGE.md index 684459ff0..5663833d5 100644 --- a/docs/USAGE.md +++ b/docs/USAGE.md @@ -405,21 +405,44 @@ the embeddings connector) are implemented and gated. Several limits decide what you can actually ask for, and each refuses by name rather than rendering something else. -In particular, the encoders being present does NOT mean image, keyframe, -reference-video or reference-audio conditioning is usable: the video engine -still refuses every one of those by name, because the request-side work between -a file on disk and a tensor the encoder accepts — image decode, aspect-fill -resize, and the H.264 CRF re-compression upstream performs before encoding -whenever the resolved CRF is not `0` and the image is at least 2 pixels on its -shorter side — is not ported. The engine also holds no -encoder to call: it materializes the VAE DECODER key filters only, so no -encoder weights are ever in memory, and the refusal names that rather than -claiming the encoder itself is missing. Two encoder-level limits are worth +**Image conditioning (image-to-video) runs at `image_crf=0`, and only there.** +Pass a first frame as binary PPM (`first_frame_path` / `first_frame_ppm`) plus +the per-generation extra `image_crf=0`; the engine decodes it, aspect-fills and +centre-crops it to each phase's own resolution, VAE-encodes it, and replaces +latent frame 0's clean tokens. `noise_aug` is the pinning strength (`1.0`, the +default, pins the frame exactly). + +`image_crf=0` must be asked for **explicitly**, and it is **out of +distribution**. Upstream re-compresses a conditioning image through H.264 at the +CRF the checkpoint's generation was trained with, and an LTX-2.5 checkpoint +resolves that to **18**. That round trip needs libx264 and no codec is vendored +here, so a non-zero CRF — including the default a caller gets by saying nothing — +is refused by name. `image_crf=0` is upstream-legal (upstream short-circuits it +and documents an explicit `0` as "skip re-compression entirely") but conditions +the model on pixels it was not trained to see. That is a render-quality cost, and +it is stated rather than applied silently. + +Keyframe, reference-image, reference-video and reference-audio conditioning are +still refused, each naming a different missing piece: a last-frame keyframe needs +the token-APPEND machinery — a keyframe is appended to the sequence with its own +positions and a rebuilt attention mask, then trimmed back off, and this engine's +phase loop is fixed at the target grid's token count — while the served +first-frame arm only REPLACES tokens that already exist; the reference arms need +the IC-LoRA's scale factors, which live in LoRA metadata this project does not +read; reference audio additionally needs the AUDIO VAE's encoder key filter, +which is not built. (Until 2026-08-13 this said a last-frame keyframe needs the +DiT's unported `keyframes_abs_pos_embedding`. That was wrong: a supplied keyframe +is appended unmarked, so the embedding never applies to it. Where the embedding +does bite is the FIRST latent frame of every render, which is a separate gap, +tracked as issue #658.) Three encoder-level limits are worth stating in advance because they are refusals rather than approximations. A reference waveform whose sample rate differs from the audio VAE's is refused rather than resampled, since upstream uses a polyphase kaiser resampler this -project does not carry. And a VAE configured with `latent_log_var: none` is -refused, because upstream itself raises on it. +project does not carry. A VAE configured with `latent_log_var: none` is +refused, because upstream itself raises on it. And a video-VAE `res_x` encoder +block that declares no `num_layers` is refused rather than defaulted, because +upstream subscripts that key and raises `KeyError` on it; no other encoder block +kind reads it. **A typed prompt works.** `--encoder` names the Gemma-4 12B text tower and `--prompt` carries the words. The tower tokenizes them with its OWN embedded @@ -457,7 +480,11 @@ refused, because a stream left unconditioned renders instead of failing. returns the trace of the last `Generate()` — whether the conditioning came from a prompt or from embeds, the prompt string, the row count and both stream widths, an FNV-1a digest over the exact f32 buffers cross-attention read, and each stream's -absmax. It is returned **by value, under the engine's own lock**, so it is safe to +absmax. When the request carried an image it also reports the CRF and strength it +was conditioned at, how many tokens the encoded image replaced, and a digest over +**those tokens as written into the state** — not over the encoder's output, so a +build that encoded an image and never placed it reads as unconditioned rather +than healthy. It is returned **by value, under the engine's own lock**, so it is safe to call from a server thread while another thread renders — but `Generate` holds that same lock for the WHOLE render, so such a call blocks for minutes rather than returning a stale answer immediately. `completed` is true only if that @@ -511,6 +538,11 @@ ltx2-gen --dit ltx-2.5-22b-distilled-fp8.safetensors \ Swap the two `--encoder*` flags and `--prompt` for `--prompt-embeds` + `--audio-prompt-embeds` to condition from files instead. +Add `--first-frame frame.ppm --image-crf 0` for image-to-video. The PPM is +binary P6 at maxval 255 (no PNG/JPEG codec is vendored); `--image-crf 0` is +required and is not the default, because omitting it resolves the checkpoint's +own CRF 18 and refuses — see the out-of-distribution note above. + `--frames` must satisfy `(frames - 1) % 8 == 0` and width/height must divide by 64 (32 for the VAE, twice that because the distilled recipe's first phase runs at half resolution). Omitting all three renders the recipe default, which is diff --git a/examples/ltx2_gen/main.cpp b/examples/ltx2_gen/main.cpp index e17bb6953..28ed2cc0a 100644 --- a/examples/ltx2_gen/main.cpp +++ b/examples/ltx2_gen/main.cpp @@ -100,6 +100,7 @@ const char* Need(int argc, char** argv, int i, const char* flag) { " [--max-phase N] [--allow-unported]\n" " [--prompt-valid-rows N] how many embed rows are real tokens\n" " [--frames N] [--width N] [--height N] [--seed N]\n" + " [--first-frame ] [--image-crf 0]\n" " [--device cpu|cuda]\n\n" "Renders LTX-2.5 (family \"ltx-2.5\") through vllm_video_engine_load +\n" "vllm_video_generate.\n\n" @@ -121,7 +122,17 @@ const char* Need(int argc, char** argv, int i, const char* flag) { "weights. The row count must then be a multiple of the connector's learnable\n" "register count (128 on the shipped files), and --prompt-valid-rows says how\n" "many of them are real: the rest are padding, and padding is REPLACED by the\n" - "learnable register table rather than ignored.\n"); + "learnable register table rather than ignored.\n\n" + "IMAGE CONDITIONING (image-to-video). --first-frame takes a binary PPM (P6,\n" + "maxval 255) and pins latent frame 0 to it: it is decoded, aspect-filled and\n" + "centre-cropped to each phase's own resolution, VAE-encoded, and written into\n" + "the clean latent. It needs --image-crf 0, and that is DELIBERATELY not the\n" + "default. Upstream re-compresses a conditioning image through H.264 at the CRF\n" + "the checkpoint's generation was trained with, which for LTX-2.5 is 18; that\n" + "round trip needs libx264 and none is vendored here, so leaving --image-crf out\n" + "resolves 18 and REFUSES by name. --image-crf 0 is upstream-legal and OUT OF\n" + "DISTRIBUTION: the model sees pixels it was not trained on. That is a quality\n" + "cost, and this tool states it rather than turning it on quietly.\n"); std::exit(code); } @@ -133,7 +144,7 @@ int main(int argc, char** argv) { std::string workdir = "/tmp/ltx2_gen", out_path, ffmpeg = "ffmpeg", device = "cuda"; // BORROWED by `vllm_video_generate`, like the extras below, so it is owned // here and pointed at only after parsing. - std::string prompt; + std::string prompt, first_frame, image_crf; // The extras are BORROWED by the load call, so the strings must outlive it. // Kept as two parallel vectors of owned strings plus the char* views the ABI @@ -174,6 +185,14 @@ int main(int argc, char** argv) { else if (f == "--prompt-valid-rows") SetExtra("prompt_embeds_valid_rows", Need(argc, argv, ++i, f.c_str())); else if (f == "--allow-unported") SetExtra("allow_unported_modules", "1"); + // Image conditioning (row LTX25-IMAGE-COND, issue #644). `--first-frame` is + // a binary PPM; `--image-crf` is the PER-GENERATION extra, so it rides + // vp.extra_* rather than mp.extra_*. Only 0 is served, and it is NOT + // defaulted here — leaving it out lets the engine resolve the checkpoint's + // own 18 and refuse, which is the point: this CLI must not be the thing that + // quietly turns an out-of-distribution render on. + else if (f == "--first-frame") first_frame = Need(argc, argv, ++i, "--first-frame"); + else if (f == "--image-crf") image_crf = Need(argc, argv, ++i, "--image-crf"); else if (f == "--device") device = Need(argc, argv, ++i, "--device"); else if (f == "--frames") vp.num_frames = std::atoi(Need(argc, argv, ++i, "--frames")); else if (f == "--width") vp.width = std::atoi(Need(argc, argv, ++i, "--width")); @@ -203,6 +222,27 @@ int main(int argc, char** argv) { mp.family = "ltx-2.5"; vp.output_dir = workdir.c_str(); if (!prompt.empty()) vp.prompt = prompt.c_str(); + if (!first_frame.empty()) vp.first_frame = first_frame.c_str(); + + // The PER-GENERATION extras are a SEPARATE array from the load-time ones, and + // conflating them is the whole failure this keeps apart: `image_crf` handed to + // the load call is an unknown LOAD extra and is refused there, which would + // read as "the flag does not work" rather than as "it goes on the other call". + std::vector gen_keys, gen_values; + if (!image_crf.empty()) { + gen_keys.emplace_back("image_crf"); + gen_values.push_back(image_crf); + } + std::vector gkeys, gvalues; + for (size_t i = 0; i < gen_keys.size(); ++i) { + gkeys.push_back(gen_keys[i].c_str()); + gvalues.push_back(gen_values[i].c_str()); + } + if (!gkeys.empty()) { + vp.extra_keys = gkeys.data(); + vp.extra_values = gvalues.data(); + vp.n_extras = static_cast(gkeys.size()); + } std::vector keys, values; keys.reserve(extra_keys.size()); diff --git a/include/vllm.h b/include/vllm.h index 24fd02683..c7c3ffea3 100644 --- a/include/vllm.h +++ b/include/vllm.h @@ -853,8 +853,16 @@ typedef struct vllm_video_params { /* Where frame_%06d.ppm + audio.wav land (created if absent). REQUIRED. */ const char* output_dir; /* v18: FAMILY-SPECIFIC per-generation settings, same parallel-array shape as - * the load-time extras. MiniMax-H3 defines none, and refuses any key it does - * not know rather than ignoring it. 0 => none. */ + * the load-time extras. Every family refuses a key it does not know rather + * than ignoring it. 0 => none. + * MiniMax-H3: none. + * LTX-2.5: "image_crf" — the H.264 CRF an image conditioning is + * re-compressed at. Only "0" is served; an LTX-2.5 checkpoint + * RESOLVES 18 when this is absent and the codec round trip is + * unported, so leaving it out refuses BY NAME rather than + * rendering. "0" is upstream-legal and out of distribution; + * see docs/USAGE.md. No ABI change was needed for it, which is + * what this parallel-array shape exists for. */ const char* const* extra_keys; const char* const* extra_values; int32_t n_extras; diff --git a/include/vllm/model_executor/models/ltx2_image_preprocess.h b/include/vllm/model_executor/models/ltx2_image_preprocess.h new file mode 100644 index 000000000..36936289a --- /dev/null +++ b/include/vllm/model_executor/models/ltx2_image_preprocess.h @@ -0,0 +1,103 @@ +// LTX-2.5 IMAGE CONDITIONING INPUT — pixels to the VAE encoder's [-1, 1] space. +// +// Row: LTX25-IMAGE-COND. Spec: .agents/specs/ltx25-image-conditioning.md §3.2. +// Issue #644. +// +// ─── WHAT THIS IS A PORT OF (file:line on BOTH sides) ──────────────────────── +// Upstream root: Lightricks/LTX-2 @ fd4ded7f, +// packages/ltx-pipelines/src/ltx_pipelines/utils/ +// OURS <- UPSTREAM +// Ltx2LoadImageAndPreprocess <- media_io/decode.py:46-79 +// Ltx2PreprocessImageCrf <- media_io/decode.py:413-435 (`preprocess`) +// Ltx2DecodePpmRgb <- media_io/decode.py:139-170 (`decode_image`) +// Ltx2ResizeAndCenterCrop <- media_io/resize.py:41-73 +// (the /127.5 - 1 map) <- media_io/range_map.py:8-9 +// Ltx2ResolveDefaultImageCrf <- constants.py:36-37, 124, 126-133 +// + blocks.py:966-983 (`ImageConditioner`) +// +// ─── THE ORDER IS LOAD-BEARING ─────────────────────────────────────────────── +// `load_image_and_preprocess` resizes in 0..255 SPACE and normalizes AFTER +// (decode.py:76-78). Bilinear interpolation is affine, so the two commute in +// exact arithmetic and NOT in floating point. This project's other PPM reader +// (`minimax_h3_video.cpp:87-120`) normalizes at decode time, which is why this +// file has its own decoder rather than calling that one: reusing it would put +// the affine map first, and no shape or finiteness check could see it. +// +// ─── WHAT IS DELIBERATELY NOT PORTED, AND WHAT THAT COSTS ──────────────────── +// * THE H.264 ROUND TRIP. `preprocess` re-compresses at the checkpoint's +// `default_image_crf` to match the compression the model was trained against +// (decode.py:413-435 -> encode_single_frame:386-400, libx264 preset=veryfast, +// rgb24 -> yuv420p, dimensions truncated to even). No codec is vendored here, +// so a non-zero CRF is REFUSED BY NAME. `crf == 0` short-circuits upstream at +// :425-426 and is served — it is upstream-legal ("including ``0`` to skip +// re-compression entirely", args.py:58-59) and OUT OF DISTRIBUTION for a 2.5 +// checkpoint, whose resolved default is 18. Both halves of that are said out +// loud rather than one of them. +// * PNG / JPEG / EXR. Only binary PPM (P6) is read — the same NAMED residual +// minimax_h3_video.cpp:84-86 already carries for this tree. EXIF orientation +// and ICC -> sRGB conversion (decode.py:143-168) have nothing to act on in a +// PPM, so they are absent rather than dropped. +// * A `maxval` other than 255. PIL rescales those; mirroring its rescale is a +// separate port, so they are refused rather than scaled by a rule nobody +// checked against PIL. +#pragma once + +#include +#include +#include + +namespace vllm { + +// `decode_image` (decode.py:139-170) for the one container this tree reads: +// binary PPM (P6), `maxval == 255`, into a uint8 [H, W, 3] buffer — the exact +// shape and dtype `np.array(image, dtype=np.uint8)` returns at :170. +std::vector Ltx2DecodePpmRgb(const std::string& field, const std::string& bytes, + int64_t* out_height, int64_t* out_width); + +// `preprocess` (decode.py:413-435). Identity at `crf == 0` (:425-426); THROWS by +// name at any other value, naming the unported codec round trip. `crf < 0` is +// refused too: upstream's only unset spelling is `None`, which +// `ImageConditioner.resolve_crf` fills in before `preprocess` ever sees it +// (blocks.py:977-983), so a negative value is not "unset" in any upstream sense. +void Ltx2PreprocessImageCrf(int64_t crf); + +// `resize_and_center_crop` (resize.py:41-73) for a single image. Takes HWC +// f32 in ANY value space and returns [channels, height, width] — the +// `1 c f h w` upstream emits at f = 1, with the two singleton axes dropped. +// +// Aspect FILL then centre crop: `scale = max(height/src_h, width/src_w)`, then +// `ceil` (upstream's own comment: avoids a negative crop offset from +// floating-point rounding), then bilinear with `align_corners=False`, then +// `crop_top = (new_h - height) // 2`. +// +// The bilinear kernel mirrors PyTorch's index map in f32, which is the dtype +// `HelperInterpLinear::compute_indices_weights` dispatches at for a float +// tensor: `real = scale * (i + 0.5) - 0.5` clamped to >= 0, `idx = floor(real)` +// capped at `src - 1`, `lambda = clamp(real - idx, 0, 1)`, right tap +// `min(idx + 1, src - 1)`. HEIGHT is the OUTER sum and WIDTH the inner one, +// matching `Interpolate::eval`'s recursion order over the dimensions +// `upsample_generic_Nd_kernel_impl` appends in. +std::vector Ltx2ResizeAndCenterCrop(const float* hwc, int64_t src_height, + int64_t src_width, int64_t channels, int64_t height, + int64_t width); + +// `load_image_and_preprocess` (decode.py:46-79) end to end: decode -> CRF -> +// f32 0..255 -> resize+crop -> `/127.5 - 1.0`. Returns [3, height, width] in +// [-1, 1], which is what `Ltx2ConvVideoEncode` takes at `frame_count = 1`. +std::vector Ltx2LoadImageAndPreprocess(const std::string& field, const std::string& bytes, + int64_t height, int64_t width, int64_t crf); + +// `PipelineParams.default_image_crf` as `detect_params` resolves it +// (constants.py:126-133): the newest generation row at or below the checkpoint's +// `model_version`, so an unrecognised NEWER version inherits the closest known +// one rather than falling back to 2.0's. Today that is `(2, 4) -> 18` +// (LTX_2_4_IMAGE_CRF, :37, :124) and everything older -> 33 (DEFAULT_IMAGE_CRF, +// :36). `components` is `Ltx2ParseModelVersion`'s output; an EMPTY one compares +// below every row, exactly as `detect_model_version` documents at :137-140. +// +// This is what makes the CRF refusal a real one rather than a formality: an +// LTX-2.5 request that does not name a CRF resolves 18 and is refused, and the +// caller has to ask for 0 knowingly. +int64_t Ltx2ResolveDefaultImageCrf(const std::vector& version_components); + +} // namespace vllm diff --git a/include/vllm/model_executor/models/ltx2_video_vae_encoder.h b/include/vllm/model_executor/models/ltx2_video_vae_encoder.h index 65faa1a29..b7fb8640c 100644 --- a/include/vllm/model_executor/models/ltx2_video_vae_encoder.h +++ b/include/vllm/model_executor/models/ltx2_video_vae_encoder.h @@ -28,7 +28,7 @@ // // ─── THE FIVE THINGS THAT FAIL SILENTLY ────────────────────────────────────── // * THE ENCODER'S DEFAULT SPATIAL PADDING IS `zeros`; THE DECODER'S IS -// `reflect` (model_configurator.py:63-67 vs :90). They read the SAME +// `reflect` (model_configurator.py:63-68 vs :92). They read the SAME // checkpoint key `spatial_padding_mode` on a flat CausalVideoAutoencoder // config, so they only diverge when the key is ABSENT — and then they diverge // silently, by a half-pixel border, in opposite directions. @@ -62,6 +62,7 @@ #include #include "vllm/model_executor/models/ltx2_audio_vae.h" // Ltx2VaeWeights +#include "vllm/model_executor/models/ltx2_loader.h" // Ltx2VaeKeyRule, nlohmann::json #include "vllm/model_executor/models/ltx2_upsampler.h" // Ltx2LatentVolume #include "vllm/model_executor/models/ltx2_video_vae.h" // Ltx2NormLayer, Ltx2PaddingMode @@ -180,4 +181,46 @@ Ltx2LatentVolume Ltx2ConvVideoEncode(const Ltx2ConvVideoEncoderConfig& config, int64_t frame_count, int64_t height, int64_t width, int64_t* out_cropped_frames = nullptr); +// ─── THE DELIVERY ROUTE (row LTX25-IMAGE-COND, issue #644) ─────────────────── +// +// Everything above is the encoder's MATH, and it landed in phase L11 with +// goldens. What did not land is any way to reach it from a checkpoint: until +// this row, `Ltx2VideoVaeEncoderKeyRules` matched NOWHERE in the tree and no +// parser produced an `Ltx2ConvVideoEncoderConfig` from a `vae` config object. +// `ltx2_video.cpp:752` materialized `Ltx2VideoVaeDecoderKeyRules()` alone, so +// the encoder was a brick with no delivery route and every conditioning arm +// refused for that reason. These three declarations are that route. + +// `VAE_ENCODER_COMFY_KEYS_FILTER` (video_vae/model_configurator.py:267-276), in +// the same first-match-wins prefix form `Ltx2VideoVaeDecoderKeyRules` uses. +// +// `per_channel_statistics.` IS IN BOTH FILTERS UPSTREAM and must be in both +// here: `Ltx2ConvVideoEncode` divides its conv output by +// `per_channel_statistics.std-of-means` (video_vae.py:336), so an encoder bag +// filtered without them cannot produce a latent in the DiT's space at all — it +// throws on the missing key rather than emitting an unnormalized one, which is +// the one mercy in this arrangement. +std::vector Ltx2VideoVaeEncoderKeyRules(); + +// Does this checkpoint carry an ENCODER half? A Comfy-split `vae/` file may hold +// the decoder alone, and the answer decides whether image conditioning is served +// or refused BY NAME. Asked of the file's declared tensor names only; no payload +// is read. +bool Ltx2CheckpointHasVideoEncoder(const std::vector& tensor_names); + +// `VideoEncoderConfigurator.from_metadata` (model_configurator.py:72-78) over +// `_prepare_video_encoder_kwargs` (:37-69), key for key. +// +// THE TWO FIELDS THAT FAIL SILENTLY IF READ FROM THE WRONG PLACE: +// * the LATENT WIDTH. On a flat `CausalVideoAutoencoder` config it is +// `vae.latent_channels`; the top-level `vae.out_channels` is the DECODER's +// RGB count and reading it builds a 3-channel-latent encoder that still runs +// (:41-43). On a nested `CausalDiffusionVAE` config it is +// `vae.encoder.out_channels` (:46-49) — the same spelling, the other object. +// * `spatial_padding_mode`. The ENCODER's default is `zeros`; the decoder's is +// `reflect` (:63-68 vs :92). Both read the same checkpoint key on a flat +// config, so they diverge only when the key is ABSENT, silently, by a +// half-pixel border, in opposite directions. +Ltx2ConvVideoEncoderConfig Ltx2ParseConvVideoEncoderConfig(const nlohmann::json& config); + } // namespace vllm diff --git a/include/vllm/multimodal/ltx2_video.h b/include/vllm/multimodal/ltx2_video.h index 0953aebff..b74f38d0f 100644 --- a/include/vllm/multimodal/ltx2_video.h +++ b/include/vllm/multimodal/ltx2_video.h @@ -234,6 +234,28 @@ inline constexpr char kLtx2PromptValidRowsExtra[] = "prompt_embeds_valid_rows"; // resolved in either direction, for the same reason. inline constexpr char kLtx2EncoderConfigPathExtra[] = "encoder_config_path"; +// ── the PER-GENERATION extra (VideoGenParams::extras) ─────────────────────── + +// The H.264 CRF the image conditioning is re-compressed at, `ImageConditioner`'s +// `resolve_crf` (ltx-pipelines/utils/blocks.py:977-983). Row LTX25-IMAGE-COND, +// issue #644. +// +// ABSENT MEANS "WHAT THE MODEL WAS TRAINED WITH", which for an LTX-2.5 +// checkpoint is **18** — `detect_params` maps a version at or above `(2, 4)` +// onto `LTX_2_4_PARAMS` and its `LTX_2_4_IMAGE_CRF` (utils/constants.py:37, +// 124, 130-133). And that round trip is NOT ported: it needs libx264 +// (media_io/decode.py:430-434 -> encode_single_frame:386-400) and no codec is +// vendored here. So the DEFAULT REFUSES, by name. +// +// `image_crf=0` is the supported value and is served. It is upstream-legal — +// `preprocess` short-circuits at `if crf == 0: return image` (decode.py:425-426) +// and an explicit 0 is documented as "skip re-compression entirely" +// (utils/args.py:58-59) — and it is OUT OF DISTRIBUTION, because the model was +// trained on images that had been through the codec. Both halves are said out +// loud rather than one of them: a caller has to ask for 0 knowingly, and gets a +// render conditioned on uncompressed pixels rather than a refusal. +inline constexpr char kLtx2ImageCrfExtra[] = "image_crf"; + // WHAT THE LAST `Generate()` ACTUALLY HANDED THE DiT's CROSS-ATTENTION. // // Every field is read off the exact f32 buffers `Ltx2ModalityInput::context` @@ -265,16 +287,22 @@ inline constexpr char kLtx2EncoderConfigPathExtra[] = "encoder_config_path"; // * the conditioning rows REVERSED, putting every caption row on the wrong // token, // -// and BOTH passed `test_ltx2_video` at 30 cases / 499 assertions with exit 0. -// The digest moved, as it must — but no assertion says WHICH value it should -// have moved to. -// -// THAT COUNT IS THIS HEAD'S, and the distinction is the point of writing it -// down. A reviewer first measured the pair at `43aa58377`, where the suite stood -// at 485 assertions; the numbers were carried forward unchanged while the suite -// grew, so the comment named a count no run of it could produce. Re-run here -// (CPU Release, mutant recompiled and relinked each leg, tree restored -// byte-for-byte and re-verified green between legs): 499/499, exit 0, both. +// and BOTH passed `test_ltx2_video` with exit 0 — at 30 cases / 499 assertions +// when the pair was last re-run. The digest moved, as it must — but no assertion +// says WHICH value it should have moved to. +// +// THE COUNT IS DELIBERATELY NOT RESTATED AS A CURRENT FIGURE, and the reason is +// the history: a reviewer first measured the pair at `43aa58377`, where the +// suite stood at 485 assertions; the numbers were carried forward unchanged +// while the suite grew, so the comment named a count no run of it could produce. +// It was re-measured at 499 (CPU Release, mutant recompiled and relinked each +// leg, tree restored byte-for-byte and re-verified green between legs) — and +// then row LTX25-IMAGE-COND (#644) added the image-conditioning cases and the +// suite moved to 32 / 550, which is exactly how the previous number went stale +// the first time. A count in a header is a MEASUREMENT OF ANOTHER FILE stored +// here, which AGENTS.md §Records names as the thing that couples every PR to +// lines it does not own. What survives is the finding — the mutations passed — +// and the SHA-dated measurement above it; `ctest` is the authority on the count. // // THE VALUE ORACLE THE COMPOSITION IS OWED. The per-brick oracles are real and // strong: the Gemma-4 tower against a running `transformers` at a measured bf16 @@ -311,6 +339,30 @@ struct Ltx2ConditioningTrace { // two prompts the SAME digest and RED any dependence check, but it would do so // for the wrong reason; this says which happened. double video_absmax = 0.0, audio_absmax = 0.0; + // ── the IMAGE conditioning (row LTX25-IMAGE-COND, issue #644) ──────────── + // + // Zero everywhere when the request carried no image. `image_tokens` is how + // many of the video stream's tokens the encoded image REPLACED in the clean + // latent, and `image_digest` is FNV-1a over THOSE TOKENS' raw f32 bytes — the + // same instrument, and with the same limits, as the two prompt digests above: + // it detects CHANGE, it does not pin VALUES. + // + // OVER THE TOKENS, NOT OVER THE ENCODER'S OUTPUT, and that choice is load + // bearing: a digest of the encoder's output answers "was an image encoded", + // which stays true of a build that encodes one and then never places it — an + // unconditioned render with a perfectly healthy trace. + // + // IT IS ALSO THE ONLY WAY TO ASK WHETHER THE ENCODER WEIGHTS WERE READ. "The + // conditioning loaded" and "the conditioning was used" are different claims, + // and a render cannot be inspected for either. Perturbing one encoder tensor + // and watching this digest move is what separates them, which is exactly the + // check `test_ltx2_video` runs. + int64_t image_tokens = 0; + uint64_t image_digest = 0; + double image_absmax = 0.0; + int64_t image_crf = 0; // the CRF this render actually preprocessed at + double image_strength = 0.0; // `ImageConditioningInput.strength` (args.py:64) + // True only once the `Generate` that produced this conditioning RETURNED. The // trace is filled immediately after the connector and BEFORE the denoise loop, // because that is the only point at which the exact buffers cross-attention diff --git a/scripts/gen-ltx2-image-cond-goldens.py b/scripts/gen-ltx2-image-cond-goldens.py new file mode 100644 index 000000000..70aa6a6e7 --- /dev/null +++ b/scripts/gen-ltx2-image-cond-goldens.py @@ -0,0 +1,698 @@ +#!/usr/bin/env python3 +"""Emit tests/vllm/multimodal/ltx2_image_cond_goldens.inc — the LTX-2.5 IMAGE +CONDITIONING parity oracle. Row LTX25-IMAGE-COND, issue #644. + +Spec: .agents/specs/ltx25-image-conditioning.md §5. + +WHAT THIS GATES, and why it is a separate generator from +`scripts/gen-ltx2-vae-goldens.py`. That script gates the VAE BRICKS. This one +gates the CHAIN a conditioning image travels: pixels -> aspect-fill resize -> +normalize -> VideoEncoder -> VideoConditionByLatentIndex -> GaussianNoiser. Every +link but the first two already had a golden; the chain did not, and a chain whose +links are each green can still be wired in the wrong ORDER — which for this +particular chain is exactly the defect that survives every shape and finiteness +check (§4 of the spec). + +Upstream sources (Lightricks/LTX-2): + ltx-pipelines/.../utils/media_io/resize.py:41-73 -> section 1, 2 + ltx-pipelines/.../utils/media_io/range_map.py:8-9 -> section 2 + ltx-pipelines/.../utils/media_io/decode.py:413-435 -> section 2 (the crf==0 branch) + ltx-core/.../model/video_vae/video_vae.py:148-336 -> section 3 + ltx-core/.../conditioning/types/latent_cond.py:22-43 -> section 4 + ltx-core/.../components/noisers.py:30-37 -> section 5 + +Usage: + python3 scripts/gen-ltx2-image-cond-goldens.py \\ + --ltx2 ~/_git/LTX-2 \\ + --out tests/vllm/multimodal/ltx2_image_cond_goldens.inc + +Needs torch + numpy + einops (CPU only). NO checkpoint and no gated download. + +UPSTREAM REVISION ANCHOR, and a DIRTY-TREE REFUSAL, exactly as +gen-ltx2-vae-goldens.py has them and for the identical reason: without a SHA +nobody can tell a PORT drift from an UPSTREAM one, and `git rev-parse HEAD` on a +dirty tree stamps a clean anchor onto numbers that commit cannot reproduce. + + Pinned revision: fd4ded7f2d88d3da713abcdd4ad41ecc4a9314ca + +Advancing the pin is a deliberate edit in BOTH places (here and +`kLtx2ImgCondUpstreamRevisionPin` in tests/vllm/multimodal/test_ltx2_image_cond.cpp). + +── THREE HARNESS ADAPTATIONS, all recorded because none changes the math ─────── + +1. `av` AND `OpenImageIO` ARE STUBBED, and the stub is a TRIPWIRE. + `ltx_pipelines.utils.media_io.decode` imports both at module scope, and + neither is installed here. The crf==0 path this row ports never reaches + either — that is its whole point — so they are registered as modules whose + every attribute is a stub object. This is NOT a way to avoid running upstream: + `preprocess(image, 0)` is executed for real and asserted to return the + IDENTICAL object it was given (`result is image`), which no stub can + manufacture, and `preprocess(image, 18)` is executed too and asserted to RAISE + inside the stubbed codec — which proves the branch is live rather than + assumed. A source-text assertion about `if crf == 0` would have been + self-confirming; this is not. + +2. THE PACKAGE `__init__` FILES ARE NOT EXECUTED. `ltx_pipelines/__init__.py` + and its parents pull in the same codec chain. The three package levels are + registered as module objects carrying only `__path__`, so `import + ltx_pipelines.utils.media_io.resize` loads the SUBMODULE from its real file + without running any `__init__.py`. The submodules themselves are executed + verbatim. + +3. `GaussianNoiser._sample_noise` DRAWS FROM THE SHARED STREAM instead of + `torch.randn`. Upstream keys its draw to a `torch.Generator`; the C++ side + consumes the same deterministic stream in the same order. This mirrors the + `torch.randn` patch gen-ltx2-vae-goldens.py already applies to the decoder. + +ORACLE IDENTITY is asserted, not assumed: `ltx_core.__file__` and +`ltx_pipelines...resize.__file__` are both checked to live under `--ltx2` before +anything runs, because a `.pth`, an editable install or a namespace-package +layout would otherwise resolve a DIFFERENT source silently. +""" + +from __future__ import annotations + +import argparse +import math +import re +import subprocess +import sys +import types +from pathlib import Path + +import numpy as np + +_MASK64 = (1 << 64) - 1 + + +# --------------------------------------------------------------------------- +# THE GOLDEN BAND, READ from the C++ suite rather than repeated here — the same +# arrangement gen-ltx2-vae-goldens.py uses, and for the same reason: a literal +# here would be a second definition of one number in a second language, and a +# widened C++ band would leave this generator certifying arms nobody checks +# against. A parse that does not find EXACTLY ONE definition is fatal. +# --------------------------------------------------------------------------- + +_GOLDEN_TOL_SOURCE = ( + Path(__file__).resolve().parents[1] + / "tests" + / "vllm" + / "multimodal" + / "test_ltx2_image_cond.cpp" +) + + +def _read_golden_tol() -> float: + text = _GOLDEN_TOL_SOURCE.read_text(encoding="utf-8") + hits = re.findall(r"^constexpr double kLtx2ImgGoldenTol = ([0-9eE.+-]+);", text, re.M) + if len(hits) != 1: + raise SystemExit( + f"expected EXACTLY ONE `constexpr double kLtx2ImgGoldenTol = ...;` in " + f"{_GOLDEN_TOL_SOURCE}, found {len(hits)} — the generator cannot assert against a " + f"band it cannot resolve" + ) + return float(hits[0]) + + +# --------------------------------------------------------------------------- +# The shared deterministic stream. Byte-for-byte the one +# scripts/gen-ltx2-vae-goldens.py uses and tests/vllm/models/test_ltx2_vae.cpp +# mirrors, so a tensor built by this script and one built by that one from the +# same NAME are the same tensor. +# --------------------------------------------------------------------------- + + +def fnv1a64(name: str) -> int: + h = 0xCBF29CE484222325 + for byte in name.encode("utf-8"): + h ^= byte + h = (h * 0x100000001B3) & _MASK64 + return h + + +def splitmix64(x: int) -> int: + x = (x + 0x9E3779B97F4A7C15) & _MASK64 + z = x + z = ((z ^ (z >> 30)) * 0xBF58476D1CE4E5B9) & _MASK64 + z = ((z ^ (z >> 27)) * 0x94D049BB133111EB) & _MASK64 + return z ^ (z >> 31) + + +def ltx_rand(name: str, count: int) -> np.ndarray: + """`count` values uniform in [-1, 1), reproducible from `name` alone.""" + seed = fnv1a64(name) + out = np.empty(count, dtype=np.float64) + for i in range(count): + u = splitmix64((seed + i) & _MASK64) + out[i] = ((u >> 11) * (2.0**-53)) * 2.0 - 1.0 + return out + + +def ltx_bytes(name: str, count: int) -> np.ndarray: + """`count` UINT8 codes — a conditioning image is uint8 out of the decoder + (`np.array(image, dtype=np.uint8)`, decode.py:170), and quantizing a float + stream afterwards would gate a different input than the one a real PPM + carries.""" + seed = fnv1a64(name) + return np.array( + [splitmix64((seed + i) & _MASK64) % 256 for i in range(count)], dtype=np.uint8 + ) + + +def param_values(name: str, shape) -> np.ndarray: + """The per-parameter role rule, IDENTICAL to gen-ltx2-vae-goldens.py's. Only + the roles a video ENCODER actually carries are reachable here; the audio and + vocoder roles that script also handles have no counterpart in this chain.""" + count = int(np.prod(shape)) if len(shape) else 1 + rank = len(shape) + if name.endswith("std-of-means"): + return ltx_rand(name, count) * 0.1 + 1.0 + if name.endswith("mean-of-means"): + return ltx_rand(name, count) * 0.1 + if name.endswith(".bias"): + return ltx_rand(name, count) * 0.05 + if rank == 1 and name.endswith(".weight"): + return ltx_rand(name, count) * 0.1 + 1.0 + return ltx_rand(name, count) * 0.1 + + +def fill_from_stream(module, prefix: str = "") -> list[tuple[str, int]]: + import torch + + state = module.state_dict() + manifest: list[tuple[str, int]] = [] + filled = {} + for name, tensor in state.items(): + values = param_values(prefix + name, tuple(tensor.shape)) + filled[name] = torch.from_numpy(values.astype(np.float32)).reshape(tensor.shape) + manifest.append((prefix + name, int(values.size))) + module.load_state_dict(filled, strict=True) + return manifest + + +# --------------------------------------------------------------------------- +# Emit helpers +# --------------------------------------------------------------------------- + + +def _cxx_float(value: float, digits: int) -> str: + if not math.isfinite(value): + raise ValueError(f"refusing to emit non-finite golden value: {value}") + text = f"{value:.{digits}g}" + if "." not in text and "e" not in text and "E" not in text: + text += ".0" + return text + + +def emit_f32(out, name: str, values) -> None: + flat = np.asarray(values, dtype=np.float32).reshape(-1).tolist() + out.write(f"inline constexpr float {name}[] = {{\n") + for i in range(0, len(flat), 6): + chunk = ", ".join(_cxx_float(v, 9) + "f" for v in flat[i : i + 6]) + out.write(" " + chunk + ",\n") + out.write("};\n\n") + + +def emit_scalar(out, name: str, value) -> None: + out.write(f"inline constexpr int64_t {name} = {int(value)};\n") + + +def emit_double(out, name: str, value) -> None: + out.write(f"inline constexpr double {name} = {_cxx_float(float(value), 17)};\n") + + +def emit_manifest(out, name: str, manifest: list[tuple[str, int]]) -> None: + out.write(f"inline constexpr const char* {name}Names[] = {{\n") + for key, _ in manifest: + out.write(f' "{key}",\n') + out.write("};\n") + out.write(f"inline constexpr int64_t {name}Counts[] = {{\n") + for i in range(0, len(manifest), 10): + out.write(" " + ", ".join(str(c) for _, c in manifest[i : i + 10]) + ",\n") + out.write("};\n\n") + + +# --------------------------------------------------------------------------- +# Reduced-dimension geometry +# --------------------------------------------------------------------------- + +# Section 1 — the resize cases. Chosen so that between them they cover: pure +# upscale, pure downscale, a source WIDER than the target aspect, a source +# TALLER than it, an exact identity, and — case 1 — a scale at which `ceil` +# disagrees with BOTH `round` and `int`: `32 * (16/24)` is 21.3333, so upstream +# resizes to 22 rows and crops 3 off the top, while a port that rounded would +# resize to 21 and crop 2. Every value in the output moves, and every shape +# check still passes. That case is asserted below rather than trusted. +# +# MEASURED, and recorded because it corrects what an earlier draft of this +# comment claimed: no source/target pair in this size range makes +# `src * scale` land just ABOVE an integer in IEEE double, which is the +# negative-crop hazard resize.py:61-62 names. The `ceil` is still load-bearing — +# for the ordinary reason above — but not for the reason upstream's comment +# gives, and asserting the reason upstream gives would have been asserting +# something false. +RESIZE_CASES = [ + # (src_h, src_w, dst_h, dst_w) + (12, 20, 16, 16), + (32, 24, 16, 16), + (16, 16, 16, 16), + (8, 8, 16, 24), + (10, 7, 16, 16), +] + +# Section 2-5 — the conditioning chain. The image is a 12x20 PPM (case 0's +# geometry, so the `ceil` branch is LIVE on the path that actually renders) and +# the target is 16x16, which the encoder's (2 patch x 2 x 2) spatial factor of 8 +# turns into a 2x2 latent. +IMAGE_SRC_H, IMAGE_SRC_W = 12, 20 +IMAGE_DST_H, IMAGE_DST_W = 16, 16 + +IMG_ENC_BLOCKS = [ + ("res_x", {"num_layers": 1}), + ("compress_space_res", {"multiplier": 2}), + ("compress_all_res", {"multiplier": 1}), +] +IMG_ENC = dict(convolution_dimensions=3, in_channels=3, out_channels=4, patch_size=2) + +# The target latent the conditioning is placed into: 3 latent frames of the +# encoder's own (channels, height, width), conditioned at index 0 — which is what +# `combined_image_conditionings` selects for `frame_idx == 0` +# (ltx-pipelines/utils/helpers.py:295-300). +COND_TARGET_FRAMES = 3 +COND_PATCH = 1 +COND_FPS = 8.0 +COND_STRENGTH = 0.7 +COND_LATENT_IDX = 0 + +# A NON-UNIT noise scale, deliberately. `noise_scale == 1` is the ONLY value at +# which ltx_core's composition and diffusers' agree (spec §3.3), so gating there +# would leave the divergence invisible. +NOISE_SCALE = 0.625 + + +# --------------------------------------------------------------------------- +# Sections +# --------------------------------------------------------------------------- + + +def section_resize(out, resize_mod) -> None: + import torch + + out.write( + "// --- section 1: resize_and_center_crop (media_io/resize.py:41-73) ---\n" + "// Aspect FILL, `ceil`, bilinear align_corners=False, then centre crop.\n" + ) + emit_scalar(out, "kLtx2ImgResizeCases", len(RESIZE_CASES)) + out.write("\n") + for index, (src_h, src_w, dst_h, dst_w) in enumerate(RESIZE_CASES): + codes = ltx_bytes(f"ltx2.imgcond.resize{index}", src_h * src_w * 3) + hwc = torch.from_numpy(codes.reshape(src_h, src_w, 3).astype(np.float32)) + y = resize_mod.resize_and_center_crop(hwc, dst_h, dst_w) + # `1 c f h w` with f == 1 for a 3-D input (resize.py:73). + assert tuple(y.shape) == (1, 3, 1, dst_h, dst_w), f"unexpected resize shape {y.shape}" + out.write(f"// case {index}: {src_h}x{src_w} -> {dst_h}x{dst_w}\n") + emit_scalar(out, f"kLtx2ImgResize{index}SrcH", src_h) + emit_scalar(out, f"kLtx2ImgResize{index}SrcW", src_w) + emit_scalar(out, f"kLtx2ImgResize{index}DstH", dst_h) + emit_scalar(out, f"kLtx2ImgResize{index}DstW", dst_w) + out.write("\n") + emit_f32(out, f"kLtx2ImgResize{index}Golden", y.numpy()) + + # The `ceil` is asserted rather than described. At least one case must have a + # dimension where ceil disagrees with BOTH round and floor, or nothing in + # this section can tell those three ports apart. + separating = [] + for index, (src_h, src_w, dst_h, dst_w) in enumerate(RESIZE_CASES): + scale = max(dst_h / src_h, dst_w / src_w) + for axis, src in (("h", src_h), ("w", src_w)): + exact = src * scale + if math.ceil(exact) != round(exact) and math.ceil(exact) != math.floor(exact): + separating.append((index, axis, exact)) + assert separating, ( + "no resize case has a dimension where `ceil` disagrees with both `round` and `floor`, " + "so section 1 cannot tell those three ports apart. Add a shape pair that does rather " + "than deleting this assertion" + ) + + +def section_preprocess(out, resize_mod, range_mod, decode_mod, tol) -> None: + import torch + + codes = ltx_bytes("ltx2.imgcond.image", IMAGE_SRC_H * IMAGE_SRC_W * 3) + image = codes.reshape(IMAGE_SRC_H, IMAGE_SRC_W, 3) + + # UPSTREAM'S OWN `preprocess`, EXECUTED. `result is image` is the assertion + # that matters: the crf==0 branch RETURNS THE ARGUMENT (decode.py:425-426), + # and no codec stub can produce object identity. + passed = decode_mod.preprocess(image=image, crf=0) + assert passed is image, ( + "preprocess(crf=0) did not return its argument — the short-circuit this whole row " + "is built on (decode.py:425-426) is not where the spec says it is" + ) + + # ...and the OTHER branch is live. Executed too, so "a non-zero CRF needs a + # codec" is measured rather than read. + reached_codec = False + try: + decode_mod.preprocess(image=image, crf=18) + except Exception: # noqa: BLE001 - the stubbed codec is what raises + reached_codec = True + assert reached_codec, ( + "preprocess(crf=18) did NOT reach the stubbed codec, so either the round trip moved or " + "the stub is answering for it — either way the CRF refusal this row ships would be " + "guarding nothing" + ) + + # decode.py:76-78, in order: f32 in 0..255 -> resize -> normalize. + as_float = torch.from_numpy(passed.astype(np.float32)) + resized = resize_mod.resize_and_center_crop(as_float, IMAGE_DST_H, IMAGE_DST_W) + normalized = range_mod.normalize_images(resized, device="cpu", dtype=torch.float32) + + # THE ORDER IS A CLAIM, so it is MEASURED — and the measurement is a NEGATIVE + # RESULT that is recorded rather than engineered away. + # + # `resize` is a convex combination and `normalize` is affine, so + # resize(normalize(x)) and normalize(resize(x)) are EQUAL in exact + # arithmetic. Their f32 difference is pure rounding and CANNOT be amplified + # by choosing a different image: measured at 1.94e-07 here, against a golden + # band of 5e-06 and against this port's own distance from torch, which is + # larger still (see kLtx2ImgPixelTol in the suite: torch's bilinear + # contracts to FMA, so no portable f32 port reproduces it bit for bit). + # + # So NO GOLDEN IN THIS FILE CAN SEE THE ORDER SWAP. The order is still + # mirrored, for the only reason left: it is upstream's. That puts it in the + # same class AGENTS.md names for a too-WIDE dtype — correct, invisible to + # every gate we own, and therefore checked deliberately once and written + # down rather than assumed to be covered. + swapped = resize_mod.resize_and_center_crop( + range_mod.normalize_images(as_float, device="cpu", dtype=torch.float32), + IMAGE_DST_H, + IMAGE_DST_W, + ) + order_gap = float((normalized - swapped).abs().max()) + assert order_gap < tol, ( + f"resize-then-normalize and normalize-then-resize now differ by {order_gap:g}, ABOVE the " + f"golden band {tol:g}. That would be new information — the two are algebraically equal, " + f"so a gap this size means one of them stopped being the affine/convex pair this comment " + f"assumes. Investigate before touching this assertion" + ) + + out.write( + "// --- section 2: load_image_and_preprocess at crf=0 (decode.py:46-79) ---\n" + "// decode -> preprocess(crf=0) -> f32 0..255 -> resize+crop -> /127.5 - 1.\n" + "// `preprocess(crf=0) is image` was ASSERTED at generation time, and\n" + "// `preprocess(crf=18)` was asserted to reach the codec and raise.\n" + ) + emit_scalar(out, "kLtx2ImgPreSrcH", IMAGE_SRC_H) + emit_scalar(out, "kLtx2ImgPreSrcW", IMAGE_SRC_W) + emit_scalar(out, "kLtx2ImgPreDstH", IMAGE_DST_H) + emit_scalar(out, "kLtx2ImgPreDstW", IMAGE_DST_W) + out.write( + "// max|resize-then-normalize - normalize-then-resize|, measured upstream. It is\n" + "// BELOW the golden band, which is the recorded NEGATIVE RESULT: the two orders\n" + "// are algebraically identical (a convex combination commutes with an affine\n" + "// map), so their f32 gap is pure rounding and cannot be amplified. No golden\n" + "// here can see the swap; the order is mirrored because it is upstream's, and\n" + "// that is written down rather than assumed to be covered.\n" + ) + emit_double(out, "kLtx2ImgPreOrderGap", order_gap) + out.write("\n") + emit_f32(out, "kLtx2ImgPreGolden", normalized.numpy()) + return normalized + + +def section_encode(out, image_5d) -> None: + from ltx_core.model.video_vae.enums import LogVarianceType, NormLayerType, PaddingModeType + from ltx_core.model.video_vae.video_vae import VideoEncoder + + enc = VideoEncoder( + encoder_blocks=IMG_ENC_BLOCKS, + norm_layer=NormLayerType.PIXEL_NORM, + latent_log_var=LogVarianceType.UNIFORM, + encoder_spatial_padding_mode=PaddingModeType.ZEROS, + **IMG_ENC, + ).eval() + manifest = fill_from_stream(enc, prefix="ltx2.imgenc.") + latent = enc(image_5d) + + out.write( + "// --- section 3: VideoEncoder over the preprocessed image " + "(video_vae.py:264-336) ---\n" + "// This is `video_encoder(image)` at ltx-pipelines/utils/helpers.py:294.\n" + ) + emit_scalar(out, "kLtx2ImgEncOutC", latent.shape[1]) + emit_scalar(out, "kLtx2ImgEncOutT", latent.shape[2]) + emit_scalar(out, "kLtx2ImgEncOutH", latent.shape[3]) + emit_scalar(out, "kLtx2ImgEncOutW", latent.shape[4]) + emit_scalar(out, "kLtx2ImgEncTemporalFactor", enc.video_scale_factors.time) + emit_scalar(out, "kLtx2ImgEncSpatialFactor", enc.video_scale_factors.height) + out.write("\n") + emit_manifest(out, "kLtx2ImgEncParam", manifest) + emit_f32(out, "kLtx2ImgEncGolden", latent.numpy()) + return latent + + +def section_condition_and_noise(out, latent, tol) -> None: + import torch + + from ltx_core.components.noisers import GaussianNoiser + from ltx_core.components.patchifiers import VideoLatentPatchifier + from ltx_core.conditioning.types.latent_cond import VideoConditionByLatentIndex + from ltx_core.tools import VideoLatentTools + from ltx_core.types import SpatioTemporalScaleFactors, VideoLatentShape + + target = VideoLatentShape( + batch=1, + channels=int(latent.shape[1]), + frames=COND_TARGET_FRAMES, + height=int(latent.shape[3]), + width=int(latent.shape[4]), + ) + tools = VideoLatentTools( + patchifier=VideoLatentPatchifier(patch_size=COND_PATCH), + target_shape=target, + fps=COND_FPS, + scale_factors=SpatioTemporalScaleFactors.default(), + causal_fix=True, + ) + base = tools.create_initial_state(device="cpu", dtype=torch.float32) + + item = VideoConditionByLatentIndex( + latent=latent, strength=COND_STRENGTH, latent_idx=COND_LATENT_IDX + ) + conditioned = item.apply_to(base, tools) + + assert not torch.equal(conditioned.clean_latent, base.clean_latent), ( + "the item must CHANGE the clean latent or section 4 gates nothing" + ) + assert not torch.equal(conditioned.denoise_mask, base.denoise_mask), ( + "the item must CHANGE the denoise mask or section 4 gates nothing" + ) + assert torch.equal(conditioned.latent, base.latent), ( + "upstream leaves the NOISY tensor untouched (latent_cond.py:38-39). diffusers does " + "NOT (pipeline_ltx2_condition.py:1002-1004); if this ever flips, spec §3.3's choice " + "has to be revisited rather than the assertion deleted" + ) + + out.write( + "// --- section 4: VideoConditionByLatentIndex (latent_cond.py:22-43) ---\n" + "// clean_latent[start:stop] = tokens; denoise_mask[start:stop] = 1 - strength.\n" + "// The NOISY tensor is deliberately untouched — that was asserted here.\n" + ) + emit_scalar(out, "kLtx2ImgCondTokens", conditioned.latent.shape[1]) + emit_scalar(out, "kLtx2ImgCondWidth", conditioned.latent.shape[2]) + emit_scalar(out, "kLtx2ImgCondTargetFrames", COND_TARGET_FRAMES) + emit_scalar(out, "kLtx2ImgCondPatch", COND_PATCH) + emit_scalar(out, "kLtx2ImgCondLatentIdx", COND_LATENT_IDX) + emit_double(out, "kLtx2ImgCondStrength", COND_STRENGTH) + emit_double(out, "kLtx2ImgCondFps", COND_FPS) + out.write("\n") + emit_f32(out, "kLtx2ImgCondClean", conditioned.clean_latent.numpy()) + emit_f32(out, "kLtx2ImgCondMask", conditioned.denoise_mask.numpy()) + + # --- section 5: the noiser, at a NON-UNIT scale. + noise = torch.from_numpy( + ltx_rand("ltx2.imgcond.noise", int(conditioned.latent.numel())).astype(np.float32) + ).reshape(conditioned.latent.shape) + + class DeterministicNoiser(GaussianNoiser): + def _sample_noise(self, latent_state): # noqa: ARG002 - shape comes from the closure + return noise + + noised = DeterministicNoiser(generator=None)(conditioned, noise_scale=NOISE_SCALE) + + # THE DIFFUSERS FORM, computed here only so the DIVERGENCE is a measured + # number in the record rather than a claim. It is NOT emitted as a golden: + # this port follows ltx_core (spec §3.3). + diffusers_latent = conditioned.latent.clone() + diffusers_latent = torch.where( + conditioned.denoise_mask.unsqueeze(-1) < 1.0, + conditioned.clean_latent, + diffusers_latent, + ) + diffusers_out = torch.lerp(diffusers_latent.float(), noise.float(), NOISE_SCALE) + divergence = float((noised.latent - diffusers_out).abs().max()) + assert divergence > tol, ( + f"the ltx_core and diffusers compositions differ by only {divergence:g} at " + f"noise_scale={NOISE_SCALE}, inside the band {tol:g} — section 5 would then not be " + f"gating the choice spec §3.3 makes. Pick a scale that separates them" + ) + + out.write( + "// --- section 5: GaussianNoiser over the conditioned state " + "(components/noisers.py:30-37) ---\n" + "// latent = lerp(latent, noise, noise_scale); latent = lerp(clean, latent, mask).\n" + "// The DOUBLE lerp, at a NON-UNIT scale — the only regime in which ltx_core and\n" + "// diffusers disagree. `kLtx2ImgNoiseDivergence` is how far apart they are here,\n" + "// measured; the suite asserts it is above the band, which is what makes this\n" + "// section able to catch a silent switch to the diffusers form.\n" + ) + emit_double(out, "kLtx2ImgNoiseScale", NOISE_SCALE) + emit_double(out, "kLtx2ImgNoiseDivergence", divergence) + out.write("\n") + emit_f32(out, "kLtx2ImgNoisedGolden", noised.latent.numpy()) + + +# --------------------------------------------------------------------------- +# Upstream loading +# --------------------------------------------------------------------------- + + +class _StubModule(types.ModuleType): + """A module whose every attribute is another stub. Registered for `av` and + `OpenImageIO`, which `decode.py` imports at module scope and the crf==0 path + never reaches. See the module docstring: the stub is a tripwire, not a + shortcut — the assertions in section 2 are what prove upstream ran.""" + + def __getattr__(self, name): + # DUNDERS ARE NOT FABRICATED. `inspect` walks `sys.modules` looking for + # `__file__` on every module while resolving a frame, and a stub that + # answers it makes `os.path.splitext` raise from inside torch's custom-op + # registration — a failure with nothing to do with this script. Anything + # a real module would not have must stay absent. + if name.startswith("__") and name.endswith("__"): + raise AttributeError(name) + stub = _StubModule(f"{self.__name__}.{name}") + setattr(self, name, stub) + return stub + + def __call__(self, *args, **kwargs): + return _StubModule(f"{self.__name__}()") + + +def load_upstream(root: Path): + core_src = root / "packages" / "ltx-core" / "src" + pipe_src = root / "packages" / "ltx-pipelines" / "src" + if not (core_src / "ltx_core" / "model" / "video_vae" / "video_vae.py").is_file(): + raise SystemExit(f"no ltx_core under {core_src}; point --ltx2 at a Lightricks/LTX-2 tree") + if not (pipe_src / "ltx_pipelines" / "utils" / "media_io" / "resize.py").is_file(): + raise SystemExit(f"no ltx_pipelines under {pipe_src}") + sys.path.insert(0, str(core_src)) + sys.path.insert(0, str(pipe_src)) + + for dep in ("av", "OpenImageIO"): + if dep not in sys.modules: + sys.modules[dep] = _StubModule(dep) + + # Register the package levels WITHOUT executing their __init__.py. + for name, rel in ( + ("ltx_pipelines", "ltx_pipelines"), + ("ltx_pipelines.utils", "ltx_pipelines/utils"), + ("ltx_pipelines.utils.media_io", "ltx_pipelines/utils/media_io"), + ): + if name in sys.modules: + continue + package = types.ModuleType(name) + package.__path__ = [str(pipe_src / rel)] + package.__package__ = name + sys.modules[name] = package + + import ltx_core # noqa: PLC0415 + import ltx_pipelines.utils.media_io.decode as decode_mod # noqa: PLC0415 + import ltx_pipelines.utils.media_io.range_map as range_mod # noqa: PLC0415 + import ltx_pipelines.utils.media_io.resize as resize_mod # noqa: PLC0415 + + # ORACLE IDENTITY, asserted rather than assumed, on BOTH packages. + for module, expected in ((ltx_core, core_src), (resize_mod, pipe_src)): + resolved = Path(module.__file__).resolve() + if not resolved.is_relative_to(expected.resolve()): + raise SystemExit( + f"{module.__name__} resolved to {resolved}, which is NOT under {expected}. " + "Refusing to generate goldens from an oracle this script did not choose." + ) + return resize_mod, range_mod, decode_mod + + +def upstream_revision(root: Path) -> str: + try: + done = subprocess.run( + ["git", "-C", str(root), "rev-parse", "HEAD"], + check=True, + capture_output=True, + text=True, + ) + except Exception: # noqa: BLE001 - a tarball checkout carries no git metadata + return "unknown" + dirty = subprocess.run( + ["git", "-C", str(root), "status", "--porcelain"], + check=True, + capture_output=True, + text=True, + ).stdout.strip() + if dirty: + raise SystemExit( + f"the LTX-2 checkout at {root} is DIRTY:\n{dirty}\n" + "Refusing to generate: `git rev-parse HEAD` would stamp a CLEAN revision anchor " + "onto goldens produced by a tree that commit cannot reproduce." + ) + return done.stdout.strip() + + +def main() -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--ltx2", required=True, type=Path, + help="a checkout of Lightricks/LTX-2 (the repo root)") + parser.add_argument("--out", required=True, type=Path) + args = parser.parse_args() + + tol = _read_golden_tol() + root = args.ltx2.expanduser().resolve() + resize_mod, range_mod, decode_mod = load_upstream(root) + revision = upstream_revision(root) + + import torch + + torch.set_grad_enabled(False) + + args.out.parent.mkdir(parents=True, exist_ok=True) + with args.out.open("w", encoding="utf-8") as out: + out.write( + "// GENERATED by scripts/gen-ltx2-image-cond-goldens.py — DO NOT EDIT BY HAND.\n" + "//\n" + "// LTX-2.5 IMAGE CONDITIONING goldens (row LTX25-IMAGE-COND, issue #644),\n" + "// produced by EXECUTING upstream Lightricks/LTX-2 at reduced dimensions on\n" + "// CPU. Weights and pixels come from the shared deterministic stream, so no\n" + "// weight byte and no image byte is checked in. Regenerate with:\n" + "// python3 scripts/gen-ltx2-image-cond-goldens.py --ltx2 \n" + "// --out tests/vllm/multimodal/ltx2_image_cond_goldens.inc\n" + "//\n" + f"// Upstream revision: {revision}\n" + "//\n" + "// See .agents/specs/ltx25-image-conditioning.md section 5.\n" + "#pragma once\n\n#include \n\nnamespace vllm_test {\n\n" + "// The upstream tree these numbers came from. The suite asserts this equals\n" + "// the SHA it pins, so regenerating against a DIFFERENT checkout fails the\n" + "// gate instead of silently replacing the oracle.\n" + f'inline constexpr const char* kLtx2ImgCondUpstreamRevision = "{revision}";\n\n' + ) + section_resize(out, resize_mod) + image_chw = section_preprocess(out, resize_mod, range_mod, decode_mod, tol) + latent = section_encode(out, image_chw) + section_condition_and_noise(out, latent, tol) + out.write("} // namespace vllm_test\n") + print(f"wrote {args.out}", file=sys.stderr) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/src/vllm/model_executor/models/ltx2_image_preprocess.cpp b/src/vllm/model_executor/models/ltx2_image_preprocess.cpp new file mode 100644 index 000000000..2e10648d3 --- /dev/null +++ b/src/vllm/model_executor/models/ltx2_image_preprocess.cpp @@ -0,0 +1,219 @@ +// LTX-2.5 image conditioning input. Row LTX25-IMAGE-COND, issue #644. +// See ltx2_image_preprocess.h for the upstream anchors and the residuals. +#include "vllm/model_executor/models/ltx2_image_preprocess.h" + +#include +#include +#include +#include +#include +#include +#include +#include + +namespace vllm { +namespace { + +[[noreturn]] void Fail(const std::string& why) { throw std::runtime_error(why); } + +// ── PyTorch's `align_corners=False` index map, in f32 ─────────────────────── +// aten/src/ATen/native/UpSample.h `area_pixel_compute_scale` / +// `area_pixel_compute_source_index`, and +// aten/src/ATen/native/cpu/UpSampleKernel.cpp +// `HelperInterpLinear::compute_indices_weights` + `guard_index_and_lambda`. +// +// The arithmetic is f32 because that is the `opmath_t` those helpers are +// dispatched at for a float tensor. Computing it in double here would be MORE +// accurate and would still be wrong: the goldens come from torch, so this port +// has to reproduce torch's rounding, not improve on it. +struct LinearTap { + int64_t lo = 0, hi = 0; + float w_lo = 1.0F, w_hi = 0.0F; +}; + +std::vector LinearTaps(int64_t src, int64_t dst) { + const float scale = static_cast(src) / static_cast(dst); + std::vector taps(static_cast(dst)); + for (int64_t i = 0; i < dst; ++i) { + float real = scale * (static_cast(i) + 0.5F) - 0.5F; + if (real < 0.0F) real = 0.0F; // the `!cubic && src_idx < 0` clamp + int64_t index = std::min(static_cast(std::floor(real)), src - 1); + float lambda = std::min(std::max(real - static_cast(index), 0.0F), 1.0F); + LinearTap& tap = taps[static_cast(i)]; + tap.lo = index; + tap.hi = std::min(index + 1, src - 1); + tap.w_lo = 1.0F - lambda; + tap.w_hi = lambda; + } + return taps; +} + +int NextPpmInt(std::istringstream& in, const std::string& field) { + // PPM comments (`#` to end of line) may appear between any two header tokens. + while (true) { + in >> std::ws; + if (in.peek() != '#') break; + std::string skip; + std::getline(in, skip); + } + int value = 0; + if (!(in >> value)) Fail(field + ": bad PPM header"); + return value; +} + +} // namespace + +std::vector Ltx2DecodePpmRgb(const std::string& field, const std::string& bytes, + int64_t* out_height, int64_t* out_width) { + std::istringstream in(bytes, std::ios::binary); + std::string magic; + in >> magic; + if (magic != "P6") { + Fail(field + + ": not a binary PPM (P6). No PNG/JPEG/EXR codec is vendored in this tree, so an image " + "conditioning must be supplied as binary PPM — the same residual the MiniMax-H3 video " + "seam carries (minimax_h3_video.cpp:84-86). Upstream's own decoder is PIL's " + "(media_io/decode.py:139-170) and reads every format PIL does."); + } + const int width = NextPpmInt(in, field); + const int height = NextPpmInt(in, field); + const int maxval = NextPpmInt(in, field); + if (width <= 0 || height <= 0) { + Fail(field + ": PPM declares a " + std::to_string(width) + "x" + std::to_string(height) + + " image"); + } + if (maxval != 255) { + // PIL rescales a non-255 maxval on the way to uint8; which rounding it uses + // is a property of PIL's PPM plugin, not of LTX. Refusing rather than + // inventing one — a rescale nobody checked against PIL would shift every + // pixel of a conditioning image by a fraction no gate here can see. + Fail(field + ": PPM maxval is " + std::to_string(maxval) + + "; only 255 is read. `decode_image` (media_io/decode.py:139-170) returns " + "`np.array(image, dtype=np.uint8)` out of PIL, and mirroring PIL's rescale for a " + "narrower or wider maxval is a separate port. Re-save the image at maxval 255."); + } + in.get(); // the single whitespace byte between the header and the payload + std::vector rgb(static_cast(width) * static_cast(height) * 3); + in.read(reinterpret_cast(rgb.data()), static_cast(rgb.size())); + if (!in) Fail(field + ": truncated PPM payload"); + if (out_height != nullptr) *out_height = height; + if (out_width != nullptr) *out_width = width; + return rgb; +} + +void Ltx2PreprocessImageCrf(int64_t crf) { + if (crf == 0) return; // decode.py:425-426 — `if crf == 0: return image` + Fail( + "image conditioning at CRF " + std::to_string(crf) + + " is not ported: the H.264 round trip `preprocess` performs at a non-zero CRF " + "(media_io/decode.py:430-434 -> encode_single_frame:386-400, libx264 preset=veryfast, " + "rgb24 -> yuv420p, dimensions truncated to even, then decode_single_frame:403-410) needs a " + "codec, and none is vendored in this tree. CRF 0 IS supported and is the only supported " + "value: upstream short-circuits it at decode.py:425-426 and documents an explicit 0 as " + "\"skip re-compression entirely\" (utils/args.py:58-59). Say so explicitly — an LTX-2.5 " + "checkpoint RESOLVES 18 when the caller leaves the CRF unset (ImageConditioner.resolve_crf, " + "blocks.py:977-983, over constants.py:37/124/130-133), so CRF 0 conditions on pixels this " + "model generation was not trained against. That is a quality cost, not a correctness one, " + "and it is stated rather than rendered silently."); +} + +std::vector Ltx2ResizeAndCenterCrop(const float* hwc, int64_t src_height, + int64_t src_width, int64_t channels, int64_t height, + int64_t width) { + if (hwc == nullptr) Fail("ltx2 resize: null input"); + if (src_height <= 0 || src_width <= 0 || channels <= 0 || height <= 0 || width <= 0) { + Fail("ltx2 resize: every dimension must be positive"); + } + + // resize.py:60-63. `scale = max(...)` is the aspect-FILL choice — the crop + // happens after, so the SHORT side is what the target must be covered by. + // `ceil` is upstream's own guard against a float rounding that would make + // new_h/new_w land just under the target and give a negative crop offset. + const double scale = std::max(static_cast(height) / static_cast(src_height), + static_cast(width) / static_cast(src_width)); + const int64_t new_h = static_cast(std::ceil(static_cast(src_height) * scale)); + const int64_t new_w = static_cast(std::ceil(static_cast(src_width) * scale)); + if (new_h < height || new_w < width) { + Fail("ltx2 resize: the aspect-fill resize produced " + std::to_string(new_h) + "x" + + std::to_string(new_w) + ", which cannot be cropped to " + std::to_string(height) + "x" + + std::to_string(width)); + } + + const std::vector rows = LinearTaps(src_height, new_h); + const std::vector cols = LinearTaps(src_width, new_w); + + // resize.py:71 — `crop_top = (new_h - height) // 2`, floor division. + const int64_t crop_top = (new_h - height) / 2; + const int64_t crop_left = (new_w - width) / 2; + + std::vector out(static_cast(channels * height * width)); + for (int64_t c = 0; c < channels; ++c) { + for (int64_t y = 0; y < height; ++y) { + const LinearTap& r = rows[static_cast(crop_top + y)]; + for (int64_t x = 0; x < width; ++x) { + const LinearTap& k = cols[static_cast(crop_left + x)]; + // WIDTH is the inner sum and HEIGHT the outer one, and each partial sum + // is rounded to f32 before the next multiply — `Interpolate::eval` + // recurses over the dimensions in the order + // `upsample_generic_Nd_kernel_impl` appends them (height, then width), + // so the row combination happens first. Folding this into one + // four-term dot product gives a different last bit. + auto at = [&](int64_t yy, int64_t xx) { + return hwc[static_cast((yy * src_width + xx) * channels + c)]; + }; + const float lo = at(r.lo, k.lo) * k.w_lo + at(r.lo, k.hi) * k.w_hi; + const float hi = at(r.hi, k.lo) * k.w_lo + at(r.hi, k.hi) * k.w_hi; + out[static_cast((c * height + y) * width + x)] = lo * r.w_lo + hi * r.w_hi; + } + } + } + return out; +} + +std::vector Ltx2LoadImageAndPreprocess(const std::string& field, const std::string& bytes, + int64_t height, int64_t width, int64_t crf) { + // decode.py:74-78, step for step and IN THIS ORDER. + int64_t src_h = 0, src_w = 0; + const std::vector rgb = Ltx2DecodePpmRgb(field, bytes, &src_h, &src_w); // :74 + Ltx2PreprocessImageCrf(crf); // :75 + + // :76 — `torch.tensor(image, dtype=torch.float32)`. Values are still 0..255. + std::vector as_float(rgb.size()); + for (size_t i = 0; i < rgb.size(); ++i) as_float[i] = static_cast(rgb[i]); + + // :77 — the resize runs in 0..255 space. + std::vector chw = Ltx2ResizeAndCenterCrop(as_float.data(), src_h, src_w, 3, height, width); + + // :78 — `normalize_images` (range_map.py:8-9), AFTER the resize. + for (float& v : chw) v = v / 127.5F - 1.0F; + return chw; +} + +int64_t Ltx2ResolveDefaultImageCrf(const std::vector& version_components) { + // `_PARAMS_SINCE_VERSION` (constants.py:130-133), newest first, and + // `detect_params`'s "the newest generation this version is at or above" + // (:166-177). Only `default_image_crf` differs between the rows, so only that + // is resolved here; a row that moved another knob would have to grow this. + struct Row { + std::vector since; + int64_t crf; + }; + static const std::vector kRows = { + {{2, 4}, 18}, // LTX_2_4_PARAMS -> LTX_2_4_IMAGE_CRF (:37, :124) + {{2, 3}, 33}, // LTX_2_3_PARAMS inherits DEFAULT_IMAGE_CRF (:36, :83-88) + }; + for (const Row& row : kRows) { + // Tuple comparison, which is what `parsed >= since` is in Python: element by + // element, and a SHORTER tuple compares below a longer one that agrees on + // the shared prefix. An empty `parsed` therefore falls through every row, + // which is `detect_model_version`'s documented "compares below every real + // version" (:138-139). + if (!std::lexicographical_compare(version_components.begin(), version_components.end(), + row.since.begin(), row.since.end())) { + return row.crf; + } + } + return 33; // DEFAULT_IMAGE_CRF, via LTX_2_PARAMS (:36, :48, :80) +} + +} // namespace vllm diff --git a/src/vllm/model_executor/models/ltx2_video_vae_encoder_load.cpp b/src/vllm/model_executor/models/ltx2_video_vae_encoder_load.cpp new file mode 100644 index 000000000..baf1cf98f --- /dev/null +++ b/src/vllm/model_executor/models/ltx2_video_vae_encoder_load.cpp @@ -0,0 +1,266 @@ +// LTX-2.5 CONV VIDEO VAE ENCODER — the LOAD path, row LTX25-IMAGE-COND (#644). +// +// Spec: .agents/specs/ltx25-image-conditioning.md §3.1. +// +// ─── WHY THIS IS ITS OWN TU ────────────────────────────────────────────────── +// The decoder's twin of this code lives in `ltx2_loader.cpp`, which is a 1400-line +// file two concurrent rows of the #644 campaign both need to touch. Putting the +// encoder's four functions there would have made that file the lock AGENTS.md +// §Records names. It is additive here instead, next to the header that declares +// the encoder it serves, and it duplicates only the three tiny config accessors +// (`ConfigGet` / `ConfigObject` / the two enum parsers) that `ltx2_loader.cpp` +// keeps in its own anonymous namespace and does not export. +// +// ─── WHAT THIS IS A PORT OF (file:line on BOTH sides) ──────────────────────── +// Upstream root: Lightricks/LTX-2 @ fd4ded7f, +// packages/ltx-core/src/ltx_core/model/video_vae/ +// OURS <- UPSTREAM +// Ltx2VideoVaeEncoderKeyRules <- model_configurator.py:267-276 +// Ltx2ParseConvVideoEncoderConfig <- model_configurator.py:37-69, 72-78 +// Ltx2CheckpointHasVideoEncoder <- (the SDOps `with_matching` prefixes of +// the same filter, asked as a question) +#include "vllm/model_executor/models/ltx2_video_vae_encoder.h" + +#include + +#include +#include +#include + +namespace vllm { +namespace { + +[[noreturn]] void Fail(const std::string& why) { + throw std::runtime_error("ltx2 video vae encoder: " + why); +} + +bool StartsWith(const std::string& value, const std::string& prefix) { + return value.rfind(prefix, 0) == 0; +} + +// `config.get(key, fallback)`, refusing a present-but-wrong-typed value rather +// than falling back — the same polarity `ltx2_loader.cpp:845-856` states for the +// decoder, and for the same reason: a checkpoint that says `"patch_size": "4"` +// means something, and treating it as absent builds a different latent grid. +template +T ConfigGet(const nlohmann::json& config, const std::string& key, T fallback, + const std::string& where) { + const auto it = config.find(key); + if (it == config.end() || it->is_null()) return fallback; + try { + return it->get(); + } catch (const std::exception&) { + Fail("'" + where + "." + key + "' is present but not the expected type (" + it->dump() + ")"); + } +} + +const nlohmann::json& ConfigObject(const nlohmann::json& parent, const std::string& key, + const std::string& where) { + static const nlohmann::json kEmpty = nlohmann::json::object(); + const auto it = parent.find(key); + if (it == parent.end() || it->is_null()) return kEmpty; + if (!it->is_object()) Fail("'" + where + "." + key + "' is not a JSON object"); + return *it; +} + +Ltx2NormLayer ParseNormLayer(const std::string& name, const std::string& where) { + if (name == "pixel_norm") return Ltx2NormLayer::kPixelNorm; + if (name == "group_norm") return Ltx2NormLayer::kGroupNorm; + Fail(where + ".norm_layer is '" + name + "'; only 'pixel_norm' and 'group_norm' are ported"); +} + +Ltx2PaddingMode ParsePaddingMode(const std::string& name, const std::string& where) { + if (name == "zeros") return Ltx2PaddingMode::kZeros; + if (name == "reflect") return Ltx2PaddingMode::kReflect; + if (name == "replicate") return Ltx2PaddingMode::kReplicate; + Fail(where + ".spatial_padding_mode is '" + name + + "'; only 'zeros', 'reflect' and 'replicate' are ported"); +} + +// `LogVarianceType` (video_vae/enums.py:9-13). The DEFAULT is `uniform` +// (model_configurator.py:62), and it is not cosmetic: it decides how many +// channels `conv_out` emits and therefore WHICH half of them the mean split +// keeps. See ltx2_video_vae_encoder.h. +Ltx2LogVarianceType ParseLogVariance(const std::string& name, const std::string& where) { + if (name == "per_channel") return Ltx2LogVarianceType::kPerChannel; + if (name == "uniform") return Ltx2LogVarianceType::kUniform; + if (name == "constant") return Ltx2LogVarianceType::kConstant; + if (name == "none") return Ltx2LogVarianceType::kNone; + Fail(where + ".latent_log_var is '" + name + "'; it is not one of the four (enums.py:9-13)"); +} + +// `[["res_x", {"num_layers": 4}], ["compress_all", {"multiplier": 2}], ...]` — +// the same pair form the decoder's list uses, read by `_make_encoder_block` +// (video_vae.py:39-145), which asks for exactly `num_layers` and `multiplier`. +// +// The encoder's list has NO `inject_noise` and NO `residual`: `_make_encoder_block` +// never reads them (contrast `_make_decoder_block`), so they are deliberately +// not parsed here. A checkpoint carrying them on an encoder block would be +// declaring something upstream ignores, and silently honouring it would build a +// module upstream cannot. +std::vector ParseEncoderBlocks(const nlohmann::json& blocks, + const std::string& where) { + if (!blocks.is_array()) Fail("'" + where + ".encoder_blocks' is not an array"); + std::vector out; + for (const nlohmann::json& entry : blocks) { + if (!entry.is_array() || entry.size() != 2 || !entry[0].is_string() || !entry[1].is_object()) { + Fail("an '" + where + ".encoder_blocks' entry is not a [name, {params}] pair: " + + entry.dump()); + } + Ltx2VideoEncoderBlock block; + block.name = entry[0].get(); + const nlohmann::json& params = entry[1]; + // `num_layers` is REQUIRED, and required by exactly one block kind. Upstream + // SUBSCRIPTS it — `num_layers=block_config["num_layers"]` (video_vae.py:55) — + // which raises `KeyError` when it is absent, and no other branch of + // `_make_encoder_block` reads it at all (:61-145). This defaulted to 1, so a + // `res_x` config upstream refuses outright built a silent one-layer + // `UNetMidBlock3D` here — the same class of wrong-shape-without-a-word defect + // that `multiplier`'s sentinel two lines below exists to prevent, resolved the + // opposite way. Made consistent 2026-08-13 (review of #657, row LTX25-IMAGE-COND). + if (block.name == "res_x" && !params.contains("num_layers")) { + Fail("a '" + where + ".encoder_blocks' res_x entry carries no 'num_layers': " + + entry.dump() + + ". Upstream subscripts it (video_vae.py:55) and raises KeyError, so a checkpoint " + "without it is one upstream cannot load either; the layer count is not derivable " + "from anything else in the config"); + } + block.num_layers = ConfigGet(params, "num_layers", 1, "encoder_block"); + // 0 is the sentinel `Ltx2VideoEncoderBlock` documents for "the upstream + // default for this block kind" (2 for every `*_x_y` / `*_res`). An ABSENT + // multiplier must stay 0 rather than become 1, which would quietly halve + // every widening block's output width. + block.multiplier = ConfigGet(params, "multiplier", 0, "encoder_block"); + out.push_back(std::move(block)); + } + if (out.empty()) Fail("'" + where + ".encoder_blocks' is empty; there is no encoder to build"); + return out; +} + +} // namespace + +std::vector Ltx2VideoVaeEncoderKeyRules() { + // `VAE_ENCODER_COMFY_KEYS_FILTER` (model_configurator.py:267-276), TRANSLATED + // into this port's rule shape — not copied rule for rule, and the difference is + // worth stating rather than glossing. + // + // Upstream's `SDOps` is two passes (loader/sd_ops.py:101-122): an `any()` over + // its four `with_matching` PREFIXES decides whether the key is admitted at all, + // and then every `with_replacement` is applied in order as a chained SUBSTRING + // replace. Upstream declares four matchings and only three replacements, + // because the bare `per_channel_statistics.` form needs admitting but not + // rewriting. + // + // This port has ONE pass: a first-match-wins prefix loop where the matched + // rule's replacement is substituted (`Ltx2LoadVaeTensors`). So the fourth rule + // below is an IDENTITY — it exists to carry upstream's fourth matching, since a + // key admitted by no rule here is dropped. ORDER MATTERS for the same reason + // upstream lists its `vae.`-prefixed spellings first: `vae.encoder.conv_in...` + // does not start with `encoder.`, so a bare-first ordering would drop it. + // + // The two shapes agree on every key any shipped checkpoint carries: all four + // upstream matchings are prefixes, no admitted key contains a second + // replacement's substring after the first has fired, and no replacement is a + // substring of another's output. They would diverge on a key needing TWO + // rewrites in one pass, which no LTX-2 VAE file produces — behaviourally + // equivalent, therefore, not verbatim. + return { + {"vae.encoder.", ""}, + {"vae.per_channel_statistics.", "per_channel_statistics."}, + {"encoder.", ""}, + {"per_channel_statistics.", "per_channel_statistics."}, + }; +} + +bool Ltx2CheckpointHasVideoEncoder(const std::vector& tensor_names) { + // Only the two ENCODER prefixes count. `per_channel_statistics.` is in the + // filter but is carried by decoder-only files too, so treating it as evidence + // of an encoder would report every Comfy-split decoder as encodable and then + // fail deep inside `Ltx2ConvVideoEncode` on a missing `conv_in.conv.weight`. + for (const std::string& name : tensor_names) { + if (StartsWith(name, "vae.encoder.") || StartsWith(name, "encoder.")) return true; + } + return false; +} + +Ltx2ConvVideoEncoderConfig Ltx2ParseConvVideoEncoderConfig(const nlohmann::json& config) { + const nlohmann::json& vae = ConfigObject(config, "vae", "config"); + if (vae.empty()) Fail("the video VAE config carries no 'vae' object"); + + // `_prepare_video_encoder_kwargs`'s two layouts (model_configurator.py:46-53). + const bool nested = vae.contains("encoder") && !vae.at("encoder").is_null(); + const nlohmann::json& enc = nested ? ConfigObject(vae, "encoder", "vae") : vae; + const std::string where = nested ? "vae.encoder" : "vae"; + + Ltx2ConvVideoEncoderConfig out; + + // `convolution_dimensions` (:56). Asserted rather than stored, exactly as the + // decoder's parser does: this port is 3-D only. + const int64_t dims = ConfigGet(enc, "dims", ConfigGet(vae, "dims", 3, "vae"), + where); + if (dims != 3) Fail("vae.dims is " + std::to_string(dims) + "; only the 3-D encoder is ported"); + + out.in_channels = ConfigGet(enc, "in_channels", 3, where); + + // THE LATENT WIDTH (:48 nested / :52 flat). Never the top-level `out_channels`. + out.out_channels = nested ? ConfigGet(enc, "out_channels", + ConfigGet(vae, "latent_channels", 128, + "vae"), + where) + : ConfigGet(vae, "latent_channels", 128, "vae"); + + // `encoder_blocks`: NESTED reads `encoder.blocks` then `encoder.encoder_blocks` + // (:49); FLAT reads `vae.encoder_blocks` (:53). Neither layout falls back to + // the other's object, so neither does this. + const nlohmann::json* blocks = nullptr; + if (nested) { + if (enc.contains("blocks") && !enc.at("blocks").is_null()) { + blocks = &enc.at("blocks"); + } else if (enc.contains("encoder_blocks") && !enc.at("encoder_blocks").is_null()) { + blocks = &enc.at("encoder_blocks"); + } + } else if (vae.contains("encoder_blocks") && !vae.at("encoder_blocks").is_null()) { + blocks = &vae.at("encoder_blocks"); + } + if (blocks == nullptr) { + // Upstream's default is `[]`, and `VideoEncoder([])` is an encoder with no + // down-blocks at all: it would run, produce a latent at the WRONG scale + // factors, and every shape downstream would still check out because the + // pipeline derives its shapes from VIDEO_SCALE_FACTORS rather than from this + // list. Refused instead. + Fail( + "the video VAE config declares no 'encoder_blocks', and upstream's default for it is an " + "EMPTY list (model_configurator.py:49, 53). An encoder with no down-blocks still runs and " + "still returns a latent — at scale factors of (1, patch_size, patch_size) rather than the " + "checkpoint's — so refusing is the only way this is visible. A decoder-only Comfy-split " + "file is the expected case here and is reported separately by " + "Ltx2CheckpointHasVideoEncoder"); + } + out.encoder_blocks = ParseEncoderBlocks(*blocks, where); + + out.patch_size = ConfigGet(enc, "patch_size", 4, where); + out.norm_layer = + ParseNormLayer(ConfigGet(enc, "norm_layer", "pixel_norm", where), where); + out.latent_log_var = + ParseLogVariance(ConfigGet(enc, "latent_log_var", "uniform", where), where); + + // `encoder_spatial_padding_mode` (:63-68): the ENCODER key first, then the + // top-level `encoder_spatial_padding_mode`, then `zeros` — NOT the decoder's + // `reflect`. All three levels are mirrored because upstream's chained + // `.get(a, config.get(b, "zeros"))` is what decides which checkpoint spelling + // wins, and collapsing it to one lookup changes that. + out.spatial_padding_mode = ParsePaddingMode( + ConfigGet( + enc, "spatial_padding_mode", + ConfigGet(vae, "encoder_spatial_padding_mode", "zeros", "vae"), where), + where); + + // norm_num_groups, norm_eps and pixel_norm_eps are NOT checkpoint keys: they + // are `VideoEncoder._DEFAULT_NORM_NUM_GROUPS` and the literals + // `_make_encoder_block` passes (video_vae.py:56, 66, 240). They keep the + // values ltx2_video_vae_encoder.h pins to those lines. Reading them from + // config would let a file move a constant no golden can see. + return out; +} + +} // namespace vllm diff --git a/src/vllm/multimodal/ltx2_video.cpp b/src/vllm/multimodal/ltx2_video.cpp index c0eaf389f..7a10e6ce2 100644 --- a/src/vllm/multimodal/ltx2_video.cpp +++ b/src/vllm/multimodal/ltx2_video.cpp @@ -26,13 +26,16 @@ #include "vllm/model_executor/models/device_pool.h" // ActivePool(b)/DevicePool::Drain #include "vllm/model_executor/models/ltx2.h" #include "vllm/model_executor/models/ltx2_audio_vae.h" +#include "vllm/model_executor/models/ltx2_conditioning.h" #include "vllm/model_executor/models/ltx2_connector.h" #include "vllm/model_executor/models/ltx2_device.h" +#include "vllm/model_executor/models/ltx2_image_preprocess.h" #include "vllm/model_executor/models/ltx2_loader.h" #include "vllm/model_executor/models/ltx2_pipeline.h" #include "vllm/model_executor/models/ltx2_text_encoder.h" #include "vllm/model_executor/models/ltx2_upsampler.h" #include "vllm/model_executor/models/ltx2_video_vae.h" +#include "vllm/model_executor/models/ltx2_video_vae_encoder.h" #include "vllm/model_executor/models/minimax_h3.h" #include "vllm/platforms/interface.h" // CurrentPlatform() — which accelerator, if any #include "vllm/tokenizer/tokenizer.h" @@ -243,7 +246,7 @@ int64_t ExtraInt(const std::map& extras, const std::st if (consumed != raw.size()) throw std::invalid_argument("trailing"); return static_cast(value); } catch (const std::exception&) { - Fail("the load extra '" + key + "' is '" + raw + "', which is not an integer"); + Fail("the extra '" + key + "' is '" + raw + "', which is not an integer"); } } @@ -441,6 +444,18 @@ struct Ltx2VideoEngine::Impl { Ltx2ConvVideoDecoderConfig video_cfg; Ltx2VaeWeights video_weights; + // The ENCODER half of the same file (row LTX25-IMAGE-COND, issue #644). Its + // absence is what every conditioning arm was refused for: before this row the + // load below materialized `Ltx2VideoVaeDecoderKeyRules()` alone and no encoder + // key filter existed anywhere in the tree, so `Ltx2ConvVideoEncode` — ported + // and gated since phase L11 — had no weights to run on. + // + // A Comfy-split `vae/` file may carry the decoder alone, so this is OPTIONAL + // and its absence is reported by name at the request rather than guessed at. + bool has_video_encoder = false; + Ltx2ConvVideoEncoderConfig video_encoder_cfg; + Ltx2VaeWeights video_encoder_weights; + Ltx2AudioDecoderConfig audio_cfg; Ltx2VaeWeights audio_weights; Ltx2VocoderBweConfig vocoder_cfg; @@ -773,8 +788,46 @@ std::unique_ptr Ltx2VideoEngine::Load(const VideoModelParams& p if (params.video_vae_path.empty()) Fail("video_vae_path is required"); { const SafetensorsFile f = SafetensorsFile::Open(params.video_vae_path); - im.video_cfg = Ltx2ParseConvVideoDecoderConfig(Ltx2ReadCheckpointConfig(f), &im.video_kind); + const nlohmann::json vae_config = Ltx2ReadCheckpointConfig(f); + im.video_cfg = Ltx2ParseConvVideoDecoderConfig(vae_config, &im.video_kind); im.video_weights = Ltx2LoadVaeWeights(f, Ltx2VideoVaeDecoderKeyRules()); + + // `ImageConditioner` builds its VideoEncoder from the SAME checkpoint with + // `VAE_ENCODER_COMFY_KEYS_FILTER` (blocks.py:956-961). It builds it lazily + // and frees it after the callable returns (:988-993); this engine keeps it + // resident instead, because the encoder is small next to the DiT and a + // conditioning image arrives per request. That is a deliberate divergence + // from upstream's lifecycle and nothing else: same filter, same + // configurator, same weights. + if (Ltx2CheckpointHasVideoEncoder(f.Names())) { + im.video_encoder_cfg = Ltx2ParseConvVideoEncoderConfig(vae_config); + im.video_encoder_weights = Ltx2LoadVaeWeights(f, Ltx2VideoVaeEncoderKeyRules()); + im.has_video_encoder = true; + + // The encoder's LATENT WIDTH against the DiT's input, asserted rather + // than assumed. `_prepare_video_encoder_kwargs` reads it from + // `latent_channels` and NOT from the top-level `out_channels` + // (model_configurator.py:41-43); a config that got that wrong builds a + // 3-channel-latent encoder that still runs, still returns a latent, and + // conditions the DiT on a tensor of the wrong width. + if (im.video_encoder_cfg.out_channels != im.dit.params.in_channels) { + Fail("the video VAE encoder emits " + + std::to_string(im.video_encoder_cfg.out_channels) + + " latent channels but the DiT takes " + + std::to_string(im.dit.params.in_channels) + + ". `_prepare_video_encoder_kwargs` reads this from `vae.latent_channels`, never " + "from the top-level `vae.out_channels`, which is the DECODER's RGB count " + "(video_vae/model_configurator.py:41-42, and the flat-layout read at :52)."); + } + // And its INPUT width, for the same reason in the other direction: the + // encoder takes RGB, and a config declaring otherwise would silently + // reinterpret the image planes. + if (im.video_encoder_cfg.in_channels != 3) { + Fail("the video VAE encoder declares " + + std::to_string(im.video_encoder_cfg.in_channels) + + " input channels; this seam supplies RGB"); + } + } } if (im.video_cfg.in_channels != im.dit.params.out_channels) { Fail("the video VAE takes " + std::to_string(im.video_cfg.in_channels) + @@ -1005,9 +1058,17 @@ VideoResult Ltx2VideoEngine::Generate(const VideoGenParams& gen) { std::lock_guard guard(im.mutex); if (gen.output_dir.empty()) Fail("output_dir is required"); - if (!gen.extras.empty()) { - Fail("unknown per-generation extra '" + gen.extras.begin()->first + - "' (this family defines none)"); + for (const auto& kv : gen.extras) { + // `image_crf` is the only per-generation extra this family defines (row + // LTX25-IMAGE-COND). Everything else is refused rather than ignored, for the + // reason `CheckKnownExtras` gives for the load side: a mistyped knob that is + // silently dropped renders the DEFAULT and looks like the feature not + // working — and for THIS knob the default is a refusal, so a typo would turn + // a served request into an unexplained one. + if (kv.first != kLtx2ImageCrfExtra) { + Fail("unknown per-generation extra '" + kv.first + "'. This family defines: " + + std::string(kLtx2ImageCrfExtra)); + } } if (!gen.prompt.empty() && !im.has_encoder) { Fail( @@ -1135,27 +1196,125 @@ VideoResult Ltx2VideoEngine::Generate(const VideoGenParams& gen) { im.trace.video_absmax = AbsMax(v); im.trace.audio_absmax = AbsMax(a); } - // Image / reference conditioning is `ImageConditioner` upstream - // (ltx-pipelines/utils/blocks.py:936-993, called at distilled.py:212). The - // ENCODER it needs is no longer what is missing — phase L11 ported it as - // `Ltx2ConvVideoEncode` — so the refusal names what actually is: this engine - // holds no encoder to call. Refused by name rather than dropped: a keyframe - // that is silently ignored renders an unconditioned clip that looks like the - // feature not working. - if (!gen.first_frame_path.empty() || !gen.first_frame_ppm.empty() || - !gen.last_frame_path.empty() || !gen.ref_image_paths.empty() || - !gen.ref_video_dir.empty() || !gen.ref_audio_path.empty() || !gen.ref_audio_wav.empty()) { + // ── conditioning on pixels (row LTX25-IMAGE-COND, issue #644) ───────────── + // + // Upstream this is `ImageConditioner` (ltx-pipelines/utils/blocks.py:936-993, + // called at distilled.py:212) feeding `combined_image_conditionings` + // (utils/helpers.py:272-308). ONE of its four arms is served here, and the + // other three are refused BY NAME rather than dropped — a keyframe that is + // silently ignored renders an unconditioned clip that looks like the feature + // not working. + // + // THESE MESSAGES ARE WRITTEN TO BE RE-CHECKABLE, and the count is now SIX + // refusals in this campaign whose stated reason turned out to be false or + // stale. Two of the six stood right here. The first said no encoder weights + // could be materialized — true when written, and what this row fixed. The + // second replaced it and blamed `keyframes_abs_pos_embedding`, which was + // verifiably NOT the blocker at the pin (see the last-frame message below for + // the three anchors that refute it), and a test had been written to assert + // that wrong reason by name. + // + // So: name the exact symbol or upstream `file:line` that would have to change + // for the refusal to become false, never a category — and where a plausible + // reason has already been ruled OUT, say so and cite what ruled it out, so the + // next reader re-checks the claim instead of re-deriving the refutation. Local + // anchors are SYMBOLS, not line numbers in this file: same-file line numbers + // drift on every edit, which is how the previous message's citation went stale. + const bool wants_image = !gen.first_frame_path.empty() || !gen.first_frame_ppm.empty(); + if (!gen.last_frame_path.empty()) { + Fail( + "a LAST-frame keyframe is not served. What is missing is the TOKEN-APPEND machinery. " + "`Ltx2ConvVideoEncode` and `Ltx2ConditionVideoByKeyframe` are both ported and gated, " + "and this engine materializes encoder weights through Ltx2VideoVaeEncoderKeyRules, so " + "none of those is the gap. The gap is that `VideoConditionByKeyframeIndex.apply_to` " + "(conditioning/types/keyframe_cond.py:36-90) APPENDS tokens to the sequence: it " + "concatenates onto `latent`, `denoise_mask`, `positions` and `clean_latent` (:79-82), " + "gives the appended tokens their own pixel coordinates offset to `frame_idx` (:46-59), " + "and rebuilds the attention mask through `update_attention_mask` (:68-76) — and then " + "`clear_conditioning` (ltx_core/tools.py:88-105) trims those extra tokens back off " + "before unpatchify. This engine cannot do any of that yet: `Ltx2LatentState` has no " + "attention-mask field at all (see the note on its declaration in ltx2_conditioning.h), " + "and the phase loop is fixed at the target grid's token count — one " + "`Ltx2VideoTokenCount(vshape, 1)` feeds the sigma schedule, the `Ltx2ModalityInput` " + "handed to the DiT, and `Ltx2VideoUnpatchify`, with the clear step an explicit identity " + "because nothing was ever appended. Serving this arm means growing that sequence " + "through the DiT and trimming it back. Conditioning on the FIRST frame needs none of " + "it, which is why that arm IS served: `VideoConditionByLatentIndex` REPLACES tokens " + "that already exist (conditioning/types/latent_cond.py:38-39) and the token count never " + "changes. WHAT IS *NOT* THE REASON, because this refusal used to say it was: " + "`keyframes_abs_pos_embedding`. A SUPPLIED keyframe is appended with `marked=False` " + "(keyframe_cond.py:84-86, whose comment says given keyframe content carries no keyframe " + "marker), and its sole consumer adds `mask * embedding` with `mask = keyframes_mask > 0` " + "(model/transformer/transformer_args.py:42-43, called once at :269) — so on exactly " + "these tokens the embedding contributes nothing, and porting it would not serve this " + "arm. The tokens that DO reach it are the target's own first latent frame, marked " + "unconditionally by `_first_frame_keyframes_mask` (ltx_core/tools.py:184-196) — which " + "is the frame the SERVED first-frame arm writes into. That omission is real and is " + "tracked as issue #658; it is not what blocks a last-frame keyframe."); + } + if (!gen.ref_image_paths.empty() || !gen.ref_video_dir.empty()) { + Fail( + "reference-image / reference-video conditioning is not served. The encoder and the " + "placement are both here — `Ltx2ConditionVideoByReference` is ported and gated — but " + "it takes a `downscale_factor` and a `temporal_scale_factor` that must match what the " + "IC-LoRA was TRAINED with (conditioning/types/reference_video_cond.py:36-37, applied at " + ":65-77), and " + "upstream carries those in the LoRA's own metadata, which this project does not read. " + "A guessed pair places the reference plausibly and wrongly, which no output check can " + "see, so it is refused instead. Use first_frame_ppm / first_frame_path for " + "image-to-video."); + } + if (!gen.ref_audio_path.empty() || !gen.ref_audio_wav.empty()) { Fail( - "keyframe / reference conditioning is not ported for this family. The video VAE " - "ENCODER itself landed in phase L11 (Ltx2ConvVideoEncode), but nothing can reach it " - "from here: this engine materializes the DECODER key filter only, so no " - "VAE_ENCODER_COMFY_KEYS_FILTER / VideoEncoderConfigurator path " - "(video_vae/model_configurator.py:72, 267) puts encoder weights in memory, and " - "upstream resolves each image conditioning's CRF against the checkpoint's " - "default_image_crf when the caller left it unset (ImageConditioner.resolve_crf, " - "ltx-pipelines/utils/blocks.py:977-983) and then re-compresses through an H.264 " - "round trip unless that CRF is 0 (media_io/decode.py:413-435, from " - "load_image_and_preprocess :75), which this build does not do. Recorded as owed."); + "reference-AUDIO conditioning is not served. `Ltx2ConditionAudioByReference` is ported " + "and gated (conditioning/types/reference_audio_cond.py:34-65), and what it needs is an " + "encoded waveform: `encode_audio` through the audio VAE's ENCODER " + "(ltx-pipelines/utils/helpers.py:264-269). This row built the VIDEO encoder's load " + "path only — there is no AUDIO_VAE_ENCODER key filter — so nothing can turn a WAV into " + "audio latents here. Recorded as owed."); + } + + // The CRF, resolved the way `ImageConditioner.resolve_crf` resolves it + // (blocks.py:966-983) over `detect_params` (utils/constants.py:166-179): from + // the CHECKPOINT's own generation when the caller left it unset. For LTX-2.5 + // that is 18, and 18 is not ported — so the DEFAULT REFUSES and a caller has + // to ask for 0 knowingly. Resolved and checked BEFORE any pixel is read, so an + // unsupported request costs nothing and reports the same thing every time. + int64_t image_crf = 0; + double image_strength = 0.0; + std::string image_bytes; + if (wants_image) { + if (!im.has_video_encoder) { + Fail( + "an image conditioning was supplied but the video VAE checkpoint at '" + + im.params.video_vae_path + + "' carries no ENCODER half: no tensor in it is named `vae.encoder.*` or `encoder.*`, " + "which is what `VAE_ENCODER_COMFY_KEYS_FILTER` matches " + "(video_vae/model_configurator.py:267-276). A Comfy-split `vae/` file holding the " + "decoder alone reads exactly like this. Supply the monolithic VAE checkpoint, or the " + "encoder file, rather than rendering unconditioned."); + } + image_crf = ExtraInt(gen.extras, kLtx2ImageCrfExtra, + Ltx2ResolveDefaultImageCrf(Ltx2ParseModelVersion(im.model_version))); + // Throws by name at any non-zero value, naming the unported codec round + // trip and saying that 0 is the supported — and out-of-distribution — one. + Ltx2PreprocessImageCrf(image_crf); + + // `ImageConditioningInput.strength` (utils/args.py:64). The seam's + // `noise_aug` is documented as "keyframe pinning strength; <= 0 => 1.0" + // (include/vllm.h), which is the same polarity: 1 pins, and the item turns + // it into `denoise_mask = 1 - strength` (latent_cond.py:41). + image_strength = gen.noise_aug > 0.0 ? gen.noise_aug : 1.0; + if (image_strength > 1.0) { + Fail("the image conditioning strength is " + std::to_string(image_strength) + + "; upstream's denoise mask is `1 - strength` (latent_cond.py:41) and a strength " + "above 1 makes it negative, which the noiser extrapolates PAST the clean latent " + "rather than toward it (components/noisers.py:33)"); + } + image_bytes = gen.first_frame_ppm.empty() ? ReadFileBytes("first_frame", gen.first_frame_path) + : gen.first_frame_ppm; + im.trace.image_crf = image_crf; + im.trace.image_strength = image_strength; } // ── geometry ────────────────────────────────────────────────────────────── @@ -1286,7 +1445,7 @@ VideoResult Ltx2VideoEngine::Generate(const VideoGenParams& gen) { video_initial = up.data; } - // ── build the two states (create_noised_state, helpers.py:428-447) ─────── + // ── build the two states (create_noised_state, helpers.py:428-445) ─────── StreamState video; video.width = vshape.channels; // patch_size 1 (VideoLatentPatchifier(1)) video.tokens = Ltx2VideoTokenCount(vshape, 1); @@ -1340,8 +1499,93 @@ VideoResult Ltx2VideoEngine::Generate(const VideoGenParams& gen) { audio.positions.assign(timings.begin(), timings.end()); } + // ── the image conditioning (issue #644) ───────────────────────────────── + // + // BEFORE THE NOISER AND AFTER THE STATE, which is upstream's order + // (`create_noised_state`, helpers.py:428-445: initial state, THEN the + // conditioning items, THEN the noiser) and is not interchangeable: the + // item writes ONLY `clean_latent` and `denoise_mask` (latent_cond.py:38-39) + // and the noiser is what composes them into the noisy tensor + // (components/noisers.py:31-34). Applying it afterwards leaves the + // conditioned tokens pinned to NOISE, with an identical clean tensor and an + // identical mask — so nothing but the noised latent itself can see it. + // + // PER PHASE, and encoded per phase, because the two-stage recipe renders its + // stages at DIFFERENT resolutions (`phase.spatial_downscale`) and upstream + // passes each stage's own height/width to `combined_image_conditionings`, + // whose `height` / `width` are per-call parameters (helpers.py:274-275) that + // distilled.py fills differently per stage: `stage_1_w, stage_1_h = width // + // 2, height // 2` at :251 passed at :255-256, against the full-resolution + // `height` / `width` at :285-286. Conditioning stage 1 only would let stage 2 re-noise + // the pinned frame away; conditioning stage 2 with stage 1's latent would + // place a half-resolution image into a full-resolution grid. + if (wants_image) { + const std::vector pixels = Ltx2LoadImageAndPreprocess( + "first_frame", image_bytes, phase_h, phase_w, image_crf); + int64_t cropped = 0; + const Ltx2LatentVolume encoded = Ltx2ConvVideoEncode( + im.video_encoder_cfg, im.video_encoder_weights, pixels, + im.video_encoder_cfg.in_channels, /*frame_count=*/1, phase_h, phase_w, &cropped); + if (encoded.frames != 1) { + Fail("the video VAE encoder returned " + std::to_string(encoded.frames) + + " latent frames for a single image; `VideoConditionByLatentIndex` places one " + "(ltx-pipelines/utils/helpers.py:294-300)"); + } + if (encoded.channels != vshape.channels || encoded.height != vshape.height || + encoded.width != vshape.width) { + Fail("the encoded image is " + std::to_string(encoded.channels) + "x" + + std::to_string(encoded.height) + "x" + std::to_string(encoded.width) + + " but phase '" + phase.name + "' needs " + std::to_string(vshape.channels) + "x" + + std::to_string(vshape.height) + "x" + std::to_string(vshape.width) + + ". Upstream raises ConditioningError on exactly this " + "(conditioning/types/latent_cond.py:25-30): the encoder's spatial factor and the " + "pipeline's VIDEO_SCALE_FACTORS must agree, and they do not."); + } + + // `Ltx2ConditionVideoByLatentIndex` writes `clean` and `mask` and reads + // `tokens` / `width`; `latent` and `positions` are carried so the struct + // is coherent rather than half-filled, not because the item consults them. + Ltx2LatentState state; + state.tokens = video.tokens; + state.width = video.width; + state.pos_dims = 3; + state.latent = video.latent; + state.clean = video.clean; + state.mask = video.mask; + Ltx2ConditionVideoByLatentIndex(&state, vshape, /*patch_size=*/1, encoded, image_strength, + /*latent_idx=*/0); + video.clean = state.clean; + video.mask = state.mask; + + // The witness, taken from the TOKENS THAT WERE WRITTEN rather than from + // `encoded` — and the difference is not cosmetic. Digesting the encoder's + // output would answer "was an image encoded", which stays true of a build + // that encodes an image and then never places it: the render would be + // unconditioned and every field here would look healthy. Digesting the + // conditioned slice of the clean latent answers "did those tokens reach + // the state", which is the question. Filled on the LAST phase, so it + // describes the conditioning the finished latent carries. + // + // IT IS STILL A CHANGE DETECTOR, not a value gate — the same limit the two + // prompt digests carry. What the placed tokens should NUMERICALLY be is + // gated against executed upstream in `test_ltx2_image_cond`, which drives + // these very functions; MEASURED, because a mutation that moved the + // composition inside this loop and left `test_ltx2_video` green is how + // this comment came to be here. + const int64_t placed = Ltx2VideoTokenCount({1, vshape.channels, 1, vshape.height, + vshape.width}, + 1); + const std::vector written( + video.clean.begin(), + video.clean.begin() + static_cast(placed * video.width)); + im.trace.image_tokens = placed; + im.trace.image_digest = DigestF32(written); + im.trace.image_absmax = AbsMax(written); + } + // The noiser draws VIDEO first, AUDIO second, from one generator - // (blocks.py:576-580 builds the video state before the audio one). + // (blocks.py:554-563 builds the video state before the audio one; :576-580, + // which this used to cite, is the TEARDOWN and proves nothing about order). const float noise_scale = static_cast(phase.noise_scale); ApplyGaussianNoise(video, state_noise.Draw(static_cast(video.latent.size())), noise_scale); diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt index 0e3d9ecb0..2e62e3580 100644 --- a/tests/CMakeLists.txt +++ b/tests/CMakeLists.txt @@ -212,6 +212,13 @@ target_compile_definitions(test_video_engine PRIVATE # LTX-2.5 L7: the family behind the VideoEngine seam and the driving loop, run # over a reduced-dimension checkpoint set written in the SHIPPED file format. vllm_cpp_add_test(test_ltx2_video vllm/multimodal/test_ltx2_video.cpp) +# LTX-2.5 image conditioning (row LTX25-IMAGE-COND, issue #644): the CHAIN from +# PPM bytes to conditioned, noised tokens, gated against upstream ltx_core AND +# ltx_pipelines executed at reduced dimensions. ltx2_image_cond_goldens.inc +# lives next to the test source. +vllm_cpp_add_test(test_ltx2_image_cond vllm/multimodal/test_ltx2_image_cond.cpp) +target_include_directories(test_ltx2_image_cond + PRIVATE ${CMAKE_CURRENT_SOURCE_DIR}/vllm/multimodal) # LTX-2.5 DiT parity gate (row MODEL-DIFFUSION-LTX25, spec phase L2). The goldens # live next to the test source, like every other generated .inc. vllm_cpp_add_test(test_ltx2 vllm/models/test_ltx2.cpp) diff --git a/tests/vllm/multimodal/ltx2_image_cond_goldens.inc b/tests/vllm/multimodal/ltx2_image_cond_goldens.inc new file mode 100644 index 000000000..19f201664 --- /dev/null +++ b/tests/vllm/multimodal/ltx2_image_cond_goldens.inc @@ -0,0 +1,1007 @@ +// GENERATED by scripts/gen-ltx2-image-cond-goldens.py — DO NOT EDIT BY HAND. +// +// LTX-2.5 IMAGE CONDITIONING goldens (row LTX25-IMAGE-COND, issue #644), +// produced by EXECUTING upstream Lightricks/LTX-2 at reduced dimensions on +// CPU. Weights and pixels come from the shared deterministic stream, so no +// weight byte and no image byte is checked in. Regenerate with: +// python3 scripts/gen-ltx2-image-cond-goldens.py --ltx2 +// --out tests/vllm/multimodal/ltx2_image_cond_goldens.inc +// +// Upstream revision: fd4ded7f2d88d3da713abcdd4ad41ecc4a9314ca +// +// See .agents/specs/ltx25-image-conditioning.md section 5. +#pragma once + +#include + +namespace vllm_test { + +// The upstream tree these numbers came from. The suite asserts this equals +// the SHA it pins, so regenerating against a DIFFERENT checkout fails the +// gate instead of silently replacing the oracle. +inline constexpr const char* kLtx2ImgCondUpstreamRevision = "fd4ded7f2d88d3da713abcdd4ad41ecc4a9314ca"; + +// --- section 1: resize_and_center_crop (media_io/resize.py:41-73) --- +// Aspect FILL, `ceil`, bilinear align_corners=False, then centre crop. +inline constexpr int64_t kLtx2ImgResizeCases = 5; + +// case 0: 12x20 -> 16x16 +inline constexpr int64_t kLtx2ImgResize0SrcH = 12; +inline constexpr int64_t kLtx2ImgResize0SrcW = 20; +inline constexpr int64_t kLtx2ImgResize0DstH = 16; +inline constexpr int64_t kLtx2ImgResize0DstW = 16; + +inline constexpr float kLtx2ImgResize0Golden[] = { + 133.555557f, 109.907425f, 68.1666489f, 123.722206f, 158.870361f, 160.444458f, + 118.388893f, 53.9444466f, 53.0f, 92.7407455f, 153.481491f, 193.27774f, + 144.574188f, 77.7221985f, 102.166641f, 177.999924f, 125.361115f, 85.0231705f, + 34.3124733f, 130.145813f, 185.097214f, 198.708344f, 167.150467f, 53.1689873f, + 81.4375f, 141.212952f, 124.175926f, 181.437439f, 141.356598f, 67.9537048f, + 63.6944504f, 121.958275f, 156.111115f, 124.627327f, 87.6041565f, 108.99305f, + 132.798615f, 163.875f, 202.393524f, 118.134262f, 106.375f, 111.046295f, + 77.342598f, 160.999908f, 139.044083f, 71.1134415f, 53.7986298f, 78.9652481f, + 200.398148f, 204.555557f, 202.923645f, 85.2384567f, 46.8333359f, 87.9097061f, + 208.615738f, 223.245361f, 124.0f, 25.9282417f, 44.1689835f, 144.833221f, + 145.527832f, 96.2268753f, 76.5972443f, 60.307888f, 109.453705f, 138.777771f, + 158.798599f, 192.78009f, 211.388885f, 210.034729f, 186.087952f, 199.606476f, + 151.0f, 107.71991f, 137.627319f, 181.166626f, 188.250015f, 156.699158f, + 81.5139618f, 42.0440178f, 77.432869f, 51.969902f, 70.2638626f, 155.449051f, + 215.192123f, 212.090286f, 122.564812f, 120.15741f, 126.125f, 142.925934f, + 173.851852f, 209.201355f, 168.101944f, 90.8171844f, 39.2431068f, 93.0161438f, + 103.613426f, 72.5185089f, 97.1805496f, 106.254631f, 174.872665f, 192.152802f, + 88.3564835f, 67.6157379f, 103.625f, 146.314819f, 146.129623f, 190.993011f, + 142.046417f, 65.4282379f, 66.0763855f, 128.134201f, 161.115753f, 173.277771f, + 200.666687f, 86.2222443f, 137.520813f, 177.381973f, 94.9189835f, 71.6782379f, + 103.25f, 128.261566f, 84.8356476f, 143.062424f, 132.946823f, 103.687462f, + 136.187469f, 128.775482f, 143.143524f, 117.277794f, 65.333313f, 126.444427f, + 174.868042f, 178.340286f, 117.766197f, 168.414352f, 149.75f, 115.497688f, + 118.182869f, 169.437439f, 175.294006f, 131.368149f, 29.979269f, 105.539268f, + 180.807861f, 188.442139f, 156.250015f, 109.583351f, 123.870361f, 151.833328f, + 164.409714f, 190.798615f, 151.5625f, 128.034729f, 202.756958f, 208.416656f, + 149.458435f, 73.2384491f, 50.9236336f, 63.9814682f, 191.918976f, 212.150452f, + 213.187546f, 95.6875305f, 68.1411972f, 103.083321f, 182.194443f, 139.972229f, + 96.625f, 109.430557f, 239.98613f, 175.27092f, 120.131989f, 84.4120407f, + 76.819458f, 97.9189606f, 160.402771f, 164.393509f, 182.958359f, 110.73613f, + 59.5254707f, 72.083313f, 172.215286f, 60.8263931f, 24.0f, 68.9953766f, + 209.365753f, 79.7154388f, 100.548485f, 162.252396f, 82.7153549f, 189.80545f, + 104.152779f, 123.754631f, 113.041649f, 198.597198f, 221.789352f, 213.583328f, + 191.506943f, 131.784729f, 102.0f, 100.412041f, 127.449074f, 45.340374f, + 78.9512711f, 141.099594f, 91.0069962f, 175.638794f, 131.710648f, 100.40741f, + 104.062469f, 209.895798f, 199.192139f, 163.451385f, 158.622681f, 113.53009f, + 117.5f, 125.833328f, 94.1666641f, 112.874977f, 122.226852f, 119.333351f, + 101.000015f, 194.608704f, 160.953705f, 123.574074f, 115.937469f, 215.104156f, + 153.428268f, 79.5972214f, 88.0740738f, 131.037033f, 160.4375f, 150.777771f, + 73.5555496f, 151.249908f, 134.247772f, 95.8054886f, 156.916611f, 188.393494f, + 172.759247f, 178.759277f, 133.055527f, 227.129623f, 126.796326f, 13.5555573f, + 6.59257507f, 190.29628f, 229.5f, 172.166656f, 58.8333282f, 141.944351f, + 107.407524f, 74.8331528f, 241.499832f, 156.703796f, 143.629639f, 81.1481323f, + 129.333313f, 187.111099f, 123.518539f, 77.2222137f, 134.333344f, 107.666672f, + 106.5f, 101.481476f, 62.9629593f, 171.777649f, 168.166748f, 103.592636f, + 62.1111526f, 100.796249f, 68.1203766f, 53.2198906f, 104.888855f, 203.407379f, + 167.465302f, 133.958328f, 188.060196f, 185.46759f, 173.0625f, 144.768524f, + 93.2870331f, 191.395721f, 213.016235f, 167.574173f, 72.3889847f, 143.446686f, + 51.995369f, 47.9768486f, 67.8055267f, 176.879593f, 153.104172f, 133.208328f, + 201.539352f, 183.113434f, 188.625f, 186.553238f, 144.793976f, 175.437469f, + 192.023148f, 171.713058f, 79.3056488f, 164.189713f, 100.125008f, 63.4328842f, + 32.9027557f, 106.606461f, 92.9074173f, 92.0624847f, 164.590286f, 108.201393f, + 155.1875f, 213.097229f, 190.319443f, 127.909798f, 120.618027f, 129.513931f, + 92.2917023f, 151.680496f, 97.097229f, 41.0300903f, 47.6527786f, 48.0231476f, + 139.018494f, 193.104172f, 130.465286f, 149.076385f, 198.3125f, 214.236115f, + 130.347214f, 120.034729f, 171.215179f, 218.819504f, 162.708389f, 125.09726f, + 133.039352f, 106.21991f, 93.305542f, 132.379623f, 170.581009f, 187.930557f, + 172.150452f, 198.168976f, 179.5625f, 155.479172f, 164.645828f, 106.465355f, + 129.312424f, 161.5047f, 87.4306259f, 134.462906f, 108.025467f, 103.587967f, + 83.5138702f, 147.032394f, 173.226852f, 174.222229f, 158.03241f, 189.328705f, + 149.375f, 111.527779f, 149.305557f, 76.4028625f, 92.6179733f, 121.178329f, + 36.8264732f, 158.629501f, 36.7986107f, 30.6851864f, 29.6874847f, 87.1874924f, + 154.152771f, 172.486115f, 106.379623f, 141.564819f, 128.9375f, 97.4120331f, + 74.4490738f, 38.3958778f, 75.5600967f, 116.268585f, 49.97229f, 169.532288f, + 78.2569504f, 33.1296196f, 63.8124771f, 146.312485f, 156.513901f, 167.069443f, + 221.879639f, 197.064819f, 152.5625f, 98.5509262f, 44.4768448f, 37.4375076f, + 38.1435127f, 56.8795738f, 112.805496f, 76.0602264f, 59.4490776f, 21.4652672f, + 58.5624657f, 166.062469f, 170.182877f, 132.451401f, 92.6342545f, 143.004623f, + 136.5f, 90.152771f, 17.9305496f, 16.3055553f, 60.7614937f, 106.877335f, + 87.3403015f, 106.858772f, 83.6851883f, 92.8263855f, 91.9374771f, 179.437485f, + 176.446777f, 114.826408f, 18.0578594f, 130.372681f, 136.3125f, 99.7013855f, + 81.0902786f, 45.5972633f, 68.8726196f, 109.412041f, 101.81945f, 137.907364f, + 158.962952f, 221.000031f, 158.090271f, 182.627304f, 171.585648f, 130.659729f, + 66.9907303f, 181.620377f, 160.5f, 123.581024f, 189.599533f, 107.291771f, + 57.2916985f, 59.8702812f, 143.944382f, 133.719925f, 213.962967f, 204.000015f, + 155.631958f, 127.391212f, 159.432861f, 199.618057f, 215.379623f, 235.564804f, + 187.5f, 119.84491f, 79.7523117f, 58.7083588f, 79.819397f, 96.6482086f, + 36.2778397f, 76.3726349f, 108.164352f, 116.571747f, 152.687531f, 73.5208511f, + 82.0254517f, 115.590271f, 136.106476f, 180.365738f, 179.5f, 152.243057f, + 115.298607f, 110.854172f, 139.41893f, 170.381958f, 156.770844f, 86.8033066f, + 64.9907379f, 85.5231171f, 164.416687f, 76.6389084f, 74.1157379f, 81.9861145f, + 55.4259224f, 112.462959f, 139.375f, 141.430557f, 121.986115f, 109.312515f, + 136.627258f, 170.770844f, 159.937515f, 118.185226f, 108.000008f, 118.370346f, + 186.500015f, 126.500023f, 136.314804f, 121.500023f, 24.9629593f, 63.4814796f, + 85.0f, 90.3888855f, 79.277771f, 46.5000381f, 76.1295547f, 103.814896f, + 26.777853f, 150.499863f, 236.833328f, 216.944458f, 181.666672f, 177.222229f, + 107.259277f, 52.4444427f, 64.4444427f, 82.2222214f, 164.5f, 221.981476f, + 163.462967f, 107.111176f, 73.9629822f, 87.9073029f, 189.388779f, 101.074173f, + 114.912041f, 95.4398193f, 78.2291641f, 86.2847137f, 129.180542f, 168.520828f, + 179.363419f, 168.900467f, 171.375f, 151.460648f, 72.1087875f, 143.430466f, + 149.171341f, 114.747711f, 84.2847519f, 67.8680725f, 59.3842583f, 77.7245331f, + 95.5416641f, 94.4305496f, 146.284714f, 191.979172f, 193.328705f, 171.476837f, + 155.625f, 116.481483f, 22.9629536f, 111.80545f, 149.074081f, 141.414398f, + 104.28476f, 116.916656f, 93.9814835f, 168.562485f, 229.131943f, 204.224548f, + 148.773163f, 106.076385f, 103.775467f, 94.7939758f, 130.3125f, 141.125f, + 56.1249924f, 41.9027939f, 79.7846451f, 149.749924f, 224.749924f, 222.92363f, + 125.314819f, 161.71524f, 236.923615f, 235.90509f, 124.189842f, 52.5347137f, + 116.317131f, 129.002319f, 177.1875f, 215.652786f, 198.430557f, 156.652832f, + 116.270874f, 94.1388626f, 116.916641f, 169.243011f, 133.372681f, 147.958328f, + 171.3125f, 215.479156f, 127.534752f, 45.5555534f, 58.0138855f, 108.569443f, + 152.8125f, 180.203705f, 178.90741f, 124.527847f, 81.2870712f, 52.2430687f, + 40.8541794f, 86.5925446f, 122.386581f, 109.743034f, 146.520828f, 140.131943f, + 125.888893f, 90.472229f, 27.7291622f, 81.8958282f, 108.75f, 114.418983f, + 103.400467f, 81.8194656f, 95.0370026f, 105.132004f, 51.5208893f, 51.6967545f, + 119.60186f, 73.7800598f, 155.368088f, 45.2754898f, 114.418953f, 159.743073f, + 55.2893524f, 76.8634262f, 78.0f, 61.4120331f, 28.44907f, 75.6527252f, + 162.238312f, 219.097305f, 138.541748f, 78.7130127f, 229.768524f, 164.460678f, + 54.9097176f, 75.3726807f, 106.71064f, 137.868057f, 161.136581f, 156.599533f, + 141.0f, 113.995369f, 74.3657379f, 158.236008f, 181.557892f, 178.236084f, + 205.458313f, 147.546356f, 213.233795f, 195.094925f, 125.80555f, 153.212952f, + 178.539337f, 182.451401f, 153.731476f, 192.990738f, 137.75f, 67.192131f, + 59.3217583f, 170.437378f, 165.854248f, 111.84259f, 115.361107f, 100.037048f, + 167.511581f, 160.435196f, 158.888901f, 147.40741f, 171.101837f, 185.326385f, + 163.39119f, 214.039352f, 145.625f, 72.5462952f, 108.842598f, 148.208282f, + 151.715302f, 121.717651f, 60.2361755f, 95.0231094f, 128.525467f, 87.9490585f, + 124.020851f, 65.9652863f, 95.7152634f, 153.423615f, 201.305557f, 213.527786f, + 168.3125f, 136.1875f, 188.6875f, 99.6320496f, 140.74292f, 195.011703f, + 72.3265152f, 147.060104f, 142.456024f, 92.7546234f, 109.812508f, 72.3125076f, + 156.45137f, 222.631958f, 181.916656f, 138.583328f, 143.1875f, 131.090271f, + 36.3680458f, 21.0902939f, 102.75679f, 178.525558f, 88.6181488f, 155.532333f, + 159.673615f, 147.409714f, 157.097229f, 118.393524f, 162.662018f, 181.701385f, + 107.145836f, 67.9791718f, 111.5f, 139.974533f, 54.5115662f, 37.9305725f, + 74.5045547f, 121.032417f, 112.513901f, 176.967529f, 156.256943f, 180.777786f, + 153.555557f, 112.629639f, 118.224525f, 113.201393f, 61.9652786f, 45.5763893f, + 120.125f, 167.439804f, 67.2546234f, 61.409729f, 79.3379288f, 111.351814f, + 146.722183f, 166.162033f, 135.666656f, 182.537079f, 98.0000076f, 58.0000076f, + 48.0740776f, 51.5f, 58.2037048f, 66.3518524f, 162.0f, 203.481476f, + 44.9629478f, 70.6110764f, 109.314766f, 149.962921f, 182.555527f, 128.129684f, +}; + +// case 1: 32x24 -> 16x16 +inline constexpr int64_t kLtx2ImgResize1SrcH = 32; +inline constexpr int64_t kLtx2ImgResize1SrcW = 24; +inline constexpr int64_t kLtx2ImgResize1DstH = 16; +inline constexpr int64_t kLtx2ImgResize1DstW = 16; + +inline constexpr float kLtx2ImgResize1Golden[] = { + 119.340904f, 176.340912f, 178.363632f, 95.9659042f, 54.3295479f, 92.2045593f, + 117.011375f, 171.011368f, 62.6477318f, 162.329544f, 105.602264f, 101.750008f, + 114.443176f, 108.340904f, 170.170441f, 125.749992f, 100.034103f, 170.227203f, + 55.5227623f, 63.6022568f, 79.1363297f, 46.2045708f, 151.590897f, 85.4886856f, + 130.20459f, 63.7273254f, 31.1591549f, 144.715881f, 110.284119f, 51.715992f, + 157.659103f, 46.0796013f, 137.25f, 69.0f, 88.625f, 38.0f, + 113.875f, 163.0f, 170.75f, 192.875f, 160.875f, 192.75f, + 105.375f, 84.375f, 199.875f, 215.25f, 133.5f, 172.5f, + 133.659088f, 70.1477051f, 51.5681877f, 155.579605f, 171.284073f, 126.352219f, + 54.1249046f, 145.147705f, 38.3408585f, 165.704529f, 114.920494f, 35.9658852f, + 219.784088f, 158.045425f, 28.6817875f, 53.6022034f, 95.7159042f, 128.852249f, + 66.2272644f, 103.420448f, 88.6136322f, 113.090912f, 144.863647f, 112.170456f, + 182.579544f, 62.3068275f, 124.10228f, 43.8863678f, 222.863632f, 133.170441f, + 71.5681839f, 150.284103f, 149.340973f, 60.7727661f, 166.102371f, 77.875f, + 115.261406f, 82.0567932f, 154.67041f, 84.681778f, 168.761353f, 56.2954025f, + 71.2612991f, 35.943161f, 88.227211f, 57.2727013f, 124.590942f, 190.249985f, + 134.875f, 164.125015f, 143.329544f, 94.954567f, 136.420471f, 228.124985f, + 131.727264f, 109.704552f, 115.181839f, 140.443192f, 68.1704483f, 81.4318161f, + 177.602264f, 150.840897f, 130.079529f, 143.170425f, 164.409119f, 198.193192f, + 159.545456f, 78.3408356f, 118.715828f, 126.045395f, 112.556839f, 114.534088f, + 143.761322f, 98.9317474f, 82.0909424f, 183.988724f, 162.715912f, 125.079575f, + 190.954651f, 140.363708f, 154.158981f, 81.8865051f, 89.7954865f, 48.920536f, + 188.897797f, 93.6138535f, 125.250137f, 149.454575f, 157.488647f, 123.329628f, + 62.1250496f, 92.1931152f, 174.659149f, 157.477142f, 91.6478806f, 145.477127f, + 29.2727394f, 56.8863945f, 70.6363754f, 141.920441f, 228.488632f, 206.97728f, + 121.784119f, 88.579567f, 153.500015f, 149.545456f, 114.284081f, 140.704529f, + 109.295486f, 133.056808f, 138.60228f, 18.5000076f, 117.125038f, 190.761353f, + 79.7954941f, 173.27269f, 89.8863831f, 207.261292f, 207.034027f, 223.636276f, + 169.625f, 58.5568161f, 75.3068237f, 105.886353f, 132.159088f, 197.374985f, + 152.852249f, 43.3181953f, 129.556686f, 122.909042f, 112.215813f, 121.284126f, + 175.522766f, 94.5569153f, 122.386444f, 133.738739f, 137.272644f, 71.8522949f, + 63.6590729f, 74.1590729f, 84.7613525f, 112.545334f, 97.7159424f, 133.238739f, + 21.6478233f, 37.6022606f, 100.636276f, 130.94313f, 201.636276f, 65.0342407f, + 169.965836f, 116.102249f, 21.1025047f, 125.215881f, 206.692993f, 230.147629f, + 124.818237f, 80.2955933f, 170.340851f, 168.772644f, 131.25f, 104.625f, + 89.125f, 74.125f, 156.625f, 130.75f, 132.375f, 102.0f, + 154.25f, 78.125f, 87.0f, 104.625f, 162.5f, 143.375f, + 90.625f, 85.75f, 44.9317474f, 169.738632f, 148.25f, 83.6477432f, + 202.772751f, 109.363647f, 160.170471f, 206.977325f, 71.6704254f, 129.590958f, + 139.363663f, 188.386444f, 159.85228f, 134.625015f, 57.4431877f, 115.125023f, + 170.920547f, 143.579514f, 60.5796242f, 84.4090652f, 77.4660797f, 75.4885712f, + 101.806679f, 78.0796432f, 152.102158f, 191.988525f, 183.079498f, 105.352371f, + 112.193344f, 138.306854f, 131.000107f, 169.284195f, 179.579544f, 152.034088f, + 110.22728f, 185.602264f, 77.8863678f, 50.4090919f, 106.988632f, 116.840904f, + 94.5340805f, 152.181824f, 34.7727318f, 167.52272f, 150.079559f, 167.295441f, + 155.238632f, 147.329544f, 190.056747f, 57.5454407f, 110.784065f, 77.4090805f, + 139.193222f, 78.3068008f, 235.4431f, 133.27272f, 75.4772644f, 206.829544f, + 132.57959f, 86.420517f, 96.8295517f, 95.0909348f, 74.2387009f, 139.079483f, + 146.375f, 132.125f, 81.0f, 105.375f, 172.75f, 115.375f, + 142.0f, 163.0f, 114.0f, 110.625f, 127.625f, 192.125f, + 91.125f, 80.625f, 149.0f, 58.25f, 101.499916f, 158.261322f, + 180.318222f, 87.0454178f, 91.5226898f, 124.999969f, 200.806824f, 73.6476593f, + 91.1931534f, 66.2045746f, 89.5000305f, 240.318222f, 133.386398f, 177.840988f, + 86.0681686f, 214.147797f, 123.431816f, 79.0113754f, 173.727264f, 162.85228f, + 99.3636398f, 131.988647f, 171.784088f, 221.079544f, 119.102272f, 147.704544f, + 101.897728f, 157.806808f, 117.875008f, 174.77272f, 136.795441f, 113.681808f, + 174.556885f, 164.875f, 175.988632f, 99.9658661f, 55.8181419f, 66.3180847f, + 174.011414f, 128.56813f, 74.5340652f, 197.909119f, 134.204559f, 92.7499695f, + 180.886337f, 57.1022263f, 172.670547f, 47.7272682f, 95.7613678f, 171.511368f, + 162.75f, 89.2841034f, 104.943207f, 99.5568008f, 72.5795593f, 63.9886665f, + 172.238663f, 159.25f, 93.6931763f, 178.034088f, 145.988647f, 29.6704578f, + 134.102249f, 141.329529f, 144.147736f, 108.727203f, 72.1817474f, 172.000046f, + 193.454514f, 137.886444f, 169.647766f, 159.386337f, 163.590851f, 75.5339966f, + 163.829605f, 142.761353f, 64.4431f, 113.465973f, 48.0454521f, 138.011429f, + 147.079651f, 138.999893f, 112.295349f, 117.852295f, 73.9205704f, 140.636337f, + 144.170486f, 200.011261f, 218.477264f, 89.477356f, 59.4887733f, 89.2160034f, + 199.443222f, 126.454422f, 204.886261f, 156.647766f, 97.2273026f, 113.624985f, + 156.113617f, 148.170456f, 160.727264f, 52.6931992f, 64.0682068f, 106.98864f, + 135.386383f, 147.89772f, 195.579529f, 199.909073f, 111.73867f, 37.1818199f, + 59.0909309f, 96.7613831f, 153.488647f, 203.465866f, 66.1818466f, 108.931839f, + 147.772736f, 122.954529f, 158.340881f, 57.7386246f, 153.85228f, 156.545456f, + 158.011353f, 216.579498f, 158.670441f, 137.840881f, 189.488602f, 113.534065f, + 149.477234f, 71.886322f, 166.125031f, 210.068268f, 102.386246f, 141.1819f, + 117.613678f, 142.431976f, 176.261368f, 155.738617f, 167.272797f, 91.0795135f, + 180.056915f, 166.181976f, 51.6363106f, 117.227386f, 81.2501526f, 72.863739f, + 135.056854f, 157.42041f, 105.46582f, 157.42041f, 193.318146f, 171.02269f, + 67.3295593f, 134.011261f, 198.136383f, 214.32933f, 239.443024f, 65.2727737f, + 108.625061f, 147.556885f, 172.0f, 123.625f, 94.625f, 111.125f, + 126.375f, 145.5f, 191.375f, 134.375f, 124.25f, 102.125f, + 215.625f, 131.25f, 98.25f, 147.75f, 185.75f, 154.625f, + 156.556824f, 126.181831f, 66.6022949f, 114.465912f, 51.5908241f, 63.6590424f, + 139.079514f, 60.9772453f, 50.3976631f, 166.409103f, 151.738602f, 198.034088f, + 67.5454559f, 185.363632f, 155.249969f, 192.95459f, 104.500053f, 138.590759f, + 148.988556f, 105.693336f, 144.693237f, 146.875f, 152.522522f, 124.193275f, + 172.465988f, 201.136398f, 172.772568f, 128.454483f, 102.76133f, 177.420441f, + 58.2727966f, 80.0909576f, 100.090912f, 141.681808f, 158.420456f, 109.840919f, + 147.113632f, 118.886353f, 151.045441f, 228.568176f, 80.7840958f, 156.806824f, + 124.636375f, 63.9545441f, 66.8863602f, 77.2840958f, 96.0113602f, 140.295456f, + 160.693207f, 184.079514f, 177.625031f, 154.443161f, 222.17038f, 174.852234f, + 31.1250114f, 46.0227356f, 116.749969f, 80.613678f, 193.068161f, 188.431854f, + 195.840897f, 68.5454559f, 64.4545746f, 161.06813f, 196.375f, 103.75f, + 221.25f, 90.375f, 114.25f, 69.625f, 104.25f, 151.875f, + 114.625f, 121.75f, 110.0f, 189.125f, 138.625f, 49.0f, + 103.125f, 121.125f, 209.147736f, 162.306854f, 102.124939f, 161.375046f, + 81.0226746f, 64.1477356f, 82.3976898f, 81.8521881f, 204.613647f, 43.0909042f, + 222.375076f, 156.886368f, 47.045414f, 106.772766f, 66.6249924f, 99.0794983f, + 81.227272f, 124.556831f, 149.079529f, 66.0000076f, 74.8068161f, 145.715912f, + 214.625f, 152.511368f, 191.090912f, 85.272728f, 106.295441f, 55.9431763f, + 129.784073f, 158.89772f, 162.159088f, 198.943176f, 156.977341f, 101.57946f, + 64.5113297f, 151.477295f, 102.818214f, 212.181824f, 53.7612305f, 118.397675f, + 52.0112877f, 179.68187f, 100.545502f, 147.45462f, 140.318253f, 89.5567932f, + 155.045471f, 193.5f, 125.727264f, 141.136353f, 90.0454636f, 153.761368f, + 159.818176f, 102.306816f, 185.329544f, 111.113625f, 101.454559f, 189.102264f, + 150.818146f, 158.795441f, 170.443192f, 46.5909195f, 167.761368f, 94.4318085f, + 73.9090805f, 169.568237f, 150.818192f, 152.079498f, 125.625f, 124.73867f, + 100.30674f, 119.227325f, 150.943161f, 108.431801f, 180.86377f, 181.000046f, + 190.136353f, 108.318207f, 123.249954f, 46.4090652f, 178.897644f, 140.477402f, + 162.806595f, 149.57959f, 191.465927f, 162.590805f, 126.897614f, 91.5796509f, + 199.193222f, 90.170578f, 187.2043f, 114.829552f, 104.238808f, 92.0342331f, + 153.545258f, 170.795349f, 190.784073f, 191.840912f, 144.999969f, 71.8750229f, + 82.9091187f, 71.8636398f, 107.318169f, 177.75f, 174.22728f, 102.465927f, + 109.943161f, 157.02272f, 86.8636703f, 71.9659348f, 54.8295364f, 119.499992f, + 105.011421f, 229.409088f, 50.6250076f, 122.113625f, 213.999969f, 122.875046f, + 113.772758f, 127.784088f, 234.215866f, 196.681763f, 32.6364136f, 50.2955246f, + 115.261368f, 117.306816f, 147.761353f, 116.568138f, 164.283981f, 172.94313f, + 81.2613907f, 89.545433f, 86.181778f, 230.931808f, 171.090897f, 136.590912f, + 59.8180695f, 62.579525f, 207.840973f, 134.340775f, 68.1249313f, 92.806778f, + 137.193192f, 91.068306f, 16.7727737f, 41.8977623f, 36.0795631f, 100.000069f, + 205.340759f, 199.647705f, 147.03418f, 60.7159538f, 155.693146f, 130.806885f, + 126.352203f, 60.5570717f, 211.613632f, 183.658875f, 141.363647f, 61.318222f, + 39.75f, 48.0f, 68.125f, 127.0f, 137.0f, 120.5f, + 192.0f, 124.5f, 109.0f, 160.0f, 62.625f, 208.5f, + 137.25f, 92.25f, 127.375f, 71.5f, 207.579636f, 113.045502f, + 206.363708f, 37.5454178f, 67.2272186f, 175.943253f, 128.147705f, 102.374969f, + 54.1931648f, 213.295486f, 79.3181992f, 182.715912f, 172.420517f, 218.47731f, + 195.47731f, 158.045517f, 158.647537f, 108.772713f, 104.261307f, 133.715805f, + 146.431671f, 123.11348f, 154.863556f, 147.636444f, 59.4205437f, 161.795502f, + 113.159058f, 139.306671f, 164.431885f, 119.909142f, 102.841034f, 166.76149f, +}; + +// case 2: 16x16 -> 16x16 +inline constexpr int64_t kLtx2ImgResize2SrcH = 16; +inline constexpr int64_t kLtx2ImgResize2SrcW = 16; +inline constexpr int64_t kLtx2ImgResize2DstH = 16; +inline constexpr int64_t kLtx2ImgResize2DstW = 16; + +inline constexpr float kLtx2ImgResize2Golden[] = { + 47.0f, 65.0f, 150.0f, 227.0f, 58.0f, 150.0f, + 134.0f, 0.0f, 167.0f, 112.0f, 233.0f, 209.0f, + 35.0f, 172.0f, 246.0f, 198.0f, 175.0f, 138.0f, + 221.0f, 237.0f, 116.0f, 104.0f, 108.0f, 243.0f, + 32.0f, 253.0f, 89.0f, 209.0f, 46.0f, 31.0f, + 14.0f, 23.0f, 73.0f, 53.0f, 209.0f, 138.0f, + 73.0f, 91.0f, 46.0f, 239.0f, 124.0f, 74.0f, + 225.0f, 218.0f, 234.0f, 225.0f, 65.0f, 109.0f, + 41.0f, 248.0f, 153.0f, 7.0f, 88.0f, 123.0f, + 171.0f, 137.0f, 216.0f, 4.0f, 16.0f, 158.0f, + 37.0f, 182.0f, 243.0f, 158.0f, 182.0f, 252.0f, + 36.0f, 228.0f, 172.0f, 1.0f, 169.0f, 80.0f, + 162.0f, 54.0f, 123.0f, 55.0f, 135.0f, 249.0f, + 233.0f, 91.0f, 200.0f, 177.0f, 88.0f, 96.0f, + 25.0f, 179.0f, 169.0f, 2.0f, 65.0f, 36.0f, + 148.0f, 6.0f, 65.0f, 204.0f, 73.0f, 93.0f, + 97.0f, 111.0f, 136.0f, 140.0f, 125.0f, 20.0f, + 88.0f, 241.0f, 181.0f, 129.0f, 171.0f, 212.0f, + 105.0f, 7.0f, 128.0f, 135.0f, 194.0f, 240.0f, + 5.0f, 14.0f, 203.0f, 23.0f, 76.0f, 60.0f, + 80.0f, 228.0f, 99.0f, 222.0f, 116.0f, 94.0f, + 117.0f, 25.0f, 174.0f, 32.0f, 17.0f, 150.0f, + 25.0f, 41.0f, 47.0f, 185.0f, 206.0f, 25.0f, + 178.0f, 18.0f, 44.0f, 109.0f, 229.0f, 148.0f, + 1.0f, 116.0f, 206.0f, 98.0f, 213.0f, 193.0f, + 25.0f, 126.0f, 226.0f, 13.0f, 105.0f, 251.0f, + 168.0f, 167.0f, 85.0f, 60.0f, 12.0f, 232.0f, + 82.0f, 14.0f, 124.0f, 127.0f, 66.0f, 244.0f, + 1.0f, 180.0f, 209.0f, 125.0f, 92.0f, 3.0f, + 169.0f, 11.0f, 41.0f, 156.0f, 122.0f, 49.0f, + 173.0f, 65.0f, 0.0f, 169.0f, 115.0f, 154.0f, + 108.0f, 40.0f, 241.0f, 86.0f, 32.0f, 239.0f, + 94.0f, 227.0f, 183.0f, 22.0f, 209.0f, 96.0f, + 107.0f, 168.0f, 171.0f, 128.0f, 147.0f, 149.0f, + 74.0f, 129.0f, 143.0f, 253.0f, 139.0f, 162.0f, + 223.0f, 168.0f, 70.0f, 160.0f, 229.0f, 73.0f, + 165.0f, 102.0f, 137.0f, 111.0f, 43.0f, 94.0f, + 10.0f, 40.0f, 225.0f, 132.0f, 180.0f, 123.0f, + 46.0f, 56.0f, 238.0f, 208.0f, 72.0f, 136.0f, + 98.0f, 177.0f, 84.0f, 162.0f, 41.0f, 216.0f, + 117.0f, 9.0f, 158.0f, 37.0f, 79.0f, 126.0f, + 135.0f, 74.0f, 95.0f, 46.0f, 85.0f, 217.0f, + 200.0f, 252.0f, 54.0f, 255.0f, 30.0f, 216.0f, + 95.0f, 245.0f, 253.0f, 9.0f, 16.0f, 191.0f, + 194.0f, 163.0f, 187.0f, 216.0f, 118.0f, 242.0f, + 243.0f, 196.0f, 71.0f, 227.0f, 14.0f, 40.0f, + 118.0f, 182.0f, 158.0f, 240.0f, 227.0f, 146.0f, + 219.0f, 89.0f, 174.0f, 19.0f, 222.0f, 216.0f, + 205.0f, 190.0f, 197.0f, 58.0f, 237.0f, 228.0f, + 106.0f, 177.0f, 216.0f, 195.0f, 168.0f, 194.0f, + 14.0f, 161.0f, 119.0f, 255.0f, 38.0f, 182.0f, + 18.0f, 199.0f, 199.0f, 162.0f, 222.0f, 71.0f, + 152.0f, 111.0f, 96.0f, 150.0f, 33.0f, 63.0f, + 72.0f, 121.0f, 226.0f, 194.0f, 48.0f, 85.0f, + 187.0f, 137.0f, 42.0f, 200.0f, 204.0f, 95.0f, + 10.0f, 122.0f, 170.0f, 116.0f, 255.0f, 40.0f, + 246.0f, 65.0f, 66.0f, 184.0f, 217.0f, 113.0f, + 247.0f, 250.0f, 179.0f, 17.0f, 248.0f, 101.0f, + 122.0f, 53.0f, 57.0f, 98.0f, 44.0f, 127.0f, + 170.0f, 32.0f, 215.0f, 21.0f, 109.0f, 1.0f, + 226.0f, 177.0f, 150.0f, 237.0f, 235.0f, 196.0f, + 8.0f, 214.0f, 0.0f, 22.0f, 134.0f, 218.0f, + 227.0f, 238.0f, 169.0f, 133.0f, 56.0f, 10.0f, + 209.0f, 78.0f, 217.0f, 127.0f, 23.0f, 237.0f, + 157.0f, 112.0f, 244.0f, 188.0f, 224.0f, 57.0f, + 176.0f, 91.0f, 237.0f, 252.0f, 159.0f, 117.0f, + 198.0f, 90.0f, 37.0f, 207.0f, 90.0f, 219.0f, + 77.0f, 244.0f, 82.0f, 195.0f, 50.0f, 109.0f, + 152.0f, 107.0f, 1.0f, 83.0f, 244.0f, 15.0f, + 97.0f, 36.0f, 37.0f, 221.0f, 235.0f, 105.0f, + 16.0f, 124.0f, 94.0f, 89.0f, 18.0f, 77.0f, + 169.0f, 244.0f, 163.0f, 45.0f, 57.0f, 109.0f, + 238.0f, 169.0f, 76.0f, 230.0f, 35.0f, 176.0f, + 150.0f, 240.0f, 145.0f, 163.0f, 12.0f, 132.0f, + 164.0f, 147.0f, 35.0f, 230.0f, 47.0f, 37.0f, + 221.0f, 242.0f, 38.0f, 148.0f, 40.0f, 185.0f, + 109.0f, 0.0f, 74.0f, 11.0f, 200.0f, 155.0f, + 4.0f, 168.0f, 251.0f, 203.0f, 128.0f, 46.0f, + 236.0f, 103.0f, 87.0f, 213.0f, 85.0f, 210.0f, + 85.0f, 167.0f, 88.0f, 123.0f, 69.0f, 254.0f, + 213.0f, 163.0f, 179.0f, 8.0f, 100.0f, 91.0f, + 217.0f, 225.0f, 160.0f, 79.0f, 76.0f, 128.0f, + 0.0f, 104.0f, 114.0f, 15.0f, 106.0f, 94.0f, + 193.0f, 116.0f, 46.0f, 12.0f, 170.0f, 42.0f, + 36.0f, 224.0f, 239.0f, 79.0f, 189.0f, 161.0f, + 192.0f, 102.0f, 11.0f, 101.0f, 59.0f, 206.0f, + 56.0f, 153.0f, 44.0f, 252.0f, 117.0f, 80.0f, + 249.0f, 138.0f, 81.0f, 84.0f, 50.0f, 224.0f, + 90.0f, 47.0f, 31.0f, 186.0f, 216.0f, 216.0f, + 74.0f, 222.0f, 250.0f, 186.0f, 210.0f, 7.0f, + 42.0f, 92.0f, 43.0f, 37.0f, 209.0f, 195.0f, + 124.0f, 9.0f, 126.0f, 59.0f, 12.0f, 165.0f, + 45.0f, 108.0f, 112.0f, 118.0f, 86.0f, 195.0f, + 103.0f, 137.0f, 105.0f, 43.0f, 160.0f, 1.0f, + 141.0f, 73.0f, 168.0f, 80.0f, 108.0f, 19.0f, + 248.0f, 42.0f, 37.0f, 9.0f, 248.0f, 199.0f, + 86.0f, 27.0f, 233.0f, 219.0f, 19.0f, 162.0f, + 115.0f, 197.0f, 6.0f, 32.0f, 153.0f, 54.0f, + 38.0f, 255.0f, 34.0f, 174.0f, 18.0f, 159.0f, + 245.0f, 225.0f, 15.0f, 223.0f, 249.0f, 96.0f, + 144.0f, 228.0f, 45.0f, 227.0f, 73.0f, 236.0f, + 41.0f, 220.0f, 10.0f, 16.0f, 85.0f, 177.0f, + 219.0f, 121.0f, 95.0f, 144.0f, 78.0f, 27.0f, + 85.0f, 158.0f, 114.0f, 116.0f, 193.0f, 114.0f, + 136.0f, 225.0f, 3.0f, 208.0f, 221.0f, 148.0f, + 68.0f, 242.0f, 178.0f, 52.0f, 76.0f, 0.0f, + 235.0f, 4.0f, 87.0f, 137.0f, 66.0f, 225.0f, + 119.0f, 26.0f, 187.0f, 7.0f, 108.0f, 153.0f, + 196.0f, 34.0f, 206.0f, 107.0f, 19.0f, 89.0f, + 103.0f, 229.0f, 45.0f, 241.0f, 216.0f, 26.0f, + 17.0f, 43.0f, 243.0f, 36.0f, 211.0f, 153.0f, + 237.0f, 222.0f, 169.0f, 107.0f, 61.0f, 49.0f, + 114.0f, 207.0f, 139.0f, 100.0f, 190.0f, 181.0f, + 223.0f, 92.0f, 58.0f, 14.0f, 53.0f, 228.0f, + 139.0f, 241.0f, 204.0f, 200.0f, 88.0f, 189.0f, + 214.0f, 228.0f, 132.0f, 167.0f, 128.0f, 175.0f, + 120.0f, 218.0f, 49.0f, 175.0f, 215.0f, 226.0f, + 112.0f, 139.0f, 122.0f, 221.0f, 55.0f, 12.0f, + 190.0f, 43.0f, 211.0f, 229.0f, 87.0f, 193.0f, + 121.0f, 25.0f, 158.0f, 88.0f, 35.0f, 23.0f, + 158.0f, 10.0f, 34.0f, 122.0f, 4.0f, 227.0f, + 174.0f, 218.0f, 124.0f, 75.0f, 147.0f, 4.0f, + 176.0f, 134.0f, 36.0f, 52.0f, 231.0f, 63.0f, + 41.0f, 2.0f, 201.0f, 60.0f, 96.0f, 178.0f, + 13.0f, 125.0f, 242.0f, 23.0f, 18.0f, 12.0f, + 109.0f, 46.0f, 203.0f, 31.0f, 117.0f, 92.0f, + 108.0f, 210.0f, 60.0f, 150.0f, 66.0f, 253.0f, + 174.0f, 0.0f, 239.0f, 154.0f, 25.0f, 186.0f, +}; + +// case 3: 8x8 -> 16x24 +inline constexpr int64_t kLtx2ImgResize3SrcH = 8; +inline constexpr int64_t kLtx2ImgResize3SrcW = 8; +inline constexpr int64_t kLtx2ImgResize3DstH = 16; +inline constexpr int64_t kLtx2ImgResize3DstW = 24; + +inline constexpr float kLtx2ImgResize3Golden[] = { + 210.0f, 210.0f, 220.0f, 230.0f, 240.0f, 232.333328f, + 224.666672f, 217.0f, 198.999985f, 181.0f, 163.0f, 156.666656f, + 150.333328f, 144.0f, 97.333313f, 50.6666222f, 4.0f, 21.6666756f, + 39.3333511f, 57.0f, 65.0000076f, 73.0000076f, 81.0f, 81.0f, + 183.666656f, 183.666672f, 177.222214f, 170.777756f, 164.333313f, 173.111115f, + 181.888885f, 190.666656f, 179.777771f, 168.888885f, 158.0f, 164.888885f, + 171.777786f, 178.666672f, 135.222214f, 91.7777405f, 48.3333397f, 50.1111145f, + 51.8888893f, 53.6666641f, 71.222229f, 88.7777939f, 106.333336f, 106.333336f, + 157.333328f, 157.333328f, 134.444427f, 111.555542f, 88.6666489f, 113.888878f, + 139.111099f, 164.333328f, 160.555557f, 156.777771f, 153.0f, 173.111115f, + 193.222229f, 213.333344f, 173.111099f, 132.888855f, 92.6666794f, 78.5555573f, + 64.4444351f, 50.3333321f, 77.444458f, 104.555588f, 131.666672f, 131.666672f, + 131.0f, 131.0f, 91.6666641f, 52.3333321f, 13.0f, 54.6666718f, + 96.3333435f, 138.0f, 141.333344f, 144.666672f, 148.0f, 181.333344f, + 214.666687f, 248.0f, 210.999985f, 173.999969f, 137.0f, 106.999985f, + 76.9999695f, 47.0f, 83.666687f, 120.333366f, 157.0f, 157.0f, + 167.333344f, 167.333344f, 140.000015f, 112.666695f, 85.3333664f, 108.222252f, + 131.11113f, 154.0f, 138.666656f, 123.333321f, 107.999977f, 153.888901f, + 199.777786f, 245.666672f, 214.0f, 182.333298f, 150.666672f, 121.333328f, + 91.9999771f, 62.6666756f, 95.2222443f, 127.777817f, 160.333328f, 160.333328f, + 203.666672f, 203.666672f, 188.333328f, 173.000015f, 157.666687f, 161.777786f, + 165.888901f, 170.0f, 135.999985f, 101.999985f, 67.9999924f, 126.444466f, + 184.888901f, 243.333328f, 216.999985f, 190.666641f, 164.333344f, 135.666656f, + 106.999977f, 78.3333359f, 106.777794f, 135.22226f, 163.666672f, 163.666672f, + 240.0f, 240.0f, 236.666672f, 233.333344f, 230.0f, 215.333328f, + 200.666656f, 186.0f, 133.333313f, 80.6666565f, 28.0f, 99.0000381f, + 170.000015f, 241.0f, 219.999985f, 198.999985f, 178.0f, 149.999985f, + 121.999969f, 94.0f, 118.333344f, 142.666687f, 167.0f, 167.0f, + 159.999969f, 159.999969f, 158.222183f, 156.444412f, 154.666626f, 158.666626f, + 162.666641f, 166.666656f, 145.333328f, 124.000015f, 102.666702f, 142.111145f, + 181.555573f, 220.999985f, 198.777756f, 176.555527f, 154.333328f, 142.777771f, + 131.222214f, 119.666679f, 126.333336f, 133.0f, 139.666656f, 139.666656f, + 79.9999847f, 79.9999847f, 79.7777557f, 79.555542f, 79.333313f, 101.999985f, + 124.666656f, 147.333328f, 157.333344f, 167.333344f, 177.333344f, 185.222229f, + 193.111115f, 201.0f, 177.555527f, 154.111084f, 130.666656f, 135.555557f, + 140.444458f, 145.333344f, 134.333328f, 123.333321f, 112.333328f, 112.333328f, + 0.0f, 5.96046448e-08f, 1.33333349f, 2.66666675f, 4.0f, 45.3333397f, + 86.6666794f, 128.0f, 169.333344f, 210.666672f, 252.0f, 228.333328f, + 204.666656f, 181.0f, 156.333328f, 131.666641f, 107.0f, 128.333344f, + 149.666687f, 171.0f, 142.333313f, 113.666641f, 85.0f, 85.0f, + 13.6666737f, 13.6666737f, 15.888896f, 18.1111183f, 20.3333416f, 62.3333473f, + 104.333351f, 146.333344f, 173.666672f, 201.0f, 228.333313f, 196.777756f, + 165.222198f, 133.666641f, 127.999985f, 122.333321f, 116.666672f, 134.000015f, + 151.333344f, 168.666656f, 154.111115f, 139.555557f, 125.000015f, 125.000023f, + 27.3333473f, 27.3333473f, 30.444458f, 33.5555687f, 36.6666832f, 79.3333588f, + 122.000031f, 164.666687f, 178.0f, 191.333313f, 204.666641f, 165.222168f, + 125.777725f, 86.3332901f, 99.6666489f, 113.000008f, 126.333344f, 139.666672f, + 153.000015f, 166.333328f, 165.888901f, 165.444473f, 165.000031f, 165.000031f, + 41.0f, 41.0f, 45.0f, 49.0f, 53.0f, 96.3333359f, + 139.666672f, 183.0f, 182.333328f, 181.666672f, 181.0f, 133.666641f, + 86.3333206f, 39.0f, 71.3333511f, 103.666695f, 136.0f, 145.333344f, + 154.666687f, 164.0f, 177.666672f, 191.333344f, 205.0f, 205.0f, + 68.6666794f, 68.6666794f, 63.1111183f, 57.5555573f, 52.0f, 84.8888855f, + 117.777771f, 150.666656f, 147.555542f, 144.444427f, 141.333313f, 129.222214f, + 117.11113f, 105.000031f, 111.555573f, 118.111122f, 124.666656f, 130.0f, + 135.333328f, 140.666656f, 149.111099f, 157.555542f, 165.999985f, 165.999985f, + 96.3333588f, 96.3333588f, 81.2222366f, 66.1111145f, 51.0f, 73.4444351f, + 95.8888702f, 118.333305f, 112.77774f, 107.222183f, 101.666626f, 124.777786f, + 147.888931f, 171.000061f, 151.777802f, 132.555557f, 113.333321f, 114.666649f, + 115.999977f, 117.333313f, 120.555527f, 123.777748f, 126.999962f, 126.999962f, + 124.0f, 124.0f, 99.3333282f, 74.6666641f, 50.0f, 62.0f, + 74.0f, 86.0f, 77.9999924f, 70.0f, 62.0f, 120.333359f, + 178.666672f, 237.0f, 191.999985f, 146.999954f, 102.0f, 99.3333282f, + 96.6666641f, 94.0f, 92.0f, 90.0f, 88.0f, 88.0f, + 239.0f, 239.0f, 183.666656f, 128.333328f, 73.0f, 87.0f, + 101.000008f, 115.0f, 76.9999847f, 38.9999924f, 1.0f, 12.0000057f, + 23.0000038f, 34.0f, 105.333366f, 176.666733f, 248.0f, 240.666656f, + 233.333313f, 226.0f, 214.666656f, 203.333328f, 192.0f, 192.0f, + 195.666672f, 195.666672f, 149.222214f, 102.777779f, 56.3333321f, 78.4444427f, + 100.555557f, 122.666672f, 107.999992f, 93.3333359f, 78.6666794f, 68.5555573f, + 58.4444466f, 48.3333359f, 91.5555649f, 134.777817f, 178.0f, 190.444443f, + 202.888901f, 215.333328f, 210.888885f, 206.444443f, 202.0f, 202.0f, + 152.333328f, 152.333328f, 114.777763f, 77.2222137f, 39.6666641f, 69.8888855f, + 100.111115f, 130.333328f, 139.000015f, 147.666672f, 156.333359f, 125.111107f, + 93.8888931f, 62.6666679f, 77.7777863f, 92.8888931f, 107.999985f, 140.222229f, + 172.444473f, 204.666656f, 207.111099f, 209.555557f, 212.0f, 212.0f, + 109.0f, 109.0f, 80.3333282f, 51.6666641f, 23.0f, 61.3333359f, + 99.6666718f, 138.0f, 170.000015f, 202.0f, 234.0f, 181.666641f, + 129.333328f, 77.0f, 63.9999924f, 50.9999886f, 38.0f, 90.0000229f, + 142.000046f, 194.0f, 203.333344f, 212.666687f, 222.0f, 222.0f, + 121.000008f, 121.000008f, 105.888893f, 90.7777939f, 75.6666946f, 99.0000229f, + 122.333351f, 145.666672f, 157.333328f, 168.999985f, 180.666641f, 159.555527f, + 138.444443f, 117.333351f, 111.55558f, 105.777802f, 100.000031f, 134.444473f, + 168.888931f, 203.333328f, 192.999985f, 182.666641f, 172.333313f, 172.333313f, + 133.0f, 133.0f, 131.444458f, 129.888901f, 128.333344f, 136.666672f, + 145.0f, 153.333328f, 144.666656f, 135.999985f, 127.333321f, 137.444443f, + 147.555557f, 157.666672f, 159.111115f, 160.555573f, 162.000015f, 178.888901f, + 195.777802f, 212.666672f, 182.666641f, 152.666626f, 122.666656f, 122.666656f, + 145.0f, 145.0f, 157.0f, 169.0f, 181.0f, 174.333328f, + 167.666656f, 161.0f, 131.999985f, 102.999992f, 74.0f, 115.333351f, + 156.666672f, 198.0f, 206.666672f, 215.333344f, 224.0f, 223.333344f, + 222.666672f, 222.0f, 172.333313f, 122.666618f, 73.0f, 73.0f, + 117.333321f, 117.333328f, 141.444443f, 165.555557f, 189.666672f, 184.333344f, + 179.0f, 173.666672f, 150.222214f, 126.777786f, 103.333344f, 128.444458f, + 153.555557f, 178.666656f, 178.333313f, 177.999969f, 177.666641f, 179.88887f, + 182.111099f, 184.333313f, 161.333313f, 138.333313f, 115.333359f, 115.333351f, + 89.6666565f, 89.6666641f, 125.888885f, 162.111115f, 198.333344f, 194.333328f, + 190.333328f, 186.333328f, 168.444427f, 150.555557f, 132.666672f, 141.555557f, + 150.444443f, 159.333328f, 149.999985f, 140.666641f, 131.333328f, 136.444443f, + 141.555557f, 146.666656f, 150.333328f, 154.0f, 157.666672f, 157.666672f, + 62.0f, 62.0000038f, 110.333336f, 158.666672f, 207.0f, 204.333328f, + 201.666672f, 199.0f, 186.666656f, 174.333328f, 162.0f, 154.666656f, + 147.333328f, 140.0f, 121.666656f, 103.333313f, 85.0f, 93.0000076f, + 101.000008f, 109.0f, 139.333359f, 169.666702f, 200.0f, 200.0f, + 47.6666603f, 47.6666603f, 79.6666565f, 111.666649f, 143.666641f, 140.77774f, + 137.888855f, 134.999969f, 133.444427f, 131.88887f, 130.333328f, 123.444427f, + 116.555534f, 109.666656f, 115.444443f, 121.222237f, 127.000023f, 135.000015f, + 143.000031f, 151.000015f, 172.888916f, 194.777802f, 216.666672f, 216.666672f, + 33.3333206f, 33.3333206f, 48.9999695f, 64.666626f, 80.3332672f, 77.2221603f, + 74.1110458f, 70.999939f, 80.2221756f, 89.4444046f, 98.6666336f, 92.2221909f, + 85.7777481f, 79.3333054f, 109.222229f, 139.11116f, 169.000046f, 177.000046f, + 185.000046f, 193.000046f, 206.444473f, 219.888931f, 233.333344f, 233.333344f, + 19.0f, 19.0f, 18.333334f, 17.666666f, 17.0f, 13.666666f, + 10.3333321f, 7.0f, 27.0000095f, 47.0000038f, 67.0f, 60.9999962f, + 55.0f, 49.0f, 103.000023f, 157.000046f, 211.0f, 219.0f, + 227.000015f, 235.0f, 240.0f, 245.0f, 250.0f, 250.0f, + 34.0000076f, 34.0000076f, 37.2222328f, 40.4444542f, 43.6666794f, 44.3333473f, + 45.0000153f, 45.666687f, 69.888916f, 94.1111374f, 118.333359f, 113.888916f, + 109.444466f, 105.000031f, 124.111122f, 143.222229f, 162.333298f, 160.444412f, + 158.555511f, 156.666626f, 185.555542f, 214.444458f, 243.333328f, 243.333313f, + 49.0000153f, 49.0000153f, 56.1111298f, 63.2222443f, 70.3333588f, 75.0000305f, + 79.6667023f, 84.333374f, 112.777832f, 141.222275f, 169.666718f, 166.777832f, + 163.888931f, 161.000046f, 145.222229f, 129.444427f, 113.666618f, 101.888824f, + 90.1110306f, 78.3332596f, 131.111084f, 183.888901f, 236.666656f, 236.666656f, + 64.0f, 64.0f, 75.0f, 86.0f, 97.0f, 105.666664f, + 114.333336f, 123.0f, 155.666687f, 188.333328f, 221.0f, 219.666672f, + 218.333344f, 217.0f, 166.333313f, 115.666618f, 65.0f, 43.3333244f, + 21.6666451f, 0.0f, 76.6667023f, 153.333405f, 230.0f, 230.0f, + 86.0f, 86.0f, 138.000015f, 190.000015f, 242.0f, 235.0f, + 227.999985f, 221.0f, 161.999969f, 102.999985f, 44.0f, 111.333366f, + 178.666672f, 246.0f, 174.666626f, 103.333267f, 32.0f, 77.666687f, + 123.333374f, 169.0f, 187.666672f, 206.333359f, 225.0f, 225.0f, + 124.666672f, 124.666672f, 152.555557f, 180.444443f, 208.333328f, 195.222214f, + 182.111099f, 169.0f, 131.888855f, 94.7777634f, 57.6666679f, 115.000023f, + 172.333344f, 229.666656f, 178.333313f, 126.999954f, 75.6666718f, 92.8889008f, + 110.111137f, 127.333328f, 155.000015f, 182.666687f, 210.333328f, 210.333328f, + 163.333344f, 163.333344f, 167.111115f, 170.888885f, 174.666656f, 155.444427f, + 136.222198f, 116.999985f, 101.777763f, 86.5555496f, 71.3333359f, 118.666687f, + 166.000015f, 213.333328f, 181.999985f, 150.666641f, 119.333344f, 108.111115f, + 96.8888702f, 85.6666565f, 122.333344f, 159.000031f, 195.666672f, 195.666656f, + 202.0f, 202.0f, 181.666656f, 161.333328f, 141.0f, 115.666664f, + 90.3333282f, 65.0f, 71.6666718f, 78.3333359f, 85.0f, 122.333351f, + 159.666672f, 197.0f, 185.666656f, 174.333328f, 163.0f, 123.333313f, + 83.666626f, 44.0f, 89.666687f, 135.333374f, 181.0f, 181.0f, + 156.999985f, 156.999985f, 152.222198f, 147.444443f, 142.666672f, 126.111115f, + 109.555565f, 93.0000153f, 105.888908f, 118.777802f, 131.666687f, 152.000015f, + 172.333344f, 192.666656f, 174.777756f, 156.888855f, 138.999985f, 117.777771f, + 96.5555344f, 75.3333511f, 91.222229f, 107.111107f, 122.999977f, 122.999969f, + 111.999992f, 111.999992f, 122.777771f, 133.555557f, 144.333328f, 136.555557f, + 128.777771f, 121.000008f, 140.11113f, 159.222229f, 178.333344f, 181.666672f, + 185.0f, 188.333328f, 163.88887f, 139.444412f, 114.999992f, 112.222221f, + 109.444443f, 106.666672f, 92.777771f, 78.8888626f, 64.9999847f, 64.9999847f, + 67.0f, 67.0f, 93.3333359f, 119.666664f, 146.0f, 147.0f, + 148.0f, 149.0f, 174.333344f, 199.666672f, 225.0f, 211.333328f, + 197.666656f, 184.0f, 152.999985f, 121.999969f, 91.0f, 106.666672f, + 122.333351f, 138.0f, 94.333313f, 50.666626f, 7.0f, 7.0f, + 79.6666718f, 79.6666718f, 99.7777786f, 119.888893f, 140.0f, 146.111115f, + 152.222229f, 158.333344f, 180.444458f, 202.555557f, 224.666672f, 194.555527f, + 164.444427f, 134.333313f, 116.888863f, 99.4444199f, 82.0f, 90.3333359f, + 98.6666565f, 106.999985f, 74.5555344f, 42.1110725f, 9.66666794f, 9.66666794f, + 92.3333359f, 92.3333359f, 106.222221f, 120.111115f, 134.0f, 145.222229f, + 156.444443f, 167.666672f, 186.555557f, 205.444443f, 224.333344f, 177.777756f, + 131.222198f, 84.6666565f, 80.777771f, 76.8888855f, 73.0f, 74.0f, + 74.9999924f, 75.9999924f, 54.7777634f, 33.5555305f, 12.333334f, 12.333334f, + 105.0f, 105.0f, 112.666664f, 120.333336f, 128.0f, 144.333344f, + 160.666672f, 177.0f, 192.666672f, 208.333328f, 224.0f, 160.999969f, + 97.9999847f, 35.0f, 44.6666718f, 54.3333435f, 64.0f, 57.6666641f, + 51.3333282f, 45.0f, 34.9999962f, 24.9999905f, 15.0f, 15.0f, + 148.666687f, 148.666687f, 134.666672f, 120.666672f, 106.666656f, 110.444427f, + 114.222198f, 117.999969f, 131.666641f, 145.333298f, 158.999969f, 116.666626f, + 74.3333054f, 32.0f, 54.3333473f, 76.6666946f, 99.0000153f, 84.7777863f, + 70.5555496f, 56.3333397f, 49.7777824f, 43.2222252f, 36.6666794f, 36.6666756f, + 192.333374f, 192.333374f, 156.666687f, 121.0f, 85.333313f, 76.5555191f, + 67.7777252f, 58.9999428f, 70.6666107f, 82.3332672f, 93.999939f, 72.3332825f, + 50.6666374f, 28.9999962f, 64.0000229f, 99.0000534f, 134.000031f, 111.888901f, + 89.777771f, 67.6666794f, 64.5555649f, 61.444458f, 58.3333549f, 58.3333511f, + 236.0f, 236.0f, 178.666656f, 121.333328f, 64.0f, 42.6666641f, + 21.3333282f, 0.0f, 9.66667175f, 19.3333359f, 29.0f, 28.0f, + 27.0f, 26.0f, 73.666687f, 121.333374f, 169.0f, 138.999985f, + 108.999969f, 79.0f, 79.3333359f, 79.6666641f, 80.0f, 80.0f, + 187.999985f, 187.999985f, 155.444427f, 122.888885f, 90.3333435f, 66.4444504f, + 42.5555611f, 18.6666756f, 43.1111374f, 67.5555878f, 92.0000305f, 94.5555878f, + 97.111145f, 99.6667023f, 108.444466f, 117.222237f, 125.999985f, 102.666641f, + 79.3332901f, 55.9999886f, 61.777771f, 67.5555573f, 73.3333282f, 73.3333282f, + 139.999954f, 139.999954f, 132.222198f, 124.444443f, 116.666695f, 90.2222366f, + 63.7777901f, 37.3333511f, 76.555603f, 115.777832f, 155.000061f, 161.111176f, + 167.22229f, 173.333405f, 143.222244f, 113.111076f, 82.9999542f, 66.3332825f, + 49.666626f, 32.9999771f, 44.2222099f, 55.4444427f, 66.6666565f, 66.6666565f, + 92.0f, 92.0f, 109.0f, 126.0f, 143.0f, 113.999992f, + 84.9999924f, 56.0f, 110.000031f, 164.0f, 218.0f, 227.666672f, + 237.333344f, 247.0f, 177.999969f, 108.999931f, 40.0f, 29.9999962f, + 19.9999905f, 10.0f, 26.6666756f, 43.3333511f, 60.0f, 60.0f, +}; + +// case 4: 10x7 -> 16x16 +inline constexpr int64_t kLtx2ImgResize4SrcH = 10; +inline constexpr int64_t kLtx2ImgResize4SrcW = 7; +inline constexpr int64_t kLtx2ImgResize4DstH = 16; +inline constexpr int64_t kLtx2ImgResize4DstW = 16; + +inline constexpr float kLtx2ImgResize4Golden[] = { + 143.956512f, 156.358017f, 191.082184f, 219.076096f, 159.57608f, 100.07608f, + 103.811142f, 124.792122f, 157.119553f, 200.793472f, 228.703125f, 198.810455f, + 168.917816f, 129.575409f, 89.5061188f, 75.1956558f, 123.086967f, 130.393341f, + 150.851242f, 166.847839f, 124.847832f, 82.8478317f, 110.658966f, 157.509506f, + 188.885864f, 204.78804f, 208.390625f, 166.894714f, 125.398788f, 104.833572f, + 85.8783951f, 79.108696f, 102.2174f, 104.428673f, 110.620255f, 114.619583f, + 90.1195755f, 65.6195679f, 117.506798f, 190.226898f, 220.652161f, 208.782608f, + 188.07814f, 134.978943f, 81.8797684f, 80.091713f, 82.250679f, 83.0217361f, + 107.434776f, 108.627037f, 111.965347f, 114.585587f, 107.870911f, 101.156219f, + 136.939529f, 184.313202f, 204.442963f, 197.328812f, 182.612762f, 140.023087f, + 97.433403f, 90.6997147f, 86.7241898f, 85.3043442f, 121.347824f, 122.879753f, + 127.169159f, 131.949722f, 143.115479f, 154.28125f, 160.567261f, 165.522415f, + 172.241867f, 180.725555f, 182.096466f, 157.387222f, 132.677979f, 113.091034f, + 93.8980942f, 87.0434799f, 105.326111f, 113.617538f, 136.833557f, 159.689514f, + 177.864792f, 196.04007f, 175.677994f, 144.80571f, 139.056396f, 158.430038f, + 171.620255f, 162.137924f, 152.655579f, 136.558411f, 119.952431f, 114.021721f, + 52.717392f, 75.6148148f, 139.727585f, 200.110733f, 212.008835f, 223.906921f, + 180.379074f, 121.735054f, 104.667801f, 129.177307f, 148.970795f, 151.472153f, + 153.973511f, 161.341034f, 169.08287f, 171.847824f, 17.7173347f, 51.2873154f, + 145.283264f, 232.907608f, 237.701111f, 242.494614f, 179.771088f, 98.6338577f, + 73.0020599f, 102.875702f, 129.22554f, 142.654892f, 156.084229f, 177.223541f, + 198.955887f, 206.717468f, 82.5f, 104.453125f, 165.921875f, 222.5f, + 215.5f, 208.5f, 149.078125f, 75.359375f, 56.765625f, 93.296875f, + 125.9375f, 144.3125f, 162.6875f, 142.671875f, 119.703125f, 111.5f, + 147.282593f, 157.618881f, 186.560455f, 212.092407f, 193.29892f, 174.505447f, + 118.385193f, 52.0849228f, 40.5292168f, 83.718071f, 122.649452f, 145.970108f, + 169.290756f, 108.120255f, 40.450428f, 16.2826309f, 173.260864f, 167.418472f, + 151.059799f, 135.186172f, 125.61824f, 116.050301f, 93.9055862f, 68.3308334f, + 61.7112617f, 74.0468826f, 88.410347f, 110.209259f, 132.008179f, 113.296867f, + 91.4693909f, 83.6738663f, 192.391296f, 169.565216f, 105.652206f, 46.5449028f, + 49.9117393f, 53.2785759f, 70.522438f, 91.5509338f, 89.4965897f, 64.3593826f, + 48.7092667f, 67.8451385f, 86.9810104f, 125.484367f, 165.477524f, 179.760803f, + 157.717468f, 137.374374f, 80.4137268f, 28.2526798f, 38.4863739f, 48.7200661f, + 69.4456329f, 93.0326309f, 99.1528625f, 87.8063354f, 80.5379562f, 88.2227554f, + 95.9075623f, 133.681381f, 173.769745f, 188.087021f, 79.0217438f, 77.0890045f, + 71.6773148f, 67.5190201f, 79.6548843f, 91.7907562f, 86.6195679f, 76.7282639f, + 93.9762268f, 138.363449f, 170.891296f, 159.934784f, 148.978256f, 138.613449f, + 128.294174f, 124.608704f, 37.9347572f, 48.0944099f, 76.5414276f, 103.558411f, + 111.985054f, 120.41169f, 103.647415f, 80.012886f, 105.297539f, 179.501358f, + 232.297546f, 206.599197f, 180.900833f, 142.281921f, 102.669151f, 88.5217209f, + 109.673874f, 112.972115f, 122.207176f, 129.917114f, 117.80027f, 105.683434f, + 120.237083f, 142.064499f, 166.112732f, 192.38179f, 206.862793f, 178.120941f, + 149.379089f, 142.159653f, 136.595764f, 134.608673f, 86.2608643f, 105.734375f, + 160.260193f, 208.403534f, 173.574722f, 138.745926f, 131.581512f, 131.96196f, + 152.158279f, 192.170517f, 217.953125f, 191.560455f, 165.167801f, 147.95993f, + 131.458557f, 125.565216f, 131.478256f, 134.17186f, 141.713989f, 148.974213f, + 152.56929f, 156.164398f, 142.586945f, 124.326088f, 124.074059f, 141.830856f, + 158.89064f, 173.394699f, 187.898773f, 184.033279f, 178.754745f, 176.869553f, + 176.695648f, 162.609375f, 123.167801f, 89.5448532f, 131.563858f, 173.582886f, + 153.592392f, 116.690224f, 95.9898148f, 91.4911804f, 99.8281403f, 155.228943f, + 210.629761f, 220.106644f, 226.050934f, 228.173904f, 187.021744f, 173.529892f, + 135.752701f, 104.025116f, 150.942245f, 197.85936f, 176.82608f, 137.260864f, + 118.010857f, 119.076065f, 127.280548f, 161.662369f, 196.044159f, 195.232346f, + 191.713348f, 190.456543f, 185.717392f, 178.611404f, 158.714676f, 143.141968f, + 183.781921f, 224.421875f, 204.135864f, 167.233688f, 156.733688f, 172.635864f, + 183.571335f, 176.295517f, 169.019699f, 150.04213f, 130.164413f, 123.065231f, + 169.739151f, 169.508163f, 168.86142f, 170.544159f, 202.510193f, 234.476227f, + 216.508835f, 184.923233f, 178.893356f, 198.419159f, 208.869568f, 186.043488f, + 163.217377f, 129.290085f, 94.5088348f, 82.0869598f, 135.82608f, 143.067932f, + 163.345108f, 183.628403f, 203.991165f, 224.353943f, 210.625671f, 187.599869f, + 180.809113f, 190.253403f, 196.288055f, 189.820648f, 183.353271f, 138.406921f, + 90.500679f, 73.391304f, 103.543503f, 116.648117f, 153.341049f, 188.134537f, + 198.235077f, 208.335632f, 198.147461f, 182.42601f, 174.931442f, 175.663757f, + 178.719467f, 190.294174f, 201.868881f, 147.798889f, 88.6793289f, 67.5652008f, + 80.5f, 90.34375f, 117.90625f, 144.03125f, 151.46875f, 158.90625f, + 148.296875f, 132.765625f, 124.890625f, 124.671875f, 132.890625f, 172.046875f, + 211.203125f, 158.75f, 99.25f, 78.0f, 57.4565277f, 64.0394135f, + 82.4714737f, 99.9280014f, 104.702461f, 109.476913f, 98.4463425f, 83.1053085f, + 74.8498764f, 73.6800385f, 87.0618286f, 153.799591f, 220.537369f, 169.70108f, + 109.820648f, 88.4347839f, 71.3695526f, 73.9408875f, 81.140625f, 87.907608f, + 89.0489197f, 90.1902237f, 89.201767f, 87.6324615f, 96.7010651f, 116.407578f, + 140.530533f, 180.847137f, 221.163727f, 176.044159f, 124.35257f, 105.891296f, + 91.8043365f, 90.2316437f, 85.828125f, 81.5489197f, 78.8858795f, 76.2228394f, + 87.122963f, 101.722137f, 131.239105f, 175.673874f, 211.522385f, 215.887878f, + 220.253387f, 181.574036f, 139.583542f, 124.586945f, 78.5217819f, 77.3634872f, + 74.1202698f, 70.7439041f, 65.6365585f, 60.529232f, 88.377037f, 125.212608f, + 157.444962f, 185.074081f, 204.206589f, 192.184845f, 180.163116f, 158.841751f, + 136.805038f, 128.934784f, 37.6521797f, 40.6379128f, 48.9979668f, 56.3009529f, + 49.8620911f, 43.423233f, 92.3580093f, 156.394699f, 176.833557f, 153.674591f, + 132.820663f, 120.418488f, 108.016312f, 112.985741f, 119.291443f, 121.54348f, + 9.39130116f, 15.3152142f, 31.9021721f, 47.3097801f, 47.3858566f, 47.4619331f, + 102.986404f, 173.633133f, 182.540085f, 129.707199f, 86.3022995f, 77.4666977f, + 68.6310959f, 83.4375f, 100.062508f, 106.000015f, 18.9565163f, 24.2010822f, + 38.8858643f, 54.6739082f, 84.8043289f, 114.934746f, 133.557053f, 149.040787f, + 147.200439f, 128.035995f, 114.386528f, 120.958534f, 127.53054f, 102.812492f, + 75.6875153f, 66.0000229f, 171.195648f, 149.504089f, 88.767662f, 38.4014969f, + 122.848503f, 207.295532f, 171.400131f, 102.684105f, 66.390625f, 62.5196991f, + 74.4945602f, 144.570648f, 214.646759f, 194.48233f, 167.376373f, 157.695663f, + 115.108711f, 109.585609f, 94.1209259f, 82.6813736f, 123.568611f, 164.455856f, + 132.527863f, 80.7411728f, 53.5781288f, 51.0387268f, 56.8858719f, 93.4837036f, + 130.081543f, 124.129097f, 114.903557f, 111.608704f, 59.0217514f, 69.6671295f, + 99.4741821f, 126.961266f, 124.288719f, 121.61618f, 93.6555786f, 58.7982368f, + 40.765625f, 39.5577469f, 39.2771797f, 42.3967514f, 45.5163231f, 53.7758293f, + 62.4307175f, 65.5217514f, 106.630394f, 107.764915f, 110.941574f, 115.062515f, + 131.459244f, 147.855957f, 111.389931f, 60.506794f, 41.546875f, 54.5101814f, + 60.2995796f, 39.7846413f, 19.2697029f, 33.9660339f, 51.3709221f, 57.5869522f, + 188.804337f, 171.868195f, 124.447014f, 84.4375076f, 140.779892f, 197.122284f, + 147.993195f, 70.0991821f, 46.859375f, 78.2737732f, 94.1990509f, 53.3308411f, + 12.4626369f, 31.0040779f, 54.1154861f, 62.3695641f, 191.34787f, 173.827454f, + 124.770378f, 83.6005249f, 144.964676f, 206.328812f, 154.20314f, 71.1256866f, + 45.8743248f, 78.4490509f, 98.3716125f, 71.9028397f, 45.4340706f, 52.8960381f, + 62.9680634f, 66.5652161f, 96.5652313f, 99.8328934f, 108.982338f, 119.171196f, + 142.872284f, 166.573364f, 123.265625f, 61.6827431f, 37.1922531f, 49.7941628f, + 66.2112808f, 96.6175308f, 127.023773f, 105.165077f, 79.2860031f, 70.0434799f, + 24.8042984f, 43.958519f, 97.590332f, 146.956528f, 140.869568f, 134.782608f, + 100.343735f, 58.1725311f, 35.6555367f, 32.7927551f, 46.5502319f, 121.248619f, + 195.947006f, 147.949051f, 90.5129166f, 70.0000153f, 83.5f, 90.765625f, + 111.109375f, 130.625f, 139.375f, 148.125f, 122.84375f, 88.28125f, + 74.609375f, 81.828125f, 97.71875f, 145.40625f, 193.09375f, 136.984375f, + 72.890625f, 50.0f, 142.195633f, 137.572678f, 124.628395f, 114.293488f, + 137.880432f, 161.467392f, 145.343735f, 118.389938f, 113.563171f, 130.863434f, + 148.887207f, 169.563858f, 190.240494f, 126.019707f, 55.2683449f, 30.0000057f, + 168.369553f, 163.695633f, 150.608688f, 140.103912f, 163.167786f, 186.231644f, + 167.208557f, 136.707184f, 130.943604f, 149.917801f, 169.185455f, 189.529205f, + 209.872955f, 137.944962f, 58.9191513f, 30.6956501f, 188.804337f, 186.168457f, + 178.788025f, 173.351196f, 193.181366f, 213.011536f, 188.961258f, 152.943604f, + 144.516968f, 163.681381f, 184.035995f, 208.754745f, 233.47348f, 153.909637f, + 66.3240433f, 35.0434761f, 187.000015f, 181.021774f, 164.282669f, 149.991913f, + 167.529968f, 185.068008f, 175.262955f, 158.000671f, 156.902161f, 171.967392f, + 186.384521f, 198.425308f, 210.466095f, 140.603302f, 64.4402237f, 37.2391319f, + 167.0f, 153.277176f, 114.853264f, 80.3179398f, 96.3342514f, 112.350548f, + 132.559113f, 153.910995f, 168.315216f, 175.771744f, 178.504074f, 163.914398f, + 149.324738f, 103.347832f, 54.9565277f, 37.673912f, 152.76088f, 136.150818f, + 89.6426468f, 47.0128822f, 54.8022919f, 62.5917053f, 97.0292206f, 138.73439f, + 162.245255f, 167.561829f, 165.627045f, 137.103943f, 108.580841f, 85.1005325f, + 62.0081367f, 53.760849f, 155.804352f, 150.879059f, 137.088287f, 122.814499f, + 102.26152f, 81.7085495f, 83.0210724f, 90.2969055f, 103.726265f, 123.309135f, + 137.760208f, 133.394714f, 129.029205f, 123.877693f, 118.665733f, 116.804314f, +}; + +// --- section 2: load_image_and_preprocess at crf=0 (decode.py:46-79) --- +// decode -> preprocess(crf=0) -> f32 0..255 -> resize+crop -> /127.5 - 1. +// `preprocess(crf=0) is image` was ASSERTED at generation time, and +// `preprocess(crf=18)` was asserted to reach the codec and raise. +inline constexpr int64_t kLtx2ImgPreSrcH = 12; +inline constexpr int64_t kLtx2ImgPreSrcW = 20; +inline constexpr int64_t kLtx2ImgPreDstH = 16; +inline constexpr int64_t kLtx2ImgPreDstW = 16; +// max|resize-then-normalize - normalize-then-resize|, measured upstream. It is +// BELOW the golden band, which is the recorded NEGATIVE RESULT: the two orders +// are algebraically identical (a convex combination commutes with an affine +// map), so their f32 gap is pure rounding and cannot be amplified. No golden +// here can see the swap; the order is mirrored because it is upstream's, and +// that is written down rather than assumed to be covered. +inline constexpr double kLtx2ImgPreOrderGap = 1.9371509552001953e-07; + +inline constexpr float kLtx2ImgPreGolden[] = { + -0.444880188f, -0.359767616f, -0.324618816f, 0.000726103783f, 0.00595498085f, 0.0880173445f, + 0.499055862f, 0.853449583f, 0.533333302f, 0.0262889862f, -0.194480777f, 0.179520249f, + -0.27726835f, -0.92374748f, -0.732026339f, -0.0727675557f, -0.360275924f, -0.459803998f, + -0.319172025f, -0.378721833f, -0.495134354f, -0.419880211f, -0.000127077103f, -0.00521057844f, + 0.097058773f, 0.142211318f, -0.0386165977f, -0.138017297f, 0.0238376856f, 0.134895086f, + -0.297929823f, -0.1120736f, -0.357661545f, -0.629357338f, -0.436329007f, -0.254048645f, + -0.480864167f, -0.656154752f, -0.490867853f, -0.478522122f, -0.108823538f, 0.288725495f, + 0.386764765f, -0.0412849188f, 0.0200794935f, 0.187073708f, -0.162962615f, -0.216757417f, + -0.344880104f, -0.723747313f, -0.575327039f, 0.32881248f, 0.0119827986f, -0.552832246f, + -0.743445992f, -0.324419022f, -0.0352941155f, 0.282625318f, 0.790250659f, 0.44466269f, + -0.151560545f, -0.581282139f, -0.210185647f, -0.264306366f, 0.213289738f, -0.19477123f, + -0.205065489f, 0.302559733f, -0.180174172f, -0.617538154f, -0.346605003f, 0.181354403f, + -0.152941167f, -0.523583889f, -0.10746187f, 0.48779881f, 0.662817836f, 0.638652802f, + 0.761383295f, 0.26140213f, -0.0770334005f, -0.0485113263f, 0.221568465f, 0.365359426f, + -0.203213394f, -0.722821355f, -0.69431746f, -0.0356391072f, -0.233823538f, -0.560639024f, + -0.297748685f, 0.344661593f, 0.230646968f, -0.119989336f, 0.117483497f, -0.263725042f, + -0.438144505f, -0.274818599f, 0.19689548f, 0.207788706f, 0.0449346304f, -0.270697057f, + -0.703413248f, -0.0556282401f, 0.0823529959f, -0.10230577f, -0.447258592f, 0.156589627f, + -0.125725329f, -0.604248822f, -0.159804344f, -0.140577376f, -0.537291229f, -0.514560819f, + -0.104684174f, -0.0494916439f, 0.411637545f, 0.487254858f, -0.305664539f, 0.0952068567f, + 0.543627501f, 0.48932457f, -0.656645f, -0.0636172891f, -0.192156255f, -0.439488649f, + 0.211927295f, 0.555010557f, 0.178830743f, 0.658641934f, 0.522113442f, 0.385584712f, + 0.184295654f, -0.0532679558f, -0.311328948f, -0.607625246f, -0.218137264f, 0.0396513939f, + -0.661873698f, -0.406100512f, -0.434422433f, -0.380120635f, 0.380227923f, -0.179411173f, + -0.138725579f, 0.280827761f, 0.490413904f, 0.597893953f, 0.114633322f, -0.193736434f, + 0.151597738f, 0.0586420298f, -0.0509803891f, -0.152578056f, -0.220842421f, -0.389651179f, + -0.414433599f, -0.330283344f, -0.190849781f, -0.334222794f, -0.131535947f, 0.0184639692f, + 0.203485847f, 0.171532393f, -0.154211998f, -0.240631878f, 0.249854803f, 0.440123439f, + 0.115196109f, -0.222240329f, -0.070461154f, -0.295370102f, -0.375163436f, -0.2901963f, + -0.0156865716f, 0.072748661f, 0.197694421f, 0.0915213823f, -0.010076046f, -0.549654901f, + -0.526398003f, -0.287745178f, -0.0463508368f, 0.17805016f, 0.0671569109f, -0.157262206f, + -0.282171369f, -0.118191957f, -0.207334518f, -0.152433753f, 0.694334626f, 0.707461834f, + -0.520279586f, -0.286583602f, 0.760838866f, 0.225617409f, -0.261691988f, -0.403758228f, + -0.0229302645f, -0.508769035f, -0.11813724f, 0.225526452f, -0.404829383f, 0.443245173f, + 0.769135833f, 0.512927651f, -0.39193809f, 0.251578808f, -0.131227314f, 0.252704978f, + 0.837309599f, 0.132153273f, -0.123892486f, -0.191884518f, -0.225817025f, -0.0820261836f, + -0.139215708f, -0.124891043f, 0.238943458f, 0.636764169f, 0.509404898f, 0.126725078f, + -0.0969496965f, 0.432406783f, -0.0303558111f, 0.0705699921f, 0.227015257f, 0.208133578f, + 0.00339508057f, -0.1610021f, -0.129466295f, 0.338943243f, -0.130392134f, -0.419880092f, + 0.601906419f, 0.648692608f, 0.160766959f, -0.27550894f, 0.230663896f, 0.208170056f, + -0.370951355f, -0.835294127f, -0.738126755f, 0.55744338f, 0.121859312f, -0.34074086f, + 0.237908483f, 0.499346375f, -0.125490189f, -0.497312963f, 0.542629004f, 0.568627477f, + 0.00217962265f, -0.448802531f, 0.300652862f, -0.422366619f, -0.0530138016f, 0.0644880533f, + -0.0958607197f, 0.560638905f, 0.273638487f, -0.201307178f, -0.385766208f, -0.287000775f, + -0.368627429f, -0.406826437f, -0.174437165f, 0.0409584045f, -0.179810643f, -0.459114313f, + -0.203486085f, -0.576615453f, 0.0853304863f, -0.318318725f, -0.508170187f, 0.337145805f, + 0.410076261f, 0.11214602f, -0.306699336f, -0.378594756f, -0.461764693f, -0.469644189f, + -0.309876502f, -0.213126421f, -0.0862202048f, -0.00395774841f, -0.0896512866f, 0.0523781776f, + -0.125508249f, -0.546859026f, -0.801797628f, 0.0892698765f, 0.440141678f, 0.293355227f, + -0.318409622f, -0.0351852179f, -0.300000012f, -0.608478606f, -0.459604204f, -0.233551443f, + -0.0799565315f, 0.0497455597f, 0.234204531f, 0.192955732f, -0.633097291f, -0.500490069f, + -0.691394508f, -0.0929921269f, 0.254793048f, 0.180991411f, -0.397984803f, 0.54537034f, + 0.0333333015f, -0.697494507f, -0.427341998f, -0.00849723816f, -0.204937696f, -0.383606553f, + 0.463888049f, -0.254465461f, -0.740177929f, -0.233714879f, 0.433442473f, -0.0487653017f, + -0.339324594f, -0.424564302f, -0.283932507f, 0.0450435877f, -0.143137276f, -0.220152497f, + 0.450871468f, 0.0215691328f, -0.455918133f, -0.789814889f, -0.713562131f, -0.570806265f, + -0.707098782f, -0.282389581f, 0.561002135f, 0.341684818f, -0.27830416f, -0.507516444f, + 0.141031265f, -0.256935358f, -0.162745118f, 0.133006454f, 0.355228782f, -0.115032136f, + -0.0420483947f, 0.218554974f, 0.0863835812f, 0.124836564f, -0.395279586f, 0.0624363422f, + 0.537309527f, 0.59032321f, 0.12578094f, -0.0838236213f, 0.430246949f, -0.163798094f, + -0.187745094f, 0.0259803534f, 0.163235307f, 0.00947737694f, 0.330282688f, 0.681681514f, + 0.351253033f, 0.157189727f, 0.0665214062f, 0.677995801f, 0.452396393f, 0.615069032f, + 0.600308776f, 0.54793036f, 0.538834572f, 0.288289785f, -0.155882359f, -0.364506125f, + 0.109713197f, 0.322657704f, 0.484422565f, 0.421187878f, -0.164868712f, -0.433187842f, + 0.0246913433f, 0.364270329f, -0.179629862f, 0.575635195f, 0.366866469f, 0.209368229f, + 0.733932376f, 0.757897615f, 0.344117641f, 0.00444805622f, 0.256445169f, 0.10697186f, + 0.420151949f, 0.683388352f, 0.0668307543f, 0.191212654f, -0.00624549389f, 0.0234932899f, + -0.378921747f, 0.202777743f, 0.35274148f, 0.290740728f, 0.161419749f, 0.539778352f, + 0.633333325f, 0.633660197f, 0.72516346f, -0.139214694f, 0.164250851f, 0.761946797f, + 0.249238014f, 0.245715261f, -0.113108218f, -0.204284668f, -0.239215791f, 0.24444437f, + 0.280573845f, 0.162472725f, 0.0806462765f, 0.40453887f, 0.478921533f, 0.530174255f, + 0.782897592f, -0.236763537f, -0.0245290399f, 0.511456609f, -0.0992913246f, -0.14039582f, + -0.132752359f, -0.206082106f, 0.0156316757f, 0.616212606f, 0.187999368f, -0.0838780999f, + 0.538035631f, 0.490831614f, -0.000980377197f, -0.29713136f, 0.25769794f, -0.112199962f, + -0.0541942716f, -0.0239825845f, -0.781426251f, -0.607153475f, 0.67421937f, 0.32997489f, + -0.416067719f, 0.232443571f, 0.502923012f, 0.547494531f, 0.459931016f, 0.582661629f, + -0.100980401f, -0.81978941f, -0.805265069f, 0.349236131f, 0.184478283f, -0.522567153f, + -0.526198268f, 0.156026959f, 0.427450895f, 0.68275249f, 0.23284328f, -0.0874181986f, + 0.302051425f, 0.578649282f, 0.222058892f, 0.00637257099f, -0.0490196347f, -0.045878768f, + -0.0814633369f, 0.609421968f, 0.328559399f, -0.252360463f, -0.0577344298f, 0.355210304f, + 0.081590414f, 0.486946106f, 0.634640813f, -0.289106548f, 0.106572032f, 0.419662356f, + -0.229357243f, -0.501688421f, -0.150490224f, 0.342084169f, 0.504756689f, 0.635566354f, + 0.209550738f, -0.284059763f, -0.076361835f, 0.269044757f, -0.113580227f, -0.102832496f, + 0.58344245f, -0.369353473f, 0.0341320038f, 0.228758216f, -0.75686276f, -0.75686276f, + -0.388235271f, 0.127668738f, 0.580827951f, 0.495860577f, -0.0930999517f, -0.690196276f, + -0.53333348f, 0.0534489155f, 0.159767509f, 0.777051568f, 0.448366284f, -0.423093498f, + 0.0111835003f, 0.583442211f, 0.56296289f, 0.650108933f, 0.835294127f, 0.852432847f, + 0.42251277f, 0.0832247734f, -0.0322440267f, 0.15773356f, 0.802613735f, 0.86172843f, + -0.382443786f, 0.0610927343f, 0.662690878f, -0.0308459401f, 0.110130668f, 0.46361649f, + 0.629593492f, 0.244698644f, 0.448039174f, 0.639106631f, 0.216448903f, -0.243027806f, + -0.311746776f, -0.192574441f, -0.197658002f, 0.0545386076f, -0.381898999f, -0.330737412f, + 0.411710501f, -0.177251041f, -0.00751644373f, 0.272440076f, 0.245007277f, -0.229212046f, + 0.0642156601f, 0.469989061f, 0.341448903f, 0.0401419401f, -0.0522695184f, -0.136310399f, + -0.535729408f, -0.494480848f, 0.122748733f, -0.289016008f, -0.211274326f, -0.6883986f, + -0.264669657f, 0.0715686083f, -0.301833689f, -0.453612924f, -0.135784328f, 0.363779902f, + 0.771187425f, 0.821023941f, 0.647912502f, 0.30379498f, -0.152995169f, -0.624563813f, + -0.108079076f, -0.416140199f, -0.153431416f, 0.0622549057f, -0.0374363661f, 0.059150219f, + 0.662763238f, 0.506626725f, 0.17892158f, 0.137745142f, 0.863235235f, 0.717102528f, + 0.577215075f, 0.279939055f, -0.47750473f, -0.758006334f, -0.446296275f, -0.41390723f, + 0.0816992521f, 0.151416183f, 0.0795025826f, 0.108006477f, 0.394153953f, 0.709331751f, + 0.515686274f, 0.197657943f, 0.13011992f, 0.648855567f, 0.541285992f, 0.181917191f, + 0.173202634f, 0.199419022f, -0.330501139f, -0.0123820901f, 0.276634097f, 0.278812647f, + 0.00352215767f, -0.192647099f, -0.0677197576f, 0.596042156f, 0.536274552f, 0.201307058f, + 0.0183006525f, 0.245261312f, 0.00615537167f, -0.182190239f, 0.517156124f, 0.350617528f, + 0.144262791f, 0.635693669f, 0.434695005f, 0.531281829f, -0.181898892f, -0.672821403f, + -0.273602128f, 0.42937541f, 0.278431416f, 0.0311909914f, 0.437872171f, -0.258604825f, + -0.714215398f, -0.658443332f, 0.319770336f, -0.322275996f, 0.0364196301f, 0.64941895f, + 0.786655903f, 0.556445241f, -0.060221374f, -0.241775692f, 0.533587575f, 0.818264365f, + 0.231372595f, -0.44048661f, -0.417247653f, 0.448582888f, 0.169990182f, -0.439161599f, + -0.075327158f, 0.00637245178f, -0.0913035274f, 0.214796424f, 0.806372881f, -0.128267765f, + -0.186274588f, 0.229302764f, 0.859567881f, 0.693264365f, 0.186764717f, -0.170334041f, + 0.123057365f, 0.606208563f, 0.48102808f, 0.102560043f, 0.0263072252f, 0.472003937f, + 0.170134425f, 0.224491477f, 0.516830325f, -0.59210217f, -0.333170116f, 0.186165452f, + 0.26839149f, 0.102087855f, -0.0323529243f, 0.00526511669f, 0.368373275f, 0.158660412f, + 0.0726941824f, 0.0777778625f, 0.103921413f, 0.684258699f, 0.527505398f, 0.515414f, + 0.0722224712f, -0.651089132f, -0.348765552f, -0.127450883f, -0.672567189f, -0.55564636f, + -0.28039217f, -0.0355846286f, -0.0123456717f, -0.578212857f, -0.713471472f, -0.464670181f, + 0.0916116238f, 0.498093367f, -0.601143777f, -0.457462072f, 0.0663400888f, -0.500108778f, + 0.148620009f, 0.817647099f, 0.742138743f, 0.270824194f, 0.201960802f, 0.0559185743f, + -0.652868629f, -0.28540349f, -0.365104914f, -0.440487266f, 0.246513486f, -0.276306629f, + -0.0828250051f, 0.0448800325f, 0.427015424f, -0.389251828f, 0.0781043768f, 0.512200475f, + 0.0886710882f, 0.0276688337f, 0.245588183f, 0.23928833f, -0.511619568f, 0.169171214f, + 0.365159988f, 0.244644284f, 0.114651561f, -0.481535316f, 0.140758872f, 0.19656837f, + 0.685839176f, -0.28729099f, -0.0620916486f, 0.188180804f, -0.26710242f, -0.323747277f, + -0.107352912f, 0.00448429585f, -0.373874366f, 0.290739894f, 0.535475612f, 0.448729396f, + 0.194553614f, -0.066339612f, -0.193028212f, -0.215105653f, 0.755555868f, -0.203049839f, + -0.142338455f, 0.111111164f, 0.0291939974f, -0.528540254f, -0.65882349f, -0.554974616f, + -0.392302155f, 0.0875811577f, 0.147131681f, 0.107770085f, 0.485402703f, 0.701815367f, +}; + +// --- section 3: VideoEncoder over the preprocessed image (video_vae.py:264-336) --- +// This is `video_encoder(image)` at ltx-pipelines/utils/helpers.py:294. +inline constexpr int64_t kLtx2ImgEncOutC = 4; +inline constexpr int64_t kLtx2ImgEncOutT = 1; +inline constexpr int64_t kLtx2ImgEncOutH = 2; +inline constexpr int64_t kLtx2ImgEncOutW = 2; +inline constexpr int64_t kLtx2ImgEncTemporalFactor = 2; +inline constexpr int64_t kLtx2ImgEncSpatialFactor = 8; + +inline constexpr const char* kLtx2ImgEncParamNames[] = { + "ltx2.imgenc.per_channel_statistics.std-of-means", + "ltx2.imgenc.per_channel_statistics.mean-of-means", + "ltx2.imgenc.conv_in.conv.weight", + "ltx2.imgenc.conv_in.conv.bias", + "ltx2.imgenc.down_blocks.0.res_blocks.0.conv1.conv.weight", + "ltx2.imgenc.down_blocks.0.res_blocks.0.conv1.conv.bias", + "ltx2.imgenc.down_blocks.0.res_blocks.0.conv2.conv.weight", + "ltx2.imgenc.down_blocks.0.res_blocks.0.conv2.conv.bias", + "ltx2.imgenc.down_blocks.1.conv.conv.weight", + "ltx2.imgenc.down_blocks.1.conv.conv.bias", + "ltx2.imgenc.down_blocks.2.conv.conv.weight", + "ltx2.imgenc.down_blocks.2.conv.conv.bias", + "ltx2.imgenc.conv_out.conv.weight", + "ltx2.imgenc.conv_out.conv.bias", +}; +inline constexpr int64_t kLtx2ImgEncParamCounts[] = { + 4, 4, 1296, 4, 432, 4, 432, 4, 216, 2, + 216, 1, 1080, 5, +}; + +inline constexpr float kLtx2ImgEncGolden[] = { + -0.241159096f, 1.22298455f, -0.218436688f, 1.04175878f, -0.0579345077f, -0.170286119f, + 0.36133939f, -0.979117632f, 0.290022969f, 0.0848331228f, -0.29933691f, -0.477867126f, + -0.224734351f, -0.0779257789f, -0.457862705f, -0.312173635f, +}; + +// --- section 4: VideoConditionByLatentIndex (latent_cond.py:22-43) --- +// clean_latent[start:stop] = tokens; denoise_mask[start:stop] = 1 - strength. +// The NOISY tensor is deliberately untouched — that was asserted here. +inline constexpr int64_t kLtx2ImgCondTokens = 12; +inline constexpr int64_t kLtx2ImgCondWidth = 4; +inline constexpr int64_t kLtx2ImgCondTargetFrames = 3; +inline constexpr int64_t kLtx2ImgCondPatch = 1; +inline constexpr int64_t kLtx2ImgCondLatentIdx = 0; +inline constexpr double kLtx2ImgCondStrength = 0.69999999999999996; +inline constexpr double kLtx2ImgCondFps = 8.0; + +inline constexpr float kLtx2ImgCondClean[] = { + -0.241159096f, -0.0579345077f, 0.290022969f, -0.224734351f, 1.22298455f, -0.170286119f, + 0.0848331228f, -0.0779257789f, -0.218436688f, 0.36133939f, -0.29933691f, -0.457862705f, + 1.04175878f, -0.979117632f, -0.477867126f, -0.312173635f, 0.0f, 0.0f, + 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, + 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, + 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, + 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, + 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, +}; + +inline constexpr float kLtx2ImgCondMask[] = { + 0.300000012f, 0.300000012f, 0.300000012f, 0.300000012f, 1.0f, 1.0f, + 1.0f, 1.0f, 1.0f, 1.0f, 1.0f, 1.0f, +}; + +// --- section 5: GaussianNoiser over the conditioned state (components/noisers.py:30-37) --- +// latent = lerp(latent, noise, noise_scale); latent = lerp(clean, latent, mask). +// The DOUBLE lerp, at a NON-UNIT scale — the only regime in which ltx_core and +// diffusers disagree. `kLtx2ImgNoiseDivergence` is how far apart they are here, +// measured; the suite asserts it is above the band, which is what makes this +// section able to catch a silent switch to the diffusers form. +inline constexpr double kLtx2ImgNoiseScale = 0.625; +inline constexpr double kLtx2ImgNoiseDivergence = 0.87261331081390381; + +inline constexpr float kLtx2ImgNoisedGolden[] = { + -0.338830173f, -0.151256427f, 0.267070055f, -0.00758534716f, 0.86769706f, 0.048100654f, + 0.112138331f, -0.0327227563f, -0.0503813364f, 0.399275422f, -0.103254408f, -0.272276074f, + 0.901173115f, -0.605140507f, -0.402147412f, -0.251273006f, -0.0780501515f, 0.290990531f, + 0.508082986f, -0.0305619389f, 0.587033451f, 0.297063053f, -0.0872664228f, -0.419992477f, + 0.184497401f, -0.302416623f, -0.286860138f, -0.430119991f, 0.191023424f, -0.0793055296f, + -0.0607184321f, 0.51918453f, 0.618465066f, -0.31636554f, 0.57936877f, 0.231961042f, + -0.445122898f, 0.183641195f, -0.206588715f, 0.441616356f, 0.00255125063f, 0.142263561f, + 0.613677382f, 0.518635511f, 0.348936856f, -0.106226586f, 0.488095999f, 0.14958632f, +}; + +} // namespace vllm_test diff --git a/tests/vllm/multimodal/ltx2_video_fixture.h b/tests/vllm/multimodal/ltx2_video_fixture.h index ec3fca7af..f991db459 100644 --- a/tests/vllm/multimodal/ltx2_video_fixture.h +++ b/tests/vllm/multimodal/ltx2_video_fixture.h @@ -628,6 +628,55 @@ inline void WriteReducedVideoVae(const vllm::Ltx2ConvVideoDecoderConfig& cfg, put("conv_out.conv.weight", {patch_out, channels, 3, 3, 3}, 0.1); put("conv_out.conv.bias", {patch_out}, 0.05); + // ── the ENCODER half (row LTX25-IMAGE-COND, issue #644) ────────────────── + // + // A monolithic LTX-2 VAE file carries both halves and the two key filters + // separate them (video_vae/model_configurator.py:255-276). Before this row the + // fixture wrote the decoder alone, so nothing in this suite could have noticed + // that no encoder key filter existed — which is a fair description of how the + // gap survived. + // + // These are written under the BARE `encoder.` prefix rather than + // `vae.encoder.`, matching the shipped Comfy-split spelling and exercising the + // third of the four encoder rules; the decoder half above is likewise bare. + auto put_enc = [&](const std::string& name, const std::vector& shape, double scale, + double offset = 0.0) { + int64_t numel = 1; + for (const int64_t d : shape) numel *= d; + entries.push_back( + {"encoder." + name, "BF16", shape, Param("ltx2.vvaeenc." + name, numel, scale, offset)}); + }; + // The FORWARD mirror of the decoder's block list, reduced to plain strided + // convolutions so the encoder runs at the latent width throughout. It must + // multiply out to the SAME (8, 32, 32) the pipeline derives every latent shape + // from, or an encoded image would not fit the grid it is placed into. + const std::vector encoder_blocks = { + "res_x", "compress_space", "compress_time", "compress_all", "compress_all", + }; + const int64_t latent = cfg.in_channels; + const int64_t patched_in = 3 * cfg.patch_size * cfg.patch_size; + put_enc("conv_in.conv.weight", {latent, patched_in, 3, 3, 3}, 0.1); + put_enc("conv_in.conv.bias", {latent}, 0.05); + for (size_t i = 0; i < encoder_blocks.size(); ++i) { + const std::string bp = "down_blocks." + std::to_string(i); + if (encoder_blocks[i] == "res_x") { + put_enc(bp + ".res_blocks.0.conv1.conv.weight", {latent, latent, 3, 3, 3}, 0.1); + put_enc(bp + ".res_blocks.0.conv1.conv.bias", {latent}, 0.05); + put_enc(bp + ".res_blocks.0.conv2.conv.weight", {latent, latent, 3, 3, 3}, 0.1); + put_enc(bp + ".res_blocks.0.conv2.conv.bias", {latent}, 0.05); + } else { + // The plain strided path reads `.conv.weight`, NOT `.conv.conv.*` + // — that second spelling belongs to the `*_res` family's + // SpaceToDepthDownsample (ltx2_video_vae.cpp). + put_enc(bp + ".conv.weight", {latent, latent, 3, 3, 3}, 0.1); + put_enc(bp + ".conv.bias", {latent}, 0.05); + } + } + // `latent_log_var` defaults to `uniform`, so conv_out emits one extra channel + // and the mean split drops it (video_vae.py:308-315). + put_enc("conv_out.conv.weight", {latent + 1, latent, 3, 3, 3}, 0.1); + put_enc("conv_out.conv.bias", {latent + 1}, 0.05); + nlohmann::json vae; vae["_class_name"] = "CausalVideoAutoencoder"; vae["dims"] = 3; @@ -651,6 +700,13 @@ inline void WriteReducedVideoVae(const vllm::Ltx2ConvVideoDecoderConfig& cfg, blocks.push_back(nlohmann::json::array({b.name, params})); } vae["decoder_blocks"] = blocks; + nlohmann::json enc_blocks = nlohmann::json::array(); + for (const std::string& name : encoder_blocks) { + nlohmann::json params = nlohmann::json::object(); + if (name == "res_x") params["num_layers"] = 1; + enc_blocks.push_back(nlohmann::json::array({name, params})); + } + vae["encoder_blocks"] = enc_blocks; nlohmann::json config; config["vae"] = vae; nlohmann::json metadata; diff --git a/tests/vllm/multimodal/test_ltx2_image_cond.cpp b/tests/vllm/multimodal/test_ltx2_image_cond.cpp new file mode 100644 index 000000000..9fd2bde9f --- /dev/null +++ b/tests/vllm/multimodal/test_ltx2_image_cond.cpp @@ -0,0 +1,765 @@ +// LTX-2.5 IMAGE CONDITIONING gate — row LTX25-IMAGE-COND, issue #644. +// +// Spec: .agents/specs/ltx25-image-conditioning.md §5. +// +// WHAT THIS GATES. The CHAIN a conditioning image travels, against EXECUTED +// upstream at reduced dimensions: PPM -> aspect-fill resize -> normalize -> +// `Ltx2ConvVideoEncode` -> `Ltx2ConditionVideoByLatentIndex` -> the noiser. +// Every link but the first two already had a golden in `test_ltx2_vae`; the +// CHAIN did not, and a chain of green links can still be wired in the wrong +// ORDER. It also gates the three things this row built to reach that chain from +// a checkpoint at all: the encoder key rules, the encoder config parser, and the +// CRF resolution that decides whether an image request is served or refused. +// +// WHAT IT CANNOT SHOW. Nothing here is a render-quality result. And one negative +// result is recorded rather than papered over: NO golden in this file can see a +// swap of the resize/normalize ORDER. `resize` is a convex combination and +// `normalize` is affine, so the two orders are equal in exact arithmetic and +// differ only by f32 rounding — measured at 1.94e-07 upstream +// (`kLtx2ImgPreOrderGap`), which is below the golden band AND below this port's +// own distance from torch. The order is mirrored because it is upstream's, and +// that fact is written down here instead of being assumed to be covered. +#include + +#include +#include +#include +#include +#include +#include + +#include + +#include "ltx2_image_cond_goldens.inc" +#include "vllm/model_executor/models/ltx2_conditioning.h" +#include "vllm/model_executor/models/ltx2_image_preprocess.h" +#include "vllm/model_executor/models/ltx2_pipeline.h" +#include "vllm/model_executor/models/ltx2_video_vae_encoder.h" + +namespace { + +// --------------------------------------------------------------------------- +// TOLERANCES, DERIVED FROM MEASUREMENT rather than picked — a band that can +// never bind reports nothing, which is the criticism test_ltx2_vae.cpp:172-180 +// already makes of its own history. Both were measured by setting them to 1e-12 +// and reading the reported `worst` on this box (CPU Release, gcc, the tree's own +// `-ffp-contract=off`): +// +// section 1, resize, 0..255 PIXEL space 6.10352e-05 (case 2, the identity, is 0) +// section 2, preprocess, [-1, 1] 4.76837e-07 +// section 3, encoded latent 2.68221e-07 +// section 4, conditioned clean 2.68221e-07 +// section 5, noised latent 1.78814e-07 +// +// `kLtx2ImgGoldenTol` covers everything in latent/normalized space at ~4x the +// worst of those. It is also the band scripts/gen-ltx2-image-cond-goldens.py +// PARSES out of this file, so the number has exactly one definition and the +// generator's own assertions are stated against the band the suite applies. +// +// `kLtx2ImgPixelTol` is section 1's, and it is two orders wider for a MEASURED +// reason worth stating because it looks like slack and is not. Torch's bilinear +// does not round the way any portable f32 expression does. The index map and the +// lambdas were probed directly with basis images and match this port BIT FOR +// BIT; the residual is in the ACCUMULATION, and it appears even on an output +// element whose width weights are exactly (1, 0) — a pure two-term +// `a*h0 + c*h1`, where dimension order cannot be the explanation. Plain-f32, +// f64-accumulate and premultiplied-weight orderings were all tried against torch +// and all three land 1 ulp away on the same elements, which is the signature of +// FMA contraction inside torch's kernel. This tree compiles with +// `-ffp-contract=off`, so it cannot reproduce that even in principle. 6.1e-05 is +// ~1 ulp at 255; the band is ~3x it, for a different libm and a different +// -march. A structural porting error moves this by orders of magnitude, not by +// ulps — a `round` instead of a `ceil` in the resize moves it to ~200. +// --------------------------------------------------------------------------- +constexpr double kLtx2ImgGoldenTol = 2e-6; +constexpr double kLtx2ImgPixelTol = 2e-4; + +// The exact upstream tree the goldens came from. Regenerating against a +// DIFFERENT checkout fails here instead of silently replacing the oracle. +constexpr const char* kLtx2ImgCondUpstreamRevisionPin = + "fd4ded7f2d88d3da713abcdd4ad41ecc4a9314ca"; + +// --------------------------------------------------------------------------- +// The shared deterministic stream — the exact mirror of the generator's +// `ltx_rand` / `ltx_bytes` / `param_values`, and byte-for-byte what +// tests/vllm/models/test_ltx2_vae.cpp uses. It is duplicated here rather than +// shared because that file is a .cpp, not a header; the duplication is made SAFE +// by `CheckManifest` below, which asserts this file's parameter set is exactly +// the state_dict the generator filled — so a copy that drifted is a failure +// rather than a silently different model. +// --------------------------------------------------------------------------- + +uint64_t Fnv1a64(const std::string& name) { + uint64_t h = 0xCBF29CE484222325ULL; + for (unsigned char byte : name) { + h ^= static_cast(byte); + h *= 0x100000001B3ULL; + } + return h; +} + +uint64_t Splitmix64(uint64_t x) { + x += 0x9E3779B97F4A7C15ULL; + uint64_t z = x; + z = (z ^ (z >> 30)) * 0xBF58476D1CE4E5B9ULL; + z = (z ^ (z >> 27)) * 0x94D049BB133111EBULL; + return z ^ (z >> 31); +} + +std::vector Ltx2Rand(const std::string& name, int64_t count) { + const uint64_t seed = Fnv1a64(name); + std::vector out(static_cast(count)); + for (int64_t i = 0; i < count; ++i) { + const uint64_t u = Splitmix64(seed + static_cast(i)); + out[static_cast(i)] = (static_cast(u >> 11) * 0x1p-53) * 2.0 - 1.0; + } + return out; +} + +// The generator's `ltx_bytes`: uint8 codes, because a conditioning image IS +// uint8 out of the decoder and quantizing a float stream afterwards would gate a +// different input than a real PPM carries. +std::vector Ltx2Bytes(const std::string& name, int64_t count) { + const uint64_t seed = Fnv1a64(name); + std::vector out(static_cast(count)); + for (int64_t i = 0; i < count; ++i) { + out[static_cast(i)] = + static_cast(Splitmix64(seed + static_cast(i)) % 256U); + } + return out; +} + +std::vector Ltx2Input(const std::string& name, int64_t count, double scale) { + const std::vector raw = Ltx2Rand(name, count); + std::vector out(static_cast(count)); + for (int64_t i = 0; i < count; ++i) { + out[static_cast(i)] = static_cast(raw[static_cast(i)] * scale); + } + return out; +} + +bool EndsWith(const std::string& text, const std::string& suffix) { + return text.size() >= suffix.size() && + text.compare(text.size() - suffix.size(), suffix.size(), suffix) == 0; +} + +std::vector Ltx2Param(const std::string& name, const std::vector& shape) { + int64_t count = 1; + for (int64_t dim : shape) count *= dim; + const size_t rank = shape.size(); + double scale = 0.1; + double offset = 0.0; + if (EndsWith(name, "std-of-means")) { + offset = 1.0; + } else if (EndsWith(name, "mean-of-means")) { + // scale 0.1, offset 0 + } else if (EndsWith(name, ".bias")) { + scale = 0.05; + } else if (rank == 1 && EndsWith(name, ".weight")) { + offset = 1.0; + } + const std::vector raw = Ltx2Rand(name, count); + std::vector out(static_cast(count)); + for (int64_t i = 0; i < count; ++i) { + out[static_cast(i)] = + static_cast(raw[static_cast(i)] * scale + offset); + } + return out; +} + +struct ParamBag { + vllm::Ltx2VaeWeights weights; + std::vector names; + std::vector counts; + + void Put(const std::string& name, const std::vector& shape) { + std::vector values = Ltx2Param(name, shape); + counts.push_back(static_cast(values.size())); + names.push_back(name); + weights.tensors[name] = std::move(values); + } +}; + +// The reduced encoder the generator built (`IMG_ENC_BLOCKS` / `IMG_ENC`). +vllm::Ltx2ConvVideoEncoderConfig ImageEncoderConfig() { + vllm::Ltx2ConvVideoEncoderConfig cfg; + cfg.in_channels = 3; + cfg.out_channels = 4; + cfg.patch_size = 2; + cfg.norm_layer = vllm::Ltx2NormLayer::kPixelNorm; + cfg.latent_log_var = vllm::Ltx2LogVarianceType::kUniform; + cfg.spatial_padding_mode = vllm::Ltx2PaddingMode::kZeros; + cfg.encoder_blocks = { + {"res_x", 1, 0}, + {"compress_space_res", 1, 2}, + {"compress_all_res", 1, 1}, + }; + cfg.prefix = "ltx2.imgenc."; + return cfg; +} + +// The parameter set `VideoEncoder(**IMG_ENC)` builds, in state_dict order. +ParamBag BuildImageEncoderParams(const vllm::Ltx2ConvVideoEncoderConfig& cfg) { + ParamBag bag; + const std::string p = cfg.prefix; + bag.Put(p + "per_channel_statistics.std-of-means", {cfg.out_channels}); + bag.Put(p + "per_channel_statistics.mean-of-means", {cfg.out_channels}); + const int64_t patched_in = cfg.in_channels * cfg.patch_size * cfg.patch_size; + bag.Put(p + "conv_in.conv.weight", {cfg.out_channels, patched_in, 3, 3, 3}); + bag.Put(p + "conv_in.conv.bias", {cfg.out_channels}); + + int64_t feature = cfg.out_channels; + for (size_t i = 0; i < cfg.encoder_blocks.size(); ++i) { + const vllm::Ltx2VideoEncoderBlock& block = cfg.encoder_blocks[i]; + const std::string bp = p + "down_blocks." + std::to_string(i); + const int64_t multiplier = block.multiplier != 0 ? block.multiplier : 2; + if (block.name == "res_x") { + for (int64_t j = 0; j < block.num_layers; ++j) { + const std::string rp = bp + ".res_blocks." + std::to_string(j); + bag.Put(rp + ".conv1.conv.weight", {feature, feature, 3, 3, 3}); + bag.Put(rp + ".conv1.conv.bias", {feature}); + bag.Put(rp + ".conv2.conv.weight", {feature, feature, 3, 3, 3}); + bag.Put(rp + ".conv2.conv.bias", {feature}); + } + } else { + // The *_res family: SpaceToDepthDownsample's conv emits + // out_channels / prod(stride); the space-to-depth fold multiplies it back. + const int64_t st = block.name == "compress_space_res" ? 1 : 2; + const int64_t ss = block.name == "compress_time_res" ? 1 : 2; + const int64_t out = feature * multiplier; + const int64_t conv_out = out / (st * ss * ss); + bag.Put(bp + ".conv.conv.weight", {conv_out, feature, 3, 3, 3}); + bag.Put(bp + ".conv.conv.bias", {conv_out}); + feature = out; + } + } + bag.Put(p + "conv_out.conv.weight", {cfg.out_channels + 1, feature, 3, 3, 3}); + bag.Put(p + "conv_out.conv.bias", {cfg.out_channels + 1}); + return bag; +} + +// NaN-hardened, for the reason issue #449 records: `std::max(worst, x)` is +// `worst < x ? x : worst`, and `worst < NaN` is false, so an all-NaN arm reduces +// to 0.0 and reports a perfect match. +double MaxAbsDiff(const std::vector& got, const float* want, size_t count) { + REQUIRE(got.size() == count); + double worst = 0.0; + for (size_t i = 0; i < count; ++i) { + const double diff = std::abs(static_cast(got[i]) - static_cast(want[i])); + if (!(diff <= worst)) worst = diff; // NaN takes this branch + } + return worst; +} + +// A binary PPM (P6) carrying `codes` as an h x w RGB payload — the container the +// engine actually reads, built here so section 2 gates the DECODER too and not +// just the arithmetic downstream of it. +std::string MakePpm(const std::vector& codes, int64_t height, int64_t width) { + std::string out = "P6\n" + std::to_string(width) + " " + std::to_string(height) + "\n255\n"; + out.append(reinterpret_cast(codes.data()), codes.size()); + return out; +} + +const std::string& ConditioningImagePpm() { + static const std::string ppm = MakePpm( + Ltx2Bytes("ltx2.imgcond.image", + vllm_test::kLtx2ImgPreSrcH * vllm_test::kLtx2ImgPreSrcW * 3), + vllm_test::kLtx2ImgPreSrcH, vllm_test::kLtx2ImgPreSrcW); + return ppm; +} + +// The latent state the generator built: the target shape, patch 1, causal fix. +vllm::Ltx2VideoLatentShape ConditioningTarget() { + vllm::Ltx2VideoLatentShape target; + target.batch = 1; + target.channels = vllm_test::kLtx2ImgEncOutC; + target.frames = vllm_test::kLtx2ImgCondTargetFrames; + target.height = vllm_test::kLtx2ImgEncOutH; + target.width = vllm_test::kLtx2ImgEncOutW; + return target; +} + +// The encoded conditioning image, as `Ltx2ConvVideoEncode` produces it. +vllm::Ltx2LatentVolume EncodeConditioningImage(const std::vector& chw) { + const vllm::Ltx2ConvVideoEncoderConfig cfg = ImageEncoderConfig(); + const ParamBag bag = BuildImageEncoderParams(cfg); + return vllm::Ltx2ConvVideoEncode(cfg, bag.weights, chw, cfg.in_channels, /*frame_count=*/1, + vllm_test::kLtx2ImgPreDstH, vllm_test::kLtx2ImgPreDstW, + nullptr); +} + +// The composition under test, factored so the mutation witnesses below can drive +// it with a deliberately WRONG `latent`, `strength` or ORDER and show the golden +// moves. Returns the noised latent. +struct Composition { + std::vector clean, mask, noised; +}; + +Composition ComposeConditioning(const vllm::Ltx2LatentVolume& conditioning, double strength, + bool condition_before_noise) { + const vllm::Ltx2VideoLatentShape target = ConditioningTarget(); + const vllm::Ltx2ScaleFactors factors; + vllm::Ltx2LatentState state = vllm::Ltx2CreateVideoLatentState( + target, vllm_test::kLtx2ImgCondPatch, factors, vllm_test::kLtx2ImgCondFps, + /*causal_fix=*/true); + + const std::vector noise = + Ltx2Input("ltx2.imgcond.noise", static_cast(state.latent.size()), 1.0); + + auto apply = [&]() { + vllm::Ltx2ConditionVideoByLatentIndex(&state, target, vllm_test::kLtx2ImgCondPatch, + conditioning, strength, + vllm_test::kLtx2ImgCondLatentIdx); + }; + auto noise_it = [&]() { + std::vector broadcast(state.latent.size()); + for (int64_t t = 0; t < state.tokens; ++t) { + for (int64_t c = 0; c < state.width; ++c) { + broadcast[static_cast(t * state.width + c)] = + state.mask[static_cast(t)]; + } + } + state.latent = vllm::Ltx2GaussianNoise( + state.latent.data(), state.clean.data(), broadcast.data(), noise.data(), + static_cast(state.latent.size()), + static_cast(vllm_test::kLtx2ImgNoiseScale)); + }; + + if (condition_before_noise) { + apply(); + noise_it(); + } else { + noise_it(); + apply(); + } + return Composition{state.clean, state.mask, state.latent}; +} + +} // namespace + +// ─── the anchor ───────────────────────────────────────────────────────────── + +TEST_CASE("ltx2 image cond: the goldens carry the PINNED upstream revision") { + CHECK(std::string(vllm_test::kLtx2ImgCondUpstreamRevision) == + std::string(kLtx2ImgCondUpstreamRevisionPin)); +} + +// ─── section 1: the resize ────────────────────────────────────────────────── + +TEST_CASE("ltx2 image cond: resize_and_center_crop matches executed upstream") { + // Five shape pairs: upscale, downscale, identity, wider-than-target and + // taller-than-target. Case 1 (32x24 -> 16x16) is the one where `ceil` + // disagrees with BOTH `round` and `int`: 32 * (16/24) is 21.333, so upstream + // resizes to 22 rows and crops 3, while a rounding port resizes to 21 and + // crops 2 — every output value moves and every shape check still passes. + struct Case { + int64_t src_h, src_w, dst_h, dst_w; + const float* golden; + size_t golden_size; + const char* name; + }; + const Case cases[] = { + {vllm_test::kLtx2ImgResize0SrcH, vllm_test::kLtx2ImgResize0SrcW, + vllm_test::kLtx2ImgResize0DstH, vllm_test::kLtx2ImgResize0DstW, + vllm_test::kLtx2ImgResize0Golden, std::size(vllm_test::kLtx2ImgResize0Golden), + "12x20 -> 16x16"}, + {vllm_test::kLtx2ImgResize1SrcH, vllm_test::kLtx2ImgResize1SrcW, + vllm_test::kLtx2ImgResize1DstH, vllm_test::kLtx2ImgResize1DstW, + vllm_test::kLtx2ImgResize1Golden, std::size(vllm_test::kLtx2ImgResize1Golden), + "32x24 -> 16x16 (the ceil case)"}, + {vllm_test::kLtx2ImgResize2SrcH, vllm_test::kLtx2ImgResize2SrcW, + vllm_test::kLtx2ImgResize2DstH, vllm_test::kLtx2ImgResize2DstW, + vllm_test::kLtx2ImgResize2Golden, std::size(vllm_test::kLtx2ImgResize2Golden), + "16x16 -> 16x16 (identity)"}, + {vllm_test::kLtx2ImgResize3SrcH, vllm_test::kLtx2ImgResize3SrcW, + vllm_test::kLtx2ImgResize3DstH, vllm_test::kLtx2ImgResize3DstW, + vllm_test::kLtx2ImgResize3Golden, std::size(vllm_test::kLtx2ImgResize3Golden), + "8x8 -> 16x24"}, + {vllm_test::kLtx2ImgResize4SrcH, vllm_test::kLtx2ImgResize4SrcW, + vllm_test::kLtx2ImgResize4DstH, vllm_test::kLtx2ImgResize4DstW, + vllm_test::kLtx2ImgResize4Golden, std::size(vllm_test::kLtx2ImgResize4Golden), + "10x7 -> 16x16"}, + }; + REQUIRE(std::size(cases) == static_cast(vllm_test::kLtx2ImgResizeCases)); + + for (size_t i = 0; i < std::size(cases); ++i) { + const Case& c = cases[i]; + INFO("resize case " << i << ": " << c.name); + const std::vector codes = + Ltx2Bytes("ltx2.imgcond.resize" + std::to_string(i), c.src_h * c.src_w * 3); + std::vector hwc(codes.size()); + for (size_t k = 0; k < codes.size(); ++k) hwc[k] = static_cast(codes[k]); + + const std::vector got = + vllm::Ltx2ResizeAndCenterCrop(hwc.data(), c.src_h, c.src_w, 3, c.dst_h, c.dst_w); + REQUIRE(got.size() == c.golden_size); + const double worst = MaxAbsDiff(got, c.golden, c.golden_size); + CAPTURE(worst); + CHECK(worst <= kLtx2ImgPixelTol); + } +} + +TEST_CASE("ltx2 image cond: an aspect-fill resize that cannot cover the target is refused") { + // Not reachable through `Ltx2LoadImageAndPreprocess` — `scale = max(...)` makes + // it unreachable by construction — which is exactly why it is asserted here: + // a guard nobody can trip is a guard nobody notices has been deleted, and this + // one is what stops a future caller with its own scale from silently indexing + // a negative crop offset. + const std::vector tiny(3 * 4 * 4, 1.0F); + CHECK_THROWS(vllm::Ltx2ResizeAndCenterCrop(tiny.data(), 0, 4, 3, 4, 4)); + CHECK_THROWS(vllm::Ltx2ResizeAndCenterCrop(nullptr, 4, 4, 3, 4, 4)); +} + +// ─── section 2: the whole preprocess, from PPM bytes ──────────────────────── + +TEST_CASE("ltx2 image cond: load_image_and_preprocess matches upstream at crf=0") { + const std::vector got = vllm::Ltx2LoadImageAndPreprocess( + "first_frame", ConditioningImagePpm(), vllm_test::kLtx2ImgPreDstH, + vllm_test::kLtx2ImgPreDstW, /*crf=*/0); + REQUIRE(got.size() == std::size(vllm_test::kLtx2ImgPreGolden)); + const double worst = + MaxAbsDiff(got, vllm_test::kLtx2ImgPreGolden, std::size(vllm_test::kLtx2ImgPreGolden)); + CAPTURE(worst); + CHECK(worst <= kLtx2ImgGoldenTol); + + // The output space, asserted rather than assumed: `Ltx2ConvVideoEncode` takes + // [-1, 1] pixels, and a port that forgot the `- 1.0` would still be finite, + // still be the right shape, and still pass every structural check. + const auto minmax = std::minmax_element(got.begin(), got.end()); + CHECK(*minmax.first >= -1.0F); + CHECK(*minmax.second <= 1.0F); + CHECK(*minmax.first < 0.0F); +} + +TEST_CASE("ltx2 image cond: the resize/normalize ORDER is below every band here") { + // THE NEGATIVE RESULT, asserted so it stays true rather than left as prose. + // A convex combination commutes with an affine map, so the two orders differ + // only by f32 rounding — `kLtx2ImgPreOrderGap` is that difference, measured + // upstream. It is below the golden band, so no value comparison in this file + // can catch a swap; and it is below `kLtx2ImgPixelTol` too, so tightening the + // band would not help either, it would only make section 1 red. + // + // If this ever fails, the two operations stopped being the affine/convex pair + // this reasoning rests on, and the ORDER becomes gateable — which is a finding + // worth acting on, not an assertion to delete. + CHECK(vllm_test::kLtx2ImgPreOrderGap < kLtx2ImgGoldenTol); + CHECK(vllm_test::kLtx2ImgPreOrderGap * 127.5 < kLtx2ImgPixelTol); +} + +// ─── the CRF refusal, and what resolves it ────────────────────────────────── + +TEST_CASE("ltx2 image cond: a non-zero CRF is refused by name, never rendered") { + CHECK_NOTHROW(vllm::Ltx2PreprocessImageCrf(0)); + for (const int64_t crf : {1, 18, 33, 51, -1}) { + INFO("crf " << crf); + CHECK_THROWS_WITH_AS(vllm::Ltx2PreprocessImageCrf(crf), + doctest::Contains("encode_single_frame"), std::runtime_error); + } + // And it is refused through the FULL entry point too, not only the helper — + // an image request that named a CRF must not reach the encoder. + CHECK_THROWS_WITH_AS( + vllm::Ltx2LoadImageAndPreprocess("first_frame", ConditioningImagePpm(), 16, 16, 18), + doctest::Contains("CRF 18"), std::runtime_error); +} + +TEST_CASE("ltx2 image cond: an LTX-2.5 checkpoint RESOLVES crf 18, so the default refuses") { + // constants.py:130-133 — the newest row at or below the version. This is what + // makes the refusal above a real one: a caller who says nothing gets 18. + CHECK(vllm::Ltx2ResolveDefaultImageCrf({2, 5}) == 18); + CHECK(vllm::Ltx2ResolveDefaultImageCrf({2, 5, 0}) == 18); + CHECK(vllm::Ltx2ResolveDefaultImageCrf({2, 4}) == 18); + CHECK(vllm::Ltx2ResolveDefaultImageCrf({3}) == 18); // newer inherits the closest known row + CHECK(vllm::Ltx2ResolveDefaultImageCrf({2, 3}) == 33); + CHECK(vllm::Ltx2ResolveDefaultImageCrf({2, 0}) == 33); + CHECK(vllm::Ltx2ResolveDefaultImageCrf({2}) == 33); + // `detect_model_version` returns () for an unset/unparseable version, which + // "compares below every real version" (:138-139) — the OLDEST fallback. + CHECK(vllm::Ltx2ResolveDefaultImageCrf({}) == 33); +} + +// ─── the PPM decoder's own refusals ───────────────────────────────────────── + +TEST_CASE("ltx2 image cond: a container this tree cannot read is refused by name") { + int64_t h = 0, w = 0; + CHECK_THROWS_WITH_AS(vllm::Ltx2DecodePpmRgb("first_frame", "\x89PNG\r\n\x1a\n", &h, &w), + doctest::Contains("binary PPM"), std::runtime_error); + // P3 is ASCII PPM — a real format, and NOT the one that is read. + CHECK_THROWS_AS(vllm::Ltx2DecodePpmRgb("first_frame", "P3\n2 2\n255\n0 0 0", &h, &w), + std::runtime_error); + // A non-255 maxval is REFUSED rather than rescaled: PIL's rescale is what + // upstream would apply and mirroring it is a separate port. + CHECK_THROWS_WITH_AS( + vllm::Ltx2DecodePpmRgb("first_frame", std::string("P6\n1 1\n15\n") + "\x01\x02\x03", &h, &w), + doctest::Contains("maxval"), std::runtime_error); + // A truncated payload must not be padded with whatever was in the buffer. + CHECK_THROWS_WITH_AS(vllm::Ltx2DecodePpmRgb("first_frame", "P6\n4 4\n255\nshort", &h, &w), + doctest::Contains("truncated"), std::runtime_error); + + // The happy path reports the FILE's own geometry, not the caller's. + const std::vector rgb = + vllm::Ltx2DecodePpmRgb("first_frame", ConditioningImagePpm(), &h, &w); + CHECK(h == vllm_test::kLtx2ImgPreSrcH); + CHECK(w == vllm_test::kLtx2ImgPreSrcW); + CHECK(rgb.size() == static_cast(h * w * 3)); +} + +// ─── the load path this row exists to build ───────────────────────────────── + +TEST_CASE("ltx2 image cond: the ENCODER key rules are the encoder's, not the decoder's") { + const std::vector rules = vllm::Ltx2VideoVaeEncoderKeyRules(); + REQUIRE(rules.size() == 4); + // model_configurator.py:267-276, rule for rule and IN ORDER — the `vae.` + // spellings must precede their bare twins or a monolithic checkpoint's + // `vae.encoder.*` matches nothing and is silently dropped. + CHECK(rules[0].match_prefix == "vae.encoder."); + CHECK(rules[0].replacement.empty()); + CHECK(rules[1].match_prefix == "vae.per_channel_statistics."); + CHECK(rules[1].replacement == "per_channel_statistics."); + CHECK(rules[2].match_prefix == "encoder."); + CHECK(rules[2].replacement.empty()); + CHECK(rules[3].match_prefix == "per_channel_statistics."); + CHECK(rules[3].replacement == "per_channel_statistics."); + + // The two filters must DISAGREE. Before this row the engine held the decoder's + // alone, and asserting they differ is what says this one is not a copy. + const std::vector decoder = vllm::Ltx2VideoVaeDecoderKeyRules(); + REQUIRE(decoder.size() == rules.size()); + bool differs = false; + for (size_t i = 0; i < rules.size(); ++i) { + if (rules[i].match_prefix != decoder[i].match_prefix) differs = true; + } + CHECK(differs); +} + +TEST_CASE("ltx2 image cond: a decoder-only checkpoint is REPORTED, not half-loaded") { + CHECK(!vllm::Ltx2CheckpointHasVideoEncoder( + {"vae.decoder.conv_in.conv.weight", "vae.per_channel_statistics.std-of-means"})); + // per_channel_statistics alone is NOT evidence of an encoder: a Comfy-split + // decoder file carries it too, and treating it as evidence would report every + // one of them as encodable and then throw deep inside the encoder instead. + CHECK(!vllm::Ltx2CheckpointHasVideoEncoder({"per_channel_statistics.mean-of-means"})); + CHECK(vllm::Ltx2CheckpointHasVideoEncoder({"vae.encoder.conv_in.conv.weight"})); + CHECK(vllm::Ltx2CheckpointHasVideoEncoder({"encoder.conv_in.conv.weight"})); + CHECK(!vllm::Ltx2CheckpointHasVideoEncoder({})); +} + +TEST_CASE("ltx2 image cond: the encoder config reads the LATENT width, not the RGB one") { + nlohmann::json vae; + vae["_class_name"] = "CausalVideoAutoencoder"; + vae["dims"] = 3; + vae["in_channels"] = 3; + vae["out_channels"] = 3; // the DECODER's RGB count + vae["latent_channels"] = 128; // the ENCODER's latent width + vae["patch_size"] = 4; + vae["norm_layer"] = "pixel_norm"; + vae["encoder_blocks"] = nlohmann::json::array({ + nlohmann::json::array({"res_x", {{"num_layers", 2}}}), + nlohmann::json::array({"compress_all_res", {{"multiplier", 2}}}), + }); + nlohmann::json config; + config["vae"] = vae; + + const vllm::Ltx2ConvVideoEncoderConfig cfg = vllm::Ltx2ParseConvVideoEncoderConfig(config); + CHECK(cfg.out_channels == 128); // NOT 3 + CHECK(cfg.in_channels == 3); + CHECK(cfg.patch_size == 4); + REQUIRE(cfg.encoder_blocks.size() == 2); + CHECK(cfg.encoder_blocks[0].name == "res_x"); + CHECK(cfg.encoder_blocks[0].num_layers == 2); + // An ABSENT multiplier stays at the 0 sentinel — becoming 1 would quietly + // halve every widening block's output width. + CHECK(cfg.encoder_blocks[0].multiplier == 0); + CHECK(cfg.encoder_blocks[1].multiplier == 2); + // THE DEFAULT THAT DIVERGES FROM THE DECODER'S. `spatial_padding_mode` is + // absent here, so the encoder must take `zeros` while the decoder takes + // `reflect` (model_configurator.py:63-68 vs :92). + CHECK(cfg.spatial_padding_mode == vllm::Ltx2PaddingMode::kZeros); + CHECK(cfg.latent_log_var == vllm::Ltx2LogVarianceType::kUniform); + + SUBCASE("a declared spatial_padding_mode wins over the default") { + config["vae"]["spatial_padding_mode"] = "reflect"; + CHECK(vllm::Ltx2ParseConvVideoEncoderConfig(config).spatial_padding_mode == + vllm::Ltx2PaddingMode::kReflect); + } + SUBCASE("the top-level encoder_spatial_padding_mode is the second lookup") { + config["vae"]["encoder_spatial_padding_mode"] = "replicate"; + CHECK(vllm::Ltx2ParseConvVideoEncoderConfig(config).spatial_padding_mode == + vllm::Ltx2PaddingMode::kReplicate); + } + SUBCASE("a NESTED CausalDiffusionVAE config reads encoder.out_channels") { + nlohmann::json nested; + nested["vae"]["latent_channels"] = 64; + nested["vae"]["encoder"]["out_channels"] = 96; + nested["vae"]["encoder"]["blocks"] = config["vae"]["encoder_blocks"]; + CHECK(vllm::Ltx2ParseConvVideoEncoderConfig(nested).out_channels == 96); + } + SUBCASE("no encoder_blocks is REFUSED, never defaulted to an empty list") { + config["vae"].erase("encoder_blocks"); + CHECK_THROWS_WITH_AS(vllm::Ltx2ParseConvVideoEncoderConfig(config), + doctest::Contains("encoder_blocks"), std::runtime_error); + } + SUBCASE("a 2-D checkpoint is refused rather than built as 3-D") { + config["vae"]["dims"] = 2; + CHECK_THROWS_AS(vllm::Ltx2ParseConvVideoEncoderConfig(config), std::runtime_error); + } + SUBCASE("an unknown latent_log_var is refused rather than mapped to the nearest") { + config["vae"]["latent_log_var"] = "gaussian"; + CHECK_THROWS_AS(vllm::Ltx2ParseConvVideoEncoderConfig(config), std::runtime_error); + } + SUBCASE("a res_x block WITHOUT num_layers is refused, as upstream's subscript is") { + // `_make_encoder_block` reads `block_config["num_layers"]` — a SUBSCRIPT + // (video_vae.py:55) — so a `res_x` block that omits it is a config upstream + // cannot load either. This parser used to default it to 1, which silently + // built a one-layer `UNetMidBlock3D` instead; two lines below it, an absent + // `multiplier` is deliberately kept at a sentinel for exactly that reason. + // The two are consistent as of the review of #657. + config["vae"]["encoder_blocks"][0][1].erase("num_layers"); + CHECK_THROWS_WITH_AS(vllm::Ltx2ParseConvVideoEncoderConfig(config), + doctest::Contains("num_layers"), std::runtime_error); + } + SUBCASE("only res_x needs num_layers; no other block kind reads it") { + // The other half of the mirror, and the half a bare "make it strict" would + // get wrong: `_make_encoder_block`'s remaining branches (video_vae.py:61-145) + // never touch `num_layers`, so requiring it everywhere would refuse configs + // upstream loads. + config["vae"]["encoder_blocks"][1][1].erase("multiplier"); + const vllm::Ltx2ConvVideoEncoderConfig lean = + vllm::Ltx2ParseConvVideoEncoderConfig(config); + REQUIRE(lean.encoder_blocks.size() == 2); + CHECK(lean.encoder_blocks[1].name == "compress_all_res"); + CHECK(lean.encoder_blocks[1].num_layers == 1); + CHECK(lean.encoder_blocks[1].multiplier == 0); + } +} + +// ─── sections 3-5: the chain ──────────────────────────────────────────────── + +TEST_CASE("ltx2 image cond: the encoder's parameter set IS the generator's state_dict") { + const ParamBag bag = BuildImageEncoderParams(ImageEncoderConfig()); + REQUIRE(bag.names.size() == std::size(vllm_test::kLtx2ImgEncParamNames)); + REQUIRE(bag.counts.size() == std::size(vllm_test::kLtx2ImgEncParamCounts)); + for (size_t i = 0; i < bag.names.size(); ++i) { + INFO("parameter " << i); + CHECK(bag.names[i] == std::string(vllm_test::kLtx2ImgEncParamNames[i])); + CHECK(bag.counts[i] == vllm_test::kLtx2ImgEncParamCounts[i]); + } +} + +TEST_CASE("ltx2 image cond: PPM to conditioned, noised tokens, against executed upstream") { + // THE WHOLE CHAIN, driven from the container bytes rather than from any + // intermediate golden — which is the one thing the per-brick goldens in + // test_ltx2_vae cannot show. + const std::vector chw = vllm::Ltx2LoadImageAndPreprocess( + "first_frame", ConditioningImagePpm(), vllm_test::kLtx2ImgPreDstH, + vllm_test::kLtx2ImgPreDstW, /*crf=*/0); + + const vllm::Ltx2ConvVideoEncoderConfig cfg = ImageEncoderConfig(); + CHECK(vllm::Ltx2VideoTemporalScaleFactor(cfg.encoder_blocks) == + vllm_test::kLtx2ImgEncTemporalFactor); + CHECK(vllm::Ltx2VideoSpatialScaleFactor(cfg.encoder_blocks, cfg.patch_size) == + vllm_test::kLtx2ImgEncSpatialFactor); + + const vllm::Ltx2LatentVolume encoded = EncodeConditioningImage(chw); + CHECK(encoded.channels == vllm_test::kLtx2ImgEncOutC); + CHECK(encoded.frames == vllm_test::kLtx2ImgEncOutT); + CHECK(encoded.height == vllm_test::kLtx2ImgEncOutH); + CHECK(encoded.width == vllm_test::kLtx2ImgEncOutW); + { + const double worst = MaxAbsDiff(encoded.data, vllm_test::kLtx2ImgEncGolden, + std::size(vllm_test::kLtx2ImgEncGolden)); + CAPTURE(worst); + CHECK(worst <= kLtx2ImgGoldenTol); + } + + const Composition composed = + ComposeConditioning(encoded, vllm_test::kLtx2ImgCondStrength, /*condition_before_noise=*/true); + + CHECK(static_cast(composed.mask.size()) == vllm_test::kLtx2ImgCondTokens); + { + const double worst = MaxAbsDiff(composed.clean, vllm_test::kLtx2ImgCondClean, + std::size(vllm_test::kLtx2ImgCondClean)); + CAPTURE(worst); + CHECK(worst <= kLtx2ImgGoldenTol); + } + { + const double worst = MaxAbsDiff(composed.mask, vllm_test::kLtx2ImgCondMask, + std::size(vllm_test::kLtx2ImgCondMask)); + CAPTURE(worst); + CHECK(worst <= kLtx2ImgGoldenTol); + } + { + const double worst = MaxAbsDiff(composed.noised, vllm_test::kLtx2ImgNoisedGolden, + std::size(vllm_test::kLtx2ImgNoisedGolden)); + CAPTURE(worst); + CHECK(worst <= kLtx2ImgGoldenTol); + } +} + +// ─── the goldens are LOAD-BEARING: witnesses, not assertions of shape ─────── + +TEST_CASE("ltx2 image cond: the conditioned latent is what the goldens depend on") { + // THIS IS THE CLASS THE CAMPAIGN KEEPS FINDING. An earlier phase's + // conditioning could be scaled x1.5 or row-REVERSED and every assertion in the + // suite stayed green. Each witness below drives the SAME composition with a + // deliberately wrong input and asserts the golden distance is enormous, so a + // reader can see that the comparisons above are sensitive to the thing they + // claim to be about — rather than inferring it. + const std::vector chw = vllm::Ltx2LoadImageAndPreprocess( + "first_frame", ConditioningImagePpm(), vllm_test::kLtx2ImgPreDstH, + vllm_test::kLtx2ImgPreDstW, /*crf=*/0); + const vllm::Ltx2LatentVolume encoded = EncodeConditioningImage(chw); + + const size_t noised_n = std::size(vllm_test::kLtx2ImgNoisedGolden); + const size_t clean_n = std::size(vllm_test::kLtx2ImgCondClean); + const size_t mask_n = std::size(vllm_test::kLtx2ImgCondMask); + + SUBCASE("the encoded latent scaled x1.5") { + vllm::Ltx2LatentVolume scaled = encoded; + for (float& v : scaled.data) v *= 1.5F; + const Composition c = + ComposeConditioning(scaled, vllm_test::kLtx2ImgCondStrength, true); + CHECK(MaxAbsDiff(c.clean, vllm_test::kLtx2ImgCondClean, clean_n) > kLtx2ImgGoldenTol); + CHECK(MaxAbsDiff(c.noised, vllm_test::kLtx2ImgNoisedGolden, noised_n) > kLtx2ImgGoldenTol); + } + + SUBCASE("the encoded latent's channels REVERSED — same values, wrong places") { + vllm::Ltx2LatentVolume reversed = encoded; + std::reverse(reversed.data.begin(), reversed.data.end()); + const Composition c = + ComposeConditioning(reversed, vllm_test::kLtx2ImgCondStrength, true); + CHECK(MaxAbsDiff(c.clean, vllm_test::kLtx2ImgCondClean, clean_n) > kLtx2ImgGoldenTol); + CHECK(MaxAbsDiff(c.noised, vllm_test::kLtx2ImgNoisedGolden, noised_n) > kLtx2ImgGoldenTol); + } + + SUBCASE("the mask as `strength` instead of `1 - strength`") { + // 1 - 0.7 = 0.3 against 0.7. Mask 0 means KEEP the clean value, so inverting + // it renders an unconditioned clip that looks like the feature not working. + const Composition c = ComposeConditioning(encoded, 1.0 - vllm_test::kLtx2ImgCondStrength, true); + CHECK(MaxAbsDiff(c.mask, vllm_test::kLtx2ImgCondMask, mask_n) > kLtx2ImgGoldenTol); + CHECK(MaxAbsDiff(c.noised, vllm_test::kLtx2ImgNoisedGolden, noised_n) > kLtx2ImgGoldenTol); + // ...and the CLEAN latent is IDENTICAL, which is why the mask needs its own + // comparison: a suite that only checked `clean` would be blind to this. + CHECK(MaxAbsDiff(c.clean, vllm_test::kLtx2ImgCondClean, clean_n) <= kLtx2ImgGoldenTol); + } + + SUBCASE("the conditioning applied AFTER the noiser") { + // Upstream conditions the CLEAN tensor and lets the noiser compose it + // (latent_cond.py:38-39 + noisers.py:31-34). Applying it afterwards leaves + // the noised latent pinned to noise instead of to the image — with the + // identical clean tensor and the identical mask, so only section 5 sees it. + const Composition c = + ComposeConditioning(encoded, vllm_test::kLtx2ImgCondStrength, /*condition_before_noise=*/false); + CHECK(MaxAbsDiff(c.clean, vllm_test::kLtx2ImgCondClean, clean_n) <= kLtx2ImgGoldenTol); + CHECK(MaxAbsDiff(c.mask, vllm_test::kLtx2ImgCondMask, mask_n) <= kLtx2ImgGoldenTol); + CHECK(MaxAbsDiff(c.noised, vllm_test::kLtx2ImgNoisedGolden, noised_n) > kLtx2ImgGoldenTol); + } +} + +TEST_CASE("ltx2 image cond: the noiser follows ltx_core, and diffusers is far away") { + // `kLtx2ImgNoiseDivergence` is max|ltx_core - diffusers| at this NON-UNIT noise + // scale, measured by the generator against both compositions. Asserting it is + // orders of magnitude above the band is what makes section 5 a gate on the + // CHOICE spec §3.3 makes, rather than a gate that would pass either way. + CHECK(vllm_test::kLtx2ImgNoiseDivergence > 1000.0 * kLtx2ImgGoldenTol); + CHECK(vllm_test::kLtx2ImgNoiseScale != 1.0); // the only scale at which they agree +} diff --git a/tests/vllm/multimodal/test_ltx2_video.cpp b/tests/vllm/multimodal/test_ltx2_video.cpp index b065fbe7b..abd7890db 100644 --- a/tests/vllm/multimodal/test_ltx2_video.cpp +++ b/tests/vllm/multimodal/test_ltx2_video.cpp @@ -40,6 +40,7 @@ #include "vllm/model_executor/models/ltx2_text_encoder.h" #include "vllm/model_executor/models/ltx2_upsampler.h" #include "vllm/model_executor/models/ltx2_video_vae.h" +#include "vllm/model_executor/models/ltx2_video_vae_encoder.h" #include "vllm/platforms/interface.h" // CurrentPlatform() — the seam the engine asks #include "vllm.h" #include "vt/backend.h" @@ -683,40 +684,307 @@ TEST_CASE("ltx2 video: the recipe comes from the CHECKPOINT's own model_version" } } -TEST_CASE("ltx2 video: keyframe and reference conditioning is refused by name") { +// A binary PPM the engine can actually condition on. Deliberately NOT the +// generation's own resolution: `load_image_and_preprocess` aspect-fills and +// centre-crops to the phase's height/width (media_io/resize.py:41-73), and an +// image that already fits would leave that untested. +std::string ConditioningPpm(int height, int width, unsigned seed) { + std::string out = "P6\n" + std::to_string(width) + " " + std::to_string(height) + "\n255\n"; + for (int i = 0; i < height * width * 3; ++i) { + out.push_back(static_cast((i * 37 + static_cast(seed) * 101) % 251)); + } + return out; +} + +// BOTH phases, which for image conditioning is not a detail: the two-stage +// recipe renders its stages at DIFFERENT resolutions, so the image is decoded, +// resized and encoded once per phase against that phase's own height and width +// (ltx-pipelines/utils/helpers.py:274-275 are the parameters; distilled.py:251, +// :255-256 and :285-286 are where the two stages pass different values). A +// `max_phase = 0` fixture would +// leave the second encode — and the whole reason the conditioning lives inside +// the phase loop — untested. +vllm::multimodal::VideoModelParams ConditioningParams(const ltx2_fixture::Paths& paths) { + vllm::multimodal::VideoModelParams mp = FixtureParams(paths); + mp.extras["upsampler_path"] = paths.upsampler; + return mp; +} + +TEST_CASE("ltx2 video: keyframe and reference conditioning is refused BY WHAT IS MISSING") { + // Row LTX25-IMAGE-COND (#644) SPLIT this refusal. It used to cover every + // conditioning kind with one message whose reason was "no encoder weights can + // be materialized here" — true when written, and no longer: this engine now + // loads them through `Ltx2VideoVaeEncoderKeyRules`, and the first-frame arm is + // served (see the case below). + // + // So each surviving refusal is held to naming a DIFFERENT missing piece. The + // point is not that the message is long; it is that a later reader can go and + // check the named symbol and find out whether the reason still holds — which + // is the thing five refusals in this campaign failed at. Workspace ws; const std::unique_ptr engine = - vllm::multimodal::LoadVideoEngine(FixtureParams(ws.paths)); - vllm::multimodal::VideoGenParams gen = FixtureGen(ws.root + "/keyframed"); - gen.first_frame_path = ws.paths.video_embeds; // any path: the refusal precedes the read - try { - (void)engine->Generate(gen); - FAIL("keyframe conditioning must be refused while no encoder is reachable from here"); - } catch (const std::exception& e) { - const std::string msg = e.what(); + vllm::multimodal::LoadVideoEngine(ConditioningParams(ws.paths)); + + auto refusal = [&](const char* what, + void (*arm)(vllm::multimodal::VideoGenParams&, const Workspace&)) { + vllm::multimodal::VideoGenParams gen = FixtureGen(ws.root + "/refused"); + arm(gen, ws); + try { + (void)engine->Generate(gen); + FAIL_CHECK(what << " must be refused, never dropped"); + return std::string(); + } catch (const std::exception& e) { + return std::string(e.what()); + } + }; + + SUBCASE("a LAST-frame keyframe names the TOKEN-APPEND machinery, not the embedding") { + const std::string msg = refusal("a last-frame keyframe", + [](vllm::multimodal::VideoGenParams& g, const Workspace& w) { + g.last_frame_path = w.paths.video_embeds; + }); + INFO(msg); + // THIS ASSERTION USED TO PIN A FALSE REASON. It required the message to + // blame `keyframes_abs_pos_embedding`, and at pin `fd4ded7f` that is not + // what blocks a supplied keyframe: `apply_to` appends it with + // `marked=False` (keyframe_cond.py:84-86) and the sole consumer adds + // `mask * embedding` (transformer_args.py:42-43, called at :269), so the + // embedding contributes exactly nothing to those tokens. Porting it would + // not serve this arm. The gate enforced the wrong thing, which is worse + // than not gating the message at all. + // + // What actually blocks it is the append: extended `positions`, + // `update_attention_mask`, extended `clean_latent` / `denoise_mask`, and + // `clear_conditioning` trimming back — none of which this engine's + // fixed-length phase loop can express. + CHECK(msg.find("update_attention_mask") != std::string::npos); + CHECK(msg.find("clear_conditioning") != std::string::npos); + CHECK(msg.find("keyframe_cond.py") != std::string::npos); + CHECK(msg.find("VAE_ENCODER_COMFY_KEYS_FILTER") == std::string::npos); + // The refuted reason may still be NAMED — it is worth telling a reader that + // it was ruled out — but never as the thing that is missing, and only next + // to the issue that tracks where the embedding really does bite (#658). + if (msg.find("keyframes_abs_pos_embedding") != std::string::npos) { + CHECK(msg.find("NOT* THE REASON") != std::string::npos); + CHECK(msg.find("#658") != std::string::npos); + } + } + SUBCASE("a reference video names the IC-LoRA metadata this project does not read") { + const std::string msg = refusal("a reference video", + [](vllm::multimodal::VideoGenParams& g, const Workspace& w) { + g.ref_video_dir = w.root; + }); + INFO(msg); + CHECK(msg.find("temporal_scale_factor") != std::string::npos); + CHECK(msg.find("LoRA") != std::string::npos); + } + SUBCASE("reference audio names the AUDIO encoder, which this row did not build") { + const std::string msg = refusal("reference audio", + [](vllm::multimodal::VideoGenParams& g, const Workspace& w) { + g.ref_audio_path = w.paths.audio_embeds; + }); INFO(msg); - CHECK(msg.find("ImageConditioner") != std::string::npos); - // A refusal whose stated REASON has gone stale is worse than a vague one: it - // sends the next reader to build something that already exists. Phase L11 - // ported the video VAE encoder, so the message may no longer claim the - // encoder is missing, and these two assertions hold it to the pieces that - // actually are — the loader path that would put encoder weights in memory, - // and the CRF re-compression upstream applies before encoding. - CHECK(msg.find("VAE_ENCODER_COMFY_KEYS_FILTER") != std::string::npos); - CHECK(msg.find("default_image_crf") != std::string::npos); - // And the QUALIFIER on that re-compression, which the two substrings above do - // not reach: `preprocess` returns the image UNTOUCHED at `crf == 0` - // (media_io/decode.py:413-435, the `if crf == 0:` early return at :425-426 — - // NOT the one at :427-428, which is the degenerate-size guard), so "re-compresses - // before encoding" is only true of a nonzero resolved CRF. Naming the round - // trip without naming its exception overstates what is unported and sends the - // next reader to build an H.264 path for a case that needs none — the same - // failure mode as a stale reason, one step subtler. Gated here so deleting the - // qualifier goes RED rather than quietly restoring the overstatement. - CHECK(msg.find("unless that CRF is 0") != std::string::npos); + CHECK(msg.find("audio VAE") != std::string::npos); + CHECK(msg.find("encode_audio") != std::string::npos); + } + SUBCASE("a non-zero CRF names the codec round trip, and says 0 is supported") { + // AND THIS IS THE DEFAULT PATH. An LTX-2.5 checkpoint resolves + // `default_image_crf = 18` (constants.py:37/124/130-133), so a caller who + // says nothing about the CRF lands here — which is what makes the + // out-of-distribution `crf = 0` arm a deliberate request rather than a + // silent downgrade. + vllm::multimodal::VideoGenParams gen = FixtureGen(ws.root + "/crf"); + gen.first_frame_ppm = ConditioningPpm(20, 28, 3); + try { + (void)engine->Generate(gen); + FAIL("an unset CRF resolves 18 for a 2.5 checkpoint and must be refused"); + } catch (const std::exception& e) { + const std::string msg = e.what(); + INFO(msg); + CHECK(msg.find("CRF 18") != std::string::npos); + CHECK(msg.find("encode_single_frame") != std::string::npos); + CHECK(msg.find("CRF 0 IS supported") != std::string::npos); + } + } + SUBCASE("an explicit non-zero CRF is refused just as an unset one is") { + vllm::multimodal::VideoGenParams gen = FixtureGen(ws.root + "/crf33"); + gen.first_frame_ppm = ConditioningPpm(20, 28, 4); + gen.extras[vllm::multimodal::kLtx2ImageCrfExtra] = "33"; + CHECK_THROWS_WITH_AS((void)engine->Generate(gen), doctest::Contains("CRF 33"), + std::runtime_error); + } + SUBCASE("a mistyped per-generation extra is refused, not ignored") { + vllm::multimodal::VideoGenParams gen = FixtureGen(ws.root + "/typo"); + gen.first_frame_ppm = ConditioningPpm(20, 28, 5); + gen.extras["image_crf_"] = "0"; + CHECK_THROWS_WITH_AS((void)engine->Generate(gen), doctest::Contains("image_crf_"), + std::runtime_error); + } +} + +TEST_CASE("ltx2 video: an image at crf 0 conditions the render, and the ENCODER weights are read") { + // The arm row LTX25-IMAGE-COND (#644) opened. Two separate claims are made + // here and they are NOT the same claim: + // + // 1. the conditioning REACHES the render — the trace reports the encoded + // image, and a different image gives a different digest; and + // 2. the ENCODER WEIGHTS are READ — perturbing ONE encoder tensor in the + // checkpoint moves the digest, with every byte of the REQUEST identical. + // + // (2) is the one that is easy to fake. A path that loaded the weights and then + // conditioned on something else — zeros, the raw pixels, a re-used decoder + // tensor — satisfies (1) completely. + Workspace ws; + const std::unique_ptr engine = + vllm::multimodal::LoadVideoEngine(ConditioningParams(ws.paths)); + auto* ltx2 = dynamic_cast(engine.get()); + REQUIRE(ltx2 != nullptr); + + vllm::multimodal::VideoGenParams gen = FixtureGen(ws.root + "/img"); + gen.first_frame_ppm = ConditioningPpm(20, 28, 1); + gen.extras[vllm::multimodal::kLtx2ImageCrfExtra] = "0"; + const vllm::multimodal::VideoResult result = engine->Generate(gen); + + const vllm::multimodal::Ltx2ConditioningTrace trace = ltx2->last_conditioning(); + CHECK(trace.completed); + CHECK(trace.image_crf == 0); + CHECK(trace.image_strength == 1.0); // noise_aug defaults to 1.0 => the frame is PINNED + // WHICH PHASE this describes is the claim, and `image_tokens > 0` did not make + // it. MEASURED: changing the guard to `wants_image && phase_index == 0` — the + // shape of an obvious refactor that hoists the per-phase decode+encode out of + // the loop — left this whole binary at 32 cases / 550 assertions / exit 0 + // while `refine`, the phase whose latent is actually rendered, ran with the + // pinned frame re-noised away. The design's own reason for living inside the + // loop (spec section 8.5) was gated by nothing. + // + // So the count is pinned to the LAST phase's per-latent-frame token count. + // This fixture's two-stage recipe runs `generate_lowres` at + // `spatial_downscale = 2` and `refine` at 1, so the latent grid doubles in + // each spatial dimension and the placed count is 1 then 4 — a per-phase value, + // which is exactly why `image_tokens == 4` falsifies a stage-1-only build. + constexpr int64_t kRefineImageTokens = 4; + CHECK(trace.image_tokens == kRefineImageTokens); + CHECK(trace.image_digest != 0); + // A conditioning that collapsed to zeros would give every image the same + // digest and still satisfy every check below it, so the magnitude is asked for + // separately — the same reason `video_absmax` exists next to `video_digest`. + CHECK(trace.image_absmax > 0.0); + // The render still produced its artifacts; conditioning is not a bypass. + CHECK(result.frame_count == 9); + + SUBCASE("the trace describes the LAST phase, and stage 1 is a different count") { + // The other half of the same claim, and the half a literal cannot make: the + // number above is not a constant of the fixture, it TRACKS the phase that + // ran last. Same request, capped at phase 0, must report stage 1's smaller + // count — and the ratio is checked between two MEASURED values rather than + // between two compile-time constants, which would assert nothing. + // `max_phase` is a LOAD-time extra, not a per-generation one, so the cap + // needs its own engine over the same fixture. + vllm::multimodal::VideoModelParams capped = ConditioningParams(ws.paths); + capped.extras[vllm::multimodal::kLtx2MaxPhaseExtra] = "0"; + const std::unique_ptr stage1_engine = + vllm::multimodal::LoadVideoEngine(capped); + auto* stage1_ltx2 = + dynamic_cast(stage1_engine.get()); + REQUIRE(stage1_ltx2 != nullptr); + vllm::multimodal::VideoGenParams lowres = FixtureGen(ws.root + "/img_stage1"); + lowres.first_frame_ppm = ConditioningPpm(20, 28, 1); + lowres.extras[vllm::multimodal::kLtx2ImageCrfExtra] = "0"; + (void)stage1_engine->Generate(lowres); + const vllm::multimodal::Ltx2ConditioningTrace stage1 = stage1_ltx2->last_conditioning(); + CHECK(stage1.image_tokens == 1); + CHECK(trace.image_tokens == 4 * stage1.image_tokens); + // And it is a DIFFERENT encode, not the same one carried forward: the image + // is resized and encoded against each phase's own height and width + // (ltx-pipelines/utils/helpers.py:274-275, per-stage h/w at + // distilled.py:251, 255-256, 285-286). + CHECK(stage1.image_digest != trace.image_digest); + } + + SUBCASE("a DIFFERENT image is a different conditioning") { + vllm::multimodal::VideoGenParams other = FixtureGen(ws.root + "/img2"); + other.first_frame_ppm = ConditioningPpm(20, 28, 2); + other.extras[vllm::multimodal::kLtx2ImageCrfExtra] = "0"; + (void)engine->Generate(other); + CHECK(ltx2->last_conditioning().image_digest != trace.image_digest); + } + + SUBCASE("the SAME image is the same conditioning") { + vllm::multimodal::VideoGenParams again = FixtureGen(ws.root + "/img3"); + again.first_frame_ppm = ConditioningPpm(20, 28, 1); + again.extras[vllm::multimodal::kLtx2ImageCrfExtra] = "0"; + (void)engine->Generate(again); + CHECK(ltx2->last_conditioning().image_digest == trace.image_digest); + } + + SUBCASE("the ENCODER's own weights decide the conditioning") { + // ONE tensor of the encoder half, perturbed in a SECOND fixture, with the + // request byte-identical. If the engine were conditioning on anything but + // the encoder's output — or had loaded the DECODER's tensors under the + // encoder's names — this digest would not move. + Workspace mutated; + const std::string path = mutated.paths.video_vae; + std::string bytes = ReadAll(path); + // The PAYLOAD is what gets perturbed, and its position is READ from the + // safetensors header rather than guessed at. An earlier revision of this + // case searched for the tensor's NAME and flipped a byte a fixed distance + // past it, which lands inside the JSON header of whatever tensor happens to + // be stored next — the file still parsed, the render still ran, and the + // digest did not move. The case failed, which is the only reason that is a + // footnote and not a false green. + REQUIRE(bytes.size() > 8); + uint64_t header_len = 0; + std::memcpy(&header_len, bytes.data(), sizeof(header_len)); + REQUIRE(8 + header_len <= bytes.size()); + const nlohmann::json header = + nlohmann::json::parse(bytes.substr(8, static_cast(header_len))); + const std::string needle = "encoder.conv_in.conv.weight"; + REQUIRE_MESSAGE(header.contains(needle), + "the fixture must carry an encoder half for this to prove anything"); + const size_t data_start = + 8 + static_cast(header_len) + + header.at(needle).at("data_offsets").at(0).get(); + REQUIRE(data_start + 1 < bytes.size()); + // bf16 is stored little-endian, so byte 0 of a word carries the mantissa's + // top bits; flipping 0x40 there moves that ONE weight by ~50% without any + // risk of manufacturing an Inf or a NaN out of the exponent — which would + // change the digest for a reason that has nothing to do with this claim. + bytes[data_start] = static_cast(bytes[data_start] ^ 0x40); + { + std::ofstream out(path, std::ios::binary | std::ios::trunc); + REQUIRE(out.good()); + out.write(bytes.data(), static_cast(bytes.size())); + } + const std::unique_ptr other = + vllm::multimodal::LoadVideoEngine(ConditioningParams(mutated.paths)); + auto* other_ltx2 = dynamic_cast(other.get()); + REQUIRE(other_ltx2 != nullptr); + vllm::multimodal::VideoGenParams same = FixtureGen(mutated.root + "/img"); + same.first_frame_ppm = ConditioningPpm(20, 28, 1); + same.extras[vllm::multimodal::kLtx2ImageCrfExtra] = "0"; + (void)other->Generate(same); + CHECK(other_ltx2->last_conditioning().image_digest != trace.image_digest); } } +TEST_CASE("ltx2 video: a request WITHOUT an image leaves the trace's image fields empty") { + // Otherwise "this render was conditioned on an image" and "this render was + // not" would be indistinguishable after the fact, which is the one question + // `Ltx2ConditioningTrace` exists to answer. + Workspace ws; + const std::unique_ptr engine = + vllm::multimodal::LoadVideoEngine(ConditioningParams(ws.paths)); + auto* ltx2 = dynamic_cast(engine.get()); + REQUIRE(ltx2 != nullptr); + (void)engine->Generate(FixtureGen(ws.root + "/plain")); + const vllm::multimodal::Ltx2ConditioningTrace trace = ltx2->last_conditioning(); + CHECK(trace.completed); + CHECK(trace.image_tokens == 0); + CHECK(trace.image_digest == 0); + CHECK(trace.image_absmax == 0.0); + CHECK(trace.image_strength == 0.0); +} + // ─── the floor under everything above ─────────────────────────────────────── // @@ -1027,6 +1295,34 @@ TEST_CASE("ltx2 video: the SHIPPED Lightricks checkpoints parse and load") { // upstream's SDOps drop it. CHECK(!weights.Has("encoder.conv_in.conv.weight")); MESSAGE("shipped conv video VAE: " << weights.tensors.size() << " decoder tensors"); + + // ...and the ENCODER half of the SAME file resolves through the other + // filter (row LTX25-IMAGE-COND, #644). This is the only place the encoder + // load path meets a real shipped checkpoint rather than the fixture, so it + // is the only place the CHANNEL arithmetic can be wrong in a way the fixture + // agrees with: `latent_channels` is 128 while the top-level `out_channels` + // is 3, and reading the second builds a 3-channel-latent encoder that runs. + REQUIRE(vllm::Ltx2CheckpointHasVideoEncoder(file.Names())); + const vllm::Ltx2ConvVideoEncoderConfig enc = + vllm::Ltx2ParseConvVideoEncoderConfig(vllm::Ltx2ReadCheckpointConfig(file)); + CHECK(enc.out_channels == 128); + CHECK(enc.in_channels == 3); + CHECK(enc.patch_size == cfg.patch_size); + // The encoder's block list must multiply out to the SAME scale factors the + // decoder's does, or an encoded image does not fit the grid it is placed in. + CHECK(vllm::Ltx2VideoSpatialScaleFactor(enc.encoder_blocks, enc.patch_size) == spatial); + CHECK(vllm::Ltx2VideoTemporalScaleFactor(enc.encoder_blocks) == temporal); + const vllm::Ltx2VaeWeights enc_weights = + vllm::Ltx2LoadVaeWeights(file, vllm::Ltx2VideoVaeEncoderKeyRules()); + CHECK(enc_weights.Has("conv_in.conv.weight")); + CHECK(enc_weights.Has("conv_out.conv.weight")); + // The encoder normalizes its output by these (video_vae.py:336), so the + // filter has to carry them even though they are not `encoder.*` keys. + CHECK(enc_weights.Has("per_channel_statistics.std-of-means")); + // And the DECODER's half must be dropped, or the two bags would collide on + // names like `conv_in.conv.weight` and bind half a model to the other half. + CHECK(!enc_weights.Has("decoder.conv_in.conv.weight")); + MESSAGE("shipped conv video VAE: " << enc_weights.tensors.size() << " encoder tensors"); } SUBCASE("the audio VAE and its BWE vocoder load and configure") { @@ -1786,11 +2082,17 @@ TEST_CASE("ltx2 video: a trace for a render that never completed says so") { // was never produced, and every field would look entirely healthy: real // prompt, non-zero absmax, plausible digests. // - // THE PROBE IS A REAL REFUSAL, not an injected one. Keyframe / reference - // conditioning is refused by name (ltx2_video.cpp, the `ImageConditioner` - // note) and that refusal sits AFTER the trace is written, so a prompted - // request carrying a reference image walks the whole encode path, fills the - // trace, and then fails — exactly the shape this flag exists to report. + // THE PROBE IS A REAL REFUSAL, not an injected one. Reference conditioning is + // refused by name (ltx2_video.cpp, the `ImageConditioner` note) and that + // refusal sits AFTER the trace is written, so a prompted request carrying a + // reference image walks the whole encode path, fills the trace, and then + // fails — exactly the shape this flag exists to report. + // + // IT IS STILL A REFUSAL AFTER ROW LTX25-IMAGE-COND (#644), which served the + // first-frame arm and would have made a `first_frame_ppm` probe stop + // refusing. The reference arm stays refused for a reason this row did not + // touch (the IC-LoRA scale factors), so the probe was moved to it rather than + // to whatever happened to still throw. Workspace ws; const vllm::multimodal::VideoModelParams mp = EncoderParams(ws.paths); const std::unique_ptr engine = @@ -1803,11 +2105,11 @@ TEST_CASE("ltx2 video: a trace for a render that never completed says so") { gen.ref_image_paths.push_back(ws.root + "/nonexistent-reference.png"); try { (void)engine->Generate(gen); - FAIL("keyframe / reference conditioning must be refused"); + FAIL("reference conditioning must be refused"); } catch (const std::exception& e) { const std::string msg = e.what(); INFO(msg); - CHECK(msg.find("reference conditioning") != std::string::npos); + CHECK(msg.find("reference-image / reference-video conditioning") != std::string::npos); } const vllm::multimodal::Ltx2ConditioningTrace trace = ltx->last_conditioning();