feat(ltx-2.5): image conditioning at crf=0, the VAE encoder load path, and the keyframe refusal repair (#644)

.agents/roadmap_v1.md

@@ -154,7 +154,7 @@ issue is not yet placed. Keyed record: update in place, never append.
 | [#435](https://github.com/mudler/vllm.cpp/issues/435) | `ROAD-V1-LTX25` | LTX-2.5: port the 21B joint video+audio DiT and generalize the video seam | feature |
 | [#560](https://github.com/mudler/vllm.cpp/issues/560) | `ROAD-V1-LTX25` | Stabilizing constants unreachable by any gate: FIVE instances found by sweep, incl. the DiT's own `norm_eps` field default | bug |
 | [#567](https://github.com/mudler/vllm.cpp/issues/567) | `ROAD-V1-LTX25` | `OpNameImpl` makes a SECOND file every new `OpId` must edit; collapse the enum and its names into one derived list | bug |
-| [#644](https://github.com/mudler/vllm.cpp/issues/644) | `ROAD-V1-LTX25` | LTX-2.5 FULL PORT campaign. Row 0 `LTX25-PROMPT-ADALN` (spec [`ltx25-prompt-adaln.md`](specs/ltx25-prompt-adaln.md)): `use_prompt_adaln_single` defaults TRUE in BOTH references and the shipped DiT carries its 18 tensors, but `ltx2_loader.cpp:988` cleared the flag unconditionally, so every render dropped the timestep-conditioned half of the prompt K/V modulation — invisible to every shape and finiteness check | feature |
+| [#644](https://github.com/mudler/vllm.cpp/issues/644) | `ROAD-V1-LTX25` | LTX-2.5 FULL PORT campaign: close every refused conditioning arm. Row 0 `LTX25-PROMPT-ADALN` (spec [`ltx25-prompt-adaln.md`](specs/ltx25-prompt-adaln.md)) restored `use_prompt_adaln_single`, which the loader cleared unconditionally so every render dropped the timestep-conditioned half of the prompt K/V modulation. Row 1 `LTX25-IMAGE-COND` (spec [`ltx25-image-conditioning.md`](specs/ltx25-image-conditioning.md)) builds the video VAE ENCODER's load path — `Ltx2VideoVaeEncoderKeyRules` existed nowhere in the tree — and serves an image at latent frame 0 at `crf = 0`; keyframe / reference / non-zero-CRF stay refused by name | feature |
 | [#615](https://github.com/mudler/vllm.cpp/issues/615) | `GATE-PR-SIZE-BINARY` | `check-pr-size` fail-closes on every binary path with no exemption route, so no golden-bearing PR can merge: it blocks #431 and post-dates the golden precedent it rejects, spec [`gate-pr-size-binary.md`](specs/gate-pr-size-binary.md) | bug |
 | [#670](https://github.com/mudler/vllm.cpp/issues/670) | `BACKEND-TENSTORRENT-MISTRAL` | Tenstorrent: allowlist `MistralForCausalLM` and gate it on-device; goldens are `transformers`-teacher-forced because vLLM has no TT backend, spec [`tenstorrent-mistral.md`](specs/tenstorrent-mistral.md) | feature |
 | [#238](https://github.com/mudler/vllm.cpp/issues/238) | `SAMPLE-LOGPROB-TOKEN-IDS` | `logprobs_mode`: three of four modes are runtime-refused stubs | bug |

CMakeLists.txt

@@ -838,6 +838,13 @@ add_library(vllm STATIC
   # LTX-2.5 (ROW MODEL-DIFFUSION-LTX25, phase L11): the conditioning items that
   # place the VAE ENCODERS' output into the denoise state.
   src/vllm/model_executor/models/ltx2_conditioning.cpp
+  # LTX-2.5 (ROW LTX25-IMAGE-COND, issue #644): the video VAE ENCODER's LOAD
+  # path — key filter and config parser, which existed nowhere before — and the
+  # pixel front-end an image conditioning arrives through. Separate TUs rather
+  # than additions to ltx2_loader.cpp, which two concurrent rows of the same
+  # campaign need to edit.
+  src/vllm/model_executor/models/ltx2_video_vae_encoder_load.cpp
+  src/vllm/model_executor/models/ltx2_image_preprocess.cpp
   # LTX-2.5 (ROW MODEL-DIFFUSION-LTX25, phase L6): the quantized loaders — the
   # FP8 and torchao-NVFP4 DiT arms, the torchao-NVFP4 text encoder, and the
   # load-time device staging GB10's ATS penalty makes the default.

docs/FEATURES.md

@@ -163,7 +163,7 @@ in `ltx2_text_encoder.cpp` is the call that would have to change.
 | Voxtral audio (`VoxtralForConditionalGeneration`) | Voxtral-Mini-3B-2507 | near-tie-robust 16/16 vs vLLM 0.25.0 | decode 0.97x (beats vLLM); encoder FORWARD 15.90x of vLLM's whole TTFT (pin 46.02 ms), or 2.89x with opt-in `VT_WHISPER_ENC_FA2=1` (costs 3 near-tie divergences vs 0). Not a TTFT ratio. Pending |
 | Whisper audio encoder | openai/whisper-small; whisper-large-v3 (Voxtral cfg) | encoder tower 77/77; large-v3 tower 203/203 | pending |
 | MiniMax-H3 DiT (`MiniMaxH3DiTModel`, vllm-omni lane) | MiniMax-H3 (33.1B video+audio) | portable 79/79; all three modalities COHERENT on Q4_K_M (§8.20); PRUNED ckpts run, Q8_0 seam 0.9941 (§8.21); ref2va grid was NVFP4 quant error, §8.9 REFUTED; GGUF/NVFP4/bf16 shards stream | FP4/Marlin landed; speed pending; no bf16 render yet. Render from the Q4_K_M GGUF, not the NVFP4 arm. Krea 2 text-to-image (roadmap C11) is scoped to reuse these DiT seams |
-| LTX-2.5 DiT (`LTX2VideoTransformer3DModel`, Lightricks lane) | LTX-2.5 (21.00B video+audio) | `SPIKE`. DiT, VAEs+ENCODERS, conditioning, pipeline, quant loaders gated at reduced dims. Prompt-side AdaLN ported, host+device. Typed prompt to Gemma-4 to cross-attn, FIXTURE-gated. A prompted render is OWED | Family `ltx-2.5`, `ltx2-gen`. ~29 GB NVFP4/GB10, FP8 ~44 GB, +~24 GB tower. FP8/torchao/1st-party NVFP4 load; `keyframes_abs_pos_embedding` alone needs `allow_unported`. DiffVAE, LoRA, image cond refused. Speed PENDING |
+| LTX-2.5 DiT (`LTX2VideoTransformer3DModel`, Lightricks lane) | LTX-2.5 (21.00B video+audio) | `SPIKE`. DiT, VAEs+ENCODERS, conditioning, pipeline, quant loaders gated at reduced dims. Prompt AdaLN host+device; prompt->Gemma-4->cross-attn FIXTURE-gated. Image chain PPM->resize->encode->place->noise. Render OWED | `ltx-2.5`/`ltx2-gen`. ~29 GB NVFP4/GB10, FP8 ~44 GB, +24 GB tower. FP8/torchao/NVFP4 load; `keyframes_abs_pos_embedding` needs `allow_unported`. IMAGE cond SERVED `crf=0`; DiffVAE/LoRA/keyframe/ref refused. Speed PENDING |
 | MiniMax-Music3 (`MiniMaxMusic3ForConditionalGeneration`, diffusers lane) | MiniMax-Music3 (8.6B Qwen3 LLM + 0.646B RVQ decoder + 2.4B fp32 DiT + DAC Flow-VAE); diffusers arm, ~28.5 GB | `ACTIVE`. Loader 1413/1413 (#714); AR half (W2/W3) and acoustic half (W4/W5: DiT, scheduler, vocoder) gated vs the real weights, waveform 88,064 samples. No greedy path upstream, so no token gate. W2 LM fwd, W6, W7 owed | Not measured. The denominator will be SGLang-Omni in its production configuration (both CUDA graphs, compiled DIT and DAV, batched seeded sampling) |
 | MTP speculator | Qwen3.6-27B, Qwen3.6-35B-A3B | token-identical to vLLM `mtp` at c1 | ~4% faster c1; +16% output tput (MoE) |
 | DFlash block-diffusion | Qwen3 (DFlash draft) | near-tie e2e 27/27 vs vLLM | 2.9x over spec-off, 1.003x vs vLLM DFlash-on |

docs/USAGE.md

@@ -435,21 +435,44 @@ the embeddings connector) are implemented and gated. Several limits decide what
 you can actually ask for, and each refuses by name rather than rendering
 something else.
 
-In particular, the encoders being present does NOT mean image, keyframe,
-reference-video or reference-audio conditioning is usable: the video engine
-still refuses every one of those by name, because the request-side work between
-a file on disk and a tensor the encoder accepts — image decode, aspect-fill
-resize, and the H.264 CRF re-compression upstream performs before encoding
-whenever the resolved CRF is not `0` and the image is at least 2 pixels on its
-shorter side — is not ported. The engine also holds no
-encoder to call: it materializes the VAE DECODER key filters only, so no
-encoder weights are ever in memory, and the refusal names that rather than
-claiming the encoder itself is missing. Two encoder-level limits are worth
+**Image conditioning (image-to-video) runs at `image_crf=0`, and only there.**
+Pass a first frame as binary PPM (`first_frame_path` / `first_frame_ppm`) plus
+the per-generation extra `image_crf=0`; the engine decodes it, aspect-fills and
+centre-crops it to each phase's own resolution, VAE-encodes it, and replaces
+latent frame 0's clean tokens. `noise_aug` is the pinning strength (`1.0`, the
+default, pins the frame exactly).
+
+`image_crf=0` must be asked for **explicitly**, and it is **out of
+distribution**. Upstream re-compresses a conditioning image through H.264 at the
+CRF the checkpoint's generation was trained with, and an LTX-2.5 checkpoint
+resolves that to **18**. That round trip needs libx264 and no codec is vendored
+here, so a non-zero CRF — including the default a caller gets by saying nothing —
+is refused by name. `image_crf=0` is upstream-legal (upstream short-circuits it
+and documents an explicit `0` as "skip re-compression entirely") but conditions
+the model on pixels it was not trained to see. That is a render-quality cost, and
+it is stated rather than applied silently.
+
+Keyframe, reference-image, reference-video and reference-audio conditioning are
+still refused, each naming a different missing piece: a last-frame keyframe needs
+the token-APPEND machinery — a keyframe is appended to the sequence with its own
+positions and a rebuilt attention mask, then trimmed back off, and this engine's
+phase loop is fixed at the target grid's token count — while the served
+first-frame arm only REPLACES tokens that already exist; the reference arms need
+the IC-LoRA's scale factors, which live in LoRA metadata this project does not
+read; reference audio additionally needs the AUDIO VAE's encoder key filter,
+which is not built. (Until 2026-08-13 this said a last-frame keyframe needs the
+DiT's unported `keyframes_abs_pos_embedding`. That was wrong: a supplied keyframe
+is appended unmarked, so the embedding never applies to it. Where the embedding
+does bite is the FIRST latent frame of every render, which is a separate gap,
+tracked as issue #658.) Three encoder-level limits are worth
 stating in advance because they are refusals rather than approximations. A
 reference waveform whose sample rate differs from the audio VAE's is refused
 rather than resampled, since upstream uses a polyphase kaiser resampler this
-project does not carry. And a VAE configured with `latent_log_var: none` is
-refused, because upstream itself raises on it.
+project does not carry. A VAE configured with `latent_log_var: none` is
+refused, because upstream itself raises on it. And a video-VAE `res_x` encoder
+block that declares no `num_layers` is refused rather than defaulted, because
+upstream subscripts that key and raises `KeyError` on it; no other encoder block
+kind reads it.
 
 **A typed prompt works.** `--encoder` names the Gemma-4 12B text tower and
 `--prompt` carries the words. The tower tokenizes them with its OWN embedded
@@ -487,7 +510,11 @@ refused, because a stream left unconditioned renders instead of failing.
 returns the trace of the last `Generate()` — whether the conditioning came from a
 prompt or from embeds, the prompt string, the row count and both stream widths, an
 FNV-1a digest over the exact f32 buffers cross-attention read, and each stream's
-absmax. It is returned **by value, under the engine's own lock**, so it is safe to
+absmax. When the request carried an image it also reports the CRF and strength it
+was conditioned at, how many tokens the encoded image replaced, and a digest over
+**those tokens as written into the state** — not over the encoder's output, so a
+build that encoded an image and never placed it reads as unconditioned rather
+than healthy. It is returned **by value, under the engine's own lock**, so it is safe to
 call from a server thread while another thread renders — but `Generate` holds that
 same lock for the WHOLE render, so such a call blocks for minutes rather than
 returning a stale answer immediately. `completed` is true only if that
@@ -541,6 +568,11 @@ ltx2-gen --dit  ltx-2.5-22b-distilled-fp8.safetensors \
 Swap the two `--encoder*` flags and `--prompt` for `--prompt-embeds` +
 `--audio-prompt-embeds` to condition from files instead.
 
+Add `--first-frame frame.ppm --image-crf 0` for image-to-video. The PPM is
+binary P6 at maxval 255 (no PNG/JPEG codec is vendored); `--image-crf 0` is
+required and is not the default, because omitting it resolves the checkpoint's
+own CRF 18 and refuses — see the out-of-distribution note above.
+
 `--frames` must satisfy `(frames - 1) % 8 == 0` and width/height must divide by
 64 (32 for the VAE, twice that because the distilled recipe's first phase runs at
 half resolution). Omitting all three renders the recipe default, which is

examples/ltx2_gen/main.cpp

@@ -100,6 +100,7 @@ const char* Need(int argc, char** argv, int i, const char* flag) {
       "                [--max-phase N] [--allow-unported]\n"
       "                [--prompt-valid-rows N]   how many embed rows are real tokens\n"
       "                [--frames N] [--width N] [--height N] [--seed N]\n"
+      "                [--first-frame <image.ppm>] [--image-crf 0]\n"
       "                [--device cpu|cuda]\n\n"
       "Renders LTX-2.5 (family \"ltx-2.5\") through vllm_video_engine_load +\n"
       "vllm_video_generate.\n\n"
@@ -121,7 +122,17 @@ const char* Need(int argc, char** argv, int i, const char* flag) {
       "weights. The row count must then be a multiple of the connector's learnable\n"
       "register count (128 on the shipped files), and --prompt-valid-rows says how\n"
       "many of them are real: the rest are padding, and padding is REPLACED by the\n"
-      "learnable register table rather than ignored.\n");
+      "learnable register table rather than ignored.\n\n"
+      "IMAGE CONDITIONING (image-to-video). --first-frame takes a binary PPM (P6,\n"
+      "maxval 255) and pins latent frame 0 to it: it is decoded, aspect-filled and\n"
+      "centre-cropped to each phase's own resolution, VAE-encoded, and written into\n"
+      "the clean latent. It needs --image-crf 0, and that is DELIBERATELY not the\n"
+      "default. Upstream re-compresses a conditioning image through H.264 at the CRF\n"
+      "the checkpoint's generation was trained with, which for LTX-2.5 is 18; that\n"
+      "round trip needs libx264 and none is vendored here, so leaving --image-crf out\n"
+      "resolves 18 and REFUSES by name. --image-crf 0 is upstream-legal and OUT OF\n"
+      "DISTRIBUTION: the model sees pixels it was not trained on. That is a quality\n"
+      "cost, and this tool states it rather than turning it on quietly.\n");
   std::exit(code);
 }
 
@@ -133,7 +144,7 @@ int main(int argc, char** argv) {
   std::string workdir = "/tmp/ltx2_gen", out_path, ffmpeg = "ffmpeg", device = "cuda";
   // BORROWED by `vllm_video_generate`, like the extras below, so it is owned
   // here and pointed at only after parsing.
-  std::string prompt;
+  std::string prompt, first_frame, image_crf;
 
   // The extras are BORROWED by the load call, so the strings must outlive it.
   // Kept as two parallel vectors of owned strings plus the char* views the ABI
@@ -174,6 +185,14 @@ int main(int argc, char** argv) {
     else if (f == "--prompt-valid-rows")
       SetExtra("prompt_embeds_valid_rows", Need(argc, argv, ++i, f.c_str()));
     else if (f == "--allow-unported") SetExtra("allow_unported_modules", "1");
+    // Image conditioning (row LTX25-IMAGE-COND, issue #644). `--first-frame` is
+    // a binary PPM; `--image-crf` is the PER-GENERATION extra, so it rides
+    // vp.extra_* rather than mp.extra_*. Only 0 is served, and it is NOT
+    // defaulted here — leaving it out lets the engine resolve the checkpoint's
+    // own 18 and refuse, which is the point: this CLI must not be the thing that
+    // quietly turns an out-of-distribution render on.
+    else if (f == "--first-frame") first_frame = Need(argc, argv, ++i, "--first-frame");
+    else if (f == "--image-crf") image_crf = Need(argc, argv, ++i, "--image-crf");
     else if (f == "--device") device = Need(argc, argv, ++i, "--device");
     else if (f == "--frames") vp.num_frames = std::atoi(Need(argc, argv, ++i, "--frames"));
     else if (f == "--width") vp.width = std::atoi(Need(argc, argv, ++i, "--width"));
@@ -203,6 +222,27 @@ int main(int argc, char** argv) {
   mp.family = "ltx-2.5";
   vp.output_dir = workdir.c_str();
   if (!prompt.empty()) vp.prompt = prompt.c_str();
+  if (!first_frame.empty()) vp.first_frame = first_frame.c_str();
+
+  // The PER-GENERATION extras are a SEPARATE array from the load-time ones, and
+  // conflating them is the whole failure this keeps apart: `image_crf` handed to
+  // the load call is an unknown LOAD extra and is refused there, which would
+  // read as "the flag does not work" rather than as "it goes on the other call".
+  std::vector<std::string> gen_keys, gen_values;
+  if (!image_crf.empty()) {
+    gen_keys.emplace_back("image_crf");
+    gen_values.push_back(image_crf);
+  }
+  std::vector<const char*> gkeys, gvalues;
+  for (size_t i = 0; i < gen_keys.size(); ++i) {
+    gkeys.push_back(gen_keys[i].c_str());
+    gvalues.push_back(gen_values[i].c_str());
+  }
+  if (!gkeys.empty()) {
+    vp.extra_keys = gkeys.data();
+    vp.extra_values = gvalues.data();
+    vp.n_extras = static_cast<int32_t>(gkeys.size());
+  }
 
   std::vector<const char*> keys, values;
   keys.reserve(extra_keys.size());

include/vllm.h

@@ -853,8 +853,16 @@ typedef struct vllm_video_params {
   /* Where frame_%06d.ppm + audio.wav land (created if absent). REQUIRED. */
   const char* output_dir;
   /* v18: FAMILY-SPECIFIC per-generation settings, same parallel-array shape as
-   * the load-time extras. MiniMax-H3 defines none, and refuses any key it does
-   * not know rather than ignoring it. 0 => none. */
+   * the load-time extras. Every family refuses a key it does not know rather
+   * than ignoring it. 0 => none.
+   *   MiniMax-H3: none.
+   *   LTX-2.5:    "image_crf" — the H.264 CRF an image conditioning is
+   *               re-compressed at. Only "0" is served; an LTX-2.5 checkpoint
+   *               RESOLVES 18 when this is absent and the codec round trip is
+   *               unported, so leaving it out refuses BY NAME rather than
+   *               rendering. "0" is upstream-legal and out of distribution;
+   *               see docs/USAGE.md. No ABI change was needed for it, which is
+   *               what this parallel-array shape exists for. */
   const char* const* extra_keys;
   const char* const* extra_values;
   int32_t n_extras;