port(ltx-2.5): L10 — a text prompt drives the 12B tower, and the oracle L3 said did not exist (#435)

.agents/porting-inventory.md

@@ -1250,7 +1250,24 @@ Examples: `examples/cli` ✅ (C-API client), `examples/server` ✅ (OpenAI serve
       (`Gemma4Weights`) is not
       wired — `ltx2_text_encoder.h` declares no tower contract, so L6 loads the two
       caption projections, the asset pack and the geometry, and VALIDATES every
-      tower module without materializing it.
+      tower module without materializing it. **CLOSED at L10 (2026-08-12):**
+      `Ltx2LoadGemmaTowerFromSafetensors` (ltx2_text_encoder.h/.cpp) materializes
+      the tower onto `Gemma4Weights`, reusing L6's
+      `Ltx2DequantTorchaoNvfp4ToBf16` unchanged. It reads the FLAT
+      `model.layers.{i}.*` names the LTX file ships — not the
+      `model.language_model.layers.{i}.*` form `gemma4_weights.cpp` reads, which
+      is why the tower could not simply go through that loader — and resolves
+      each layer's geometry from `layer_types`, `global_head_dim` and
+      `num_global_key_value_heads` rather than from the stored tensor widths,
+      which are HALF the logical ones under NVFP4. It REFUSES by name: a PLE
+      config against a checkpoint with no PLE tensors, `num_kv_shared_layers`
+      != 0, a `v_proj` present or absent against what `attention_k_eq_v`
+      declares, and a module in neither the BF16 nor the torchao-NVFP4 form.
+      The Gemma config is an INPUT and is never inferred: the shipped
+      `vonkaiser` build has NO `__metadata__` at all, and `layer_types`,
+      `global_head_dim`, `num_global_key_value_heads`, `attention_k_eq_v` and
+      both `rope_parameters` entries each move every hidden state while leaving
+      the tensor set byte-identical.
     * **Spec:** [ltx-2.5 spec](specs/ltx-2-5.md) §1.4 and §6 (L6). Lifecycle:
       shipped (host + load-time device staging). Owner: the LTX-2.5 row.
     * **OWED, FOUND 2026-08-12 by the phase-L8 GB10 run (entry 20 below).** The

docs/FEATURES.md

@@ -146,14 +146,20 @@ speed-pending, which [BENCHMARKS.md](BENCHMARKS.md) tracks.
 ### Standalone and non-registered lanes
 
 These run through dedicated forwards, not the `REGISTER_VLLM_MODEL` registry, so
-they sit outside the gated list above.
+they sit outside the gated list above. One caveat the LTX-2.5 row is too narrow
+to carry: its text tower's prompt tokenization mirrors upstream only while the
+checkpoint's tokenizer `post_processor` adds nothing. The shipped one is MEASURED
+empty, so this port's plain encode plus an explicit BOS prepend matches
+upstream's `add_special_tokens=True` today; a checkpoint with a non-empty
+`post_processor` would tokenize differently here, and `Ltx2TokenizeGemmaPrompt`
+in `ltx2_text_encoder.cpp` is the call that would have to change.
 
 | Lane | Tested checkpoint(s) | Correctness gate | Speed vs reference |
 |---|---|---|---|
 | Voxtral audio (`VoxtralForConditionalGeneration`) | Voxtral-Mini-3B-2507 | near-tie-robust 16/16 vs vLLM 0.25.0 | decode 0.97x (beats vLLM); encoder FORWARD 15.90x of vLLM's whole TTFT (pin 46.02 ms), or 2.89x with opt-in `VT_WHISPER_ENC_FA2=1` (costs 3 near-tie divergences vs 0). Not a TTFT ratio. Pending |
 | Whisper audio encoder | openai/whisper-small; whisper-large-v3 (Voxtral cfg) | encoder tower 77/77; large-v3 tower 203/203 | pending |
 | MiniMax-H3 DiT (`MiniMaxH3DiTModel`, vllm-omni lane) | MiniMax-H3 (33.1B video+audio) | portable 79/79; all three modalities COHERENT on Q4_K_M (§8.20); PRUNED ckpts run, Q8_0 seam 0.9941 (§8.21); ref2va grid was NVFP4 quant error, §8.9 REFUTED; GGUF/NVFP4/bf16 shards stream | FP4/Marlin landed; speed pending; no bf16 render yet. Render from the Q4_K_M GGUF, not the NVFP4 arm. Krea 2 text-to-image (roadmap C11) is scoped to reuse these DiT seams |
-| LTX-2.5 DiT (`LTX2VideoTransformer3DModel`, Lightricks lane) | LTX-2.5 (21.00B video+audio) | `SPIKE`. DiT, both VAE decoders and ENCODERS, conditioning items, pipeline and quantized loaders gated vs upstream at reduced dims. With the connector wired, renders e2e at 320x192/25f: a coherent scene, valid MP4+WAV | Family `ltx-2.5` via `ltx2-gen`. ~29 GB NVFP4/GB10, FP8 ~44 GB. FP8 and torchao-NVFP4 load, first-party NVFP4 does not. Text tower, DiffVAE, LoRA, image conditioning refused AT THE ENGINE. Speed PENDING |
+| LTX-2.5 DiT (`LTX2VideoTransformer3DModel`, Lightricks lane) | LTX-2.5 (21.00B video+audio) | `SPIKE`. DiT, both VAE decoders and ENCODERS, conditioning items, pipeline and quantized loaders gated vs upstream at reduced dims. With the connector wired, renders e2e at 320x192/25f: a coherent scene, valid MP4+WAV | Family `ltx-2.5` via `ltx2-gen`. ~29 GB NVFP4/GB10, FP8 ~44 GB. FP8 and torchao-NVFP4 load, first-party NVFP4 does not. Text tower RUNS; `encoder_path`, DiffVAE, LoRA, image cond refused AT THE ENGINE. Speed PENDING |
 | MTP speculator | Qwen3.6-27B, Qwen3.6-35B-A3B | token-identical to vLLM `mtp` at c1 | ~4% faster c1; +16% output tput (MoE) |
 | DFlash block-diffusion | Qwen3 (DFlash draft) | near-tie e2e 27/27 vs vLLM | 2.9x over spec-off, 1.003x vs vLLM DFlash-on |
 | DeepSeek-V4 MTP | DeepSeek-V4-Flash (nextn head) | lossless 5/5; real-model weight-blocked | pending |

docs/USAGE.md

@@ -369,8 +369,21 @@ the audio VAE's is refused rather than resampled, since upstream uses a polyphas
 kaiser resampler this project does not carry. And a VAE configured with
 `latent_log_var: none` is refused, because upstream itself raises on it.
 
-**There is no prompt.** The Gemma-4 12B text tower is not ported, so nothing can
-turn words into the conditioning the caption projections consume. Conditioning
+**There is no prompt**, and this page said the wrong reason for it. The Gemma-4
+12B text tower IS ported and RUNS: a prompt string tokenizes with the tokenizer
+the checkpoint ships as a tensor, the tower is materialized and forwarded, and
+both caption projections emit their conditioning streams — all of it gated in CI
+against a running upstream tower rather than against invariants derived from its
+own output. One tokenization detail is a KNOWN DIVERGENCE rather than a mirror,
+and it is checkpoint-conditional: upstream tokenizes through the HuggingFace
+`__call__` with its default `add_special_tokens=True`, so it runs the tokenizer's
+post_processor, while this port calls the plain encode and prepends BOS by hand.
+On the shipped checkpoint the two are identical — its post_processor declares an
+EMPTY special-token map, measured on the shipped file rather than assumed — so
+nothing is lost today. A checkpoint whose post_processor DID add tokens would
+tokenize differently here. What still blocks a prompt is the hop from there into
+cross-attention, which "A second behaviour changed" below states precisely.
+Conditioning therefore still
 comes from `--prompt-embeds` plus `--audio-prompt-embeds`: rows of
 little-endian f32, 4096 wide for the video stream and 2048 for the audio stream,
 with the same row count in both. Supplying a `--prompt` is refused, and supplying
@@ -456,6 +469,19 @@ because that would hand back a lower quality render as if it were the one you
 asked for. Keyframe and reference conditioning is refused for the same reason: it
 runs through the video VAE's encoder, and only the decoder is ported.
 
+A second behaviour changed and is worth stating precisely, because the reason a
+prompt is refused has moved. The Gemma-4 text tower now RUNS: a prompt string
+tokenizes with the tokenizer the checkpoint ships as a tensor, the torchao-NVFP4
+tower is materialized and forwarded, all 49 hidden states feed the multi-layer
+aggregation, and both caption projections produce the 4096-wide video and
+2048-wide audio conditioning streams. What is still missing is the hop from there
+into cross-attention: upstream routes each stream through an
+`Embeddings1DConnector` first, and while that connector's math is ported, its
+weights ship inside the DiT file and are still among the modules the DiT loader
+refuses. So `encoder_path` is still refused — with a message that now names the
+connector weights rather than the tower — and `has_encoder()` is still false.
+Conditioning meanwhile comes from `prompt_embeds_path`.
+
 ### Muse Glimmer: exactly what has been checked
 
 `MuseGlimmerForCausalLM` / `MuseGlimmerForConditionalGeneration` are not in that
@@ -1729,9 +1755,12 @@ routes stay unregistered.
 
 ## LTX-2.5: reproducing the DiT parity gate
 
-**There is no LTX-2.5 render path yet.** What ships today is the DiT's layout and
-forward, and there is no text encoder, no VAE, no pipeline and no `/v1/videos`
-route for it — asking the video engine for LTX-2.5 will not work. The C++ surface
+**This section is the DiT layer's own parity gate, not the render path.** It used
+to open by saying there was no render path, no text encoder, no VAE, no pipeline
+and no `/v1/videos` route for LTX-2.5; every one of those has since landed, and
+what the video engine can actually be asked for is under "LTX-2.5: what runs, and
+what it cannot do" above. What follows reproduces the DiT layer's gate from
+source. The C++ surface
 is `include/vllm/model_executor/models/ltx2.h`, and it refuses by name every arm it
 does not carry (a non-f32 stream dtype, the 19B caption-projection checkpoint form,
 keyframe absolute-position embeddings, the video-only / audio-only model types).
@@ -1775,6 +1804,50 @@ python3 scripts/gen-ltx2-pipeline-goldens.py \
 cmake --build build --target test_ltx2_pipeline && ./build/tests/test_ltx2_pipeline
 ```
 
+### The Gemma-4 text tower gate, and the interpreter it needs
+
+The text tower is gated against the UPSTREAM HuggingFace implementation built and
+run at reduced dimensions. It needs a `transformers` that registers
+`gemma4_unified` in `CONFIG_MAPPING` — **5.8 or newer; 5.3.0 does not have it and
+fails in a way that reads exactly like "Gemma-4 is unsupported"**. The generator
+refuses such an interpreter by name rather than emitting goldens from a tower it
+could not build.
+
+```sh
+/path/to/venv/bin/python scripts/gen-ltx2-gemma-tower-goldens.py \
+  --out tests/vllm/models/ltx2_gemma_tower_goldens.inc
+cmake --build build --target test_ltx2_text_encoder && ./build/tests/test_ltx2_text_encoder
+```
+
+No checkpoint and no download: the reduced config comes from
+`tests/vllm/models/ltx2_gemma4_text_config.json`, which is the
+`__metadata__["gemma_config"]` of the official bf16 text encoder, and every weight
+is rebuilt on both sides from the deterministic stream. The tolerance is not a
+constant — the generator MEASURES how far upstream's own answer moves between f32
+and bf16 and emits that per state as the bound.
+
+Two more gates want the real checkpoint. The prompt-token goldens are regenerated
+from the tokenizer the text encoder ships **as a tensor**, and the end-to-end case
+dequantizes the 12B tower to roughly 24 GB of host bf16, so it is opt-in rather
+than checkpoint-presence gated:
+
+```sh
+TE=$CHECKPOINT_ROOT/ltx-2.5/vonkaiser-fp8-nvfp4/text_encoders/gemma4-12b-with-proj-nvfp4-torchao.safetensors
+/path/to/venv/bin/python scripts/gen-ltx2-prompt-tokens-goldens.py \
+  --text-encoder "$TE" \
+  --out tests/vllm/models/ltx2_prompt_tokens_goldens.inc
+
+# real vocab, token-exact vs HuggingFace
+CHECKPOINT_ROOT=... ./build/tests/test_ltx2_text_encoder --test-case="ltx2 prompt: REAL*"
+
+# the full 12B vertical: ~33 GB host, minutes of CPU
+CHECKPOINT_ROOT=... VLLM_CPP_LTX2_TOWER_E2E=1 \
+  ./build/tests/test_ltx2_text_encoder --test-case="ltx2 e2e*"
+```
+
+`VLLM_CPP_LTX2_TEXT_ENCODER` names the file directly when it does not sit under
+`CHECKPOINT_ROOT` at the path above.
+
 Recipes resolve on an EXACT `(pipeline_kind, model_version)` pair and refuse
 anything else by name rather than defaulting, because a plausible but wrong sigma
 schedule or guidance scale renders a video instead of failing. The pairs that

include/vllm/model_executor/models/gemma4.h

@@ -251,6 +251,30 @@ std::vector<int32_t> Gemma4GenerateGreedyViaRegistry(
     vt::Queue& queue, int max_new_tokens,
     std::vector<float>* out_margins = nullptr);
 
+// The FULL-attention layers' "proportional" rope cos|sin table, on host in f32 —
+// the exact values the forward builds (BuildProportionalRopeCache rounds them to
+// bf16 to match the q/k it rotates; nothing else differs). Returns
+// [max_pos + 1, head_dim]: the first head_dim/2 columns are cos and the second
+// half sin, over the head_dim/2 DISTINCT angle pairs, mirroring upstream's
+// `emb = cat((freqs, freqs))` with each angle stored once
+// (`Gemma4UnifiedTextRotaryEmbedding.forward`, modeling_gemma4_unified.py:259-275
+// — the `cat` itself is :271; the inv_freq it consumes comes from
+// modeling_rope_utils.py:187-254).
+//
+// This is a GATE SURFACE, and it exists because of a measurement rather than a
+// preference. `partial_rotary_factor` decides how many angle pairs are rotated
+// and how many are zero-padded to identity, and it is the one field on this path
+// that the tower's hidden states cannot resolve: on the reduced LTX tower
+// fixture, forcing it from the config's 0.25 to 1.0 displaces the worst hidden
+// state by 1.09e-01 against that state's measured bf16 noise floor of 9.99e-02
+// — a ratio of 1.09, inside the tolerance the states are gated at — and a LARGER
+// fixture makes it worse, not better (0.65 at head_dim 16/32, seq 32), because
+// bf16 accumulation noise grows at least as fast as the rope contribution. So
+// the states are the wrong instrument and this table is the right one: f32, no
+// accumulation, compared element-wise against the oracle's own rotary embedding.
+std::vector<float> Gemma4ProportionalRopeCosSin(const HfConfig& config,
+                                                int64_t head_dim, int64_t max_pos);
+
 // Wrap already-loaded Gemma-4 weights in the registered LoadedModel so a caller
 // that owns the weights (the mm e2e gate) can drive ModelRegistry::Forward without
 // re-reading the checkpoint. `Make` OWNS the moved weights; `Borrow` does NOT own