L9B: the LTX-2.5 render driver, --video-family, and the first shipped-weights render

docs/FEATURES.md

@@ -153,7 +153,7 @@ they sit outside the gated list above.
 | Voxtral audio (`VoxtralForConditionalGeneration`) | Voxtral-Mini-3B-2507 | near-tie-robust 16/16 vs vLLM 0.25.0 | decode 0.97x (beats vLLM); encoder FORWARD 15.90x of vLLM's whole TTFT (pin 46.02 ms), or 2.89x with opt-in `VT_WHISPER_ENC_FA2=1` (costs 3 near-tie divergences vs 0). Not a TTFT ratio. Pending |
 | Whisper audio encoder | openai/whisper-small; whisper-large-v3 (Voxtral cfg) | encoder tower 77/77; large-v3 tower 203/203 | pending |
 | MiniMax-H3 DiT (`MiniMaxH3DiTModel`, vllm-omni lane) | MiniMax-H3 (33.1B video+audio) | portable 79/79; all three modalities COHERENT on Q4_K_M (§8.20); PRUNED ckpts run, Q8_0 seam 0.9941 (§8.21); ref2va grid was NVFP4 quant error, §8.9 REFUTED; GGUF/NVFP4/bf16 shards stream | FP4/Marlin landed; speed pending; no bf16 render yet. Render from the Q4_K_M GGUF, not the NVFP4 arm. Krea 2 text-to-image (roadmap C11) is scoped to reuse these DiT seams |
-| LTX-2.5 DiT (`LTX2VideoTransformer3DModel`, Lightricks lane) | LTX-2.5 (21.00B video+audio) | `SPIKE`. DiT, both VAE decoders, both VAE ENCODERS + mel front-end, the conditioning items, the pipeline and the quantized loaders gated vs upstream at reduced dims: max\|diff\| 1.2e-07 DiT, 1.8e-06 VAE decode, 6.0e-07 VAE encode, 2.8e-06 pipeline. STRUCTURAL e2e only | Fits ONE GB10 at ~29 GB NVFP4. FP8 + torchao-NVFP4 arms load. CPU f32, CUDA bf16 device-resident. Text tower, DiffVAE, LoRA, image conditioning all still refused AT THE ENGINE. FP4-resident arm OWED. Speed PENDING |
+| LTX-2.5 DiT (`LTX2VideoTransformer3DModel`, Lightricks lane) | LTX-2.5 (21.00B video+audio) | `SPIKE`. DiT, both VAE decoders and ENCODERS, conditioning items, pipeline and quantized loaders gated vs upstream at reduced dims. Renders e2e at 128x128/9f, valid MP4+WAV; frames are NOT a scene | Family `ltx-2.5` via `ltx2-gen`. ~29 GB NVFP4 on one GB10. FP8 and torchao-NVFP4 load; first-party NVFP4 does not. Text tower, connector, DiffVAE, LoRA and image conditioning refused AT THE ENGINE. Speed PENDING |
 | MTP speculator | Qwen3.6-27B, Qwen3.6-35B-A3B | token-identical to vLLM `mtp` at c1 | ~4% faster c1; +16% output tput (MoE) |
 | DFlash block-diffusion | Qwen3 (DFlash draft) | near-tie e2e 27/27 vs vLLM | 2.9x over spec-off, 1.003x vs vLLM DFlash-on |
 | DeepSeek-V4 MTP | DeepSeek-V4-Flash (nextn head) | lossless 5/5; real-model weight-blocked | pending |
@@ -267,7 +267,7 @@ Build with `-DVLLM_CPP_VULKAN=ON`; off by default.
 | Multiple engines in one process (build, destroy, rebuild) | ✅ resident device state is owned by the weights, so a new engine never inherits a freed one's pointers | ✅ | ✅ | ✅ |
 | LoRA adapters | ☐ CPU brick only | ✅ | ✅ | ✅ |
 | Embedding / pooling endpoints | ◐ `/v1/embeddings` live (task=embed; score/rerank/classify pending) | ✅ | ✅ | ✅ |
-| OpenAI video generation `/v1/videos` (Sora shape) | ✅ `model`/`size`/`seconds` aliases + `GET /{id}/content`; `input_reference` and the `metadata` video/audio references condition the render | ◐ (vllm-omni, its own request shape) | ☐ | ☐ |
+| OpenAI video generation `/v1/videos` (Sora shape) | ✅ `model`/`size`/`seconds` aliases + `GET /{id}/content`; `input_reference` and `metadata` references condition the render; `--video-family` pins the family (default DETECT), `--video-extra K=V` carries family knobs | ◐ (vllm-omni, its own request shape) | ☐ | ☐ |
 | Flat C ABI for embedding in other languages | ✅ versioned | ☐ | ☐ | ✅ |
 
 #### C-ABI capability coverage <!-- abi-capability-table:begin -->

docs/USAGE.md

@@ -346,36 +346,98 @@ tokens quietly.
 This is a deliberate state, not a bug: registering the architecture is what lets
 the config parse and weight-name mapping be tested before the forward exists.
 
-### LTX-2.5 has no user-facing entrypoint yet
-
-LTX-2.5 is being ported in phases. Its two VAE decoders, its two VAE ENCODERS
-with the mel front-end, the conditioning items that place encoded latents into
-the token stream, and its pipeline layer (the sigma schedule, the diffusion
-steps, guidance, the latent spatial x2 upsampler, the duration head and the
-embeddings connector) are implemented and gated, but no CLI flag, server
-endpoint or C ABI call reaches them, so there is nothing to run here yet and no
-request shape to document. Do not infer from [FEATURES](FEATURES.md) that a
-render is available.
-
-In particular, the encoders being present does NOT mean image, keyframe,
-reference-video or reference-audio conditioning is usable: the video engine
-still refuses every one of those by name, because the request-side work between
-a file on disk and a tensor the encoder accepts — image decode, aspect-fill
-resize, and the H.264 CRF re-compression upstream performs before encoding — is
-not ported. Two encoder-level limits are worth stating in advance because they
-are refusals rather than approximations. A reference waveform whose sample rate
-differs from the audio VAE's is refused rather than resampled, since upstream
-uses a polyphase kaiser resampler this project does not carry. And a VAE
-configured with `latent_log_var: none` is refused, because upstream itself
-raises on it.
-
-One behaviour is worth stating in advance, because it decides what you get when
-the entrypoint does arrive. LTX-2.5 ships two video decoders behind one
-checkpoint field. The convolutional one is implemented; the higher quality
-diffusion one (`NADiffusionDecoder`) is not, and asking for it fails with a
-message naming the missing neighborhood-attention kernel. It never falls back to
-the convolutional decoder, because that would hand back a lower quality render as
-if it were the one you asked for.
+### LTX-2.5: what runs, and what it cannot do
+
+LTX-2.5 is reachable as video family `ltx-2.5`, through the same
+`vllm_video_engine_load` / `vllm_video_generate` C ABI that serves MiniMax-H3,
+and through the `ltx2-gen` example that drives it. Its two VAE decoders, its two
+VAE ENCODERS with the mel front-end, the conditioning items that place encoded
+latents into the token stream, and its pipeline layer (the sigma schedule, the
+diffusion steps, guidance, the latent spatial x2 upsampler, the duration head and
+the embeddings connector) are implemented and gated. Several limits decide what
+you can actually ask for, and each refuses by name rather than rendering
+something else.
+
+The encoders being present does NOT mean image, keyframe, reference-video or
+reference-audio conditioning is usable: the video engine still refuses every one
+of those by name, because the request-side work between a file on disk and a
+tensor the encoder accepts — image decode, aspect-fill resize, and the H.264 CRF
+re-compression upstream performs before encoding — is not ported. Two
+encoder-level limits are worth stating in advance because they are refusals
+rather than approximations. A reference waveform whose sample rate differs from
+the audio VAE's is refused rather than resampled, since upstream uses a polyphase
+kaiser resampler this project does not carry. And a VAE configured with
+`latent_log_var: none` is refused, because upstream itself raises on it.
+
+**There is no prompt.** The Gemma-4 12B text tower is not ported, so nothing can
+turn words into the conditioning the caption projections consume. Conditioning
+comes from `--prompt-embeds` plus `--audio-prompt-embeds`: rows of
+little-endian f32, 4096 wide for the video stream and 2048 for the audio stream,
+with the same row count in both. Supplying a `--prompt` is refused, and supplying
+only one of the two files is refused, because a stream left unconditioned renders
+instead of failing.
+
+**The DiT config is required when the checkpoint does not carry one.** The
+shipped `vonkaiser` FP8 transformer has no `__metadata__` at all, and the values
+a config decides are ones no tensor shape encodes: `frequencies_precision` and
+`av_ca_timestep_scale_multiplier` move every RoPE angle and every audio/video
+modulation. Defaulting them resolves a different model from the same file, so
+the loader refuses and `--dit-config` supplies LTX-2.5's declared values.
+
+```sh
+ltx2-gen --dit  ltx-2.5-22b-distilled-fp8.safetensors \
+         --dit-config ltx-2.5-transformer-config.json \
+         --model-version 2.5 --allow-unported \
+         --video-vae ltx-2.5-video-vae-conv-bf16.safetensors \
+         --audio-vae ltx-2.5-audio-vae-bf16.safetensors \
+         --upsampler ltx-2.5-latent-spatial-upscaler-x2-bf16-1.0.safetensors \
+         --prompt-embeds video_prompt_embeds.f32 \
+         --audio-prompt-embeds audio_prompt_embeds.f32 \
+         --frames 25 --width 320 --height 192 --seed 20260812 \
+         --device cuda --workdir /tmp/ltx25 --out /tmp/ltx25/video.mp4
+```
+
+`--frames` must satisfy `(frames - 1) % 8 == 0` and width/height must divide by
+64 (32 for the VAE, twice that because the distilled recipe's first phase runs at
+half resolution). Omitting all three renders the recipe default, which is
+1024x1536 at 121 frames and is a much larger request than it looks.
+
+`--upsampler` is what the distilled recipe's second phase needs. Without it that
+phase refuses rather than skipping: its three-step refinement is what makes the
+upscaled latent valid, and decoding the half-resolution latent instead would hand
+back a smaller clip that looks like a completed request. `--max-phase 0` stops
+after the first phase deliberately.
+
+On the server, `--video-family ltx-2.5` pins the family instead of detecting it,
+and `--video-extra KEY=VALUE` (repeatable) carries the same family-specific load
+knobs the flags above map onto. Both are described under
+[the server's video flags](#video-family-and-family-specific-load-knobs).
+
+**Two things about that command are worth knowing before you run it.**
+
+*It is bounded by memory well below the recipe's own defaults.* Staging the
+21.00B FP8 transformer costs about 44 GB on a 119 GB GB10, and the render then
+runs a forward per denoise step on top of that. A 128x128 clip at 9 frames
+completes comfortably through both distilled phases; a 320x192 clip at 25 frames
+consumed a further 58 GB and had to be stopped. Unified memory makes those host
+bytes, and this class of box reboots rather than OOM-killing, so start small and
+grow. The recipe default (1024x1536 at 121 frames) is far beyond what one GB10
+holds today.
+
+*It cannot yet render a scene, and that is conditioning, not the model.* The
+text tower is not ported and neither is the embeddings connector that sits
+between it and the DiT's cross-attention, so whatever you supply as prompt
+embeds reaches the transformer unprocessed. A completed render at the sizes
+above is a structurally valid MP4 of smooth colour fields, not a depiction of
+anything. Treat it as an end-to-end plumbing check.
+
+LTX-2.5 ships two video decoders behind one checkpoint field. The convolutional
+one is implemented; the higher quality diffusion one (`NADiffusionDecoder`) is
+not, and asking for it fails with a message naming the missing
+neighborhood-attention kernel. It never falls back to the convolutional decoder,
+because that would hand back a lower quality render as if it were the one you
+asked for. Keyframe and reference conditioning is refused for the same reason: it
+runs through the video VAE's encoder, and only the decoder is ported.
 
 ### Muse Glimmer: exactly what has been checked
 
@@ -1268,6 +1330,40 @@ The library never spawns a process, so generation and muxing enter through a
 caller-supplied `VideoRunner` callback (`examples/server/main.cpp` supplies one
 that invokes `ffmpeg`, path configurable with `--video-ffmpeg`).
 
+### Video family, and family-specific load knobs
+
+`/v1/videos` serves whichever video family the `--video-dit` checkpoint belongs
+to. By default the family is **detected** from what the checkpoint holds, and
+that is unchanged.
+
+`--video-family NAME` pins it instead. Two registered families exist,
+`minimax-h3` and `ltx-2.5`, and a name outside that set is refused at argument
+parsing, before the text model loads, with the registered names printed. It is
+never a hint: a declared family that cannot load the checkpoint fails loudly
+rather than falling back to detection, because a checkpoint handed to the wrong
+family does not fail, it renders noise.
+
+`--video-extra KEY=VALUE`, repeatable, carries a family's own load knobs. LTX-2.5
+cannot load without `dit_config_path` and `audio_prompt_embeds_path`; MiniMax-H3
+defines `partition`, for which `--video-partition` remains the documented alias.
+A bare `KEY` with no `=` is refused rather than read as an empty value, and a
+`--video-extra partition=X` contradicting `--video-partition Y` is refused rather
+than resolved by whichever assignment ran last. A family refuses any key it does
+not define, so a mistyped knob is an error instead of a silently defaulted
+render.
+
+```sh
+vllm-server --model /path/to/text-model \
+  --video-family ltx-2.5 \
+  --video-dit ltx-2.5-22b-distilled-fp8.safetensors \
+  --video-vae ltx-2.5-video-vae-conv-bf16.safetensors \
+  --audio-vae ltx-2.5-audio-vae-bf16.safetensors \
+  --video-prompt-embeds video_prompt_embeds.f32 \
+  --video-extra dit_config_path=ltx-2.5-transformer-config.json \
+  --video-extra audio_prompt_embeds_path=audio_prompt_embeds.f32 \
+  --video-extra model_version=2.5 --video-extra allow_unported_modules=1
+```
+
 ## Consuming it as a library (C ABI)
 
 Link `libvllm` (static or shared) and include [`include/vllm.h`](../include/vllm.h).

examples/CMakeLists.txt

@@ -189,6 +189,47 @@ if(VLLM_CPP_SERVER)
     add_test(NAME test_server_help COMMAND $<TARGET_FILE:server> --help)
     set_tests_properties(test_server_help PROPERTIES
       PASS_REGULAR_EXPRESSION "--max-num-seqs N.*--max-num-batched-tokens N.*--cuda-profile-graph-replays N.*--cuda-profile-graph-batch N.*--benchmark-shutdown-fifo F.*--enable-force-include-usage.*--\\[no-\\]enable-prefix-caching")
+
+    # --video-family (row MODEL-DIFFUSION-LTX25 / #435). Black-box, because the
+    # flag table lives in an anonymous namespace inside server_main.cpp and there
+    # is no unit seam onto it: the binary IS the surface under test. Both cases
+    # exit during ParseArgs, so neither loads a model, a checkpoint or a byte of
+    # weights, and both run in milliseconds.
+    #
+    # 1. AN UNREGISTERED NAME IS REFUSED, and the refusal NAMES what is
+    #    registered. Before the flag existed this printed "unknown argument
+    #    '--video-family'" instead, which is the RED.
+    add_test(NAME test_server_video_family_unregistered
+      COMMAND $<TARGET_FILE:server> --model /nonexistent-model-dir
+              --video-family not-a-registered-family)
+    set_tests_properties(test_server_video_family_unregistered PROPERTIES
+      PASS_REGULAR_EXPRESSION "--video-family 'not-a-registered-family' is not a registered video family. Registered families:.*ltx-2\\.5.*minimax-h3")
+
+    # 2. A REGISTERED name is ACCEPTED — and specifically, the check consults the
+    #    REGISTRY rather than a list written out here, so the family LTX-2.5
+    #    registers from its own file passes. Proven by getting PAST the family
+    #    check to the next validation in ParseArgs (the tool-parser dialect),
+    #    with the family refusal absent from the output. A hardcoded list that
+    #    omitted ltx-2.5 REDs on the FAIL expression, and a check accidentally
+    #    dropped altogether REDs on it too (the unknown-argument spelling).
+    add_test(NAME test_server_video_family_registered
+      COMMAND $<TARGET_FILE:server> --model /nonexistent-model-dir
+              --video-family ltx-2.5
+              --tool-call-parser definitely-not-a-registered-parser)
+    set_tests_properties(test_server_video_family_registered PROPERTIES
+      PASS_REGULAR_EXPRESSION "unknown tool-call parser \"definitely-not-a-registered-parser\""
+      FAIL_REGULAR_EXPRESSION "not a registered video family;unknown argument")
+
+    # 3. --video-extra takes KEY=VALUE, and a bare KEY is REFUSED rather than
+    #    read as an empty value. An extra that silently became "" is how a
+    #    mistyped knob renders the default and looks like the feature not
+    #    working — the same failure the families' own unknown-extra refusals
+    #    exist to stop, and it would be defeated one layer earlier here.
+    add_test(NAME test_server_video_extra_needs_a_value
+      COMMAND $<TARGET_FILE:server> --model /nonexistent-model-dir
+              --video-extra dit_config_path)
+    set_tests_properties(test_server_video_extra_needs_a_value PROPERTIES
+      PASS_REGULAR_EXPRESSION "--video-extra takes KEY=VALUE, got 'dit_config_path'")
   endif()
 endif()
 
@@ -211,6 +252,16 @@ add_executable(minimax-h3-gen minimax_h3_gen/main.cpp)
 target_link_libraries(minimax-h3-gen PRIVATE vllm::shared)
 vllm_cpp_set_warnings(minimax-h3-gen)
 
+# ltx2-gen: an LTX-2.5 render end to end, the SECOND family behind the generic
+# video seam and a THIN client of the same C ABI. It exists rather than a flag on
+# minimax-h3-gen because LTX-2.5 cannot load without the v18 family-specific
+# extras (the audio-stream prompt embeds, the DiT config the shipped FP8
+# checkpoint does not carry, the latent upsampler) and that example drives
+# neither `family` nor the extras arrays. Row MODEL-DIFFUSION-LTX25 / #435.
+add_executable(ltx2-gen ltx2_gen/main.cpp)
+target_link_libraries(ltx2-gen PRIVATE vllm::shared)
+vllm_cpp_set_warnings(ltx2-gen)
+
 # vulkan-gemm-ab: the VK-C tactic A/B — cooperative matrix vs the portable scalar
 # GEMM, SAME BINARY with VT_VULKAN_COOPMAT as the only variable. Built only in a
 # Vulkan build, because it links the Vulkan context directly to report which