diff --git a/docs/FEATURES.md b/docs/FEATURES.md index 43682ee23..7f3b776bb 100644 --- a/docs/FEATURES.md +++ b/docs/FEATURES.md @@ -153,7 +153,7 @@ they sit outside the gated list above. | Voxtral audio (`VoxtralForConditionalGeneration`) | Voxtral-Mini-3B-2507 | near-tie-robust 16/16 vs vLLM 0.25.0 | decode 0.97x (beats vLLM); encoder FORWARD 15.90x of vLLM's whole TTFT (pin 46.02 ms), or 2.89x with opt-in `VT_WHISPER_ENC_FA2=1` (costs 3 near-tie divergences vs 0). Not a TTFT ratio. Pending | | Whisper audio encoder | openai/whisper-small; whisper-large-v3 (Voxtral cfg) | encoder tower 77/77; large-v3 tower 203/203 | pending | | MiniMax-H3 DiT (`MiniMaxH3DiTModel`, vllm-omni lane) | MiniMax-H3 (33.1B video+audio) | portable 79/79; all three modalities COHERENT on Q4_K_M (§8.20); PRUNED ckpts run, Q8_0 seam 0.9941 (§8.21); ref2va grid was NVFP4 quant error, §8.9 REFUTED; GGUF/NVFP4/bf16 shards stream | FP4/Marlin landed; speed pending; no bf16 render yet. Render from the Q4_K_M GGUF, not the NVFP4 arm. Krea 2 text-to-image (roadmap C11) is scoped to reuse these DiT seams | -| LTX-2.5 DiT (`LTX2VideoTransformer3DModel`, Lightricks lane) | LTX-2.5 (21.00B video+audio) | `SPIKE`. DiT, both VAE decoders, both VAE ENCODERS + mel front-end, the conditioning items, the pipeline and the quantized loaders gated vs upstream at reduced dims: max\|diff\| 1.2e-07 DiT, 1.8e-06 VAE decode, 6.0e-07 VAE encode, 2.8e-06 pipeline. STRUCTURAL e2e only | Fits ONE GB10 at ~29 GB NVFP4. FP8 + torchao-NVFP4 arms load. CPU f32, CUDA bf16 device-resident. Text tower, DiffVAE, LoRA, image conditioning all still refused AT THE ENGINE. FP4-resident arm OWED. Speed PENDING | +| LTX-2.5 DiT (`LTX2VideoTransformer3DModel`, Lightricks lane) | LTX-2.5 (21.00B video+audio) | `SPIKE`. DiT, both VAE decoders and ENCODERS, conditioning items, pipeline and quantized loaders gated vs upstream at reduced dims. Renders e2e at 128x128/9f, valid MP4+WAV; frames are NOT a scene | Family `ltx-2.5` via `ltx2-gen`. ~29 GB NVFP4 on one GB10. FP8 and torchao-NVFP4 load; first-party NVFP4 does not. Text tower, connector, DiffVAE, LoRA and image conditioning refused AT THE ENGINE. Speed PENDING | | MTP speculator | Qwen3.6-27B, Qwen3.6-35B-A3B | token-identical to vLLM `mtp` at c1 | ~4% faster c1; +16% output tput (MoE) | | DFlash block-diffusion | Qwen3 (DFlash draft) | near-tie e2e 27/27 vs vLLM | 2.9x over spec-off, 1.003x vs vLLM DFlash-on | | DeepSeek-V4 MTP | DeepSeek-V4-Flash (nextn head) | lossless 5/5; real-model weight-blocked | pending | @@ -267,7 +267,7 @@ Build with `-DVLLM_CPP_VULKAN=ON`; off by default. | Multiple engines in one process (build, destroy, rebuild) | ✅ resident device state is owned by the weights, so a new engine never inherits a freed one's pointers | ✅ | ✅ | ✅ | | LoRA adapters | ☐ CPU brick only | ✅ | ✅ | ✅ | | Embedding / pooling endpoints | ◐ `/v1/embeddings` live (task=embed; score/rerank/classify pending) | ✅ | ✅ | ✅ | -| OpenAI video generation `/v1/videos` (Sora shape) | ✅ `model`/`size`/`seconds` aliases + `GET /{id}/content`; `input_reference` and the `metadata` video/audio references condition the render | ◐ (vllm-omni, its own request shape) | ☐ | ☐ | +| OpenAI video generation `/v1/videos` (Sora shape) | ✅ `model`/`size`/`seconds` aliases + `GET /{id}/content`; `input_reference` and `metadata` references condition the render; `--video-family` pins the family (default DETECT), `--video-extra K=V` carries family knobs | ◐ (vllm-omni, its own request shape) | ☐ | ☐ | | Flat C ABI for embedding in other languages | ✅ versioned | ☐ | ☐ | ✅ | #### C-ABI capability coverage diff --git a/docs/USAGE.md b/docs/USAGE.md index 91566a501..b61d53df6 100644 --- a/docs/USAGE.md +++ b/docs/USAGE.md @@ -346,36 +346,98 @@ tokens quietly. This is a deliberate state, not a bug: registering the architecture is what lets the config parse and weight-name mapping be tested before the forward exists. -### LTX-2.5 has no user-facing entrypoint yet - -LTX-2.5 is being ported in phases. Its two VAE decoders, its two VAE ENCODERS -with the mel front-end, the conditioning items that place encoded latents into -the token stream, and its pipeline layer (the sigma schedule, the diffusion -steps, guidance, the latent spatial x2 upsampler, the duration head and the -embeddings connector) are implemented and gated, but no CLI flag, server -endpoint or C ABI call reaches them, so there is nothing to run here yet and no -request shape to document. Do not infer from [FEATURES](FEATURES.md) that a -render is available. - -In particular, the encoders being present does NOT mean image, keyframe, -reference-video or reference-audio conditioning is usable: the video engine -still refuses every one of those by name, because the request-side work between -a file on disk and a tensor the encoder accepts — image decode, aspect-fill -resize, and the H.264 CRF re-compression upstream performs before encoding — is -not ported. Two encoder-level limits are worth stating in advance because they -are refusals rather than approximations. A reference waveform whose sample rate -differs from the audio VAE's is refused rather than resampled, since upstream -uses a polyphase kaiser resampler this project does not carry. And a VAE -configured with `latent_log_var: none` is refused, because upstream itself -raises on it. - -One behaviour is worth stating in advance, because it decides what you get when -the entrypoint does arrive. LTX-2.5 ships two video decoders behind one -checkpoint field. The convolutional one is implemented; the higher quality -diffusion one (`NADiffusionDecoder`) is not, and asking for it fails with a -message naming the missing neighborhood-attention kernel. It never falls back to -the convolutional decoder, because that would hand back a lower quality render as -if it were the one you asked for. +### LTX-2.5: what runs, and what it cannot do + +LTX-2.5 is reachable as video family `ltx-2.5`, through the same +`vllm_video_engine_load` / `vllm_video_generate` C ABI that serves MiniMax-H3, +and through the `ltx2-gen` example that drives it. Its two VAE decoders, its two +VAE ENCODERS with the mel front-end, the conditioning items that place encoded +latents into the token stream, and its pipeline layer (the sigma schedule, the +diffusion steps, guidance, the latent spatial x2 upsampler, the duration head and +the embeddings connector) are implemented and gated. Several limits decide what +you can actually ask for, and each refuses by name rather than rendering +something else. + +The encoders being present does NOT mean image, keyframe, reference-video or +reference-audio conditioning is usable: the video engine still refuses every one +of those by name, because the request-side work between a file on disk and a +tensor the encoder accepts — image decode, aspect-fill resize, and the H.264 CRF +re-compression upstream performs before encoding — is not ported. Two +encoder-level limits are worth stating in advance because they are refusals +rather than approximations. A reference waveform whose sample rate differs from +the audio VAE's is refused rather than resampled, since upstream uses a polyphase +kaiser resampler this project does not carry. And a VAE configured with +`latent_log_var: none` is refused, because upstream itself raises on it. + +**There is no prompt.** The Gemma-4 12B text tower is not ported, so nothing can +turn words into the conditioning the caption projections consume. Conditioning +comes from `--prompt-embeds` plus `--audio-prompt-embeds`: rows of +little-endian f32, 4096 wide for the video stream and 2048 for the audio stream, +with the same row count in both. Supplying a `--prompt` is refused, and supplying +only one of the two files is refused, because a stream left unconditioned renders +instead of failing. + +**The DiT config is required when the checkpoint does not carry one.** The +shipped `vonkaiser` FP8 transformer has no `__metadata__` at all, and the values +a config decides are ones no tensor shape encodes: `frequencies_precision` and +`av_ca_timestep_scale_multiplier` move every RoPE angle and every audio/video +modulation. Defaulting them resolves a different model from the same file, so +the loader refuses and `--dit-config` supplies LTX-2.5's declared values. + +```sh +ltx2-gen --dit ltx-2.5-22b-distilled-fp8.safetensors \ + --dit-config ltx-2.5-transformer-config.json \ + --model-version 2.5 --allow-unported \ + --video-vae ltx-2.5-video-vae-conv-bf16.safetensors \ + --audio-vae ltx-2.5-audio-vae-bf16.safetensors \ + --upsampler ltx-2.5-latent-spatial-upscaler-x2-bf16-1.0.safetensors \ + --prompt-embeds video_prompt_embeds.f32 \ + --audio-prompt-embeds audio_prompt_embeds.f32 \ + --frames 25 --width 320 --height 192 --seed 20260812 \ + --device cuda --workdir /tmp/ltx25 --out /tmp/ltx25/video.mp4 +``` + +`--frames` must satisfy `(frames - 1) % 8 == 0` and width/height must divide by +64 (32 for the VAE, twice that because the distilled recipe's first phase runs at +half resolution). Omitting all three renders the recipe default, which is +1024x1536 at 121 frames and is a much larger request than it looks. + +`--upsampler` is what the distilled recipe's second phase needs. Without it that +phase refuses rather than skipping: its three-step refinement is what makes the +upscaled latent valid, and decoding the half-resolution latent instead would hand +back a smaller clip that looks like a completed request. `--max-phase 0` stops +after the first phase deliberately. + +On the server, `--video-family ltx-2.5` pins the family instead of detecting it, +and `--video-extra KEY=VALUE` (repeatable) carries the same family-specific load +knobs the flags above map onto. Both are described under +[the server's video flags](#video-family-and-family-specific-load-knobs). + +**Two things about that command are worth knowing before you run it.** + +*It is bounded by memory well below the recipe's own defaults.* Staging the +21.00B FP8 transformer costs about 44 GB on a 119 GB GB10, and the render then +runs a forward per denoise step on top of that. A 128x128 clip at 9 frames +completes comfortably through both distilled phases; a 320x192 clip at 25 frames +consumed a further 58 GB and had to be stopped. Unified memory makes those host +bytes, and this class of box reboots rather than OOM-killing, so start small and +grow. The recipe default (1024x1536 at 121 frames) is far beyond what one GB10 +holds today. + +*It cannot yet render a scene, and that is conditioning, not the model.* The +text tower is not ported and neither is the embeddings connector that sits +between it and the DiT's cross-attention, so whatever you supply as prompt +embeds reaches the transformer unprocessed. A completed render at the sizes +above is a structurally valid MP4 of smooth colour fields, not a depiction of +anything. Treat it as an end-to-end plumbing check. + +LTX-2.5 ships two video decoders behind one checkpoint field. The convolutional +one is implemented; the higher quality diffusion one (`NADiffusionDecoder`) is +not, and asking for it fails with a message naming the missing +neighborhood-attention kernel. It never falls back to the convolutional decoder, +because that would hand back a lower quality render as if it were the one you +asked for. Keyframe and reference conditioning is refused for the same reason: it +runs through the video VAE's encoder, and only the decoder is ported. ### Muse Glimmer: exactly what has been checked @@ -1268,6 +1330,40 @@ The library never spawns a process, so generation and muxing enter through a caller-supplied `VideoRunner` callback (`examples/server/main.cpp` supplies one that invokes `ffmpeg`, path configurable with `--video-ffmpeg`). +### Video family, and family-specific load knobs + +`/v1/videos` serves whichever video family the `--video-dit` checkpoint belongs +to. By default the family is **detected** from what the checkpoint holds, and +that is unchanged. + +`--video-family NAME` pins it instead. Two registered families exist, +`minimax-h3` and `ltx-2.5`, and a name outside that set is refused at argument +parsing, before the text model loads, with the registered names printed. It is +never a hint: a declared family that cannot load the checkpoint fails loudly +rather than falling back to detection, because a checkpoint handed to the wrong +family does not fail, it renders noise. + +`--video-extra KEY=VALUE`, repeatable, carries a family's own load knobs. LTX-2.5 +cannot load without `dit_config_path` and `audio_prompt_embeds_path`; MiniMax-H3 +defines `partition`, for which `--video-partition` remains the documented alias. +A bare `KEY` with no `=` is refused rather than read as an empty value, and a +`--video-extra partition=X` contradicting `--video-partition Y` is refused rather +than resolved by whichever assignment ran last. A family refuses any key it does +not define, so a mistyped knob is an error instead of a silently defaulted +render. + +```sh +vllm-server --model /path/to/text-model \ + --video-family ltx-2.5 \ + --video-dit ltx-2.5-22b-distilled-fp8.safetensors \ + --video-vae ltx-2.5-video-vae-conv-bf16.safetensors \ + --audio-vae ltx-2.5-audio-vae-bf16.safetensors \ + --video-prompt-embeds video_prompt_embeds.f32 \ + --video-extra dit_config_path=ltx-2.5-transformer-config.json \ + --video-extra audio_prompt_embeds_path=audio_prompt_embeds.f32 \ + --video-extra model_version=2.5 --video-extra allow_unported_modules=1 +``` + ## Consuming it as a library (C ABI) Link `libvllm` (static or shared) and include [`include/vllm.h`](../include/vllm.h). diff --git a/examples/CMakeLists.txt b/examples/CMakeLists.txt index 01b513e31..f351412b9 100644 --- a/examples/CMakeLists.txt +++ b/examples/CMakeLists.txt @@ -189,6 +189,47 @@ if(VLLM_CPP_SERVER) add_test(NAME test_server_help COMMAND $ --help) set_tests_properties(test_server_help PROPERTIES PASS_REGULAR_EXPRESSION "--max-num-seqs N.*--max-num-batched-tokens N.*--cuda-profile-graph-replays N.*--cuda-profile-graph-batch N.*--benchmark-shutdown-fifo F.*--enable-force-include-usage.*--\\[no-\\]enable-prefix-caching") + + # --video-family (row MODEL-DIFFUSION-LTX25 / #435). Black-box, because the + # flag table lives in an anonymous namespace inside server_main.cpp and there + # is no unit seam onto it: the binary IS the surface under test. Both cases + # exit during ParseArgs, so neither loads a model, a checkpoint or a byte of + # weights, and both run in milliseconds. + # + # 1. AN UNREGISTERED NAME IS REFUSED, and the refusal NAMES what is + # registered. Before the flag existed this printed "unknown argument + # '--video-family'" instead, which is the RED. + add_test(NAME test_server_video_family_unregistered + COMMAND $ --model /nonexistent-model-dir + --video-family not-a-registered-family) + set_tests_properties(test_server_video_family_unregistered PROPERTIES + PASS_REGULAR_EXPRESSION "--video-family 'not-a-registered-family' is not a registered video family. Registered families:.*ltx-2\\.5.*minimax-h3") + + # 2. A REGISTERED name is ACCEPTED — and specifically, the check consults the + # REGISTRY rather than a list written out here, so the family LTX-2.5 + # registers from its own file passes. Proven by getting PAST the family + # check to the next validation in ParseArgs (the tool-parser dialect), + # with the family refusal absent from the output. A hardcoded list that + # omitted ltx-2.5 REDs on the FAIL expression, and a check accidentally + # dropped altogether REDs on it too (the unknown-argument spelling). + add_test(NAME test_server_video_family_registered + COMMAND $ --model /nonexistent-model-dir + --video-family ltx-2.5 + --tool-call-parser definitely-not-a-registered-parser) + set_tests_properties(test_server_video_family_registered PROPERTIES + PASS_REGULAR_EXPRESSION "unknown tool-call parser \"definitely-not-a-registered-parser\"" + FAIL_REGULAR_EXPRESSION "not a registered video family;unknown argument") + + # 3. --video-extra takes KEY=VALUE, and a bare KEY is REFUSED rather than + # read as an empty value. An extra that silently became "" is how a + # mistyped knob renders the default and looks like the feature not + # working — the same failure the families' own unknown-extra refusals + # exist to stop, and it would be defeated one layer earlier here. + add_test(NAME test_server_video_extra_needs_a_value + COMMAND $ --model /nonexistent-model-dir + --video-extra dit_config_path) + set_tests_properties(test_server_video_extra_needs_a_value PROPERTIES + PASS_REGULAR_EXPRESSION "--video-extra takes KEY=VALUE, got 'dit_config_path'") endif() endif() @@ -211,6 +252,16 @@ add_executable(minimax-h3-gen minimax_h3_gen/main.cpp) target_link_libraries(minimax-h3-gen PRIVATE vllm::shared) vllm_cpp_set_warnings(minimax-h3-gen) +# ltx2-gen: an LTX-2.5 render end to end, the SECOND family behind the generic +# video seam and a THIN client of the same C ABI. It exists rather than a flag on +# minimax-h3-gen because LTX-2.5 cannot load without the v18 family-specific +# extras (the audio-stream prompt embeds, the DiT config the shipped FP8 +# checkpoint does not carry, the latent upsampler) and that example drives +# neither `family` nor the extras arrays. Row MODEL-DIFFUSION-LTX25 / #435. +add_executable(ltx2-gen ltx2_gen/main.cpp) +target_link_libraries(ltx2-gen PRIVATE vllm::shared) +vllm_cpp_set_warnings(ltx2-gen) + # vulkan-gemm-ab: the VK-C tactic A/B — cooperative matrix vs the portable scalar # GEMM, SAME BINARY with VT_VULKAN_COOPMAT as the only variable. Built only in a # Vulkan build, because it links the Vulkan context directly to report which diff --git a/examples/ltx2_gen/main.cpp b/examples/ltx2_gen/main.cpp new file mode 100644 index 000000000..1da8931dc --- /dev/null +++ b/examples/ltx2_gen/main.cpp @@ -0,0 +1,230 @@ +// ltx2-gen — an LTX-2.5 video+audio render, end to end, as a THIN CLIENT of the +// public C ABI (include/vllm.h) and NOTHING else, per the ONE SURFACE directive +// (ARCH-ONE-SURFACE ROW 2). Row MODEL-DIFFUSION-LTX25 phase L9B, issue #435. +// +// WHY A SECOND GENERATION EXAMPLE AND NOT A FLAG ON minimax-h3-gen. The ABI's +// video slice went family-generic at v18: `family` selects the model family and +// two parallel string arrays carry the FAMILY-SPECIFIC load knobs +// (vllm.h:735-747). `minimax-h3-gen` predates that and drives neither, and +// LTX-2.5 CANNOT LOAD without three of them — the audio-stream prompt embeds, +// the DiT config the shipped FP8 checkpoint does not carry, and (for the second +// distilled phase) the latent spatial upsampler. This file exists to name those +// knobs as flags rather than to hand a user a `--extra key=value` grab bag, and +// it is the smallest thing that can drive a real render. +// +// WHAT IT DOES NOT DO. It composes no ffmpeg command line of its own and encodes +// nothing: `vllm_video_mux_argv` builds the argv and this file exec's it, which +// is the ratified process boundary (2026-08-03). It carries no model logic — no +// noise stream, no schedule, no dtype choice — because all of that is the +// library's and a second copy here would be a parallel path. +// +// CONDITIONING IS PROMPT-EMBEDS ONLY, and that is a stated limitation rather +// than a design: the Gemma-4 12B text tower is owed (spec .agents/specs/ltx-2-5.md +// §1.4), so `--prompt` does not exist here. The engine refuses a prompt by name +// rather than rendering the supplied embeddings as though they were it. Both +// streams are conditioned or neither: LTX-2.5 cross-attends at TWO widths +// (4096 video, 2048 audio) and one of them alone leaves a stream unconditioned, +// which renders instead of failing. +#include +#include + +#include +#include +#include +#include +#include + +#include "vllm.h" + +namespace { + +// The ONE process spawn, in examples/ by decision: the library composed `args`; +// this runs it. No shell — the argv is exec'd directly. +int RunFfmpeg(const std::vector& args) { + std::vector argv; + argv.reserve(args.size() + 1); + for (const std::string& a : args) argv.push_back(const_cast(a.c_str())); + argv.push_back(nullptr); + const pid_t pid = fork(); + if (pid < 0) { + std::fprintf(stderr, "error: fork failed\n"); + return -1; + } + if (pid == 0) { + execvp(argv[0], argv.data()); + _exit(127); + } + int status = 0; + if (waitpid(pid, &status, 0) < 0) { + std::fprintf(stderr, "error: waitpid failed\n"); + return -1; + } + if (WIFSIGNALED(status)) { + std::fprintf(stderr, "error: ffmpeg died on signal %d\n", WTERMSIG(status)); + return -1; + } + return WIFEXITED(status) ? WEXITSTATUS(status) : -1; +} + +const char* Need(int argc, char** argv, int i, const char* flag) { + if (i >= argc) { + std::fprintf(stderr, "error: missing value for %s\n", flag); + std::exit(2); + } + return argv[i]; +} + +[[noreturn]] void Usage(int code) { + std::fprintf( + stderr, + "usage: ltx2-gen --dit --video-vae --audio-vae \n" + " --prompt-embeds --audio-prompt-embeds \n" + " --workdir DIR [--out ] [--ffmpeg PATH]\n" + " [--dit-config ] REQUIRED when the DiT\n" + " carries no __metadata__\n" + " [--model-version 2.5] [--pipeline-kind distilled_two_stage]\n" + " [--upsampler ] phase 2 needs it\n" + " [--max-phase N] [--allow-unported]\n" + " [--frames N] [--width N] [--height N] [--seed N]\n" + " [--device cpu|cuda]\n\n" + "Renders LTX-2.5 (family \"ltx-2.5\") through vllm_video_engine_load +\n" + "vllm_video_generate. Conditioning is PROMPT-EMBEDS: both files are rows of\n" + "little-endian f32, the video one 4096 wide and the audio one 2048, with the\n" + "SAME row count. There is no --prompt: the Gemma-4 text tower is not ported and\n" + "the engine refuses a prompt rather than silently rendering these embeddings\n" + "as if they were it.\n"); + std::exit(code); +} + +} // namespace + +int main(int argc, char** argv) { + vllm_video_model_params mp = vllm_video_model_params_default(); + vllm_video_params vp = vllm_video_params_default(); + std::string workdir = "/tmp/ltx2_gen", out_path, ffmpeg = "ffmpeg", device = "cuda"; + + // The extras are BORROWED by the load call, so the strings must outlive it. + // Kept as two parallel vectors of owned strings plus the char* views the ABI + // takes, built once after parsing. + std::vector extra_keys, extra_values; + auto SetExtra = [&](const char* key, std::string value) { + for (size_t i = 0; i < extra_keys.size(); ++i) { + if (extra_keys[i] == key) { + extra_values[i] = std::move(value); + return; + } + } + extra_keys.emplace_back(key); + extra_values.push_back(std::move(value)); + }; + + for (int i = 1; i < argc; ++i) { + const std::string f = argv[i]; + if (f == "--dit") mp.dit_path = Need(argc, argv, ++i, "--dit"); + else if (f == "--video-vae") mp.video_vae_path = Need(argc, argv, ++i, "--video-vae"); + else if (f == "--video-vae-config") mp.video_vae_config_path = Need(argc, argv, ++i, f.c_str()); + else if (f == "--audio-vae") mp.audio_vae_path = Need(argc, argv, ++i, "--audio-vae"); + else if (f == "--audio-vae-config") mp.audio_vae_config_path = Need(argc, argv, ++i, f.c_str()); + else if (f == "--prompt-embeds") mp.prompt_embeds_path = Need(argc, argv, ++i, f.c_str()); + else if (f == "--audio-prompt-embeds") + SetExtra("audio_prompt_embeds_path", Need(argc, argv, ++i, f.c_str())); + else if (f == "--dit-config") SetExtra("dit_config_path", Need(argc, argv, ++i, f.c_str())); + else if (f == "--model-version") SetExtra("model_version", Need(argc, argv, ++i, f.c_str())); + else if (f == "--pipeline-kind") SetExtra("pipeline_kind", Need(argc, argv, ++i, f.c_str())); + else if (f == "--upsampler") SetExtra("upsampler_path", Need(argc, argv, ++i, f.c_str())); + else if (f == "--duration-head") + SetExtra("duration_head_path", Need(argc, argv, ++i, f.c_str())); + else if (f == "--max-phase") SetExtra("max_phase", Need(argc, argv, ++i, f.c_str())); + else if (f == "--allow-unported") SetExtra("allow_unported_modules", "1"); + else if (f == "--device") device = Need(argc, argv, ++i, "--device"); + else if (f == "--frames") vp.num_frames = std::atoi(Need(argc, argv, ++i, "--frames")); + else if (f == "--width") vp.width = std::atoi(Need(argc, argv, ++i, "--width")); + else if (f == "--height") vp.height = std::atoi(Need(argc, argv, ++i, "--height")); + else if (f == "--seed") { + vp.seed = static_cast(std::strtoull(Need(argc, argv, ++i, "--seed"), nullptr, 10)); + vp.has_seed = 1; + } + else if (f == "--workdir") workdir = Need(argc, argv, ++i, "--workdir"); + else if (f == "--out") out_path = Need(argc, argv, ++i, "--out"); + else if (f == "--ffmpeg") ffmpeg = Need(argc, argv, ++i, "--ffmpeg"); + else if (f == "--help" || f == "-h") Usage(0); + else { + std::fprintf(stderr, "error: unknown argument: %s\n", f.c_str()); + Usage(2); + } + } + if (mp.dit_path == nullptr) Usage(2); + if (device == "cuda") mp.device = 1; + else if (device != "cpu") { + std::fprintf(stderr, "error: --device must be cpu or cuda\n"); + return 2; + } + // DECLARED, never detected. Detection would also resolve this checkpoint, but + // an explicit family is what makes an FP8-vs-NVFP4 comparison a statement + // about the two files rather than about what a detector happened to claim. + mp.family = "ltx-2.5"; + vp.output_dir = workdir.c_str(); + + std::vector keys, values; + keys.reserve(extra_keys.size()); + values.reserve(extra_values.size()); + for (size_t i = 0; i < extra_keys.size(); ++i) { + keys.push_back(extra_keys[i].c_str()); + values.push_back(extra_values[i].c_str()); + } + if (!keys.empty()) { + mp.extra_keys = keys.data(); + mp.extra_values = values.data(); + mp.n_extras = static_cast(keys.size()); + } + + vllm_video_engine* engine = nullptr; + if (vllm_video_engine_load(&mp, &engine) != VLLM_OK) { + std::fprintf(stderr, "error: %s\n", vllm_last_error()); + return 1; + } + // Which family actually loaded, from the handle rather than from the request: + // spec §3.1 requires every artifact to name what produced it. + std::fprintf(stderr, "ltx2-gen: family=%s dit=%s\n", vllm_video_engine_family(engine), + mp.dit_path); + + vllm_video_result out; + if (vllm_video_generate(engine, &vp, &out) != VLLM_OK) { + std::fprintf(stderr, "error: %s\n", vllm_last_error()); + vllm_video_engine_free(engine); + return 1; + } + std::fprintf(stderr, " wrote %d frames (%dx%d @ %d fps) + %s (%d Hz)\n", out.frame_count, + out.width, out.height, out.fps, out.audio_path, out.sample_rate); + + int status = 0; + if (!out_path.empty()) { + const std::string pattern = std::string(out.frame_dir) + "/frame_%06d.ppm"; + vllm_video_mux_params mx = vllm_video_mux_params_default(); + mx.frames = pattern.c_str(); + mx.audio_path = out.audio_path; + mx.output_path = out_path.c_str(); + mx.fps = out.fps; // the RECIPE's frame rate, not the mux default + char** mux_argv = nullptr; + int32_t mux_argc = 0; + if (vllm_video_mux_argv(&mx, &mux_argv, &mux_argc) != VLLM_OK) { + std::fprintf(stderr, "error: %s\n", vllm_last_error()); + vllm_video_result_free(&out); + vllm_video_engine_free(engine); + return 1; + } + std::vector args(mux_argv, mux_argv + mux_argc); + if (!args.empty()) args[0] = ffmpeg; + vllm_video_mux_argv_free(mux_argv, mux_argc); + status = RunFfmpeg(args); + if (status == 0) { + std::printf("wrote %s\n", out_path.c_str()); + } else { + std::fprintf(stderr, "ffmpeg exited %d\n", status); + } + } + + vllm_video_result_free(&out); + vllm_video_engine_free(engine); + return status; +} diff --git a/src/vllm/entrypoints/openai/server_main.cpp b/src/vllm/entrypoints/openai/server_main.cpp index d1c23733b..02f0c1fb0 100644 --- a/src/vllm/entrypoints/openai/server_main.cpp +++ b/src/vllm/entrypoints/openai/server_main.cpp @@ -28,6 +28,7 @@ // this binary is only built + smoke-tested against a synthetic engine (see // tests/vllm/entrypoints/openai/test_api_server.cpp). The wiring below is the // same either way. +#include #include #include #include @@ -84,6 +85,7 @@ #include "vllm/model_executor/models/model_registry.h" #include "vllm/multimodal/minimax_h3_video.h" #include "vllm/multimodal/parakeet_transcription.h" +#include "vllm/multimodal/video_engine.h" #include "vllm/tokenizer/tokenizer.h" #include "vllm/version.h" #include "vllm/v1/core/kv_cache_utils.h" @@ -194,6 +196,18 @@ struct Args { int video_encoder_max_layers = 50; std::string video_ffmpeg = "ffmpeg", video_device = "cuda"; std::string video_partition; // served partition (fl2va|ref2va); see the #77 guard + // The video model FAMILY to load. EMPTY keeps detection, which is the default + // and what every pre-L9B invocation gets. See the --video-family block below + // for why it now exists. + std::string video_family; + // FAMILY-SPECIFIC load knobs, `--video-extra KEY=VALUE`, repeatable. Pinning a + // family is useless if that family's required load knobs are unreachable: + // LTX-2.5 cannot load without `dit_config_path` (the shipped FP8 DiT carries + // no __metadata__) and `audio_prompt_embeds_path` (its audio stream conditions + // at a second width), and neither has — or should have — a dedicated flag on a + // family-generic server. `--video-partition` remains the documented alias for + // the H3 key "partition". + std::vector> video_extras; // Keep-quant is the library seam's DEFAULT arm; --video-dequant-bf16 selects // the bf16 dequant/stream arm (the throughput trade the gen example ships). // --video-keep-quant is still accepted (it names the default). @@ -362,6 +376,20 @@ Args ParseArgs(int argc, char** argv) { a.video_device = NextArg(argc, argv, i, argv[0]); } else if (flag == "--video-partition") { a.video_partition = NextArg(argc, argv, i, argv[0]); + } else if (flag == "--video-family") { + a.video_family = NextArg(argc, argv, i, argv[0]); + } else if (flag == "--video-extra") { + const std::string kv = NextArg(argc, argv, i, argv[0]); + const std::string::size_type eq = kv.find('='); + // A bare KEY is refused rather than read as KEY="": an extra whose value + // silently became empty is how a mistyped knob renders the default and + // looks like the feature not working, which is the same failure the + // families' own unknown-extra refusals exist to stop. + if (eq == std::string::npos || eq == 0) { + std::cerr << "server: --video-extra takes KEY=VALUE, got '" << kv << "'\n"; + Usage(argv[0], 2); + } + a.video_extras.emplace_back(kv.substr(0, eq), kv.substr(eq + 1)); } else if (flag == "--video-keep-quant") { // the seam's default arm; accepted for pre-fold CLI compatibility a.video_dequant_bf16 = false; @@ -469,6 +497,24 @@ Args ParseArgs(int argc, char** argv) { std::cerr << "server: --cuda-profile-graph-batch exceeds --max-num-seqs\n"; Usage(argv[0], 2); } + // The declared video family, checked HERE for the same reason the parser + // dialects below are: `LoadVideoEngine` would refuse an unregistered name + // anyway, but only after the TEXT model has loaded, so a typo would cost a + // multi-GB load instead of a second. The REGISTRY is the authority — never a + // literal list here — so a family added in its own file is accepted with no + // edit to this one, which is the whole point of the registration seam. + if (!a.video_family.empty()) { + const std::vector registered = + vllm::multimodal::RegisteredVideoFamilies(); + if (std::find(registered.begin(), registered.end(), a.video_family) == + registered.end()) { + std::cerr << "server: --video-family '" << a.video_family + << "' is not a registered video family. Registered families:"; + for (const std::string& name : registered) std::cerr << " " << name; + std::cerr << "\n"; + Usage(argv[0], 2); + } + } // Validate a NAMED parser dialect here, before the (multi-GB) model load, so a // typo costs a second rather than a full load. "auto" cannot be checked yet — // it resolves against the chat template — but detection only ever returns @@ -911,12 +957,37 @@ int VllmServerMain(int argc, char** argv) { vmp.prompt_embeds_path = args.video_prompt_embeds; // The H3-specific partition rides in the generic extras (LTX-2.5 L1). if (!args.video_partition.empty()) vmp.extras["partition"] = args.video_partition; + // ...and every other family-specific knob rides there too, from + // --video-extra KEY=VALUE. Applied AFTER the partition alias so the two + // spellings of one key cannot disagree silently: a --video-extra + // partition=X that contradicts --video-partition Y is refused by name + // rather than resolved by whichever assignment ran last. + for (const auto& kv : args.video_extras) { + const auto existing = vmp.extras.find(kv.first); + if (existing != vmp.extras.end() && existing->second != kv.second) { + throw std::runtime_error("server: --video-extra " + kv.first + "=" + kv.second + + " contradicts the value already supplied for '" + kv.first + + "' ('" + existing->second + + "'). Refusing rather than preferring one."); + } + vmp.extras[kv.first] = kv.second; + } vmp.device = args.video_device == "cuda" ? 1 : 0; vmp.dequant_bf16 = args.video_dequant_bf16 ? 1 : 0; vmp.encoder_max_layers = args.video_encoder_max_layers; - // The family is DETECTED from the checkpoint; no --video-family flag is - // invented here until a second family exists to disambiguate. + // --video-family PINS the family; empty keeps detection, which is what + // every invocation before this flag existed got and still gets. The flag + // exists now because a SECOND family is registered (LTX-2.5), and the two + // shipped LTX DiTs are separate files whose comparison is only a statement + // about the files if the family is declared rather than inferred. It is + // never a hint: an unregistered name was already refused at ParseArgs, and + // a declared family that cannot load the checkpoint fails loudly instead + // of falling back to detection. + vmp.family = args.video_family; video_engine = vllm::multimodal::LoadVideoEngine(vmp); + std::cerr << "server: video family " + << (args.video_family.empty() ? "DETECTED" : "DECLARED (--video-family)") + << "\n"; std::cerr << "server: /v1/videos on (family=" << video_engine->family() << ", device=" << args.video_device << (args.video_dequant_bf16 ? ", dequant-bf16" : ", keep-quant") << ")\n";