port(ltx-2.5): L9C -- the connector on the render path, and a pool that is drained - #574
port(ltx-2.5): L9C -- the connector on the render path, and a pool that is drained#574localai-bot wants to merge 2 commits into
Conversation
…at is drained
FOLLOWING_AGENTS_PROTOCOL
Two gaps stood between the L9B render and a real scene at a real resolution.
This closes the second one and makes the first REACHABLE rather than closing it.
THE CONNECTOR WAS NEVER CALLED. `Ltx2ConnectorForward` landed at L5, gated
against upstream on five arms, and was called by NOTHING outside its own test.
The render path handed the prompt-embeds file straight to the DiT's
cross-attention, and the two `*_embeddings_connector` families the shipped DiT
carries -- 129 tensors each, in the file -- were refused as "unported" and
stepped over. Upstream never does that: `EmbeddingsProcessor` runs an 8-layer
1-D transformer over the caption projections before the DiT sees them
(embeddings_processor.py:70-95), and those weights ship INSIDE the DiT file
because upstream's key ops rewrite them into the text encoder
(encoder_configurator.py:331-346). So they load here, beside the DiT and not
inside it, and the render applies them.
PRESENCE DECIDES, not a flag: the connector runs exactly when the checkpoint
carries it, which is the rule `DetectLtx2Video` already uses on the DiT. Half a
connector is refused -- conditioning the picture on eight transformer layers and
the soundtrack on none is not a degraded render, it is two different requests.
THE CONFIG IS READ, AND ONE OF ITS VALUES IS NOT NEAR ITS DEFAULT. LTX-2.5
declares `connector_positional_embedding_max_pos = [4096]` where
`Embeddings1DConnector`'s class default is `[1]`, and `get_fractional_positions`
DIVIDES the token index by it (rope.py:132-141). No shape can see the
difference; the default is every RoPE angle wrong. `Ltx2ParseConnectorConfig`
mirrors both configurators key for key, including the audio one's fallback onto
the VIDEO spellings, and deliberately does NOT read the DiT's
`positional_embedding_theta`, because neither configurator passes it.
THE PROCESSOR IS A SEPARATE PORT, and it carries the thing the connector's own
goldens cannot see: upstream RIGHT-PAD-SORTS the features before the connector
("Connectors expect right-padded input"), because the register table is indexed
by ABSOLUTE position. A left-padded batch handed straight in puts registers
where caption tokens belong and is finite, correctly shaped and conditioned on
nothing. It also zeroes the VIDEO encoding at masked positions and NOT the audio
one; that asymmetry is upstream's and is mirrored rather than tidied.
WHERE THE TWO REFERENCES DISAGREE, reported rather than silently picked:
`diffusers` folds that sort INTO the connector
(pipelines/ltx2/connectors.py) and its comment claims that matches "the original
LTX implementation" -- true only because `ltx_core` sorts one level up, in the
processor. They compose to the same function and differ in which module owns it.
This follows `ltx_core`, so `Ltx2ConnectorForward` stays a faithful port of
`Embeddings1DConnector`.
THE POOL WAS NEVER DRAINED. MiniMax-H3 calls `ActivePool()->Drain()` at the
denoise-to-decode boundary and records why (minimax_h3_pipeline.cpp:546-563);
the LTX path called it nowhere, and GB10's `device_pool_cap_bytes` is 0, so
every activation size class a phase touched was retained and unreusable by the
next phase at twice the resolution. On unified memory those are host bytes and
this box reboots instead of OOM-killing. The drain lands at every phase
boundary, which is also the denoise-to-decode boundary for the last phase.
`VLLM_LTX2_POOL_DRAIN=0` exists ONLY so the A/B that measures it runs on one
binary; it is not a configuration.
WHAT IS STILL SYNTHETIC, stated so it cannot be read as closed: the Gemma-4
tower is not ported, so what ENTERS the connector is still whatever the caller
put in the prompt-embeds file. `prompt_embeds_valid_rows` exists because a file
carries no tokenizer mask and padding is not inert -- it is what the register
table replaces.
Issue: #435
Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [Claude Code]
FOLLOWING_AGENTS_PROTOCOL Two bounded waits on $HOME/gpu.lock, `flock -w 2700` each, both timed out while other coordinators held the box. Waiting is normal and stealing is not, so the two arms are recorded OWED with the scripts that run them already on the box, rather than left as an absence somebody has to rediscover. The one that matters: `test_minimax_h3` and `test_capi` are UNRESOLVED, not green. The first baseline pass ran the BINARIES directly and each printed a doctest summary and then died -- `test_capi` `4 cases | 51 skipped` against a brief baseline of 55, `test_minimax_h3` `38 | 41 skipped` against 79. doctest counts the cases it never reached as "skipped", so the summary reads like a run that chose to skip them. That is the third time this campaign has recorded a green that means nothing, and it is why the second pass drives ctest and keeps the FULL output instead of grepping a bare binary. The other is the arm that would settle where L9B's ~58 GB came from at L9B's own geometry: L9B's binary, L9B's arguments. It cannot run on the L9C binary, which refuses 32 conditioning rows because that is not a multiple of the connector's 128 learnable registers. Issue: #435 Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
|
Follow-up commit Known and NOT introduced here, so a reviewer does not chase them:
The drain added here is a LIFETIME concern and #516 is a KEYING one, and they do not interact: |
|
Landed on Reviewed PASS before landing. The two suites this PR left explicitly UNRESOLVED are resolved, and they were infrastructure rather than a regression: Every upstream claim was verified at One correction to the record: the drain evidence leaned on "GPU 0%" as if load-bearing. Utilization measures compute, not memory — a Closing rather than merging because its base |
2 participants
Phase L9c of the LTX-2.5 campaign. Two gaps stood between L9B's render and a
real scene at a real resolution. One of them was real and is closed. The other
turned out not to be there, and saying so is the more useful half of this PR.
Issue: #435. Branch base:
row/LTX25-L9B-RENDER@9cf9fbf4(PR #548).Gap 1 — the connector was never called
Ltx2ConnectorForwardlanded at L5, was gated against upstream on five arms, andits ONLY caller was its own test. The render handed the prompt-embeds file
straight to the DiT's cross-attention, and the two
*_embeddings_connectorfamilies the shipped DiT carries — 129 tensors each, verified against the
checked-in manifest — were reported as "unported" and stepped over.
Upstream never does that.
EmbeddingsProcessorruns an 8-layer 1-D transformerover the caption projections before the DiT sees them
(
embeddings_processor.py:70-95), and those weights ship inside the DiT file onlybecause upstream's key ops rewrite them into the TEXT ENCODER
(
encoder_configurator.py:331-346). So they load beside the DiT, not inside it,and the render applies them.
Presence decides, not a flag — the same rule
DetectLtx2Videoalready uses onthe DiT. Half a connector is refused: conditioning the picture on eight
transformer layers and the soundtrack on none is not a degraded render.
The config value that is nowhere near its default
LTX-2.5 declares
connector_positional_embedding_max_pos = [4096]whereEmbeddings1DConnector's class default is[1], andget_fractional_positionsDIVIDES the token index by it (
rope.py:132-141). No shape can see thedifference; the default is every RoPE angle wrong.
positional_embedding_thetaisdeliberately NOT read from the DiT config even though one is declared, because
neither configurator passes it — reading it would be a re-invention, not a port.
Where the two references disagree, and where they agree surprisingly
Disagree:
ltx_coreright-pad-sorts the features in the PROCESSOR before theconnector;
diffusersfolds that sort INTO the connector and claims it matches"the original LTX implementation" — true only because
ltx_coredoes it one levelup. They compose to the same function. This port follows
ltx_core, soLtx2ConnectorForwardstays a faithful port ofEmbeddings1DConnector.Agree, and it looks like a bug:
_to_binary_maskisencoded_mask < 0.000001. An additive mask holds0.0for KEPT and-finfo(f32).maxfor PADDED, and BOTH satisfy it — so the mask is one everywhereand the video-only multiply that follows is an identity.
diffuserswrites theidentical comparison. Checked on both BEFORE mirroring, because the intent-reading
(
>= 0) is the opposite at padded positions. Gated so "fixing" it REDs.Gap 2 — the pool drain, and what it is actually worth
The drain lands at every phase boundary, which is also the denoise-to-decode
boundary for the last phase, mirroring
minimax_h3_pipeline.cpp:546-563.Same-binary A/B at 320x192 / 25 frames (
VLLM_LTX2_POOL_DRAIN, a documentedmeasurement lane in the shape
VT_POOL_BYPASSalready takes):The two output directories are byte-identical (
diff -r -qrc=0), so thedrain is numerically inert — measured, not asserted. It reclaims 0.03 GiB after
generate_lowresand 0.08 GiB afterrefine. It is correct, it is cheap, andit is NOT what unlocks a bigger render.
L9B's ~58 GB runaway did not reproduce. At the same geometry, with the drain
explicitly OFF, MemAvailable is flat at 75.4 GiB for the whole run and the render
completes — where L9B's arm ran
--max-phase 0, i.e. LESS work, and had to bestopped at a 17.2 GB floor. Nothing in this diff plausibly removes a
device-memory runaway. The arm that decides it re-runs L9B's own binary with
L9B's own arguments on an idle box.
And the ladder is not bounded by memory. At 320x192/25f the process sat at
0% GPU and ~110% CPU for most of 23 minutes:
Ltx2VideoDecodeis a HOST path.MiniMax-H3 has the device analogue and LTX-2.5 does not. That is the next
traceable step, and no ceiling is declared.
The frames ARE a scene
Under LTX-2.5's DECLARED config (
frequencies_precision=float64,av_ca_timestep_scale_multiplier=1000), the shipped 21.00B FP8 DiT at 320x192/25frames renders a temporally coherent photorealistic clip: one subject, consistent
identity and background, frame-to-frame motion. L9B's frames at the same settings
were smooth colour fields.
The statistics could not tell them apart, and that is a finding about the
instrument. Measured with L9B's own analyzer, unchanged: neighbour |dx| / sd
0.093 here against L9B's 0.033, block-mean ratios 1.012 / 1.193 against
1.036 / 1.194. Nearly identical. Only looking separated them.
It is not YOUR scene. With
--prompt-valid-rows 24, 104 of the 128conditioning rows are the connector's own trained
learnable_registers, and theother 24 are synthetic N(0, 0.2). The Gemma-4 tower is still owed, so nothing
here encodes a prompt. The audio is not silence (rms 26.8) and nothing further is
claimed about it.
Evidence
RED-first was taken as MUTATIONS, because three of the four things this phase adds
are new symbols and a tree with the tests and without them does not compile.
M1 (drop the connector's output): 3 render cases red. M2 (identity sort):
padding-side case red. M3 (
>= 0mask): binary-mask case red. M4 (report theconnector unported again): loader case red plus 19 engine cases. Restore verified
byte-for-byte; all three suites green again.
🤖 Generated with Claude Code