fix(ltx-2.5): the keyframe refusal named a FALSE reason, and a test pinned it (#644) - #666
Merged
Merged
Conversation
…learing it (#654) Restores the prompt-side AdaLN path. `use_prompt_adaln_single` was set FALSE unconditionally at `ltx2_loader.cpp:988` while both references default it TRUE (`transformer_ltx2.py:1185`; `model_configurator.py:76,138`). The shipped DiT carries the tensors, our loader refused them by name, and a real render needs `allow_unported=1` — which silently cleared the flag. So every render ran with the timestep-conditioned half of the text conditioning removed, keeping only the static `prompt_scale_shift_table`. Measured on the SHIPPED checkpoint by the reviewer, not on the synthetic fixture: the timestep term is **1439% (video) / 1593% (audio)** of the static table by RMS. The term dominates; the table is the perturbation. Pre-row renders applied `context*(1 + ~0.018 rms) + ~0.018` where upstream applies `context*(1 + ~0.25 rms) + ~0.25`. The producer is `modality.sigma`, not `timesteps` (`transformer_args.py:274-277`) — a difference no shape check can see, so it has its own mutation. `allow_unported` is fixed structurally rather than documented: the three `= false` assignments are gone, replaced by an invariant that the resolved flag must equal whether the file carries `prompt_adaln_single`. A config disagreeing with the shapes is refused instead of forced into agreement, so the opt-in can no longer disable a ported feature. Five mutations RED on the right case each, plus two novel ones from the reviewer proving host and device are independently instrumented. Reviewed PASS. Operator gate on the merged tree: BUILD_EXIT=0, zero warnings, 423 of 423 ran, all four known -j flakes green serially. Owed as follow-up (#644): the spec and commit still quote the synthetic-fixture ratios; `ltx2.h:47-49` asserts the checkpoint does not carry `keyframes_abs_pos_embedding`, which is false for the FP8 DiT; and the model-matrix row still calls the prompt K/V timestep-INDEPENDENT. Issue: #644, #435 FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
…, and rescope W1/W2 (#608, #649) (#643) FOLLOWING_AGENTS_PROTOCOL The row said "Qwen-Coder XML, Mistral, pythonic, and remaining parsers", "Our code: -", INVENTORIED, with a spec that was never written. All three parsers it named by title were implemented, alongside 38 others. This is a RECORD REPAIR, not an achievement: nothing new ships here. Three counts were wrong and two of them cancelled, which is why they survived a spec, a merge and a report. Ours is 41, not 40 -- the audit's regex used [a-z0-9_]+ and silently dropped the hyphenated granite-20b-fc, while test_detect.cpp:221 has asserted 41 since MODEL-MUSE-GLIMMER-W7. Upstream is 44, not 43. And two of OUR names exist in no upstream registry at the pin (qwen3, a local alias, and muse_glimmer, the off-pin unmerged-PR exception recorded in porting-inventory.md §16). 44 - 5 = 39 shared, + 2 = 41. The old "40 = 43 - 3" balanced only because two errors cancelled. The five upstream-only names are confirmed, but the characterisation of them was wrong in the dangerous direction -- "plain Python" reads as portable. Reading the method bodies rather than the imports: GptOssToolParser is a declared stub whose methods raise NotImplementedError and defer to Harmony; both Cohere classes hard-ImportError without the out-of-tree cohere_melody package and delegate all parsing to it. So only `inkling` is portable from vLLM source. W1/W2 are RESCOPED accordingly -- one ordinary port, one from-scratch write gated on the upstream test, and two names that each owe a recorded decision before any code. State INVENTORIED -> PARTIAL, not ANCHOR-BACKFILL: that state asserts a missing leaf spike and the spec is committed, while PARTIAL's "known to omit upstream behavior" is exactly true of the five. Summary counts re-derived from the rows. Reviewed twice. The first review FAILed it on four record findings -- a docs/USAGE.md anchor that rotted during this branch's own rebase (#641 moved that file by 572 lines), the portability characterisation, a defect flagged only in a PR body, and a backwards precedent citation. All four repaired by a fresh implementer; #649 filed for the stale TOOLS-CALLING-CORE counts. The scoped re-review returned PASS with no findings. Records only: 4 markdown files, no src/, no tests/, no checker weakened. Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
…FUSAL it carries (#607) (#642) FOLLOWING_AGENTS_PROTOCOL The multimodal seam map covered the input pipeline and the tower but not the config that decides whether either runs. 43 of the 157 official recipes pass --language-model-only and we reject it. I filed #607 saying the flag "skips loading the vision encoder", implying a boolean we could expose through an existing seam. The pinned oracle says otherwise, and the correction changes the port. multimodal.py:78's own docstring: "disables all multimodal inputs by setting all modality limits to 0. Equivalent to setting --limit-mm-per-prompt to 0 for every modality." The encoder skip is a CONSEQUENCE, at interfaces.py:293, which builds the tower inside no_init_weights + StageMissingLayer when ALL limits are zero; upstream's own docstring at :268 says so. Any route to zero limits gets it, so porting the boolean alone would invent a path upstream does not have. A fresh review then found the map still incomplete on the half that changes BEHAVIOUR. validate_num_items (context.py:409-428) raises VLLMValidationError from parse_mm_data (:461) and chat_utils.py:662, folded through allowed_mm_limits (:392-405) -- so upstream --language-model-only REFUSES every multimodal request, absent --enable-mm-embeds. That undercut my own claim that "L2 without L3 is honest and shippable": as first written, L2 would have accepted an image request upstream rejects, on the flag's PRIMARY observable axis, while my caveat guarded only "frees VRAM". The refusal now sits in L1, where the limits' own semantics belong, and L2's honesty is conditional on it rather than asserted. Our baseline is nothing, not partial: no MultimodalConfig, no limit_per_prompt, nothing gating tower construction, verified by searching for the behaviour rather than the name. This is a port, and #607's title now says so. Placed on ENG-MM-INPUT-PIPELINE rather than a new row -- that row is already "the C++ mirror of vllm/multimodal/". Open PRs were diffed as well as issues before writing; no branch touched either symbol. One second-order finding worth the record: qwen3_next.py:325 proves the flag is not purely a memory knob -- upstream uses it to select the FUSED QK-norm+RoPE+gate path while ours is unconditionally fused. That is the serving-side twin of #414. Spec plus two roadmap lines. No code, no row-state change. Reviewed twice; the scoped re-review returned PASS. Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
Brings in `65e79eee5` (row 0, LTX25-PROMPT-ADALN) and `67e53e716`. Two keyed records needed a hand resolution rather than the automatic three-way result, because both sides had added a row under the SAME key: * `.agents/roadmap_v1.md` — each branch appended its own `#644` row, which auto-merged into the issue being listed twice and `check-agent-record` failing. Resolved by taking one row per issue, naming both campaign rows (`LTX25-PROMPT-ADALN` and `LTX25-IMAGE-COND`) and both specs in it. * `docs/FEATURES.md` — the LTX-2.5 lane row conflicted. Combined: the prompt-side AdaLN from main, the served image-conditioning arm from this branch. The combined text overran the 220-char cell and 600-char row entry budgets, so the forensics stay in the two specs and the row keeps the keyed projection, which is what that page is. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
…eam (#553, #603, #651, #652) main is RED on the DSR ratchet and has been since the LTX-2.5 landing: two kCUDA sites at ltx2_video.cpp:531-551 put kcuda at 2 against a baseline of 0, so every branch cut from main since inherits the red and each session re-derives that it is pre-existing. PROVENANCE, stated rather than claimed: this repair was written by a third session and left unpushed - git ls-remote showed zero refs, so it was invisible to gh pr list and to every ownership question any of us could ask. A peer found it, verified it independently, and ceded it. I did not write it. It is adopted here through the protocol to unblock main, with a fresh review by someone who is neither its author nor its finder. THE LOAD-BEARING CHECK. kcuda goes 2 -> 0 while check-device-leakage.py (md5 9a4617a690e383d5a5c628ac3ec48111) and device-leakage-baseline.json (md5 45ec32f4e08ab1c5ccf1f6ef236895b0) are BYTE-IDENTICAL to origin/main, and git grep ltx2 over the checker is empty. So this is a repair, not a raised threshold - the checker's own message forbids the latter in terms, and most "green now" claims do not separate the two. vt_ifdef stays 32; total 34 -> 32. It takes the platform-seam route the checker's message prefers (CurrentPlatform() / device_type()), not the ALLOWLIST escape. Only vt::Device{kCUDA, index} becomes vt::Device{accelerator, index}; the CreateQueue/multi-GPU labelling comment at :539-556 is untouched. MUTATIONS, four, each anchor asserted count == 1 and the tree restored by md5: * revert the seam -> RED, kcuda=3 * --write-baseline after it -> REFUSES: "REFUSING to write a HIGHER baseline" * hand-edit the baseline JSON to 35/3 -> checker PASSES. The ratchet's only defence against a raised baseline is a human noticing a 2-line diff, which is why md5 identity and not the exit code is the real check. Filed. * drop the accelerator == kCPU clause, keeping only TryGetBackend(...) == nullptr -> test_ltx2_video.cpp:386 FATAL, 502 -> 501 assertions. TryGetBackend(kCPU) is non-null, so without that clause a CPU-only build SILENTLY ACCEPTS device = 1. The guard is real and is now measured. Gate: ctest -N registers 423, the serial run examined 423/423, 0 failed, BUILD_EXIT=0, zero warnings. Registration is byte-identical to main, so the denominator cannot have drifted. agent-preflight.sh on committed HEAD: all gates green. test_cpu_x86_llamacpp_floor fails under load on unmodified main too and passes at low load; environmental, reproduced on both sides. Also corrects a defect in the original commit message: the M3a precedent is src/vllm/entrypoints/model_loader.cpp:75-104, not model_executor/ - which is ambiguous rather than merely unqualified, since a model_executor/model_loader/ directory also exists. DELIBERATELY EXCLUDED: the author's live tip 8f26ee9 additionally carries an unaligned-load change to ltx2_loader.cpp (#627) that arrived mid-review and was never gated here. That file has four live rows on it right now, and its comment claims fifteen casts across nine loaders while LoadUnaligned appears in four files and six call sites. It wants its own scoped review rather than riding a red-main repair. Owed and filed, neither blocking: #659, the repair adopts M3a's seam but not its companion supports_model_architecture guard, so a partial backend that could decline by name now fails later in a kernel bind; #660, the kcuda bucket is the token grep \bkCUDA\b, so minimax_h3_video.cpp:221-226 spelling it static_cast<vt::DeviceType>(1) counts as zero and launders the same defect past the same gate. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Claude:claude-opus-5 [ClaudeCode]
…inned it (#644) Review findings on #657. Two blocking, four minor. F1 — the last-frame keyframe refusal blamed `keyframes_abs_pos_embedding`, and that is not what blocks it at pin `fd4ded7f`. Three anchors refute it: * `keyframe_cond.py:84-86` appends a SUPPLIED keyframe with `marked=False`, commented "given keyframe content is ordinary image guidance, not a generated keyframe slot, so it carries no keyframe marker"; * its sole consumer is `transformer_args.py:42-43` (called once at `:269`), `mask = (keyframes_mask > 0); return hidden_states + mask * embedding` — so with `marked=False` it adds NOTHING, and porting it would not serve the arm; * the contrast was backwards. `tools.py:184` + `_first_frame_keyframes_mask` (`:186-195`) mark the target's FIRST latent frame unconditionally, so the tokens that actually reach the embedding are the ones the SERVED first-frame arm writes into — which the message called "needs none of that". That gap is real and is issue #658; it is not fixed here and nothing here contradicts it. The named gate did not fire either: the refusal lives in `ltx2.cpp`, and `Ltx2AdoptDeclaredDitParams` clears the flag on every real load under `allow_unported_modules` because the shipped DiT does carry the tensor. Worse, `test_ltx2_video.cpp` had been written to REQUIRE the wrong reason by name. The message now names the token-append machinery — extended `positions` (`keyframe_cond.py:46-59`), `update_attention_mask` (`:68-76`), extended `clean_latent` / `denoise_mask` (`:79-82`) and `clear_conditioning` trimming back (`tools.py:88-105`) — against a phase loop fixed at the target grid's token count, and says why the first-frame arm needs none of it. The assertion now requires THAT, and permits the refuted reason only where it is labelled as ruled out and paired with #658. Local anchors are symbols, not same-file line numbers, which is how the previous citation went stale. Sixth refusal in this campaign whose stated reason was false; the header comment now says six and says why. RED old message, new assertions: 4 failed, exit 1 GREEN new message: 32 cases / 565 assertions, exit 0 F2 — spec section 8.5 names "conditioning stage 1 only would let stage 2 re-noise the pinned frame away" as the REASON the design conditions inside the phase loop, and gated it with nothing. MEASURED: `if (wants_image && phase_index == 0)` left the suite at 32 cases / 550 assertions / exit 0 — `image_digest` / `image_tokens` are overwritten each phase and the only assertion was `image_tokens > 0`. This is the shape of the obvious optimization (hoist the per-phase decode+encode out of the loop), so it would have landed green. The trace now pins the LAST phase's per-latent-frame token count, and a `max_phase = 0` engine over the same request pins stage 1's, with the 4x relation checked between two MEASURED values rather than two compile-time constants. RED mutant: `CHECK( 1 == 4 )` twice plus the digest equality, 6 failed, exit 1 GREEN 32 cases / 565 assertions, exit 0 F3 — nine mis-cited anchors, each re-verified against the pin before changing. `blocks.py:576-580` is the TEARDOWN, so the conditioning-order claim now cites `create_noised_state` (`helpers.py:428-445`) and the video-before-audio claim cites `blocks.py:554-563`. Also corrected: `ImageConditioner` build `:985-986` / build-and-free `:988-991`; `helpers.py:274-275` are parameter declarations, with the per-stage h/w at `distilled.py:251, :255-256, :285-286`; `model_configurator.py:41-43`→`:41-42`; decoder padding default `:90`→`:92`; `reference_video_cond.py:74, 80`→`:36-37` applied at `:65-77`; `reference_audio_cond.py:33`→`:34`; `detect_params` `:166-177`→`:166-179`. F4 — `Ltx2VideoVaeEncoderKeyRules` claimed "rule for rule". Upstream's `SDOps` (`loader/sd_ops.py:101-122`) is an `any()` admission over four matchings plus a chained substring replace over three replacements; this port is one first-match prefix pass, so its fourth rule is an identity carrying upstream's fourth matching. The comment now says behaviourally equivalent, and names the case where the two shapes would part. F5 — `ParseEncoderBlocks` defaulted `num_layers` to 1 while upstream SUBSCRIPTS `block_config["num_layers"]` (`video_vae.py:55`) and raises `KeyError`, and two lines below it `multiplier`'s sentinel is deliberately strict. A `res_x` block without `num_layers` is now refused; no other block kind reads the key, so requiring it everywhere would have refused configs upstream loads, and both halves are gated. RED guard removed: "did NOT throw at all!", 1 failed, exit 1 GREEN 15 cases / 223 assertions, exit 0 F6 — the three assertions comparing compile-time constants are left as they are, correctly labelled documentation. Noted so the count is read honestly; the new F2 assertions deliberately compare two measured values instead. USAGE carried the same false keyframe reason and is corrected, with the correction dated and #658 named so a reader can check it. USAGE also gains the `res_x` / `num_layers` requirement. FEATURES keeps its keyed row within budget. Gate, at this tree, CPU-only Release, CUDA=OFF: BUILD_EXIT=0, no ENOSPC, no BFD assertion ctest -N 424 ctest -j 8 423/424; `test_engine_core_proc` is the documented -j starvation and passes serially, exit 0 test_ltx2_video 32 cases / 565 assertions, exit 0 (was 32 / 550) test_ltx2_image_cond 15 cases / 223 assertions, exit 0 (was 15 / 198) test_ltx2_vae 36 / 3039 | test_ltx2 35 / 2435 | test_ltx2_loader 26 / 4826 test_ltx2_pipeline 37 / 2382 | test_ltx2_device 15 / 523 — all unchanged, exit 0 `LTX2_CHECKPOINT_ROOT` is unset here, so `test_ltx2_video`'s shipped-checkpoint subcase SKIPPED and is not part of those counts. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
BUILD still marked MXFP4 as planned and said all GPU GGUF weights expand. The implemented Marlin and keep-quant paths make both claims stale. STATUS also retained the old 30-entry registry count after the guarded public list grew to 37. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Codex:gpt-5 [Codex] Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
…(W1, #332) (#473) Implements the vt::Backend graph-capture seam on hipGraph (W1 of #332), mirroring src/vt/cuda/cuda_backend.cu call for call. Merged with the row's performance rationale REFUTED and recorded as such. W3 measured capture at +3.2% / +0.6% / -1.0%, not the ~2.2-3x §1 predicted, and the spec's D7 plus the inline note in §1 now say so rather than leaving a live rationale for the next agent to re-derive. That refutation was reported by the contributor against their own interest, which is the behaviour this protocol exists to produce. Merged anyway on the seam argument, which is independent of the decode number: graph capture had exactly one real implementation (CUDA), and a one-implementation abstraction is unproven. hipGraph is the cheapest available second. Runtime cost today is zero -- RocmPlatform does not override support_static_graph_mode(), so nothing in the engine reaches the new code. Review verified isolation three ways: every decode-graph call site ANDs SupportsGraphCapture() with support_static_graph_mode(); rocm_backend.hip appears zero times in a CPU build's compile_commands.json; and there is zero drift on every touched file across the 194-commit gap. A mutation of the EndCaptureGraph seam signature turns the new test red, so it genuinely guards seam drift on machines with no AMD hardware. Known-broken windows-msvc-* are the PR-only arm (#584), not this change. Carried forward: #523's per-call hipMalloc/hipFree and hipStreamSynchronize are illegal under hipGraph capture and must be reconciled before this capability could ever be switched on.
…den is binary, so no golden-bearing PR could merge (#615) (#619) Retires the fail-closed binary guard in scripts/check-pr-size.py (#615), and classifies .agents/oracles/* (#668). WHY THE BINARY GUARD GOES. It errored on any path git reports as binary, with no exemption route. Parity goldens ARE binary, so no PR capturing an oracle golden could merge -- blocking #431 and every future golden across the ROCm, Tenstorrent and Nemotron lanes. The guard also contradicted this checker's own model (the SITE_ASSET note says binaries "take the `asset` class the same way any other shipped artwork does"), post-dated the golden precedent it rejected by one day (450a1b6 2026-08-10 vs 971d550 2026-08-09), and a commit already on main (5080983, 16 .npy files) trips it. A rule nothing in the tree can satisfy is not enforcing a standard. Explicit classification is KEPT and still refuses an unclassified binary, now with a message naming the real defect instead of an unfixable property of the file. No allowlist was added: a roster of blessed binary paths is a shared must-write surface, which is the lock AGENTS.md forbids. This removes no size rule because none was left -- the per-class budgets were retired 2026-08-10. Nothing in this file measures a diff. Reviewed by a fresh agent that did not write it: every claimed number reproduced, all four sabotage mutations were caught (reintroduce the guard, break classification, disable the evidence contract, break the PR-required check), and seven adversarial commits confirmed the checker still refuses an unclassified binary, a checker change without evidence, and a binary posing as its own evidence. Its scope finding -- that goldens classify as `product`, so this admits binaries to src/ and scripts/ too -- is recorded in the spec as an accepted trade rather than argued away. ALSO FIXES #668. The secondary-oracle registry landed .agents/oracles/<id>.md with no pattern here, so all eight files were unclassified and classify_path RAISED on unmodified main -- meaning any PR recording an oracle pin was refused by a required check. Takes the same `procedure` class as .agents/specs/ and .agents/claims/, which it is structurally identical to. Deliberately a pattern, not a directory exemption: a non-.md or nested path there still fails closed. device-leakage and sanitize-cpu are red on main itself and inherited by the merge; this PR touches no C++. Both are being fixed separately. Closes #615. Closes #668.
…633) (#635) FOLLOWING_AGENTS_PROTOCOL The pin protocol covered one repository, so every architecture living only in `vllm-project/vllm-omni` had no oracle it could be gated against. #650 landed the oracle registry mid-flight and superseded this row's original design (a second pin block inside `upstream-sync.md`), so the spec was REWRITTEN onto the registry rather than patched, and says so at the top. `.agents/oracles/vllm-omni.md` is deliberately untouched: it already reads `pin = UNPINNED`, `gateable = no`, `evidence = #633`, which is exactly true. `upstream-sync.md` now points at the registry and carries only what the registry file cannot state on its own — that the two pins may legitimately DISAGREE (vllm-omni requires vLLM 0.27.0+ against our 0.26.0.dev0 parity pin), that an omni-gated number is therefore never evidence about the core pin's surface, and that an omni pin advance does not re-open the vLLM-side binding grids provided the omni oracle is isolated in its own virtualenv. `core_commit_used` is deliberately NOT added to the checker-enforced schema: the pin is UNPINNED, so there is no value to record. It lands with the measurement. `ENGINE_ROWS` 153 -> 154, with mutation evidence that is load-bearing against the exact bad merge this branch hit: main bumped the same constant 152 -> 153 for a different row, so both sides read 153 and the merge looked clean. Drop the row, rewind the constant and the rollup until every count agrees, and 1 of 53 tests fails: that one. NO product code. This is a spec and its records. Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
…ing lane (#634) (#637) FOLLOWING_AGENTS_PROTOCOL vLLM-Omni supports IndexTTS-2.5 and we had no record of it. Every audio path we ship CONSUMES audio (Parakeet, Voxtral, `audio_processor.cpp`); nothing synthesizes it. Rows both registered architectures -- `IndexTTS2TalkerForConditionalGeneration` (stage 0, ~0.8B GPT-2 AR talker) and `IndexTTS2S2MelDecoder` (stage 1: EnhancedCodec -> S2Mel CFM/DiT -> BigVGAN, 22.05 kHz) -- at `registry.py` @ `bbe6ccc512a404a2df8c977ea29003002f2683e8`. NO product code. This is a spec and its records; W1-W7 are all unstarted. Nothing runs, no audio is generated, and the lane is not reachable from the ABI or the server. That unreachability is now SCOPE rather than a footnote, verified in the tree: `include/vllm.h` is at ABI v18 with zero speech entry points, and `audio/speech`/`audio/voices` have zero hits across `src/`, `include/` and `examples/`. The spec mirrors the video lane that solved the same problem for a generative modality -- a `SpeechEngine` seam like `VideoEngine`, ABI v19 with `vllm_synthesize` named symmetrically to the existing `vllm_transcribe`, both routes through that seam, the example as a thin ABI client -- and the seam must admit the ~10 other omni TTS architectures, not just this one. Three things pinned down rather than left to be discovered: reference audio is MANDATORY (upstream has no text-only synthesis, so w2v-bert-2.0 + MaskGCT + CAMPPlus are required surface); no token-exact e2e gate is possible (a seed drives both AR sampling and per-request CFM noise), so the binding gate is per-stage numerics vs the checkpoint's own remote code plus token-exact mel codes at fixed seed/c1/fixed batch composition, with an ASR-round-trip ratchet that is explicitly neither a correlation gate nor a count-based tolerance; and we mirror vLLM-Omni even though it itself deviates from IndexTeam's `num_beams=3` default. Two reuse claims were CORRECTED after LTX-2.5 merged: it generalized the VIDEO engine, not the vocoder, and the BigVGAN 1-D core is already shared by H3 and LTX-2.5 from `minimax_h3.h`. This lane would be the THIRD consumer, so W1 shrank from "generalize" to "relocate to a neutral home". Both rows land `INVENTORIED`, unclaimed and blocked on the UNPINNED vllm-omni oracle. MODEL 370 -> 372, with mutation evidence for the two-architecture shape. Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
… against the corrected bound (#496) (#675) FOLLOWING_AGENTS_PROTOCOL W2 of KERNEL-SSM-MAMBA (#496): the CUDA arm of vt::Mamba2ChunkScan, vt::Mamba2StateUpdate and vt::RmsNormGatedGroup. Supersedes #566 and #592, which had diverged -- one carried the evidence resolving the merge precondition, the other the F1/F2 repair, and neither contained the other. F1: the declared equivalence contract named the elementary functions as the ONLY admitted source of device-vs-host divergence. Host C++ is pinned -ffp-contract=off and nothing passes --fmad=false to nvcc, so acc += xv * bv contracts on device and not on host. Stated model 3.5*K*u, contraction adds ~1.0*K*u, bound was 4*(K+2)*u -- provable only for K <= 18 while the driver shapes run at K = 200. Repaired by carrying the term to 5*(K+2)*u with the arithmetic shown in the header, all three test comments and 8.3. -fmad=false was rejected deliberately: it is a per-TU flag on a header included by the hot GDN TU. Repo-wide gap filed as #591. F2: the header claimed the device kernels stay memory-safe under a contract violation, but the validator checks metadata shape and dtype only. Two dropped checks were memory-unsafe -- an out-of-bounds WRITE past a cudaMallocAsync allocation and an out-of-bounds READ of initial_states -- and the stated reason for omitting them (a D2H plus a stream sync) did not apply, since both values are already in registers and the decode kernel does exactly that clamp for free. Repaired with both the clamps and a narrowed claim enumerating all six dropped checks. A third hole the review did not name was found and closed: seq_idx[0] < 0 indexes passed at chunk -1. All four owed gates discharged by measurement, operator-run on the gate host, each stamping its own lock-acquire time, load and disk. nvcc compile for sm_121a clean with CUTLASS, FA2 and Marlin confirmed ENABLED in the configure log rather than assumed. The three CUDA arms 12/2095, 10/5965, 12/3723 all SUCCESS. compute-sanitizer memcheck ERROR SUMMARY 0 errors. And the 9-mutation re-sweep against the MOVED bound: 9 of 9 CAUGHT. The re-sweep is the item a reader would most likely have waved through, and M6 is why it could not be: it aborts at exit 134 while printing "assertions: 2577 | 2577 passed | 0 failed" -- a clean assertions line on a FAILING run -- and is caught only because the harness reads the exit code. Two of the original eight mutations did not COMPILE under -Werror=all-warnings and were being scored as caught; a mutation that will not build is a suite that never ran. test_minimax_h3 is attributed rather than waived. Reproduced standalone on an idle box under the lock at TEST_EXIT=139, it is #486 with root cause #516, signature-for-signature. The independent baseline that PASSED was row/pool-device-key, the branch that FIXES #516, so the baseline carried a fix and this branch does not carry a defect. CI is REMOTE_UNVERIFIED, not green: every run on the predecessor branches ended cancelled, including a repo-wide mass cancellation of 20 runs across 7 branches, and cuda-fat-build never completed -- which is why the compile was run directly on the gate host. Windows reds are the main baseline (#514, #584). The row stays INVENTORIED. No lifecycle move, no measurement claimed, no performance result: a host reference plus its device arm is not a speed number. Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
… a merged pair (#517) (#684) FOLLOWING_AGENTS_PROTOCOL W2 of MODEL-TEXT-nemotron-h (#517): the non-gated relu^2 MoE expert. NemotronH's expert has no gate half -- ckpt_names=("up_proj","down_proj",""), where the empty third entry IS the absent gate. So this is not a new merged pair. MergedGemmGroup describes N GEMMs sharing an operand with a fused epilogue; with N = 1 there is nothing to merge and an arity-1 descriptor would name a fusion that does not exist. It lands as the existing grouped projection plus the activation we lacked, with the reasoning recorded next to the seam it excludes, and a fresh review confirmed that is a correct reading rather than a parallel path with a good story. The finding that justifies the row: folding the routed scale into the per-expert weight SURVIVED green in the first review, yet differs from upstream bitwise on 13276 of 20000 rows -- and it is the most likely W4 mistake, because Laguna legitimately does exactly that fold, legal there only because Laguna passes no shared expert. Now pinned bitwise at 10 and 4 ULP separation. Two conflicts on the re-merge. include/vt/ops.h had BOTH sides appending an OpId before kCount -- kLtx2 from the LTX-2.5 campaign and kMoeRelu2 here -- resolved by union with main's entry first so no existing id shifts. And op_provider.cpp auto-merged CLEANLY yet did not compile: main added an exhaustive OpName switch with no default, documented as failing the build when an OpId is appended without being named. That is the gate working exactly as designed; the alternative is a new op with no canonical spelling that nothing would have caught. cuda_moe.cu compiled against the merged ops.h with real nvcc on the gate host, BUILD_EXIT=0 WARNINGS=0 ENOSPC=0, closing the review's F1 -- whose original justification, that neither file had changed since the last GPU compile, was factually false: ops.h moved +180/-1 across six merges. Post-merge: 0 diagnostics under -Werror, relu2 12/81 and router 14/941 both SUCCESS, ten mutations caught. Squash-merged deliberately: the branch carries an inherited trailer-less merge commit that a local --no-ff would put on main. Row stays INVENTORIED. No lifecycle move, no measurement claimed. Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
…ech seam, family refusal (#634) (#681) FOLLOWING_AGENTS_PROTOCOL FOUR of the seven work items in .agents/specs/indextts-2-5.md. This does NOT make IndexTTS-2.5 runnable: the lane detects its checkpoint and REFUSES, naming the stages that are missing. No audio is synthesized by this change and none is claimed. W1 — the shared BigVGAN 1-D core moves out of `minimax_h3.h` into `vllm::vocoder1d`. It lived there because H3 ported it first; LTX-2.5 reused rather than copied it, and `ltx2_audio_vae.cpp:223-230` records why ("each copy keeps its own green gate while the two audio VAEs drift apart"). IndexTTS-2.5 is the THIRD consumer. 28 call sites retargeted; `MaterializeWeightNorm` stays with H3, which is its only caller. Gated twice, because the failures differ: a structural guard (one declaration site, one definition site, no retired spellings) is the ONLY thing that can see a fork, since a fresh copy agrees on the day it is made; and 10 cases of HAND-COMPUTED arithmetic, not values captured from the implementation, which would pass just as happily if the helper were uniformly wrong. `test_minimax_h3` 79/79 and `test_ltx2_vae` 36/36 prove the move is behaviour-preserving. W2 — the GPT-2 talker backbone, ported from `gpt2.py` @ 555967922 with the reference checkout verified at that SHA before any line was cited. The oracle is a TP=1 restatement (upstream's module needs a KV cache, a device and an initialized distributed environment), the technique gen-minimax-h3-goldens.py already uses. Hidden states within 2e-5, logits within 5e-5, argmax TOKEN-EXACT. The Conv1D transpose and causality each get their own case because both fail silently: goldens are emitted in upstream [in, out] orientation so the loader must transpose, and a non-causal model stays fluent. W6a — the `SpeechEngine` seam, mirroring `VideoEngine`, shaped for a FAMILY because vLLM-Omni carries ~10 further TTS architectures behind this one. Family registration — an IndexTTS-2.5 checkpoint now resolves to a named family and refuses with the missing pieces spelled out, which is the AGENTS.md contract for an unimplemented arm. Also records the MEASURED component inventory (gpt 17,171 LOC, s2mel 15,011, utils 18,265, codec 1,930) and one correction that changes the plan: the talker is built with `spk_cond_mode="campplus"`, so CAMPPlus is UPSTREAM of it and W3 cannot be deferred. STILL OPEN: W3 (w2v-bert-2.0, MaskGCT, CAMPPlus), W4 (EnhancedCodec, S2Mel), W5 (compose), W6b (routes + ABI). No e2e render, and any parity claim additionally needs the vLLM-Omni oracle, still UNPINNED (#633). Verified locally: six suites green (10/4/7/4/79/36 cases) plus the structural guard; check-test-registration, check-agent-record, check-doc-checkpoint and check-pr-size green. Every gate was mutation-proven, and TWICE a mutation's build failed under -Werror so the old binary re-ran and printed SUCCESS; compile status is now checked alongside every mutation result. Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
…tream itself (#634) (#689) FOLLOWING_AGENTS_PROTOCOL First increment of W3. Does NOT make IndexTTS-2.5 runnable; the lane still refuses by name. CAMPPlus is the speaker-style encoder on the mandatory reference-audio path and sits UPSTREAM of stage 0 — the talker is built with `spk_cond_mode="campplus"` (infer_v2_5.py:138) and consumes its style vector. A STRONGER ORACLE THAN W2's: `campplus/layers.py` has no vllm dependency, so the generator loads it BY FILE PATH and executes the REAL classes. W2 needed a TP=1 restatement because upstream wanted a KV cache and a distributed environment; here there is none, so there is no risk of being faithful to the wrong thing. Four primitives, each a silent-failure mode: StatsPool (UNBIASED N-1 std; the biased form differs ~0.2% at T=250, reads as noise, moves a style vector), BatchNorm1dEval (RUNNING statistics; batch statistics still normalize and are a different model), SegPooling (ceil_mode expand-then-truncate, goldens at T=250 so the trailing PARTIAL segment is exercised) and CamLayer. 4 cases / 8 assertions, all within 1e-5 of upstream. MUTATION EVIDENCE with compile status alongside every result: biased std, floor instead of ceil_mode, and dropping the global-mean term each go RED at compile_err=0; restored tree returns to SUCCESS. The compile check is there because this campaign already produced TWO false readings where a mutation failed to build under -Werror and the old binary re-ran printing SUCCESS. A killed mutation run also left the tree dirty mid-sequence — the file is untracked so `git diff` could not see it, and it was caught by grepping for the mutated line. Restoration is verified by content now, not assumed. STILL OPEN: CAMPPlus's FCM 2-D front end and 52-layer CAMDenseTDNN stack; then w2v-bert-2.0 and EnhancedCodec; then W4 S2Mel, W5 compose, W6b routes/ABI. No e2e render, and any parity claim needs the still-UNPINNED oracle (#633). Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
… gate caught (#634) (#690) FOLLOWING_AGENTS_PROTOCOL Adds TransitLayer, DenseLayer, CAMDenseTDNNLayer and CAMDenseTDNNBlock on top of the primitives already on main, gated against `campplus/layers.py` executed DIRECTLY (no restatement). 8 cases / 16 assertions, within 1e-5. A REAL DEFECT, found by the gate rather than review: upstream's final dense uses `config_str='batchnorm_'`, which get_nonlinear expands to a SINGLE batchnorm with affine=false and NO relu. The first implementation applied one unconditionally, clamping every negative component of the style vector to zero — a plausible embedding from the wrong model, invisible to any shape check, and the talker consumes exactly that vector. Two ordering traps also gated, each producing a correctly-shaped wrong tensor: TransitLayer applies its nonlinear BEFORE the projection while DenseLayer applies it AFTER, and the block's cat([x, layer(x)], dim=1) growth means appending in the wrong order still yields the right shape. MUTATION EVIDENCE with compile status per result: forced relu, prepend-instead-of- append, and swapped transit order each RED at compile_err=0. The relu mutation's first attempt reported compile_err=1 and a meaningless SUCCESS — the third time this campaign that a mutation failed to build under -Werror and the old binary re-ran, which is why compile status is now reported beside every result. REMAINING: CAMPPlus's FCM 2-D front end and forward assembly; then w2v-bert-2.0 and EnhancedCodec; then W4 S2Mel, W5 compose, W6b. No e2e render. Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
…racle is not vLLM (#672) (#679) * spec(MODEL-MUSIC-MUSIC3): scope MiniMax-Music3, the first row whose oracle is not vLLM (#672) MiniMaxAI/MiniMax-Music3 is text-to-music: lyrics plus a structured description in, a multi-minute stereo song out. It is absent from the pinned vLLM (no source files; the registry carries only the MiniMax M2/M3 TEXT architectures), absent from vLLM `main`, and absent from vllm-omni. So it is the first row to exercise the fallback-oracle rule added in #647, and its primary oracle is an OPEN diffusers PR. W0 ONLY. Spec committed before implementation, per AGENTS.md. No src/, include/ or tests/ change is in scope here; W1 onward are dispatched to fresh implementers from this spec rather than written in the coordinating session. GEOMETRY MEASURED, NOT INFERRED. Every component config.json read, plus the safetensors headers by HTTP range request -- the same no-payload technique LTX-2.5 L0 used. What that corrected, twice: * transformer is 231 tensors / 1.240B params per shard, ALL F32. The card's "2.4B" is right and the 9.73 GB on disk is fp32 storage, NOT a 4.9B bf16 model. Reading the byte count alone would have sized it at double. * condition_encoder has FOUR tensors -- layer_scale, layer_weight_logits and a proj. It is a learned weighted mix over 8 LLM hidden layers, not the encoder tower its name suggests. Also measured: rvq_depth_decoder 47 tensors / 0.646B BF16, vocoder 121 / 0.054B F32 carrying weight_g/weight_v, so weight-norm must be folded at load. THE LARGEST BRICK IS ALREADY BUILT. language_model is `Qwen3ForCausalLM` at 36L / 4096 / 32 heads / 8 KV / head_dim 128 -- our LANDED architecture exactly -- retrained on a 200 000-entry music vocabulary. Vocabulary size is a config value, not an architecture change. TWO PACKAGINGS, ONE SET OF WEIGHTS, and that had to be established rather than assumed, because the two oracles disagree about which one they serve. SGLang-Omni loads ONLY the native arm: checkpoint.py:35-56 resolves qwen_7B/qwen_7B, flowmatching_vae.pth and dav.pth, and pulls the RVQ decoder out of the Qwen shards by the model.audio_decoder. prefix. diffusers loads the six modular components. convert_minimax_music3_to_diffusers.py@c6da9936 :29-38,47-189 settles it -- load the three native artefacts, rename tensors, save. A pure re-layout, no numerical step. Therefore the diffusers arm is the port target (~28.5 GB resident, no pickle, no trust_remote_code, per-component gateable) AND SGLang-Omni stays a valid e2e cross-check instead of an incomparable second model. W1 verifies that equality against actual tensors rather than inheriting it from this reading. THE DTYPE POLARITY IS UPSTREAM'S. The conversion script defaults --dtype float32 (:267), gives transformer/condition_encoder/vocoder that dtype (:208-211) and forces the RVQ decoder to bf16 regardless (:214) -- matching the measured headers exactly, and SGLang-Omni's README confirms both placements run the acoustic stage in FP32. So fp32 there is a resolved upstream choice, not the too-wide accident AGENTS.md warns a token gate cannot see. It is mirrored, and annotated as this. ONE CONTRADICTION LEFT OPEN ON PURPOSE. The model card and SGLang-Omni's README say 32 kHz stereo; every config says 44100 (vocoder sampling_rate, and condition_encoder output_sampling_rate with hop 512). Both cannot be right and the difference is a resample -- which no tensor-parity check on the latents would catch, so it is exactly the class of defect that survives a green gate. W0 resolves it from source before ANY waveform golden is captured. It is recorded as unresolved rather than guessed. ORACLES PINNED. diffusers advances from UNPINNED to the PR #14456 head c6da9936 -- the head SHA and not the branch name, because huggingface:minimax-music3-integration can be rebased under us and "whatever the branch was that day" is not reproducible. SGLang-Omni gets its OWN record at 748a0b43: it is a third repository with its own cadence and its own registry, and folding it into sglang.md would have hidden that a model can be in one and in neither of the others. Both stay gateable = no -- read, never executed here -- which is the honest state and what W0 exists to change. GATE SHAPE, stated up front because conflating the halves is the failure mode: the LLM half emits discrete RVQ codes and gets a real token-exact gate; the acoustic half is a flow-matching loop with no logits and no sampler, so it gets per-stage tensor parity at fixed seed and reduced dimensions, following H3. A correlation coefficient is NOT a gate there -- Pearson is scale-invariant, so a uniformly scaled latent passes it while sounding wrong. Records: MODEL row ratchet 370 -> 371 (the test that mutates it to expected-1 keeps the bump load-bearing), checklist entry + rollup SPIKE 8 -> 9, the 11 non-pin architectures become 12, roadmap issue table, and a claim file. Gate: scripts/agent-preflight.sh -- all checkers and all suites ok EXCEPT test_cpu_x86_llamacpp_floor, a contention-sensitive harness that failed identically (NO_QUIET_WINDOW, exit 4) on an unmodified tree at base SHA cefacd2 under load average 35. Pre-existing and environmental, verified rather than assumed. Issue: #672, #647 FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code] * spec(MODEL-MUSIC-MUSIC3): the 32 kHz / 44100 split is a STAGE BOUNDARY, not a contradiction (#672) W0 owed this from source before any waveform golden could be captured. Resolved, and the answer is that both numbers were right about different points in the pipeline. THE VOCODER NATIVELY EMITS 44100 Hz STEREO, and it is derivable rather than merely declared -- which matters, because a declared sample rate in a config is exactly the kind of value that goes stale without anything noticing. The condition encoder's output_sampling_rate 44100 / output_hop_length 512 set a latent frame rate of 86.133 Hz (condition_embedder_minimax_music3.py:40-41), the decoder applies one ConvTranspose1d per upsampling_ratios entry for 8*8*4*2 = 512x (minimax_music3_vocoder.py:84,92-95), and 86.133 * 512 = 44100. The declared sampling_rate matches its own convolution stack. SGLang-Omni's independent implementation lands on the identical numbers (dav.py:94,115), which is the strongest form of confirmation available here: two codebases that were not copied from each other. DIFFUSERS RETURNS THAT UNRESAMPLED (modular_pipeline.py:32-36, decoders.py:84-92). SGLANG-OMNI'S SERVER RESAMPLES 44100 -> 32000 ON DELIVERY (constants.py:18-19, acoustic.py:55-58,422-431). So "32 kHz stereo" in the model card and in SGLang-Omni's README is neither wrong nor stale: it is the reference SERVER's delivery rate. diffusers' own docs state the split in as many words. DECISION: goldens are captured at 44100 stereo -- the model's native generative rate, resample-free, and what the primary oracle hands the caller. The 32 kHz form is a downstream delivery transform, gated separately if SGLang-Omni byte parity is ever wanted. That transform is NOT free, and the spec says so rather than leaving it to be discovered: acoustic.py:58 passes no lowpass_filter_width, rolloff or resampling_method, so reproducing those bytes means reproducing torchaudio's DEFAULT sinc filter, not converting 44.1 -> 32 by any correct method. And the reason this had to be settled before the first golden, not after: a latent-tensor parity check sits entirely upstream of the resample call and cannot see the difference. A golden captured at the wrong rate would be wrong by a resample while every tensor gate stayed green -- the class of defect that survives a green gate, which is the one this project spends its evidence discipline on. Also corrected: the 24000/960 pair in condition_encoder/config.json is the AR stage's 25 Hz frame rate, unrelated to output. It reads like a third sample rate and is not one. W0 now owes only the oracle stand-up. The §1.1 stop condition is removed because it is discharged; the W5 row names 44100 stereo instead of "the resolved rate". Issue: #672 FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code] --------- Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
…AND aarch64 (#634) (#701) FOLLOWING_AGENTS_PROTOCOL CAMPPlus's head treats the [1, feat_dim, T] spectrogram as a 2-D image, so this adds genuine Conv2d/BatchNorm2d and BasicResBlock. Gated against `campplus/layers.py` executed directly: 9 cases / 19 assertions, within 1e-5. THE TRAP THIS PINS: BasicResBlock strides `(stride, 1)` (layers.py:224) — FREQUENCY only, TIME untouched. Striding both still yields a well-formed tensor at half the frame rate, which every later layer accepts. Goldens confirm 16x20 -> 8x20 empirically, and the case asserts the WIDTH is unchanged alongside the values, so a symmetric stride cannot pass by matching numbers on a smaller tensor. CROSS-ARCHITECTURE VERIFICATION: the suite was rebuilt from source and re-run on kairos-4db2 (NVIDIA Thor, aarch64, g++ 13.3, no cmake — compiled directly) and reports the same 9/9 and 19/19. The goldens were generated on x86_64 with torch, so this establishes the port's numerics are not x86-specific and no FMA-contraction difference breaks the 1e-5 tolerance on another arch. MUTATION EVIDENCE with compile status per result: striding both axes, dropping the post-residual relu, and dropping the residual add each RED at compile_err=0; restored tree SUCCESS, verified by grepping the restored line. Generator's hard-coded upstream path generalized to $CAMPPLUS_LAYERS. REMAINING in CAMPPlus: the forward assembly over 3 blocks of 12/24/16 layers. Then w2v-bert-2.0, EnhancedCodec, S2Mel, compose, routes/ABI. NO e2e render and none claimed; the lane still refuses by name. Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
…l what is not paired test(#669): pin that the token budget SPLITS a prefill wave, and label what is not paired A measured claim of mine was wrong and this row is the disproof. I measured our 35B prefill as INVARIANT to --max-num-batched-tokens (0.6814s at 8192 vs 0.6828s at 2048, c4) while vLLM's swung 30%, and concluded we were failing to chunk. We are not. The budget reaches the scheduler -- chain verified hop by hop from server_main.cpp:400,806 through model_loader.cpp:626-641,704-717,1051-1058 to scheduler.cpp:233-234,465 -- and a composition probe shows chunking happening: budget 8192 gives one 4096-token step; budget 2048 gives 2048 / 2048 / 5, with the clamp firing mid-prompt (a=1, b=1, c=1024, d=1022). We are invariant because 1x4096 and 2x2048 are THE SAME TOTAL PREFILL WORK. The encoder-decoder disable cannot fire, for a stronger reason than "the condition is false": SchedulerConfig::PostInit has ZERO call sites outside its own test, so it is never invoked on the engine path at all. No timing gate can separate "chunking works and the totals are equal" from "chunking never happened" -- which is why the test pins the COMPOSITION. Its 1022-token remainder is load-bearing: a reviewer's mutation that skips the running loop keeps total_num_scheduled_tokens at exactly 2048 and is caught only by that value. Six mutations RED, three of them the reviewer's own. WHAT IS ESTABLISHED, and what is not: * intra-pin, paired: the pin's own 4096-token forward costs 42% more wall clock than two 2048s for identical work (0.8371/0.8147/0.8148 vs 0.5789/0.5789/0.5774; spreads 2.7%/0.3%; n=3; interleaved; clock pinned per leg; boot_id logged; oracle asserted). Our invariance likewise (1.4%/1.0%). So the 1.21x we appeared to win at 8192/c4 is us beating an upstream pathology, not headroom. * cross-arm: NOT paired and not differenceable. `ours` came from the chunkprobe barrier-client with --num-blocks 2048; `pin` from mnbt.sh's curl spawn-loop with --gpu-memory-utilization 0.6; never interleaved with each other. The residual "+3.0% for us vs +11% for the pin" is a HYPOTHESIS. * --language-model-only was PIN-ONLY, not a shared condition. * the ours@8192/c1 cell is a mean of TWO legs; its rep-1 leg recorded 6135.86s -- an artifact of a wedged harness whose resume ledger re-imported it, which the reducer correctly flagged NOT ESTABLISHED on a 34902x spread. Two upstream anchors corrected: the cudagraph cap is 64, not 512 (both budgets still tie), and "upstream's flat 2048" is a pydantic TESTING default that EngineArgs._set_default_args overrides to 8192/16384 on a >=70 GiB GPU. GB10 is one, so our MoE-8192 may BE upstream's effective default -- the framing that choosing it flattered us against a "true" 2048 bar is withdrawn. Filed, not fixed: #682, the sweep gave our arm 8,192 tokens of KV against the pin's 1,819,368 -- a 222x asymmetry that suppressed the very batching being varied. Provisioned, our arm ran slightly FASTER, so the 0.85x is conservative rather than flattering, but it is owed a re-measure before being banked. No product change: the scheduler already mirrors upstream. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Claude:claude-opus-5 [ClaudeCode]
…n exposed (#634) (#706) FOLLOWING_AGENTS_PROTOCOL Completes CAMPPlus, the speaker-style encoder the talker consumes (spk_cond_mode="campplus"). FCM 2-D head -> TDNN -> three CAMDenseTDNN blocks (12/24/16) -> transits -> StatsPool -> dense, 12 cases / 27 assertions against the real upstream class executed directly. Weights are keyed by upstream state_dict NAME with the 815-tensor MANIFEST emitted into the goldens, so a tensor one side builds and the other does not is a failure; a missing tensor throws BY NAME, gated. A GATE HOLE, found by mutation and closed: changing the TDNN head's padding 2->1 compiled cleanly and the suite still PASSED, because it changes the frame count 20 -> 19 and StatsPool averages over time, so the pooled embedding absorbs it. Closed by tapping the post-TDNN activation upstream and exposing it via ForwardTrace — channels, frame count and values are gated where pooling cannot hide them. Both mutations now RED at compile_err=0. Also caught for the fourth time this campaign: a failed -Werror build re-ran the previous binary and printed SUCCESS, spotted because the CASE COUNT did not change while the compile-error count was 1. REMAINING: w2v-bert-2.0, EnhancedCodec, S2Mel, compose, routes/ABI. No e2e claim. Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
) (#707) FOLLOWING_AGENTS_PROTOCOL Starts w2v-bert-2.0, the semantic front end IndexTTS-2.5 runs over the reference clip (infer_v2_5.py:174). Ports the parts that make the encoder a CONFORMER: the macaron feed-forward and the depthwise convolution module. 3 cases / 181 assertions, within 1e-5. ORACLE: `transformers` executed DIRECTLY — the class IndexTTS itself instantiates, so no restatement. Admissible under AGENTS.md §"When vLLM has no implementation": a model's own reference implementation. THE TRAP: the conv module pads (kernel-1, 0) — LEFT ONLY, so a frame never sees the future. A symmetric pad yields an identically shaped output of the same length, so neither a shape nor a length check finds it. Gated on upstream values AND by an independent causality probe: perturbing the LAST frame must leave every earlier output bit-identical while the last one moves. Also pinned: swish not gelu, and the GLU gate orientation. MUTATION EVIDENCE with compile status: symmetric pad, gelu-for-swish and swapped GLU halves each RED at compile_err=0. The gelu mutation's first attempt did not APPLY (anchor count 0, Swish being a one-liner) and printed a meaningless SUCCESS; the anchor assertion caught it. DEFERRED deliberately: relative-key self-attention, so it lands whole. Then EnhancedCodec, S2Mel, talker head, compose, routes/ABI. No e2e claim. Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
…they carry — L1 (#607) FOLLOWING_AGENTS_PROTOCOL 43 of the 157 official vLLM recipes pass --language-model-only and we rejected it. L1 ports the mechanism behind that flag: per-modality input limits, and the REFUSAL that makes a limit a limit. Precedence mirrored literally from multimodal.py:321-336 -- language_model_only returns 0 for EVERY modality BEFORE the map is consulted, else the map entry, else 999. The case that distinguishes the orderings is a modality absent from the map, and it is pinned. The refusal mirrors context.py:392-405 (the min() fold), :409-428 (message and its conditional hint, character-identical including the backticks), and both call sites, with the enable_mm_embeds escape in both spellings. The judgement call worth recording: the refusal throws vllm::v1::InputValidationError, MOVED into its own header rather than a fresh multimodal exception class. api_server.cpp already maps that type to HTTP 400, mirroring upstream's ValueError -> BadRequestError. A bespoke class would have passed its own tests and then surfaced a too-many-images request as a 500. The review confirmed exactly one definition, no include cycle, and all 17 TUs seeing the identical type -- and stated the honest limit: the 400 path is unproven END TO END until L2 wires a call site, so L2's reviewer must treat that as an owed gate rather than an inherited pass. Scope is L1 only, verified clean in both directions: no serve flags, no ABI field, no tower gating, no kernel gate -- and the refusal was NOT deferred out. chat_mm.cpp is untouched, so its first-image-and-break truncation stands, filed as #686 and named in the PR body and USAGE.md rather than silently left. Wiring it is L2, which is what makes #686 the reason L2 exists. Reviewed on aarch64 (a CI-supported lane; the dev box was at 100% disk and load ~100 and was not used). Clean build 1262/1262 under -Werror with zero warnings; ctest 424/425, the sole failure test_async_llm passing 1/1 serially. 8 mutations run, 8 caught, none survived. The reviewer discarded its own first sweep after finding cp -a preserved mtimes and ninja had skipped a rebuild, so one result came from a stale binary carrying the previous mutation. The implementer's self-declared coverage gap is real and was judged correct layering rather than a hole: the precedence is MultiModalConfig's own contract and its own suite owns it. docs/USAGE.md was forced by check-doc-checkpoint classifying any include/vllm/ edit as user-facing; the entry states no request is limited or refused today, so a reader learns strictly less capability than exists. Row stays READY: no lifecycle move, L2-L4 owed. Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
…engine (#672) (#700) `multimodal::SpeechEngine` did not exist when this spec was written. The IndexTTS-2.5 lane landed it on 2026-08-13, along with `Vocoder1D`, while W0 was in review. AGENTS.md is explicit that a capability not reachable through the shared surface is not done, and that a seam is EXTENDED rather than forked -- so the scope changes before an implementer builds the wrong thing, not after. IT FITS BETTER THAN IT LOOKS. `SpeechResult` already carries `channels`, and already documents `sample_rate` as "the family's native rate ... rather than a resampled one, so the caller decides whether to resample". That is exactly §1.1's finding arrived at independently: 44100 stereo is the model's rate, and SGLang-Omni's 32 kHz is a caller-side transform. `requires_reference_audio()` exists so a server can refuse before staging weights; Music3 returns false where IndexTTS-2 returns true, which is the difference between a voice cloner and a text-to-music model expressed in the seam rather than around it. THE ONE GENUINE GAP IS `SpeechGenParams`, which carries a single `text` field because IndexTTS-2 synthesises one utterance. Music3 takes TWO distinct inputs -- lyrics with section tags, and a structured music description -- plus generation controls (duration/frames, denoise steps, CFG). Packing both into `text` behind a separator would be a private protocol inside a shared struct, which is precisely the fork the rule exists to prevent. W6 extends the struct ADDITIVELY and leaves IndexTTS-2.5 byte-identical: a field an existing family ignores costs it nothing, while a second parallel params struct costs every future family a choice. If the extension cannot be made additive, that is a NEEDS_DECISION, not a fork. ALSO RECORDED: `SpeechEngine` is not on the ABI yet. include/vllm.h is at v18 and exposes the video engine but no `vllm_speech_*` surface, and no open PR adds one (checked the 15 open PRs). So W6 owns that surface, its version bump, and the example HTTP server as a thin client of it -- and the W5 row now names `Vocoder1D` as the thing to route through rather than leaving "the vocoder" to be invented. W5 and W6 both gain the same obligation as a done-when: H3 and IndexTTS-2.5 behaviour stays byte-identical. Touching a seam another lane owns is allowed; moving its numerics silently is not. Issue: #672 FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code] Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
…#709) FOLLOWING_AGENTS_PROTOCOL Ports Wav2Vec2BertSelfAttention with position_embeddings_type="relative_key", gated against `transformers` executed directly. 5 cases / 184 assertions, within 1e-5. THREE THINGS THAT FAIL QUIETLY, each mutated: THE CLAMP IS ASYMMETRIC (64 left, 8 right). A symmetric clamp still produces well-formed attention and diverges only for keys further ahead than right_max. The goldens use T=12 precisely so the RIGHT clamp bites; at T<=9 the mutation would pass. THE DISTANCE IS KEY MINUS QUERY. The reverse is a mirrored bias that still attends and still trains. THE RELATIVE TERM IS SCALED SEPARATELY by 1/sqrt(d), after the scores already were. Also pinned: the encoder is NOT causal — a case perturbs the last frame and requires the FIRST output to move, the opposite of the conv module's guarantee gated in the same suite. MUTATION EVIDENCE with compile status: sign flip and missing scale RED at compile_err=0. The clamp mutation first read compile_err=1 (right_max unused under -Werror) and printed a void SUCCESS — the fifth occurrence this campaign. Generator fix: self_attn is called with KEYWORD args so a positional forward_pre_hook sees an empty tuple; the preceding layer_norm output is tapped. Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
…er (#634) (#710) FOLLOWING_AGENTS_PROTOCOL Composes the three gated parts into Wav2Vec2BertEncoderLayer and gates the WHOLE block against `transformers` executed directly. 6 cases / 186 assertions, within 2e-5. WHY THE ASSEMBLY NEEDS ITS OWN CASE: the macaron 0.5 factors live in the LAYER, not in FeedForward, so every piecewise case already on main passes with or without them. Green parts do not imply a green whole. The conv module's absent pre-norm is the same trap in reverse: it normalizes internally, so adding one here double-normalizes and still produces a well-formed block. MUTATION EVIDENCE with compile status: dropping the first macaron 0.5, double-normalizing before the conv module, and dropping the final layer norm each RED at compile_err=0; restored tree SUCCESS. The w2v-bert ENCODER LAYER is complete. Remaining: model-level wrapper (feature projection + layer stack), then EnhancedCodec, talker head, length regulator, S2Mel. No e2e claim. Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
…#712) FOLLOWING_AGENTS_PROTOCOL Completes w2v-bert-2.0, the stage that turns the reference clip into the features EnhancedCodec quantizes. 8 cases / 190 assertions against `transformers` executed directly. FeatureProjection normalizes THEN projects. The encoder is the layer stack and NOTHING else: embed_positions is None for relative_key, and there is NO final layer norm after the stack — the trap worth naming, since most encoders end with one and adding it yields a correctly shaped tensor only a value comparison can reject. MUTATION EVIDENCE with compile status: adding a final norm, and projecting before normalizing, each RED at compile_err=0. PROCESS NOTE: this change's worktree lost a ref-lock race against a concurrent session, so `cd` failed and the chained commands ran in the SHARED CHECKOUT, editing two tracked files. Caught on the next command and reverted; checkout clean. A failed `cd` does not stop a chain from running wherever the shell already was. Pipeline: w2v-bert DONE, CAMPPlus DONE, vocoder core DONE, GPT-2 backbone DONE. Remaining: EnhancedCodec, talker head, length regulator, S2Mel, compose, routes/ABI. No e2e claim. Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
…d no registry face (#608 W1) FOLLOWING_AGENTS_PROTOCOL W1 was scoped as "port a parser". The Inkling dialect was ALREADY ported and oracle-gated -- src/vllm/parser/inkling.cpp, inkling_config() with its arg carver, get_parser_engine("inkling"), golden scenarios in test_parser_engine_assembly. What was missing was the tool-parser REGISTRY FACE: get_tool_parser("inkling") returned nullptr, so --tool-call-parser inkling threw at startup and a fully ported dialect was unreachable. 41 -> 42 names, 37 -> 38 families. The change is ParserEngineToolAdapter (the TOOL half of upstream's make_adapters, twin of the reasoning adapter already shipped) plus InklingEngineToolParser, the factory branch and the tool_parser_names() entry together, and ParserRequestFromChatCompletion moved beside ParserRequest so the adapter and the serving path share one projection instead of two copies. The mutation that drops tool_choice from that projection reds 20 cases -- that is what gates the move. No detect.cpp marker row, for a reason no other excluded family has: the table matches a CHAT TEMPLATE string, and Inkling has no jinja template at the pin (rendering is vllm/renderers/inkling_encoding.py). Its marker collides with nothing; a row would look harmless while being unreachable. A secondary-oracle premise was REFUSED rather than followed. I briefed that GptOssToolParser's stub meant vLLM implements nothing for gpt-oss, so SGLang was admissible under the new registry rule. It is not: vllm/parser/harmony.py is a 358-line vLLM-owned HarmonyParser(DelegatingParser) with its own test, and the stub delegates INSIDE vLLM. Following my brief would have consulted a secondary oracle where the primary speaks. Recorded in the spec so nobody re-runs the check. Reviewed, FAILed on the record, repaired, and the corrections are larger than the original claim. "Upstream never constructs the adapter" was false -- tests/parser/engine/test_inkling.py:487 TestRegisteredAdapters does exactly that -- and that sentence was load-bearing, because it justified never looking outside two test classes. Four more cases ported, one declined with its reason; the split is 22 cases, 19 PORTED / 3 AUTHORED, not the 15/18 first reported. The authored CONTENT-seed case survives but its rationale was also wrong: the seed lives only in extract_tool_calls_streaming, not in the delegating non-streaming path. Two gaps left OPEN and recorded rather than papered over. finish_streaming() is functionally ungated -- mutating it to return nullopt leaves all 22 cases green; the repairer authored a streaming twin to try to move it, it did not, and the case was DELETED rather than ship a false rationale. And README's counts are stale by two waves, but check-doc-checkpoint REFUSES the fix ("Co-edited public projections never justify README churn") and refuses a README-only fix too, so #704 needs a landing-source change or an argued exception. #703 records that --reasoning-parser inkling still throws while the tool parser now resolves. Verified on aarch64 (kairos-4db2, CPU-only; the dev box was at 98% disk). Clean build 0 warnings under -Werror, ctest 432/433 with the one failure passing serially, test_tool_parser_inkling 22/22 133/133. Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
…c codes (#634) (#713) FOLLOWING_AGENTS_PROTOCOL Ports Amphion's FactorizedVectorQuantize (use_l2_normlize=True) — the `semantic_codec.quantize(...)` of infer_v2_5.py:293, whose DISCRETE output is the semantic code the talker consumes. Gated against that module executed directly. 3 cases / 17 assertions. INDICES ARE GATED EXACTLY: an index off by one codebook entry is a different utterance, not a small numerical error. THE NORMALIZATION IS SEARCH-ONLY: distances use L2-normalized encodings and codebook rows, but decode_code returns the RAW row. Returning the normalized row leaves every index identical and moves only z_q, so one case pins indices and the other pins values. Weight-normed projections carry the legacy weight_g/weight_v spelling; g*v/||v|| is pinned by a hand-computed case. ONE MUTATION SURVIVED, AND SHOULD HAVE. Removing the encoding normalization is a provable NO-OP: dist = |e|^2 - 2 e.c + |c|^2, |e|^2 is constant per frame and a normalized codebook makes |c|^2 = 1, so argmin reduces to argmax(e.c), which scaling e cannot change. This repo's rule is that an unmoved mutation usually marks a FIXTURE GAP, so it was checked rather than assumed; it is an algebraic identity, and the reasoning is recorded at the call site. Other mutations RED at compile_err=0: returning the normalized row, and dropping the sqrt from weight-norm. REMAINING: EnhancedCodec's VocosBackbone encoder, then talker head, length regulator, S2Mel. No e2e claim. Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
… (#715) FOLLOWING_AGENTS_PROTOCOL Completes EnhancedCodec: VocosBackbone (ConvNeXt-1D) joins the factorized VQ already on main. 2 cases / 91 assertions against upstream executed directly. THE OUTPUT IS [T, dim], not [dim, T] — the final layer norm runs on the transposed tensor and is never transposed back. Same element count either way, so the layout is asserted alongside the values. A TOLERANCE THAT ACCEPTED A WRONG ACTIVATION, found by mutation and fixed in three steps, each correcting a wrong hypothesis: 1. Swapping exact-erf GELU for tanh-approx PASSED at 2e-5. First guess was small activations, so the INPUT scale went 1.0 -> 4.0. No effect: the LayerNorm before pwconv1 normalizes input scale away entirely, so the pre-activation magnitude is set by the WEIGHTS. 2. Raised pwconv1's weight scale 0.3 -> 1.0, moving |x| from ~0.85 to ~2.8 where the GELU forms differ most. Still passed. 3. MEASURED instead of guessing again: the true end-to-end difference is 1.41e-5, genuinely below the 2e-5 bound. Tightened to 2e-6, which the correct port clears comfortably; the mutation now goes RED. A bound must be set by what the DEFECT costs, not by what feels safe. Fixture and tolerance are annotated with the measurement. Four of six pipeline stages complete. Remaining: talker head, length regulator, S2Mel. No e2e claim. Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
…type policy the oracle corrected (#672) (#714) * feat(MODEL-MUSIC-MUSIC3): W1 — the modular loader, and a count the artifact corrected (#672) Phase W1 of .agents/specs/minimax-music3.md: load the six-component diffusers-arm checkpoint and assert its geometry. No forward, no scheduler, no audio; those are W2-W7 and nothing here produces a song. WHAT LANDS * `MiniMaxMusic3ResolveCheckpoint` resolves the diffusers layout and refuses everything else BY NAME. The refusal that matters is the NATIVE arm (`qwen_7B/` + `flowmatching_vae.pth` + `dav.pth`): it holds every weight this port needs in a layout nothing here reads, so it is the one input that would otherwise LOOK loadable. It is named as the native arm, told which components the diffusers arm wants, and pointed at upstream's conversion script. ONE marker is enough to diagnose it, so a partially staged native tree cannot fall through to the generic "components missing" message. * `MiniMaxMusic3LoadConfig` parses all six configs, refusing a `_class_name` mismatch and every absent key rather than defaulting. `rotary_dim`, `tie_word_embeddings` and the four condition-encoder rate keys move a model without moving a tensor shape, so a default there is a different model with no error. * `EnumerateMiniMaxMusic3*Tensors` walks upstream's own `__init__` module for module at diffusers PR #14456 head `c6da9936`, and `MiniMaxMusic3AccountTensors` accounts a component's files against it: MISSING, UNACCOUNTED, wrong SHAPE and wrong DTYPE are four separately named refusals, and it returns counts so a gate can say how many things it looked at. * The fp32/bf16 split is ENFORCED per component, not tolerated. A dtype that is too WIDE is numerically correct, so no token or golden gate can see it. * `MiniMaxMusic3LoadVocoderWeights` FOLDS the weight norm at load, through `vocoder1d::MaterializeWeightNorm`, so no `_g`/`_v` name survives and nothing downstream can read the direction `v` as if it were the weight. THE SHARED SEAM, AND WHY IT MOVED The fold lived in `minimax_h3.h` as `MiniMaxH3MaterializeWeightNorm`. MiniMax-Music3's vocoder is its second consumer, so it moved to `vocoder1d::MaterializeWeightNorm` on exactly the precedent `vocoder1d.h` records for itself: a model's header is not a home for something two lanes share, and no numeric test can see a fork, because a fresh copy agrees on the day it is made and only drifts later. The 21-line body is BYTE-IDENTICAL modulo the parameter rename; `test_minimax_h3` passes 57395 assertions unchanged. The rename is not cosmetic. The argument was `out_channels`, which is true for an `nn.Conv1d` weight `[C_out, C_in, K]` and FALSE for the `nn.ConvTranspose1d` weight `[C_in, C_out, K]` that four of this vocoder's thirty convolutions carry. torch reduces over dimension 0 either way, so it is now named `dim0`. `tests/scripts/test_vocoder1d_single_home.py` gained the symbol, and its regex alternation with it: adding it to SYMBOLS alone reported 0 definitions, which is the guard catching an incomplete edit rather than a defect in the guard. A COUNT I GOT WRONG, RECORDED The header and the test I wrote first both said the vocoder has 20 weight-normed convolutions. The spec never states that number; the artifact says 30: `conv_in` and `conv_out`, plus per block one `conv_t1` and three residual units of two convs each (7 x 4 = 28). 2 + 30*3 + 29 snake alphas = 121, the component's whole tensor count. The test asserting 20 went RED against the real manifest and the number was corrected to the measurement, not the other way round. EVIDENCE RED first, against a stub whose every entry point returned nothing: 19 test cases, 6 passed, 13 failed; 91 assertions, 51 failed. The 6 that passed are the pure-manifest gates, which assert against the artifact and need no loader. GREEN: 19/19 cases, 1361/1361 assertions with no checkpoint; 1381/1381 with `VLLM_CPP_MUSIC3_CHECKPOINT` set, accounting all 1012 tensors of the real 27 GB tree (transformer 441 over 2 shards, language_model 399 over 4, vocoder 121, rvq_depth_decoder 47, condition_encoder 4). MUTATIONS. 47 in-suite mutations are proven to fire: 20 wrong-shape, 10 wrong-dtype, 10 missing/extra, 7 config-disagreement. The relocated seam was mutated in a scratch copy twice and BOTH suites went red: reducing over the whole tensor instead of per dim-0 slice (music3 19 failed, h3 10 failed), and dropping the normalization (music3 5, h3 9). Tree restored byte-for-byte, md5 verified, both green again. The manifest is the shipped checkpoint's own safetensors headers, 1012 entries of names/dtypes/shapes and not one weight byte, so CI never needs the asset. `scripts/gen-minimax-music3-manifest.py` regenerates it and cross-checks each sharded component's headers against its index rather than globbing. Full clean rebuild, 430/430 ctest passed, 0 failed. `test_cpu_x86_llamacpp_floor` is the known load-dependent flake: it failed once mid-run and passed 3/3 on the identical tree afterwards. NOT CLAIMED: no value parity against any oracle. W1 is shapes, dtypes and refusals. The spec's `## Now` and `Status` are deliberately NOT advanced: W0's oracle is still not gateable, and W1 needs no oracle because it gates against the checkpoint's own headers. FOLLOWING_AGENTS_PROTOCOL Issue: #672 Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code] * fix(MODEL-MUSIC-MUSIC3): the on-disk dtype set is NOT the runtime policy (#672) Correcting spec section 2.1, which this branch had mirrored. The oracle now runs, and it refutes the reading: the dtypes the converter WRITES are not a runnable configuration for the pipeline that consumes them. VERIFIED AT THE PIN, not taken on report. At diffusers c6da9936 the only two casts in the whole pipeline are `denoise.py:83` (condition encoder output -> transformer.dtype) and `decoders.py:84` (latents -> vocoder.dtype). Nothing casts on the way IN: `denoise.py:82` is `components.condition_encoder(block_state.frame_hiddens[...].to(device))`, a device move with no dtype move. So the condition encoder and the depth decoder consume the language model's hidden states UNCAST, and the on-disk set raises RuntimeError: Input type (c10::BFloat16) and bias type (float) should be the same from `condition_embedder_minimax_music3.py:64`. The two lines above it are the tell and they are why this is a real invariant rather than an oversight: `:61` and `:63` cast `layer_weight_logits` and `layer_scale` with `.to(hidden_states.dtype)`, so the learned MIX follows the hidden states, but `self.proj` is a MODULE and is never cast, so it cannot. WHAT CHANGES * `MiniMaxMusic3ResolveRuntimeDtypes` / `MiniMaxMusic3CheckRuntimeDtypes` enforce dtype(language_model) == dtype(rvq_depth_decoder) == dtype(condition_encoder) and refuse a violation BY NAME, listing all three with their dtypes and the upstream line that would otherwise fail. Upstream's own message names a bias dtype and never says which component disagreed with which, which is exactly why the refusal belongs at load. * `kBf16ArFp32Acoustic` is the gated configuration: bf16 AR half, fp32 acoustic half. `kAsStored` returns the on-disk set UNREPAIRED and reports it as not runnable, so the failure stays reproducible against the oracle's `--dtype-policy on-disk`. Promoting the condition encoder to bf16 there would hide the finding the arm exists to preserve. * The acoustic half is deliberately NOT constrained: each of the transformer and the vocoder is reached through an explicit cast, so each may differ from the AR half and from the other. The gate proves all four variations pass. WHAT DOES NOT CHANGE. The per-component ON-DISK dtypes in the enumeration, and the account that refuses a file disagreeing with them. Those are facts about the artifact, measured from the headers, and they were never the thing in dispute. What was wrong was presenting that set as the runtime policy; the header said so in as many words and now says the opposite, with the oracle finding cited rather than section 2.1. fp32 on the acoustic half still needs no apology under AGENTS.md's too-wide rule: the oracle runs fp32 there too. EVIDENCE. RED first against a stub that refused nothing: 21 cases, 2 failed; 1393 assertions, 23 failed. GREEN: 21/21 and 1393/1393 without a checkpoint, 1413/1413 against the real 27 GB tree. The new gate proves all three single -component AR deviations are refused by name, that the on-disk set is reported not-runnable, and that the on-disk set it claims is the one the committed manifest actually measured, so the claim is about THIS checkpoint. Full build 0 warnings, 433/433 ctest. FOLLOWING_AGENTS_PROTOCOL Issue: #672 Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code] --------- Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
…634) (#719) FOLLOWING_AGENTS_PROTOCOL S2Mel's DiT conditions on the timestep/style vector through ADAPTIVE LAYER NORM rather than cross-attention. Ports that mechanism plus the FinalLayer using it. 4 cases / 48 assertions against upstream executed directly. modulate(x, shift, scale) = x * (1 + scale) + shift. THE `1 +` IS THE DETAIL: without it the modulation centres on 0 instead of 1, which still trains and still generates. Pinned by a hand-computed case, not only the oracle — with scale = 0 the result must be the IDENTITY plus shift. THE FINAL NORM HAS NO AFFINE PARAMETERS (elementwise_affine=False), and the manifest confirms it: linear.* and adaLN_modulation.1.* exist, norm_final.weight/bias do not. A case asserts exactly zero-mean, unit-variance rows. CHUNK ORDER IS [shift, scale], pinned by effect: zero the modulation weight, set shift = 0 and scale = -1, and x * (1 + -1) + 0 collapses to zero so the output is exactly the linear bias. Swapped, it would not be. The first version of that case asserted identical output ROWS, which was wrong — layer norm makes rows zero-MEAN, not equal. The oracle comparison passed throughout, so the failure was the TEST's, and it was rewritten around the zero-collapse rather than loosened. MUTATION EVIDENCE with compile status: dropping the `1 +`, swapping chunk order, and a phantom affine gamma each RED at compile_err=0. Remaining: S2Mel's DiT attention blocks, the talker head, composition. No e2e claim. Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
…at accepted all-zeros on bf16 (#517) (#718) FOLLOWING_AGENTS_PROTOCOL W4 of MODEL-TEXT-nemotron-h (#517): the NemotronHForCausalLM hybrid forward -- 23 Mamba2, 6 GQA attention at {5,12,19,26,33,42}, 23 non-gated relu^2 MoE. The implementation was inherited from a rescued WIP whose author's session died mid-build with nothing compiled and no RED-first evidence, and its gate turned out to be VACUOUS. On the bf16 arm -- the released checkpoint's dtype -- the band was 6e-2 ABSOLUTE against an attention reference whose peak is 1.06e-2, so a block returning all zeros passed. Only the f32 arms were gating anything. The fresh review proved that by experiment rather than accepting the report: it reconstructed the inherited gate, made the attention mixer return zeros, and watched BOTH bf16 arms accept it. Against the repaired gate the same mutant fails on all four arms. Had this landed as rescued, W4 would have shipped a suite that could not distinguish a working forward from a dead one in the dtype the model actually runs. Repairs: peak-relative bands; every comparison self-certifies by REQUIREing that its own band rejects all-zeros; the attention fixture was degenerate (logits ~0.09 gives a near-uniform softmax, blind to RoPE, causality AND scale); and the SSM-dtype guard moved to the stored state, because A = -exp(A_log) decays downstream separation to 3.16e-5, below the gate's own resolution. 161 -> 254 assertions. A conflation this operator introduced is corrected. Folding routed_scaling_factor into the ROUTER is invisible to the model-level gate -- 13/13, 254/254, separation 1.91e-07 -- because norm_topk_prob renormalisation cancels a constant factor, which layer.py:291-300 states outright by forcing the router factor to 1.0 "so it ends up being a nop". The "13276 of 20000 rows" figure was always about a DIFFERENT fold, vt::MoeCombine's own semantics, gated bitwise elsewhere. Spec 6c separates them. Seams: vt::FusedChain for both residual add+RMSNorm sites (reverting to the inherited hand-calls REDs check-fusion-consistency); ModelRegistry::Forward. dense_attn::AttnBlock is recorded NOT APPLICABLE rather than skipped -- it needs PagedKvCache, slot_mapping, block_table and loader-built weights, the same boundary kimi_linear_forward.cpp sits on, and W6 owns it. The forward composes vt::Attention rather than hand-rolling a parallel path. 15 mutations all RED, the table reproducing numerically under independent re-application. ctest 433/433, clean -Werror, Debug arm identical, preflight green. Reproduced on Thor sm_110 with CUTLASS correctly disabled: forward 13/254, scaffold 12/38285, Mamba2 suites matching GB10 exactly. That proves a CUDA-enabled build on a second architecture and NOT GPU execution of the forward, which asserts a CPU queue by design -- a caveat drawn in the spec and never upgraded. Owed and stated plainly: there is no weight loader, so nothing materialises the 18487 tensors, no checkpoint runs, and the forward refuses by name. W5, W6 and W7 remain. No speed claim. Row stays INVENTORIED. Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
…#382 (#564) Commits the spec for the ROCm head_dim=128 decode arm, the ROCm half of #382. Zero source files; the implementation is not merged and this spec says so explicitly. Every code and upstream anchor it cites was verified exact at the pinned oracle during review: bf16_decode_opt at rocm_paged_attn.hip:1684 matches its quoted snippet verbatim, the EPL static_asserts at 268/289, the launch switches at 1875-1904 with only 8/16 instantiations, the five-combination fallback dispatch at 1937-1947, the CUDA counter-claim at cuda_paged_attn.cu:321/329/2796, and CALL_CUSTOM_LAUNCHER_BLK_HEAD upstream at the pin. Review repairs landed on the branch before merge. The blocking one: the spec opened with "Landed the ROCm d=128 decode arm" when nothing had landed -- git log -S'VT_ATTN_DECODE_D128' -- src/vt/rocm/ is empty on main and rocm_paged_attn.hip still gates on d == 256 || d == 512. Merging that text would have put a false "landed" on main, where the next agent greps for the flag, finds nothing, and cannot tell "never merged" from "reverted" from "renamed". Renamed to "Result on the implementation branch" with a banner quoting the two commands that show it, and section 4 moved to future tense for the same reason. Also added the two sections AGENTS.md requires and the spec lacked -- Risks and decisions, and Stop conditions -- drawn from material already in the spec rather than invented: the bf16-tie reduction-order risk that is why the arm ships default OFF, the single-board provenance of the 3.53x, the sm_110 1.6x-SLOWER reversal recorded but deliberately unreconciled, the dangling VT_ATTN_DECODE_WMMA forward reference, and the unquantified residual #488 gap. Plus a base-SHA mismatch and an upstream line anchor that had drifted by two. The spec is honest about what it could not gate: it labels its own throughput table "indicative, not the 2-3x-idle-reproduced standard" and states that no post-change per-call oracle re-measure was run. Known-baseline failures only: windows-msvc-* are the PR-only arm (#584), and sanitize-cpu (address,undefined) is red on main itself (#674, fix pending in #688). This PR changes only Markdown.
…, rotary (#634) (#721) FOLLOWING_AGENTS_PROTOCOL The DiT's transformer is gpt-fast's: every block is conditioned by an AdaptiveLayerNorm around an RMSNorm with rotary position embeddings. Gated against gpt-fast executed directly. 5 cases / 30 assertions. TWO CONVENTIONS THAT DIFFER FROM THEIR NEIGHBOURS IN THE SAME MODEL: * This AdaptiveLayerNorm is `weight * norm(x) + bias` — NO `1 +`, unlike adaln::Modulate used by the FinalLayer and already on main. Two adaLN conventions coexist; porting one over the other is silent. The mutation applying the FinalLayer's convention here goes RED. * The rotary pairs ADJACENT components, not halves. Half-split is more common and yields a rotation that is smooth, norm-preserving and wrong. RMSNORM DOES NOT SUBTRACT THE MEAN, pinned structurally as well as against the oracle: for a row of identical positive values x / rms(x) = 1 everywhere, where LayerNorm gives 0. The rotary gets a value comparison AND a norm-preservation property, together — a rotation preserves pair magnitude under EITHER pairing, so the property alone cannot distinguish them. FIDELITY NOTE: precompute_freqs_cis returns BFLOAT16, so the rotary table carries ~3 decimal digits before any rotation runs; the tolerance reflects that. MUTATION EVIDENCE with compile status: wrong adaLN convention and half-split pairing each RED at compile_err=0. A comment-only NO-OP control ran in the same batch and stayed green, showing the harness is not simply failing on any edit. Remaining: assembling the S2Mel block, the talker head, composition. No e2e claim. Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
FOLLOWING_AGENTS_PROTOCOL Composes the primitives on main into gpt-fast's TransformerBlock as the DiT uses it, gated WHOLE against upstream executed directly. 7 cases / 34 assertions. h = x + attention(attention_norm(x, c)) out = h + feed_forward(ffn_norm(h, c)) BOTH RESIDUALS ARE FULL, unlike the macaron HALVES in the w2v-bert Conformer already on main — two block types with different residual conventions in one lane. The halving mutation goes RED. SwiGLU gates on W1 not W3, and rotary applies to q and k but NEVER to v. A GENERATOR BUG THE GATE CAUGHT: two cases began failing with ZERO failed assertions — they THREW rather than asserted, because kQ was emitted at the wrong size. Cause: the new block section looped `for n, q in ff.named_parameters()`, SHADOWING the rotary input q. Reading only the assertion line would have shown "12 passed | 0 failed" and looked fine; the CASE count exposed it, and after the fix assertions went 12 -> 34 because ~22 had been skipped. This is the repo's recorded `grep assertions:` trap met from the generator side, so mutation results here are reported by CASE COUNT. MUTATION EVIDENCE with compile status: swapping W1/W3, skipping rotary on K, and halving the attention residual each RED at compile_err=0. S2Mel's block is complete. Remaining: the talker head, then composition. No e2e claim. Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
…now has gated numerics (#634) (#724) FOLLOWING_AGENTS_PROTOCOL The talker (UnifiedVoice, model_v2.py) runs the GPT-2 backbone already on main over a sequence assembled from text tokens, mel codes and a speaker latent. This ports that assembly. 5 cases / 46 assertions. ITS POSITION EMBEDDINGS ARE NOT GPT-2's wpe: LearnedPositionEmbeddings is a SEPARATE learned table added before the backbone runs, so the backbone's own wpe applies as well. Treating them as one silently halves the positional signal. get_fixed_embedding(ind) is the INCREMENTAL-DECODE path: at step n the position is n. Returning row 0 every step makes every generated frame believe it is the first, which still decodes to audio and destroys the prosody. NO GOLDEN FILE, deliberately: this layer is pure INDEXING and a captured tensor would only restate its own table. The fixture builds a table whose row r holds r*10 + d so a row's identity is readable from any element, and the cases assert WHICH row was read. Capacity is bounded, since text_pos_embedding.emb.num_embeddings is the talker's real request limit. MUTATION EVIDENCE with compile status, by CASE COUNT: always returning row 0, dropping the capacity bound, and dropping the position add each RED at compile_err=0. MILESTONE: every one of the six pipeline stages now has its numerics ported and gated against upstream. What remains is different in KIND — composition, the checkpoint loader, and only then a render. No e2e claim; the lane still refuses by name. Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
…yout the gate caught (#634) (#725) FOLLOWING_AGENTS_PROTOCOL Wires the six gated stages into ONE path in upstream's order (infer_v2_5.py:569-660) and gates the composition. 8 cases / 433 assertions. WHAT THIS IS: a STRUCTURAL composition at reduced dimensions on synthetic weights. It proves the stages connect, each output shape is the next input shape, a change at the front reaches the back, and a mismatch throws BY STAGE. NOT a quality result, NOT parity, NOT a render. A REAL BUG THE GATE CAUGHT: style_in was built frame-major (T, F) and handed to StatsPool, which expects CHANNEL-MAJOR [C, T], so the pooled "channels" mixed features with frames. Every shape assertion passed and the statistics looked plausible. It surfaced because TWO MUTATIONS SURVIVED the first version of this gate — constant prompt condition, and codec bypass — since propagation cases only prove SOME path carries influence, not that each stage's output is consumed. Two isolation cases fixed it: a TIME-PERMUTED clip has identical per-channel statistics so the style must be unchanged while the condition reorders (which is what exposed the transpose, since the codes rotate with identical multisets); and nearest interpolation COPIES samples, so every prompt-condition value must appear verbatim in the QUANTIZED tensor. Fixture notes: the first permutation used a REVERSAL, but this fixture's codes are palindromic (8 11 8 8 11 8 8 11 8) so reversing is a genuine no-op — a rotation preserves statistics just as exactly and does reorder. The transposed layout is now itself a mutation, so the bug cannot return unnoticed. MUTATION EVIDENCE with compile status, by CASE COUNT: constant prompt condition, codec bypass and transposed style layout each RED at compile_err=0. STATE: all six stages ported, gated and composed. Before any render: the CHECKPOINT LOADER and the vLLM-Omni oracle (#633). The lane still refuses by name. Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
… the ACTIVE contract (#672) (#723) W0 is done and W1 has landed, so the row's state was stale at `SPIKE` while real loader code sat in the tree. Advancing it is what surfaced two things worth recording rather than quietly fixing. THE ACTIVE CONTRACT IS ENFORCED, AND THE SPEC DID NOT MEET IT. Moving the row to `ACTIVE` turned check-agent-record red on nine missing structured fields -- Scope, Upstream chain, Our baseline, Port map, Tests to port, Gates, Dependencies, Work breakdown, Risks/decisions. A `SPIKE` spec is not held to that; an `ACTIVE` one is, which is the checker doing exactly its job. My first attempt added them as PROSE SECTIONS and stayed red on all nine, including the two the spec already had -- because the checker does not read headings at all, it reads TABLE ROWS whose first cell names the field. Fixed by writing the structured record in the shape `minimax-h3.md` already uses, rather than by arguing with the gate. Worth stating plainly: the failure was mine reading the requirement as prose, and the checker's message was right both times. SECOND, A WORDING FIX THAT COST TWO IMPLEMENTERS. §3 said both pins "go into `.agents/oracles/` in W0", which reads as future work. `sglang-omni.md` has existed since #679, yet two separate agents reported it as still owed -- the same agent twice. A record surface described in the future tense reads as a TODO no matter what the tree holds, so it is now present tense with links, plus a note saying why. That is a small edit that stops a recurring false report. WHAT THE ROW NOW CLAIMS, and nothing beyond it: oracle gateable with 13 committed per-stage goldens (#708), modular loader gated 1413/1413 against the real 27 GB checkpoint with all 1012 tensors accounted and the native arm refused by name (#714). Checklist mark 📋 -> 🚧; rollup SPIKE 9 -> 8, ACTIVE 9 -> 10. The lifecycle change owes STATUS, BENCHMARKS and the spec's `## Now`, and all three say the same thing the row does. BENCHMARKS records every axis PENDING with NONE OWED YET and says why: W0/W1 are correctness scaffolding, the oracle capture was CPU-only because dgx.casa was down, and there is no forward pass to time. A row that has landed code is exactly where a speed claim tends to appear by implication, so the absence is written down rather than left to be inferred. Both public rows were first written over the house-style limits (a 402-char BENCHMARKS cell, and STATUS at 45 oversized cells against a 44 ratchet that may only shrink). Shortened to fit rather than lowering a ratchet to accommodate prose. Issue: #672 FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code] Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
…from the shipped checkpoint (#634) (#727) FOLLOWING_AGENTS_PROTOCOL Reads IndexTTS-2.5's own config.yaml and repository manifest instead of the recipe page, pins the dimensions in indextts2_config.h, and gates them against a COMMITTED copy of the config so the check is reproducible offline. 4 cases. FOUR FINDINGS — two settle open questions, two ADD scope: THE MODEL RUNS AT TWO SAMPLE RATES. Talker mel front end 24 kHz / 100 bins; S2Mel and vocoder 22.05 kHz / 80 bins; OUTPUT is 22.05 kHz. Conflating them produces audio at the wrong SPEED rather than an error. THE LANGUAGE QUESTION IS SETTLED AND THE RECIPE PAGE WAS WRONG: the shipped tokenizer is multilingual_zh_ja_yue_char_del.tiktoken — zh, ja, yue. This spec had flagged the disagreement as unresolved. Nothing may claim Spanish or Arabic. THE TOKENIZER IS TIKTOKEN, with no tokenizer.json — the constraint already recorded for Kimi-Linear. A QWEN-0.6B EMOTION MODEL SHIPS INSIDE THE CHECKPOINT (qwen0.6bemo4-merge/ with its own safetensors, tokenizer and config), plus feat1/feat2 matrices and an emo_condition_module. A second language model inside a TTS lane, NOT in this spec's scope and covered by no work item. Also: BigVGAN and w2v-bert-2.0 are NOT in this repository; they fetch separately into hf_cache/ at first run. No e2e claim. This is the contract a checkpoint loader is built AGAINST. Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
…686) FOLLOWING_AGENTS_PROTOCOL The residual list grouped "multiple images" with video parts and *_embeds under "the parse tolerates them but they are not routed". That is accurate for video, audio and the embeds spellings -- nothing is sent, and the user notices. It is NOT accurate for multiple images: chat_mm.cpp:256-262 locates the FIRST image_url part and breaks, so a three-image request is SERVED WITH ONE, no error, no warning. The distinction is the whole point. "Not routed" implies a visible failure; truncation is the case where the user gets a confident answer about a subset of what they sent and cannot tell. AGENTS.md requires an unimplemented arm to be "refused with a message naming the missing piece", and upstream does exactly that -- validate_num_items (context.py:409-428) raises VLLMValidationError. The line is SPLIT rather than reworded, because the two halves now have different states: video/audio/embeds remain not-routed; images are truncated and tracked as #686. #607's L1 has since ported the refusal (BaseProcessingInfo::ValidateNumItems), so wiring this call site to it is L2 -- which makes #686 the reason L2 exists rather than a separate design. Found while verifying #607 L1, which surfaced the prior art it inherits. Records only: one markdown file. Every red on this PR is inherited from main and tracked -- sanitize-cpu (#730, verified failing on main's own run at a3aa02e), windows-msvc-* (#648/#503), audit-live-rows (#731, the Music3 ACTIVE row, which fails on a clean main checkout). A one-file docs diff reaches none of them. Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
…rd that would have caught my fork was never wired (#634, #728) (#734) Read the shipped IndexTTS-2.5 checkpoint's `qwen0.6bemo4-merge/model.safetensors` header by HTTP range request (2 MB, no weights) and its `config.json`. The result CORRECTS the finding recorded in #727. That directory was recorded as unscoped work, on the assumption that a second language model inside a TTS lane implied a second port. It does not. It is 310 BF16 tensors of stock `Qwen3ForCausalLM`: hidden 1024, 28 layers, GQA 16 query heads over 8 KV heads at head_dim 128, intermediate 3072, vocab 151936, tie_word_embeddings true. `config.json` names that architecture literally, this tree registers it in `qwen3_dense.cpp`, and `qwen3_weights.cpp:168` already has the tying branch that explains the absent `lm_head.weight`. What stays in scope is the IndexTTS-specific wiring around it: feat1/feat2 and emo_condition_module. `test_indextts2_emotion_arch_covered.py` pins that reduction against a committed manifest so it cannot rot in prose. Five mutations, five caught, control green: a tensor we do not read, an unregistered architecture, lm_head present under tying, a drifted layer count, o_proj losing its GQA width. No compile step, so no stale-binary confound. Wiring those gates into CI immediately found a real defect of my own (#728). `fvq.cpp` had forked `vocoder1d::MaterializeWeightNorm` -- byte-equivalent, but without that one's check that `g` carries one magnitude per dim-0 slice. Two definitions of a fold with one home. `test_vocoder1d_single_home.py` catches it exactly as designed and reported `MaterializeWeightNorm has 2 definitions`; it had simply never been added to ci.yml, so it only ran when invoked by hand, as had `test_indextts2_config_contract.py`. A gate that is not wired is not a gate. The fork is deleted and its three call sites retarget the single home. The public projection for this lane was also missing from every earlier merge: `check-doc-checkpoint.py --commit 73d217d` fails on that already-merged commit. FEATURES.md moves the TTS row from "not started" to a ◐ that says "assembled, never run", and USAGE.md records that `/v1/audio/speech` does not exist, so a server answers 404 at the route table rather than failing at runtime. Verified locally: test_fvq 17/17, test_indextts2_pipeline 433/433, test_minimax_music3_loader 1393/1393, all compile_err=0; the three newly wired Python gates green; check-public-doc-tables and doc-checkpoint --staged green. audit-live-rows stays red, verified IDENTICAL on clean origin/main and filed as #733 rather than absorbed here. Still not done: no checkpoint loader, no render, no route, and any parity claim needs the vLLM-Omni oracle, still UNPINNED (#633). FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code] Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
…cope they CORRECT upward (#634) (#735) Read the tensor manifests of `gpt.pth`, `codec.pth` and `s2mel.pth` without downloading 4.0 GiB of weights. A torch save file is a ZIP whose small `*/data.pkl` names every tensor while the gigabytes sit in separate blobs, so `scripts/read-torch-manifest.py` fetches the central directory and that pickle by range request and unpickles it with a stub Unpickler. 1712 tensors, no torch, a few hundred KB. The record is committed. It CONFIRMS four constants in `indextts2_config.h` from a source independent of the file they were read from. `test_indextts2_config_contract.py` compares the header to `config.yaml`; both sides share a source, so it proves transcription, not truth. This compares the header to the WEIGHTS: kTalkerDim 1280 is emo_layer's square, kStyleDim 192 is spk_emb_proj's input, kVocosDim 384 and kVocosIntermediateDim 2048 are the codec decoder's ConvNeXt widths, and kCodecHiddenSize 1024 is that decoder's input. It also CORRECTS this morning's correction, upward. #734 recorded the emotion work as "surrounding wiring: feat1/feat2 and emo_condition_module", reasoning from the fact that the emotion LANGUAGE model is stock Qwen3. That part holds. But `gpt.pth` carries two further networks that no work item covers and this tree does not already have: a Conformer encoder at width 512 (`emo_conditioning_encoder`, 38 patterns, rel-pos MHA with pos_bias_u/v, macaron FFs, depthwise-15 conv module, a Conv2d-subsampling front end at [512, 261632]) and a Perceiver resampler (`emo_perceiver_encoder`, learned latents, to_q/to_kv/ to_out, GEGLU at 2730). An architecture name settles what a MODEL costs and settles nothing about what a CHECKPOINT contains. The same reading names what the reduced-dim gates do not model, all of them green over a smaller network than the checkpoint holds: a WaveNet conditioning stack inside the CFM estimator, U-Net skip linears across DiT depth, a SECOND timestep embedder, the whole conditioning front end, three length-regulator tensors, `net.gpt_layer`, and the codec encoder. `codec.pth` also ships `optimizer.state`, so part of its 0.57 GiB is training residue. Six mutations, six caught, control restored and verified byte-wise: two header constants moved, the language embedding's 107 rows, a vanished DiT skip_linear, a shrunken emotion Conformer, and a drifted tensor total. Pure data and Python, so no stale-binary confound. Also records that the talker embeds 107 language ids while the tokenizer is zh/ja/yue, which bears on what this lane may CLAIM and is asserted rather than noted. audit-live-rows stays red, identical on clean origin/main, tracked as #733. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code] Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
… the backbone prefix, refuse unsupported MoE layouts (#490) FOLLOWING_AGENTS_PROTOCOL Qwen/Qwen3.8-2.4T-A95B declares Qwen3_5MoeForCausalLM / model_type qwen3_5_moe_text -- the same GDN-hybrid MoE backbone we already run token-exact on Qwen3.6-35B-A3B. head_dim 256, linear k/v head dim 128, full_attention_interval 4, partial_rotary_factor 0.25, rope_theta 1e7, mtp 1 and vocab 248320 all match; the differences are pure scale and all config-driven. WHAT THIS DELIVERS: both architecture strings dispatch; the backbone weight namespace resolves for either spelling; mixed and empty indices are refused; an unsupported MoE expert layout is refused with a message naming the missing piece; the dense arm routes BF16/FP8/NVFP4 by tensor presence; the gated rows stay byte-identical. WHAT IT DOES NOT DELIVER, AND A CORRECTION. An earlier revision of this row asserted that the tensor names being "identical modulo the prefix" meant the loader body was already correct. The names are identical; the conclusion did not follow. LoadQwen3_5Moe -> LoadMoeExpertsInto -> LoadNvfp4Raw is a QUANTIZED checkpoint loader: it requires per-expert .weight U8 + .weight_scale F8_E4M3 + .weight_scale_2. The published bf16 checkpoint has 3-D STACKED experts and ZERO scale tensors, so it dies at lm_head before reaching them. This was invisible because our gated 35B row loads the requantized nvidia/Qwen3.6-35B-A3B-NVFP4, not a published Qwen bf16 repo -- the MoE loader has never read one. That false claim had reached the spec, a shipped header, docs/STATUS.md, model-matrix.md and three commit messages. It is struck everywhere, the debt is recorded the right way round (the quantized arm is the one implemented; the bf16/3-D-stacked arm is OWED, porting-inventory.md deviation 17(e)), and the previously-false closing condition is corrected on all seven surfaces. The dense-vs-MoE asymmetry is stated wherever the limitation appears. AHEAD OF THE PIN, DELIBERATELY. The parity pin 555967922 has only the ForConditionalGeneration entries; the text-only arms arrived upstream in vllm#50210 / ad5d29db7, post-pin. A forward port of one upstream PR; it does not advance the pin. One tracked divergence: a mixed index is REFUSED here where upstream WeightsMapper normalizes. EVIDENCE. CPU: clean -Werror build, serial ctest 431/431, test_qwen3_8_text_only 7 cases / 747 assertions. GPU (dgx GB10 sm_121a) at the reviewed head 1cc951c: SACRED test_qwen36_paged_engine 315/315, test_qwen27_paged_engine 235/235, test_qwen3coder_paged_engine 138/138 -- real runs, zero skips, goldens byte-identical. test_qwen35_plain_weights was a SKIP (4 assertions, Qwen/Qwen3.5-4B not cached) and is not counted as coverage. Reviewed independently four times. Reviews caught, and this row fixes: three MoE prefix mutations that survived the full 396-test suite; a fixture that was not the published config; the inverted debt record; and a latent UBSan misaligned-load class (#627) that the new test exposed in the shared loader -- fixed by adopting the existing vt::LoadUnaligned seam rather than a private copy. OWED, RECORDED NOT WAIVED: the bf16 / 3-D-stacked MoE expert arm; a re-run of the SACRED gates against this final head (they were captured at 1cc951c and the merges since changed no row source -- an argument, not a run); and the 2.4T run gate, unmeetable at ~4.8 TB bf16 against 128 GB unified. Nothing here claims a generated token from that checkpoint. CI: the red windows-msvc lanes (#646) and sanitize-cpu address,undefined (ltx2_loader.cpp misaligned load, #627) are pre-existing on main and reproduce on six other open PRs; neither is attributable to this row. Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
…ge the manifest found missing (#634) (#738) `s2mel.pth` carries `net.cfm.estimator.wavenet.*` and the shipped config sets `s2mel.DiT.final_layer_type: wavenet`, so this stack is on the render path. It was absent from the port map until #735 read the checkpoint manifest, and the CFM/DiT suites were green over a network without it. Reading upstream also explains three other names #735 listed separately. `t_embedder2`, `conv1` and `conv2` are not loose ends: they are the rest of this same final layer, which owns a second timestep embedder at the wavenet width (`diffusion_transformer.py:145-153`, index-tts @4f8792ff). The spec's table now says so instead of listing them as four unrelated gaps. Ported from `indextts/s2mel/modules/wavenet.py` class WN. Three details the gate holds, each of which is a plausible wrong guess: - the convolutions are upstream `SConv1d`, whose default `pad_mode` is REFLECT. Zero padding is the natural assumption and it is wrong. - ONE conditioning projection is computed outside the loop and each layer reads its own `2 * hidden` window of it; recomputing per layer, or reading layer 0's window every time, both look reasonable and both change the output. - the mask multiplies the residual update and the final output but NOT the input to `in_layers`, so a masked run is not a shorter run. A golden with three trailing zero frames pins that asymmetry, and asserts the kept frames DIFFER from the unmasked run, since the mask feeds back through the residual. The weight-norm fold calls `vocoder1d::MaterializeWeightNorm`, the single home restored in #734, rather than growing a third copy. RED first: the body shipped as a `VT_CHECK(false, ...)` so the failure was the unimplemented stage rather than an ambiguous missing-header compile error, and `compile_err=0` was recorded next to it. GREEN: 3 cases, 133 assertions. Six mutations, six caught, all with compile_err=0, control restored and verified by md5: reflect padding to zero, the tanh and sigmoid halves swapped, the conditioning slice pinned to layer 0, the residual left unmasked, the skip half read from the residual half, and the output left unmasked. A seventh attempt was VOID twice before it compiled -- once because the mutation left `ReflectIndex` unused under -Werror, once because shell escaping corrupted the edit -- and both times the stale binary printed SUCCESS. test_cpu_x86_llamacpp_floor fails as NO_QUIET_WINDOW at load 113 from this session's own builds, not from this change. audit-live-rows stays red, tracked as #733. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code] Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
…idth coupling upstream hides (#634) (#739) Ports `diffusion_transformer.py:243-253` (index-tts @4f8792ff): the long skip, conv1, the second timestep embedder, the wavenet, the long residual, the final adaLN layer and conv2. With #738 this makes the DiT's whole post-transformer path complete, and it CLOSES four separate entries the manifest had listed as unrelated gaps: `skip_linear`, `t_embedder2`, `conv1` and `conv2` are all one final layer. It composes `wavenet::Forward`, `cfm::TimestepFeatures` and `adaln::FinalLayer` rather than reimplementing any of them. Building the oracle found a coupling that the shipped config HIDES. `final_layer` is constructed at the wavenet width but conditioned on `t1`, which the DiT embeds at its own hidden width, so the wavenet final-layer path only composes when the two are equal. Both are 512 upstream, so nothing there ever exercises it; setting them differently in the generator raised `mat1 and mat2 shapes cannot be multiplied (1x8 and 6x12)` from upstream's own module. We now refuse unequal widths with a message naming which two numbers disagree, rather than inheriting a silent coincidence, and a case asserts the refusal. The oracle constructs the real upstream DiT at reduced dims and drives its own module objects, so only the SEQUENCE is restated -- once, beside the upstream line numbers it copies. Gated 4 cases / 33 assertions, and the three behavioural cases are chosen to be discriminating rather than merely non-vacuous: perturbing ONLY `x` must move the output (without the concat the tail is a function of `x_res` alone and still produces a mel of the right shape), `t1` must move it with the raw timestep held fixed, and the raw timestep must move it through `t_embedder2`. Six mutations, six caught, all compile_err=0, control restored and verified by md5: the long skip dropping x, res_projection swapped for conv1, the wavenet conditioned on t1 instead of t2, the transpose into the wavenet made a no-op, SiLU dropped from t_embedder2, and the output emitted frame-major. Still unported on this stage: `layers.N.skip_in_linear`, the PER-LAYER U-Net skip across DiT depth (`uvit_skip_connection: true`), and the conditioning front end. No render yet. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code] Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
… upstream rather than read off the formula (#634) (#741) `uvit_skip_connection: true` in the shipped config makes the first half of the transformer push its output onto a stack and the second half pop one, and the checkpoint carries a `layers.N.skip_in_linear` per receiving layer. With #738 and #739 this completes the DiT path. The routing is pure index logic, and every plausible misreading still produces a model that runs: FIFO instead of LIFO, `>=` instead of `>`, pushing a layer's INPUT instead of its output. So it was not derived by reading `gpt_fast/model.py:154-190`. `scripts/gen-dit-skip-schedule.py` drives upstream's own Transformer with the layers instrumented and records which layer actually received which layer's output, at six depths. The gate compares against those recordings, which are quoted in the test with their provenance. At the shipped depth 13: layers 0-5 emit, 7-12 receive, layer 7 takes layer 5's output and layer 12 takes layer 0's, and layer 6 in the middle does neither. The recording also found an upstream ASYMMETRY worth keeping. Emitters test `i < n/2` and receivers `i > n/2`, so at EVEN depth there is one more emitter than receiver and the earliest skip is pushed and never popped -- at depth 12, layer 0's output is simply discarded. `Schedule::orphaned` reports it. A port that "fixed" this would be a different model, so it is preserved and asserted rather than corrected. The concatenation order is pinned by hand-computed cases rather than a golden: `cat([x, skip])` reads the two halves with distinguishable multipliers, and the reversed call is asserted to give a DIFFERENT answer, so the order is proven rather than merely exercised. RED first, all six cases throwing on an unimplemented body with compile_err=0. GREEN 6 cases / 52 assertions. Seven mutations, seven caught, all compile_err=0, tree verified byte-identical afterwards: `>=` for receivers in both the list build and the routing loop, `<=` for emitters, FIFO instead of LIFO, emitting before the layer instead of after, the skip half reading x's columns, and the orphan count suppressed. Two earlier attempts were reported VOID by the harness on a non-unique and an absent anchor rather than silently mutating a neighbour. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code] Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
…s with POSIX stat FOLLOWING_AGENTS_PROTOCOL `src/vllm/multimodal/video_engine.cpp` reached Windows with unguarded POSIX `stat` -- `:21` `#include <sys/stat.h>`, `:57-60` `IsDir`, `:62-65` `Exists` -- which `check-windows-portability.py:1675-1688` flags under `full_source_posix`, i.e. every scanned source and not only the platform-boundary set. It reddened `windows-msvc-cpu` and `windows-msvc-vulkan` on EVERY open pull request in the repository, across five unrelated lanes. It hid because those jobs are `skipped` on `main` pushes (#584), so a green main carried no information. Repaired at the SOURCE, never the checker: `IsDir`/`Exists` now take the `std::error_code` <filesystem> overloads through a file-local path helper, preserving `::stat`'s return-false-for-an-uninspectable-path behaviour that the THROWING overloads would have turned into a `filesystem_error` escaping a registry query. The helper adopts `fs_io.cpp:31`'s spelling byte-for-byte and deliberately NOT `minimax_h3_sharded.cpp:55`'s, which throws on malformed UTF-8 under _WIN32 -- that would reintroduce on Windows the exception this removes on POSIX. The wider divergence is #687. RED first, and the honest RED is worth recording: a missing-path test would have passed with and without the guarantee, because ENOENT does not throw. The real probe is a 300-character path component (ENAMETOOLONG). Reviewed PASS by a fresh reviewer who reproduced the premise from another lane's CI log, confirmed by sha256 that the checker was untouched, and REJECTED the implementer's probe for the one surviving mutation -- "fails to refute is not unreachability" -- then established unreachability from the library mechanism instead: `IsDir` is TU-local with one call site gated by `Exists`, and libstdc++ implements both via `f(status(p))`, so `exists()` can only return true when the stat already succeeded. Operator gate at the merged tree: WINPORT_EXIT=0, BUILD_EXIT=0, 0 errors, 0 ENOSPC, REGISTERED=444, test_video_engine 12 cases / 260 assertions exit 0. Two ctest failures both attributed by measurement, not assumption: `test_op_parity` is #737 (a null in the Music3 golden added today -- proven by moving that one golden aside: 10/10 pass, assertions 70 -> 123), and `test_openai_conformance` is a load artifact at load average 115 that passes alone, exit 0, 48 assertions. This does NOT turn the Windows lane green on its own. Two independent defects were stacked; with this one gone the build reaches `M_PI`, which MSVC does not define -- #720, fixed by #732. Closes #664. Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5[1m] [claude-code]
…s::pi_v<double> FOLLOWING_AGENTS_PROTOCOL The SECOND of two stacked Windows defects, both from the LTX-2.5 lane. #664 (landed as f8cbc23) removed the POSIX `stat` one; with that gone the MSVC build reaches `M_PI`, a POSIX extension that MSVC's <cmath> defines only behind `_USE_MATH_DEFINES`: ltx2.cpp(72,58) / (557,69) / (572,76): error C2065: 'M_PI': undeclared identifier ltx2.cpp(72,15): error C2737: 'kBeta': const object must be initialized Those four are the ENTIRE `error C` set in both job logs. The issue's own scope was wrong in both directions, and the implementer corrected it with a positive control in the same grep. `git grep -- src include` never looked in tests/, so it MISSED `tests/vllm/models/test_vocoder1d.cpp`, which also uses M_PI unguarded; and it counted `ltx2_audio_vae.cpp` and `ltx2_video_vae.cpp`, which compile fine behind their own `#ifndef M_PI` (the latter's define being dead -- zero uses). All four sites now use `std::numbers::pi_v<double>` and the two hand-rolled defines are gone; extra spellings of one constant are #687's disease. The gate for this ALREADY EXISTED and was ALREADY RED at base, naming all four files: `test_check_windows_portability.py` asserts no `\bM_PI\b` under src/ or tests/. It never fired because that suite runs in NO workflow (#680, #646). So LTX-2.5 landed M_PI on top of a gate that was already objecting and nothing was listening -- which, with #584, is why two Windows defects stacked unseen. MSVC's header condition was reproduced on Linux with `-U_GNU_SOURCE -D_ISOC99_SOURCE`, reporting ltx2.cpp at the same LINE AND COLUMN as the real CI log, and nothing for the two guarded VAEs -- confirming their guards by measurement. Stated precisely as a g++/glibc proxy, not MSVC. Independent corroboration on the target toolchain: MSVC compiled `vocoder1d.cpp`, `minimax_h3_video_vae.cpp` and `mla_attention.cpp` cleanly in the same log, and all three already use `std::numbers::pi_v<double>`. Value identity established three ways: bit pattern 400921fb54442d18 for all three spellings and for the derived sqrt(2/pi) and pi/2 in float and double; token-sequence equality over 41,191 tokens; and 7 golden suites, 185 cases / 17,378 assertions, identical in both arms. Reviewed PASS by a fresh reviewer whose independently re-derived TU list matched, who verified the line-and-column match against the job log itself, and who caught a real overstatement: `test_vocoder1d.cpp` is NOT a lane blocker, because `build-windows-release.ps1:263-273` builds a fixed focused target list that never reaches it. Operator gate: M_PI hits 0 tree-wide (positive control: 8 std::numbers files), BUILD_EXIT=0, 0 errors, 0 ENOSPC, REGISTERED=443, windows-portability suite 71 passed / 1 failed -- the survivor being #680's, red at base too. The one ctest failure is #737, main's own Music3 golden red. Honest limit: after this and #664 the lane has never gotten past the LIBRARY compile, so link, the eight focused test executables, packaging and the smoke run remain unproven on Windows. This removes the entire remaining `error C` set visible in both job logs; it does not prove Windows is green. Closes #720. Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5[1m] [claude-code]
… to read pickle in the engine (#634) (#742) Upstream ships `.pth`: a ZIP around a Python pickle. Loading it needs a reader this tree does not have, and the answer is not to write one. Pickle executes arbitrary code by construction, so a reader in the engine would run an attacker-controllable program inside the process that serves users, and every other lane here already loads safetensors or GGUF. The conversion therefore happens offline, once, and the engine loads the result through the reader it already has. `scripts/convert-indextts2-checkpoint.py` flattens the nested state dicts with '.', which is exactly the naming `indextts2_pth_manifest.json` records, so the converted names ARE the manifest's names and the manifest can check the conversion. Run against the real checkpoint on the NAS, all three files matched their recorded tensor counts: 456, 972, 284. It also measured something the manifest could only hint at. `codec.pth` is **75% optimizer state** -- 729 of its 972 tensors are training residue -- and dropping it takes that file from 579.16 MiB to 192.99 MiB. The drop is reported with a count, never silent, and the prefix is gated from BOTH sides: every optimizer key in the manifest must match it, and no weight in `gpt.pth` or `s2mel.pth` may. A prefix widened from "optimizer." to "opt" would quietly eat model weights, which is exactly the mutation that proves the second check. The conversion needs torch and 4 GiB of weights, so CI cannot run it. `tests/scripts/test_indextts2_convert.py` holds the part where a silent mistake is unrecoverable -- which tensors survive, under which names -- using fakes, with neither torch nor the checkpoint, because a dropped weight looks exactly like a weight that was never there. It also asserts the flattener survives the ints, strings and Nones that sit beside weights in a `.pth`. 8 cases. Five mutations, five caught, script verified byte-identical after: the drop prefix widened to "opt", the drop prefix emptied, the flattener joining with '/' instead of '.', the flattener crashing on non-tensor leaves, and one source silently omitted. This unblocks the checkpoint loader; it is not the loader, and there is still no render. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code] Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
…er we mostly own on hardware we do not (#699) (#705) `dots-studio/dots3-note-prev` is a 280B-total / 16B-active multimodal MoE: text + image + video + audio understanding, 512K context, an MTP head. vLLM added it on `main` in vllm#51255 (`vllm/models/dots3_note/`, 15 files, ~5.7k LoC) and was still patching it the day before this scope (vllm#52172, 2026-08-13). W0 ONLY. Spec committed before implementation, per AGENTS.md. No src/, include/ or tests/ change is in scope; W1 onward are dispatched to fresh implementers from this spec rather than written in the coordinating session. THE PORT IS CHEAPER THAN THE PARAMETER COUNT SUGGESTS, because upstream builds the text tower by subclassing DeepSeek: `Dots3NoteModel(DeepseekV32Model)`, `Dots3NoteDecoderLayer(DeepseekV32DecoderLayer)`, `Dots3NoteMoE(DeepseekV2MoE)`, `Dots3NoteFullAttention(DeepseekV2MLAAttention)`. Our gated MLA (deepseek_v2.cpp, SACRED 8/8), the DSA lightning indexer (deepseek_v4_dsa.cpp), the noaux_tc sigmoid router, the 256-expert grouped MoE, qwen3_vl_vision.cpp (image+video STRICT 32/32) and voxtral.cpp (audio 16/16) already carry most of it. WHAT IS ACTUALLY NEW, and the first item is the campaign: * Sliding-window MLA. 33 of 46 layers are SWA (window 513) over MLA with a SECOND geometry: 64 heads, swa_kv_lora_rank 1024, swa_qk_nope_head_dim 192, own rope_theta 5e4, is_neox_style=False. Upstream needed two Triton kernels plus its own MLA metadata builder. Every MLA path we have is full attention over a paged cache (deepseek_v4_registry.cpp sets is_hybrid=false). * A heterogeneous MLA KV spec: latent rows 576 (full) vs 1088 (SWA), which upstream reconciles by padding the physical row. * The headwise sigmoid attention gate, an extra RMSNorm over the 64-dim rope-only k slice, and the two apply_mla_qkv_lora_rescale scalars. * A pyramid MoE ViT (all our vision towers are dense) and the `dots` audio stem (conv2d stem, RoPE, RMSNorm, SwiGLU, 6000 positions, 60s chunking). * The GGUF k-quant arm, which is owed and has no prior art: llama.cpp has no dots3_note, so the converter is ours and there is no quant-matched llama.cpp bar. FOUR CONFIG TRAPS THE CHECKPOINT DOES NOT CARRY, all set in Dots3NoteConfig.__init__ and all numerically silent. n_group=1 / topk_group=1, where upstream's own comment says inheriting DeepSeek-V3's 8/4 grouping "changes the selected experts at every MoE layer" and our router is gated at V3's grouped dims; indexer_rope_interleave=True (GPT-J adjacent pairs) where V3.2 and therefore our indexer default to split-half NeoX; num_nextn_predict_layers=1. A port reading only config.json gets all of them wrong and still emits plausible text. TWO BLOCKERS, RECORDED RATHER THAN WORKED AROUND. Beyond-pin: our parity pin is 555967922 (0.26.0.dev0) and the checkout at that SHA carries only dots_ocr.py, verified. Gating needs a pin advance, against an implementation upstream is still changing. Oracle-memory-infeasible, and this is the decisive one. Measured via the HF API: 131 language shards + model-vision.safetensors 13.7 GB + model-audio.safetensors 1.77 GB = ~576 GB bf16; the fp8 sibling ~290 GB; upstream's recipe is 8xH100. GB10 is ~119 GiB and Thor ~122 GiB RAM with 123 GiB free disk, so the vLLM oracle cannot run this model here at any published precision, and no smaller dots3-note exists in the dots-studio org. Same wall deepseek-v4-flash hit at 156.7 GiB, three times worse. Our own arm is plausible at <=2 bpw, but an arm with no oracle beside it yields no parity number, so every performance axis is an open gap by construction. THOR IS DESIGNATED, AND ITS LIMITS ARE MEASURED. Developer direction names 192.168.68.23 as the e2e CUDA host. Probed read-only 2026-08-14: 14 cores, 122 GB RAM, 123 GiB free on /home, but no nvcc, no cmake, no ninja, no venv, and nvidia-smi refuses under non-interactive ssh. So W0.5 provisions it, and the spec states plainly that designating a host does not change the arithmetic above: Thor can carry our low-bit arm, the sm_110 axis and every unit gate, not a 290 GB oracle. The spec ends on a three-way decision that is the developer's, not the implementer's: rent 8xH100, accept unit-gated bricks with the e2e gate recorded as owed, or park at W0. No ceiling is declared and the gap stays open. Records: MODEL ratchet 375 -> 377 in check-agent-record.py, bumped because two rows EXIST (Dots3NoteForCausalLM at SPIKE, Dots3NoteMTPModel at INVENTORIED), never to make a transition pass; the beyond-pin ratchet 371 -> 373 and the rollup 375 -> 377 in model-matrix.md; the #699 row in roadmap_v1.md; a FEATURES.md "inventoried but blocked" line. AND THE RATCHET COLLISION FIRED, which is worth recording because the record predicted it. This branch first took 373 -> 375; while it was in review the #490 branch took 373 -> 375 as well, for the two Qwen3.5 text-only arms. BOTH sides read 375, neither was right, and an auto-merge keeping either would have left model-matrix.md internally consistent while silently short two real architectures -- exactly the hazard test_music3_and_indextts_rows_both_survive_their_collision was written for. The merged tree holds four new rows, so the number is RE-DERIVED to 377 by counting the matrix after the merge. Both conflicted keyed records (model-matrix.md, check-agent-record.py) were reset to origin/main byte-for-byte and this row's scoped edits reapplied on top, per the keyed-record rule; #490's two rows are verified present and the only non-dots3 lines that differ from main are the eleven count lines this change owes. Gates: check-agent-record OK (MODEL=375), check-model-checklist OK, check-public-doc-tables OK, test_agent_record 55/55, doc-checkpoint --staged OK, now-current --staged OK. `agent-preflight.sh --staged` is green except test_cpu_x86_llamacpp_floor, which is #618's load-dependent NO_QUIET_WINDOW (exit 4 instead of 2) and REPRODUCES ON A CLEAN TREE AT THE BASE SHA a3aa02e at loadavg ~50-70. Not caused by this change, which touches no code that test exercises. CHECKER CHANGE, WITH THE MUTATION EVIDENCE THE pr-size GATE DEMANDED. The MODEL ratchet bump is a checker-semantics change, so it now carries test_dots3_rows_are_inside_the_model_ratchet. Measured, not asserted: RENAMING the MTP row leaves the count at 375, touches no claim, keeps every other check green, and REDs only the new assertion -- which is exactly the hazard a count cannot see. Two further mutations are recorded in the test's docstring as WEAKER evidence rather than presented as its force: INVENTORIED -> SPIKE is already caught by the claim-ownership rule, and SPIKE -> ACTIVE by the structured-spec rule. Every mutation was reverted and model-matrix.md restored byte-for-byte (md5 5c47ef2c045dd64c3b7b32a2aebf01a1 before and after). That last mutation also surfaced a constraint worth recording rather than rediscovering: an ACTIVE row owes this spec seven structured headings (Scope, Upstream chain, Our baseline, Port map, Tests to port, Gates, Dependencies) that a SPIKE row does not. Spec section 8.1 now says so, so the W1 lifecycle move budgets for it. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
…ghts (#634) (#743) First time any IndexTTS stage meets its actual checkpoint. `indextts2_s2mel_loader` binds the converted `s2mel.safetensors` to the ported `dit_tail`, and the shipped 512-wide tower runs: 80 x 8 mel, values in [-1.14, -0.31]. Dimensions are resolved FROM THE WEIGHTS, never from a config: hidden 512 and in_channels 80 come out of `skip_linear.weight` being [512, 592], the wavenet's 8 layers are counted, kernel 5 is read off `in_layers.0.weight_v`, and `t_embedder2.mlp.0.weight` being [512, 256] is what says the sinusoidal width is 256. A config disagreeing with the weights would be the config that is wrong. Three things are recorded rather than assumed: the tower is F32 throughout (a rare case where upstream really does store f32), `weight_g` ships as [out, 1, 1] so flattening is asserted rather than presumed, and `dilation_rate` is NOT in the checkpoint at all -- upstream ships 1, the shapes cannot distinguish it, and the code says so instead of implying the checkpoint proved it. MUTATION TESTING FOUND TWO REAL HOLES IN THIS GATE, which is the point of it. Deleting the dtype check survived, and so did making a missing tensor return an empty vector instead of throwing. Both passed because the fixtures never exercised those paths: the "missing tensor" case happened to fail inside the SHAPE lookup, never the data read, and no case ever fed a non-F32 tensor. Closing the second was itself two mistakes deep. An I64 tensor throws on BYTE LENGTH rather than dtype, so it kills the mutation for the wrong reason; and an otherwise-incomplete fixture throws on the missing tensors instead. The case now uses a COMPLETE checkpoint with exactly one I32 tensor -- four bytes, exactly F32's width -- so only the dtype check can catch it. All six mutations caught, file verified byte-identical. One more instrument bug worth recording. The real-weights case first reported mel values of 1e16 and I nearly filed it against the port. The port was fine: `0.01F * ((i % 37) - 18)` with a size_t `i` is UNSIGNED arithmetic, so every value below 18 wrapped to ~1.8e19 and the tower was fed 9e16. A stage-isolating harness showed `x_res` was already enormous before anything ran, and the wavenet was returning a perfectly sane 0.085. The inputs are now cast and, more to the point, ASSERTED before anything downstream is believed. Gated 8 cases: 36 assertions synthetic, 4782 with the real checkpoint attached via VLLM_CPP_INDEXTTS2_S2MEL. Without that variable the real-checkpoint cases skip LOUDLY, so "never checked" cannot read as "passed". This is not a render. The mel here is driven by synthetic transformer output, because the DiT blocks and the conditioning front end are still unported, and correctness against vLLM-Omni still needs the oracle pin (#633). FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code] Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
…r that is DEAD in 2.5 (#634) (#745) Ports `diffusion_transformer.py:206-226` (index-tts @4f8792ff): cond_projection, the four-way concatenation, the style broadcast, the classifier-free-guidance zeroing, and cond_x_merge_linear. The shipped `cond_x_merge_linear.weight` is [512, 864], and 864 is exactly 512 + 80 * 2 + 192, so the concatenation order and widths are pinned by the weight itself rather than by my reading of the code. Two things contradict what a reader expects, and both are now recorded where someone will hit them. `cond_in_module` is FORCED to `cond_projection` upstream: the `content_type` switch that would have selected `cond_embedder` is commented out. So `cond_embedder` is present in `s2mel.pth` and DEAD in 2.5. A port that "restored" the switch, which is exactly what reading the surrounding code suggests, would consult a tensor this model never uses. `class_dropout` is NOT a training-only path. `mask_content` sets it at inference, and it zeroes everything AFTER the first `in_channels` columns -- keeping x, dropping prompt, cond and style. That IS the CFG unconditional branch that `cfm::EulerStepCfg` consumes, so it is ported as a first-class argument rather than an artefact. Gated against upstream on BOTH branches, and the behavioural cases are chosen to separate near-misses rather than merely to be non-vacuous: under the unconditional branch, perturbing x must move the output while perturbing prompt, cond or style must NOT; under the conditional branch all four must move it; and style is proven BROADCAST rather than read per frame by shifting the whole vector and requiring every frame to move by the same delta. 5 cases / 128 assertions. Six mutations, six caught, all compile_err=0, file verified byte-identical: raw cond used instead of the projection, the concat order swapped, x read frame-major instead of channel-major, the unconditional branch zeroing x as well, the unconditional branch keeping style, and style indexed by frame instead of broadcast. Still no render: the DiT BLOCKS between this front end and the ported tail are not wired, and correctness against vLLM-Omni needs the oracle pin (#633). FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code] Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
…imator is COMPLETE front to tail (#634) (#748) Ports `gpt_fast/model.py:161-191` (Transformer.forward): N blocks, the U-Net skip routing across them, and the final AdaptiveLayerNorm. With #738, #739, #741 and #745 the S2Mel DiT now exists end to end -- front end, stack, tail -- each gated against upstream's own modules. This is composition only. The block is `dit::Block`, the routing is `dit_skip::Plan`, the merge is `dit_skip::ApplySkip`, and nothing here reimplements any of them. The rotary table is passed IN, emitted by the generator from upstream's own `freqs_cis`, so the gate measures composition rather than a second copy of `precompute_freqs_cis`. One upstream quirk is recorded rather than tidied: a `skip_in_linear` is built on EVERY layer when `uvit_skip_connection` is set, including layers that never receive a skip, so the checkpoint carries unused ones. They load and are left alone. A case asserts that layer 0's skip_in_linear cannot affect the output at all, while a receiving layer's must. 3 cases / 52 assertions against upstream Transformer.forward. Five mutations, five caught, all compile_err=0, file byte-identical after: the skip merged against itself instead of the popped value, emitting a layer's INPUT instead of its output, FIFO instead of LIFO, the final norm dropped, and the receive moved after the block. A SIXTH mutation was withdrawn as badly designed rather than reported as a gate hole. Pushing before the block WITHOUT removing the push after it leaves the stack holding both, and because a layer's input IS the previous layer's output, every pop still returned the same tensor -- a genuine no-op, not a defect the gate missed. Rewritten to emit the input INSTEAD of the output, it fails. Still no render: the talker, the reference encoders and BigVGAN are unported, so nothing yet produces the conditioning this estimator consumes, and correctness against vLLM-Omni needs the oracle pin (#633). FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code] Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Merges the PINNED SHA `32d82c64d`, not the ref. `origin/main` is shared with other worktrees in this checkout, and it advanced from `5e646d98c` to `32d82c64d` mid-merge when another session fetched — which made main's own newer files (the `dit_front` / `dit_stack` IndexTTS2 cluster, #745/#748) read as "silently dropped by the merge" in every `git diff origin/main` afterwards. They were never dropped; they are simply not in `5e646d98c`. That first attempt was aborted rather than patched, because the repair it invited — restoring files from the newer main into a tree merged from the older one — produced a CMakeLists listing TUs the tree did not contain. Merging an immutable SHA is what makes the verification below mean anything. Three conflicts, all resolved against that SHA: * `tests/vllm/multimodal/test_ltx2_video.cpp` — both sides added a different include in the same sorted run: this branch's `ltx2_video_vae_encoder.h` and main's `vllm/platforms/interface.h`. Both are kept, in sorted order. Proved by COMPILING, not by reading: `BUILD_EXIT=0` and the suite runs. * `.agents/roadmap_v1.md` and `docs/FEATURES.md` — keyed records, resolved by taking main's file WHOLESALE and reapplying this row's key onto it, never by accepting the automatic three-way result. Each file is now byte-identical to `32d82c64d` apart from exactly one line: the `#644` row and the LTX-2.5 DiT row. An earlier merge in this campaign auto-merged two `#644` rows into a duplicate that `check-agent-record` caught; taking one side wholesale is what makes that shape impossible rather than merely unlikely. The combined `#644` row names both campaign rows and both specs. The FEATURES row carries main's prompt-AdaLN facts plus this row's served image-conditioning arm, inside the 220-char cell and 600-char entry budgets. `docs/USAGE.md`, `CMakeLists.txt` and `tests/CMakeLists.txt` auto-merged, and were each verified against the pinned SHA rather than trusted: both CMake files are PURELY ADDITIVE (zero removed lines; this branch adds the two `ltx2_video_vae_encoder_load` / `ltx2_image_preprocess` TUs and the one `test_ltx2_image_cond` registration), and USAGE's only removal is the paragraph this row supersedes, with main's own additions still present. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
Collaborator
Author
Merged current
|
localai-bot
added a commit
that referenced
this pull request
Aug 14, 2026
…, and the FALSE refusal reason it exposed (#644) FOLLOWING_AGENTS_PROTOCOL Image conditioning for LTX-2.5 at crf=0, the VAE encoder load path that did not exist, and the repair of that feature's own review findings. THE FINDING WORTH READING: the keyframe refusal named a FALSE reason, and a test had pinned it. It blamed `keyframes_abs_pos_embedding`. Refuted at Lightricks/LTX-2 @ fd4ded7f, and the decisive link was one the original refutation had not cited -- `conditioning/mask_utils.py:101`, `new = torch.ones if marked else torch.zeros`, with `:97-98` returning None for an unmarked append onto a None mask. The sole consumer `transformer_args.py:42-43` is `hidden + mask * embedding`, called ONCE at `:269`, so the appended keyframe contributes exactly nothing. Meanwhile `tools.py:184` + `_first_frame_keyframes_mask` mark the TARGET's first latent frame unconditionally -- the tokens the served arm writes into. The contrast was backwards. The new message names the real reason, labels the old one "WHAT IS *NOT* THE REASON" with #658, and switches local anchors from same-file line numbers to SYMBOLS, which is how the previous citation went stale. The second finding was a gate that caught nothing: mutating to `if (wants_image && phase_index == 0)` left the suite byte-for-byte at 32/550 exit 0. The replacement pins the last phase's per-latent-frame count against a `max_phase = 0` engine -- two MEASURED values from two different engines, not two literals. Re-mutated at review: 32/33 cases, 570/576 assertions, exit 1. THIS IS A RECONSTRUCTION, and why matters. The predecessor PR's base branch was the row's own branch rather than main -- an operator error, ancestry read instead of baseRefName -- so merging it squashed onto that branch and FLATTENED AWAY the `merge: origin/main` commit beneath it. Measured consequence: the merge base fell 67 commits back, the row's apparent diff became 252 files / 42,426 insertions, and merging main produced 19 conflicts including add/add on files that are purely main's. The row was rebuilt from a pinned main SHA instead. So the review had to prove BOTH directions, and did: - nothing of main's in: the complete 19-file list, no path filter, with a positive control (ltx2 = 13) and a negative one; gpt2/talker/w2vbert/ music3_oracle/bigvgan/website/indextts/lenreg all ZERO - nothing of the row's out: the substantive +/- payload of the row against the main it was last gated on is BYTE-IDENTICAL to the reconstruction -- 4471 lines each, sha256 8e6ba77f… both, differing only in 16 hunk headers Counts attributed three ways: `test_ltx2_video` 33/576, where main's fc903b8 (#674) contributed +1 case / +11 assertions and the row +2 cases; fc903b8's own commit body records 31/513, and 502+63 = the pre-reconstruction 565. Operator gate at the exact pushed SHA, main pinned to an immutable SHA, with the leak check INSIDE the gate so it cannot pass silently: 19 row files, 0 main-only names, positive control 13. BUILD_EXIT=0, 0 errors, 0 ENOSPC, REGISTERED=463, ctest 463/463 CTEST_EXIT=0, `test_ltx2_video` 33/576 and `test_ltx2_image_cond` 15/223, both exit 0. Non-blocking follow-up filed as #794: six mis-cited upstream anchors, two pointing at blank lines. Every substantive claim is true at the pin; only the line numbers are wrong. #769's duplicate FEATURES key is carried at multiplicity 2 -- neither introduced nor multiplied. Supersedes #657 and #666. Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5[1m] [claude-code]
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Repairs the two blocking findings from the FAIL review of #657, plus its four
minor ones. Issue: #644 (row 1,
LTX25-IMAGE-COND). Spec:.agents/specs/ltx25-image-conditioning.md.Review the fix commit
eb58fd58fon its own.456e32c6cis only themerge of
origin/mainforward (row 0's prompt-side AdaLN, #654) with two keyedrecords resolved by hand.
F1 (blocking) — the refusal named a reason that is FALSE at the pin, and a test pinned it
The last-frame keyframe refusal blamed
keyframes_abs_pos_embedding. Atfd4ded7fthat is not what blocks it:keyframe_cond.py:84-86appends a supplied keyframe withmarked=False,commented "given keyframe content is ordinary image guidance, not a generated
keyframe slot, so it carries no keyframe marker".
transformer_args.py:42-43(called once at:269), ismask = (keyframes_mask > 0); return hidden_states + mask * embedding. Withmarked=Falseit adds nothing — porting the module would not serve the arm.tools.py:184+_first_frame_keyframes_mask(
:186-195) mark the target's first latent frame unconditionally, so thetokens that actually reach the embedding are exactly the ones the served
first-frame arm writes into — which the message called "needs none of that".
The named gate does not fire either:
Ltx2AdoptDeclaredDitParamsclears the flagon every real load under
allow_unported_modules, because the shipped DiT doescarry the tensor.
And
test_ltx2_video.cpphad been written to require the wrong reason byname, so the gate enforced the defect.
The message now names the token-append machinery — extended
positions(
keyframe_cond.py:46-59),update_attention_mask(:68-76), extendedclean_latent/denoise_mask(:79-82),clear_conditioningtrimming back(
tools.py:88-105) — against a phase loop fixed at the target grid's tokencount, and says why the first-frame arm needs none of it. The assertion requires
that, and permits the refuted reason only where it is labelled as ruled out and
paired with #658. Local anchors are symbols rather than same-file line numbers,
which is how the previous citation went stale.
#658 is not fixed here and nothing here contradicts it — the message and the
spec both name it as the place the embedding really does bite.
F2 (blocking) — an unrecorded survivor on the design's own load-bearing claim
Spec §8.5 names "conditioning stage 1 only would let stage 2 re-noise the pinned
frame away" as the reason for the design, and gated it with nothing.
Reproduced.
if (wants_image && phase_index == 0)left the suite at32 cases / 550 assertions / exit 0 —
image_digest/image_tokensareoverwritten each phase and the only assertion was
image_tokens > 0. That is theshape of the obvious optimization (hoist the per-phase decode+encode out of the
loop), so it would have landed green.
The trace now pins the last phase's per-latent-frame token count, and a
max_phase = 0engine over the same request pins stage 1's, with the 4x relationchecked between two measured values rather than two compile-time constants.
CHECK( 1 == 4 )twice + the digest equality; 6 failed, exit 1Minor findings
F3 — nine mis-cited anchors, each re-verified against the pin before
changing.
blocks.py:576-580is the teardown, so the conditioning-orderclaim now cites
create_noised_state(helpers.py:428-445) and thevideo-before-audio claim cites
blocks.py:554-563. Also corrected:ImageConditionerbuild:985-986/ build-and-free:988-991;helpers.py:274-275are parameter declarations, with per-stage h/w atdistilled.py:251, :255-256, :285-286;model_configurator.py:41-43→:41-42;decoder padding default
:90→:92;reference_video_cond.py:74, 80→:36-37applied at
:65-77;reference_audio_cond.py:33→:34;detect_params:166-177→:166-179.F4 — the "rule for rule" claim is replaced by what is actually true.
Upstream's
SDOps(loader/sd_ops.py:101-122) is anany()admission over fourmatchings plus a chained substring replace over three replacements; this port is
one first-match prefix pass, so its fourth rule is an identity carrying
upstream's fourth matching. Behaviourally equivalent on every shipped key, and
the comment names the case where the two shapes would part.
F5 —
num_layersdefaulted to 1 while upstream subscriptsblock_config["num_layers"](video_vae.py:55) and raisesKeyError. Ares_xblock without it is now refused; no other block kind reads the key, so requiring
it everywhere would refuse configs upstream loads, and both halves are gated.
RED with the guard removed: "did NOT throw at all!", 1 failed, exit 1. GREEN:
15 cases / 223 assertions, exit 0.
F6 — the three compile-time-constant assertions are left as they are,
correctly labelled documentation. Noted so the count is read honestly; the new F2
assertions deliberately compare two measured values instead.
Docs
docs/USAGE.mdcarried the same false keyframe reason — corrected, dated, with#658 named so a reader can check it — and gains the
res_x/num_layersrequirement.
docs/FEATURES.mdkeeps its keyed row inside the cell and entrybudgets.
Gate
CPU-only, Release,
VLLM_CPP_CUDA=OFF, at the pushed tree.LTX2_CHECKPOINT_ROOTis unset on this box, sotest_ltx2_video'sshipped-checkpoint subcase SKIPPED and is not part of those counts.
🤖 Generated with Claude Code