MODEL-NEMOTRON-H W3 land: the whole scaffold as ONE commit, with the docs/FEATURES.md clause the per-commit rule requires (#517) - #576
Merged
Conversation
…enumerated, KV-shaped, and its loader honors `quantized` (#517) FOLLOWING_AGENTS_PROTOCOL W3 of .agents/specs/nemotron-h-model.md, landed as ONE commit. It makes the architecture known; it runs nothing. The forward REFUSES BY NAME (W4, itself blocked on the Mamba2 SSD CUDA arm #496 W2) and the GGUF arm REFUSES BY NAME (W7, owed per §5b). WHY ONE COMMIT. `row/MODEL-NEMOTRON-H-W3-FIX` @ `cb37239d4` (PR #572) was green on every gate except `doc-checkpoint range`, which fails PER-COMMIT on `3981de6a4`: `check-doc-checkpoint.py:79` classifies any `src/vllm/model_executor/models/` path as `feature_surface`, `:311-316` requires `docs/FEATURES.md` in the SAME commit, and `commits_in_range` (`:369-375`) walks `rev-list --reverse --no-merges origin/main..HEAD`, so W3C's own `docs/FEATURES.md` edit in `3295d0c1a` cannot cover a later commit. `ci.yml:343` runs the identical invocation, so CI was red too. The checker has no exemption mechanism, deliberately, and weakening it was never on the table. `main` is never force-pushed, so `3981de6a4` could not be amended; the history is rebuilt here as one squashed commit off current `origin/main` with the `docs/FEATURES.md` clause included -- what the squash-merge would have produced anyway. Same precedent as W3B -> W3C on this row. TREE EQUIVALENCE, PROVEN. `git diff cb37239` against this tree lists six files: the four that `72e661ae4` touches (the one `origin/main` commit the fix branch lacked, #442 DSPARK), plus `docs/FEATURES.md` (one line) and `.agents/specs/nemotron-h-model.md` (§5e, below). Every `src/`, `tests/`, `CMakeLists.txt` and `docs/USAGE.md` path is byte-identical to `cb37239d4`. ENUMERATION, the hard number: 18487 of 18487 released tensors CLAIMED by a named consumer. 0 unaccounted, 0 invented, 0 refused. Gated both directions against the committed headers-only projection of the released index, plus a live case that re-verifies that projection against the checkpoint itself when VT_NEMOTRON35_SNAPSHOT names it. Four things the plan had wrong or unstated, settled against source and disk: 1. layers_block_type is the source of truth -- but vLLM VENDORS a NemotronHConfig with the OPPOSITE polarity (hybrid_override_pattern as the ctor arg, layers_block_type derived). That class is imported for TYPE ANNOTATION only; the runtime object comes from transformers AutoConfig, where num_hidden_layers is a property over layers_block_type whose setter discards the checkpoint's value. Both spellings parse; for the SCHEDULE pair the modern one wins (see finding 2 below -- the `mamba_*` scalars are the OPPOSITE). 2. moe_latent_size does NOT need three states: upstream's predicate is `getattr(config, "moe_latent_size", None) is not None`, so ABSENT and `null` are the same state. std::optional covers both; a real value refuses (§0). 3. The shared detail::ResolveMambaSsmCacheDType is the WRONG reader here and silently HALVES the recurrent state. It is keyed on `mamba_ssm_dtype` (Qwen3.5's spelling); NemotronH ships `mamba_ssm_cache_dtype`, so it returned the CONVOLUTION dtype. Caught by the KV gate as page_size_bytes() 1085440 vs an expected 2134016 -- the SSM state at bf16 instead of f32. Not a Qwen bug; two families, two config keys. Resolved locally, reason recorded at the call site. 4. Upstream's default conv layout is "SD" = (state_len, dim) while ours is (dim, state_len). Same bytes; the local convention is kept so the shared runner sees one orientation, and the divergence is commented rather than left for W4 to rediscover. KV: two groups carrying REAL per-layer names -- 6 attention layers on FullAttentionSpec(2 kv heads, head_size 128), 23 mamba layers on MambaSpec {{6144,3},{64,64,128}} / {bf16,f32}. conv_dim == 6144 is falsified straight off disk by conv1d.weight BF16 [6144,1,4]. The names are load-bearing: kv_cache_utils.cpp:979 and kv_cache_interface.cpp:151-158 both multiply a group's page by layer_names.size(), so a one-element tag under-counts by 23x and 6x. SCOPE: no per-module quant algorithm is resolved -- that is W1, already on main. W3 reads four coarse falsifiable keys (quant_method, quant_algo, kv_cache_scheme, the `mtp*` entry in `ignore`) and derives scale companions STRUCTURALLY; the 18487-tensor gate is what proves the derivation. A non-ModelOpt producer refuses by name. One piece of debt is recorded in the code: a dense `mlp` block's quantized companion layout is DERIVED, because no in-scope checkpoint ships one. nemotron_h.h lives under src/, not include/vllm/: W3 ships nothing on the public ABI and include/vllm/ is a USER_USAGE_PREFIXES surface (#515). THE FOUR REVIEW FINDINGS folded in (they were `3981de6a4`, the commit the per-commit rule fired on): 1. `ClaimMamba` hard-coded its FP8 companions. Every other claimer (`ClaimNvfp4`, `ClaimMoe`, `ClaimMlp`, `ClaimAttention`'s `fp8_kv`) gates on `quantized`; `ClaimFp8` did not. The released config MINUS `quantization_config` -- the shape a released bf16 NemotronH safetensors checkpoint actually ships, which spec 5b already owes -- parsed without refusal and enumerated 92 tensors that checkpoint does not have (23 mamba blocks x 2 projections x 2 companions), first `backbone.layers.0.mixer.in_proj.weight_scale`. Nothing consumes the map yet, so no gated claim was wrong -- but an unimplemented arm is refused BY NAME, never silently mis-enumerated. `quantized` is now threaded through `ClaimFp8`/`ClaimMamba` and both call sites (backbone `quantized`, MTP `mtp_quantized`). The released checkpoint is quantized and its MTP schedule is {attention, moe}, so the 18487-tensor gate is unchanged. 2. `Get{Int,Double,Bool}Aliased` preferred the MODERN key. Upstream does the OPPOSITE for the `mamba_*` scalars: `configuration_nemotron_h.py:145-155` is `self.n_groups = kwargs.pop("mamba_n_groups") if "mamba_n_groups" in kwargs else self.n_groups`, which OVERWRITES an already-populated dataclass field, so LEGACY wins. Re-derived by RUNNING transformers @ 7d06b1a5, not by reading it: `NemotronHConfig(n_groups=8, mamba_n_groups=4, conv_kernel=4, mamba_d_conv=7)` -> `n_groups=4, conv_kernel=7`. The SCHEDULE pair is genuinely modern-wins (`:158-165` consults the pattern only when the list is None), and is left alone. Upstream disagrees with itself between the two families; both polarities are now mirrored and both are stated where they are implemented. Worse than the behavior was the RECORD: the comment asserted "the modern key wins" as if it were upstream's rule, which is what would mislead the next porter. No released checkpoint ships both spellings of one field, so this was a mirroring defect, not a live one -- which is exactly why it needed a test. 3. "the unported ARMS refuse by name" had one SUBCASE (GGUF). `ForwardNemotronHForCausalLM` is an unconditional `VT_CHECK`, which throws `std::runtime_error`, so it is directly callable with a stub `LoadedModel`. Now asserted, including that the message names W4 and the spec. 4. `NemotronHBlockName` had zero call sites. `BlockFromName` now maps both directions through it and builds its refusal's expected-list from the enum, so a fifth block kind cannot arrive with a message that still lists four. RED-BEFORE, preserved from the history this squash discards. The W3 gate was written against declared-but-empty implementations and run before any of it worked (Release -Werror, doctest 2.5.2): [doctest] test cases: 10 | 2 passed | 8 failed | 0 skipped [doctest] assertions: 1258 | 1186 passed | 72 failed | [doctest] Status: FAILURE! failing in exactly the intended shape -- CHECK( 0 == 52 ) for the schedule depth, CHECK( {} == {5, 12, 19, 26, 33, 42} ) for the attention indices, REQUIRE( 0 == 2 ) for the KV groups, "did NOT throw at all!" for the GGUF refusal, 18487 unclaimed tensors for the enumeration. RED-before for findings 1 and 2 on the pre-fix tree was 12 cases / 9 failed assertions, reporting `92` companions with that exact first name, and `8 == 4` / `4 == 7` / `2 == 9` / `128 == 77` / `true` / `0.001 == 0.5` / `0.1 == 0.6` / `0.0001 == 0.7`. RECORDS. docs/FEATURES.md gains an honest scaffold row (registering an architecture IS a feature-surface change, and check-supported-models.py binds the two), the stale "Inventoried but blocked" entry is removed so the page does not say two things, and the three registered counts move 37->38 / 31->32. docs/USAGE.md gains one row in "Architectures that resolve but refuse to run": check-doc-checkpoint classifies the root CMakeLists.txt as a USER_USAGE file, and the obligation is real rather than an artifact -- pointing the CLI at a NemotronH checkpoint now resolves and refuses by name instead of reporting an unsupported architecture. THE ONE NEW LINE vs `cb37239d4`, and why it is not the wording §5e proposed. The draft was "claims the bare weights and none of the 92 FP8/NVFP4 scale companions". 92 is the count `ClaimMamba` was leaking, NOT the number of scale companions in the model -- the released checkpoint carries thousands across `ClaimNvfp4`/`ClaimMoe`/`ClaimMlp` and the fp8-KV pair -- so naming 92 would imply the architecture has 92 in total. What the gate asserts is stronger and simpler: an unquantized producer claims a strict SUBSET of the quantized arm's names with ZERO companions of any kind (`.weight_scale`, `.weight_scale_2`, `.input_scale`, `.k_scale`, `.v_scale`). Landed: 18487/18487 tensors claimed; a bf16 config claims that set minus its scale companions. Nothing runs yet (spec #517, blocked on #496) `check-public-doc-tables.py` caps a table CELL at 220 chars and this row's cell was already 211, so the clause is paid for out of THIS row's budget, exactly as that checker's `MAX_ROW_CHARS` comment directs. What moved out is "het-KV shapes match `mamba2_state_shape`" -- a forensic anchor already carried by spec §5c and by "KV-shape gated", which stays in the cell -- plus "0 unaccounted" and "released", both implied by "18487/18487". Finding 5's `:171`/`:173` count drift is deliberately NOT touched; it is pre-existing and stays open for its own issue. test_model_registry.cpp PINS the registry contents four ways; all four are reconciled, never weakened: the two size assertions 37->38, both byte-order arch sequences, kExampleConfigArchitectures, both pinned "Supported architectures:" refusal strings, and a new explicit NemotronH branch asserting is_hybrid TRUE / supports_multimodal FALSE instead of letting it fall into the pure-text-full-attention else-branch. test_model_loader_gguf.cpp's canonical arch string gains the entry too. GATES, ALL RE-RUN ON THIS SQUASHED TREE -- a squash is a new tree and inherited numbers are void. Local CPU-only host (VLLM_CPP_CUDA=OFF, GNU 13.3, Ninja), disk 68-69G free / 85% used at every measurement: Release -Werror, CLEAN full build 1213/1213 targets, 0 `warning:` and 0 `error:` lines in the captured log test_nemotron_h_scaffold offline 12/12 cases, 38284/38284, SUCCESS with VT_NEMOTRON35_SNAPSHOT 12/12 cases, 39152/39152, SUCCESS Debug arm (-g0, asserts unmasked) identical on both, SUCCESS full ctest -j4 100% passed, 0 failed out of 403 (skipped: test_modelopt_mixed_precision_ checkpoint, test_voxtral_e2e -- neither has its asset here) doc-checkpoint over origin/main..HEAD ok (the gate this commit exists for) MUTATION-PROVEN, all fifteen re-applied to THIS tree (each alone, rebuilt, run, restored, SHA-256 re-verified, working-tree-vs-index diff proven empty): layers_block_type "moe" -> kAttention at BlockFromName 7 cases / 12 asserts kMoe claims attention tensors (enumeration switch) 2 cases / 4 asserts LayerIndices shifts every index by +1 2 cases / 2 asserts conv_dim drops the 2*n_groups*state_size term 4 cases / 8 asserts SSM cache dtype collapsed to the conv dtype 1 case / 2 asserts mamba dt_bias left UNCLAIMED 2 cases / 3 asserts MTP final_layernorm dropped 2 cases / 3 asserts attention KV group collapsed to ONE layer tag 1 case / 3 asserts the `mtp*` ignore entry not honored 3 cases / 4 asserts mamba KV group collapsed to ONE layer tag 1 case / 2 asserts MTP enorm dropped 2 cases / 3 asserts the forward returns {} instead of refusing 1 case / 2 asserts NemotronHBlockName mislabels kMoe 8 cases / 0 asserts ClaimFp8 ignores `quantized` again 1 case / 1 assert the aliased getters prefer MODERN again 1 case / 8 asserts all fifteen `Status: FAILURE!`. Two are worth keeping in view. The `mtp*` mutation drives the assertion COUNT UP by 1036 (38284 -> 39320) while the gate goes red, so a changed count is itself the signal. And `NemotronHBlockName` prints `assertions: 28 | 28 passed | 0 failed` beside `8 failed` test cases -- the cases THREW, so `grep 'assertions:'` reads that red gate as clean. Read `Status:`. Spec §5e records the landing, the tree-equivalence proof, the wording decision and all of the above, since squashing discards the multi-commit history that carried it. Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
Collaborator
Author
CI note — the two Windows failures are the
|
localai-bot
pushed a commit
that referenced
this pull request
Aug 13, 2026
…o, spec re-merged by hand again (#517) Third re-merge of this land-prep. `origin/main` moved from `72e661ae4` to `51ec6bed5` while the re-gate was running, and one of the two new commits is `c6b240edd` (#576) -- W3 of THIS SAME ROW, which rewrote 349 lines of the same keyed record. `.agents/specs/nemotron-h-model.md` was merged BY HAND again, per AGENTS.md. Git's automatic resolution was DISCARDED: `git checkout origin/main -- <spec>` first, then the three W2 regions re-applied with uniqueness-asserted anchors. Verified block by block (md5 per `##` heading), **16 blocks: 13 BYTE-IDENTICAL to main, 3 differ and all three are W2's own**: identical : preamble, 0, 1, 3, 4, 5, 5a, 5b, 5c, 5d, 5e, 6, 8 W2's own : 2 (one table row), 6a (new), 7 (Now) That carries main's W3 sections §5c/§5d/§5e whole, §4's three W1 subsections whole, and §5a/§5b whole. §7 is the one place all three Ws speak: reconciled to "W1 and W3 have LANDED, W2 is in re-review", not overwritten. The only W2 edit to §2 is the routed-scale anchor the repair pass re-verified against the pin: `apply_routed_scale_to_output=True` at `nemotron_h.py:234`, factor `:233` (main still carried the pre-repair `:246`). Everything else auto-merged clean, and the delta vs `origin/main` is exactly this branch's eight W2 files. `tests/CMakeLists.txt` carries both W3's `test_nemotron_h_scaffold` (:365) and W2's `test_ops_moe_nongated_relu2` (:1187). `src/vt/cuda/cuda_moe.cu` includes only `vt/ops.h` plus CUDA/std headers, and neither it nor `vt/ops.h` is touched by this merge, so the CUDA arm the fresh reviewer compiled and GPU-verified on GB10 at `e2d68404` is unchanged. FOLLOWING_AGENTS_PROTOCOL Refs #517. Refs #576. Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Issue: #517 (spec
.agents/specs/nemotron-h-model.md, rowMODEL-TEXT-nemotron-h-nemotron-hfor-causal-lm)Supersedes #572 /
row/MODEL-NEMOTRON-H-W3-FIX@cb37239d4. Same content,one commit, plus the one
docs/FEATURES.mdclause that branch had no authorityto write. Close #572 in favor of this.
Why this branch exists
cb37239d4was green on every gate exceptdoc-checkpoint range:The rule is unconditional and per-commit:
check-doc-checkpoint.py:79classifies any
src/vllm/model_executor/models/path asfeature_surface,:311-316requiresdocs/FEATURES.mdin the SAME commit, andcommits_in_range(
:369-375) walksrev-list --reverse --no-merges origin/main..HEAD— so W3C'sown
docs/FEATURES.mdedit in3295d0c1acannot cover a later commit.ci.yml:343runs the identical invocation, so CI was red for the same reason.There is no exemption mechanism and weakening the checker was never on the table.
mainis never force-pushed, so3981de6a4could not be amended. The history isrebuilt here as one squashed commit off current
origin/maincarrying thewhole W3 + W3-FIX content with the docs clause included — which is what the
squash-merge would have produced anyway, and which satisfies the per-commit rule
trivially. Same precedent as W3B → W3C on this row.
Tree equivalence vs
cb37239d4git diff cb37239d4against this tree lists exactly six files:.agents/benchmark-record.md,.agents/specs/dspark-spec-decode.md,docs/BENCHMARKS.md,docs/STATUS.mdorigin/mainre-merge — these are72e661ae4(#442 DSPARK), the one commit the fix branch lackeddocs/FEATURES.md.agents/specs/nemotron-h-model.mdEvery
src/,tests/,CMakeLists.txtanddocs/USAGE.mdpath isbyte-identical to
cb37239d4.The
docs/FEATURES.mdclause§5e proposed "claims the bare weights and none of the 92 FP8/NVFP4 scale
companions". That is misleading: 92 is what
ClaimMambawas leaking(23 mamba × 2 projections × 2), not the number of scale companions in the model —
the released checkpoint carries thousands across
ClaimNvfp4/ClaimMoe/ClaimMlpand the fp8-KV pair. What the gate actually asserts is stronger: anunquantized producer claims a strict SUBSET of the quantized arm's names with
ZERO companions of any kind. Landed:
check-public-doc-tables.pycaps a table CELL at 220 chars and this row's cellwas already at 211, so the clause is paid for out of this row's own budget,
exactly as that checker's
MAX_ROW_CHARScomment directs. What moved out is"het-KV shapes match
mamba2_state_shape" (a forensic anchor already carried byspec §5c, and "KV-shape gated" stays in the cell) plus "0 unaccounted" and
"released", both implied by "18487/18487". Finding 5's
:171/:173countdrift is deliberately untouched — pre-existing, owed its own issue.
Gates, all re-run on this squashed tree
A squash is a new tree, so no number is inherited. Local CPU-only host
(
VLLM_CPP_CUDA=OFF, GNU 13.3, Ninja), disk 68–69G free / 85% used at everymeasurement.
-Werror, clean full buildwarning:and 0error:lines in the captured logtest_nemotron_h_scaffold, offlineStatus: SUCCESS!test_nemotron_h_scaffold,VT_NEMOTRON35_SNAPSHOTliveSUCCESS!-g0, asserts unmasked), offline / liveSUCCESS!ctest -j4test_modelopt_mixed_precision_checkpoint,test_voxtral_e2e— neither asset is here)scripts/agent-preflight.sh --stageddoc-checkpointoverorigin/main..HEADMutation proof — all fifteen re-applied to this tree
Each alone, rebuilt, run, restored, SHA-256 re-verified, working-tree-vs-index
diff proven empty. All fifteen
Status: FAILURE!:layers_block_type"moe"→kAttentionatBlockFromNamekMoeclaims attention tensors (enumeration switch)LayerIndicesshifts every index by +1conv_dimdrops the2*n_groups*state_sizetermdt_biasleft UNCLAIMEDfinal_layernormdroppedmtp*ignoreentry not honoredenormdropped{}instead of refusingNemotronHBlockNamemislabelskMoeClaimFp8ignoresquantizedagainTwo are instructive. The
mtp*mutation drives the assertion COUNT up by1036 while the gate goes red — a changed count is itself the signal. And
NemotronHBlockNameprintsassertions: 28 | 28 passed | 0 failedbeside8 failedtest cases, because the cases THREW:grep 'assertions:'readsthat red gate as clean. Read
Status:.Scope
W3 makes the architecture known; it runs nothing.
ForwardNemotronHForCausalLMrefuses by name (W4, itself blocked on the Mamba2 SSD CUDA arm #496 W2), and the
GGUF arm refuses by name (W7, owed per spec §5b). No lifecycle transition — the
row stays where it is.
🤖 Generated with Claude Code