diff --git a/.agents/specs/nemotron-h-model.md b/.agents/specs/nemotron-h-model.md index 4b13b56d4..c7113ba4a 100644 --- a/.agents/specs/nemotron-h-model.md +++ b/.agents/specs/nemotron-h-model.md @@ -94,9 +94,45 @@ memory format against the oracle explicitly. | MTP | `models/nemotron_h_mtp.py::NemotronHMTP` (`registry.py:638`) | | MIXED_PRECISION resolution | `layers/quantization/modelopt.py:2280-2450`, per-layer lookup `:2416-2445` | -The `quantized_layers` lookup is **direct name first, then shard prefix** -(`modelopt.py:2426`, `:2437`). Mirror both, in that order; a merged/sharded -local name that resolves only by prefix is the case that will bite. +**Correction (W1, 2026-08-12).** This section previously said the +`quantized_layers` lookup is "direct name first, then shard prefix +(`modelopt.py:2426`, `:2437`)". That understated upstream and was a defect in +this spec, not in upstream. `_resolve_quant_algo` (`modelopt.py:2412-2487`) has +**five** strategies and tries them in this order: + +1. **Direct lookup** (`:2424-2427`) over `_quantized_layer_prefix_candidates`. +2. **Packed/fused lookup** (`:2429-2447`): unfuse via the model's + `packed_modules_mapping`, collect each shard's algo **across all base + candidates**, and **raise `ValueError`** if the shards of one fused layer + disagree. +3. **Prefix lookup** (`:2449-2453`): any `quantized_layers` key starting with + `prefix + "."`, returning the first in map order. +4. **The `.experts` special case** (`:2455-2461`): a `FusedMoE` layer's prefix + is `...moe.experts` while ModelOpt lists `...moe.up_proj` / `...moe.down_proj`, + so the container falls back to its parent. +5. **`fused_projection_shards` fallback** (`:2463-2486`): `qkv_proj -> + (q_proj, k_proj, v_proj)` and `gate_up_proj -> (gate_proj, up_proj)` for + configs that list shard names with no `packed_modules_mapping` registered. + The algo set is rebuilt **per candidate** here, unlike strategy 2, and it + raises on disagreement too. + +`_quantized_layer_prefix_candidates` (`:2489-2505`) itself yields the prefix, a +bare `lm_head` when the prefix ends in `.lm_head` (the real checkpoint stores a +BARE `lm_head` key), and the `language_model.model.` <-> `model.language_model.` +swap, de-duplicated in order. + +Exclusion is separate and is checked **first**, in `get_quant_method` +(`:2515-2522`): `is_layer_excluded` (`:145-181`) runs `is_layer_skipped` +(`quant_utils.py:510-572`, which raises on a partially-excluded fused layer), +then a legacy substring rule kept for pre-0.39 exports, then `fnmatch` +wildcards — which is how the real `ignore` entry `mtp*` covers the whole MTP +head. + +**Measured on the real config** (all 5981 `quantized_layers` entries and all 72 +`ignore` entries): the histogram is exactly `{W4A16_NVFP4: 5935, FP8: 46}`, +every entry resolves by strategy **1** alone, and no entry collides with the +`ignore` list. Strategies 2-5 are therefore not reachable from this checkpoint +and are covered by synthetic fixture entries instead — see W1 below. Config note: `nemotron_h.py` reads `config.hybrid_override_pattern`, which newer transformers exposes as a property derived from `layers_block_type` @@ -143,6 +179,242 @@ cannot. | **W6** | **GB10 e2e token gate vs the pinned oracle** | token-exact greedy, identical prompts/counts/batching/sampling; oracle identity asserted | #496 W2 (CUDA), W4, W5 | | **W7** | GGUF k-quant / i-quant arm through the shared GGUF loader (see §5b); refused by name until it lands | quant-matched load + token gate | W4 | +### W1 progress — the resolver has LANDED (2026-08-12) + +`src/vllm/model_executor/layers/quantization/modelopt_mixed_precision.h`, +header-only, mirroring `modelopt.py` at the pin. It sits under `src/` rather +than beside its two `include/vllm/...` siblings on purpose: +`check-doc-checkpoint` classifies the whole `include/vllm/` prefix as a +user-facing surface and demands `docs/USAGE.md` move with it, and nothing this +header does is user-facing yet — it is not on the `include/vllm.h` ABI and no +loader calls it, so that edit would have documented nothing. **W3 promotes it to +`include/` when it becomes part of the consumed surface**, and pays the public +document obligation that genuinely applies then. Parses either config shape, +resolves a module prefix through all five strategies plus the exclusion pass, +and returns a TYPED `ModuleQuant` (algo, which strategy answered, group size) +rather than a raw string. + +Two gate arms, deliberately two binaries so the opt-in one cannot mask the +always-on one: + +- `test_modelopt_mixed_precision` — always-on, curated fixture at + `tests/fixtures/modelopt_mixed_precision/curated_config.json`, whose real + entries are copied verbatim from the checkpoint and whose synthetic entries + are each annotated with the strategy they exist to reach. **26 cases, 167 + assertions** (22 / 137 as first landed; see the W1 repair note below). +- `test_modelopt_mixed_precision_checkpoint` — exhaustive, reads the real + 1.3 MB `config.json` from `$CHECKPOINT_ROOT/nemotron-3.5-lightning-30b-nvfp4`, + asserts all 5981 + 72 entries and the exact histogram, plus the standalone + `hf_quant_config.json` the repair below added. **3 cases, 12181 assertions, + GREEN** (2 / 12145 as first landed; the checkpoint is staged on the NAS and + reachable from the CPU box, so this ran rather than skipping). Exits 77 — + CTest *Skipped*, with a loud banner naming the exact export — when the + checkpoint is absent. + +**One deliberate divergence from upstream**, recorded here because it is a +policy choice and not a port: an algorithm that resolves to something this +consumer does not implement is **refused by name**. Upstream's +`get_quant_method` falls through to `UnquantizedLinearMethod()` for anything +outside {FP8, NVFP4, W4A16_NVFP4, MXFP8} — including `FP8_PB_WO` and +`FP8_PER_CHANNEL_PER_TOKEN`, which are entries of its own `QUANT_ALGOS`. Silent +dequantization is numerically correct and therefore invisible to a token gate, +which is precisely the stop condition §0 names. A prefix simply ABSENT from +`quantized_layers` is not this case and stays unquantized, as upstream. + +**Still owed by later W's:** nothing consumes the resolver yet — no loader, no +`get_quant_method` equivalent, no kernel selection. W3 wires it. + +### W1 repair — the fresh review returned FAIL (2026-08-12) + +Reviewed at `f5c901ce`; repaired on `row/MODEL-NEMOTRON-H-W1-FIX`. The review +confirmed all five resolution strategies against a verbatim upstream +transcription (63,787 + 6,493 prefixes, **0 diffs**) and found one real defect +plus four guarantees the suite did not actually hold. + +**The defect: `Parse` refused the driver checkpoint's own `hf_quant_config.json` +(HIGH).** `Parse` gated on `ExtractQuantAlgo`, which mirrors +`_extract_modelopt_quant_algo` (`modelopt.py:245-263`) and requires a top-level +`quant_method` starting with `"modelopt"`. That precondition belongs to the +SELECTION hook — `override_quantization_method` uses it to tell a ModelOpt +`quantization_config` apart from a compressed-tensors one in the same field of +the same `config.json`. Upstream's PARSER, `from_config` (`:282-367`), never +inspects `quant_method` at all: it dispatches on the SHAPE. The real file has +top-level keys exactly `{producer, quantization}` and names no `quant_method` +anywhere, so the header threw +`std::invalid_argument: not a MIXED_PRECISION config (quant_algo="", quant_method="")` +on the one config `get_config_filenames()` (`:265-267`) actually points at, +while upstream parsed it and resolved all 5981 entries. Detection and parsing +are now separate: `ShapeQuantAlgo` for `Parse`, `ExtractQuantAlgo` for +`IsMixedPrecision`. + +**Four guarantees the tests did not hold.** Each survived the reviewer's +mutation with the suite green; each now has a case that catches it, proven by +re-applying that exact mutation: + +| Guarantee | Mutation that used to survive | Now caught by | +|---|---|---| +| Scans run in INSERTION order (`ordered_json`, header divergence 3) | load the fixture with plain `nlohmann::json` | `synthetic.layers.2.self_attn` resolves FP8, not W4A16_NVFP4 — sorted, `k_proj` comes first | +| Strategies 3 and 4 return the FIRST matching child | return the LAST match | new `synthetic.layers.8.moe.{a,b}_proj`, the first parent whose first and last child DISAGREE | +| Strategy 2's algo set spans ALL base candidates, unlike strategy 5 | rebuild the set per candidate | new `language_model.model...q_proj` / `model.language_model...k_proj` pair — the union raises, a per-candidate rebuild returns FP8 and never sees the second spelling | +| `is_layer_skipped`'s `experts` branch (`quant_utils.py:559-565`) | invert `e.find(prefix)`, or delete the branch | an `ignore` naming ONE expert child must exclude the whole container | + +**`FnMatch` vs CPython, measured not assumed.** Two differential sweeps against +`fnmatch.fnmatchcase`: 3,183,165 pairs (names ≤3, patterns ≤5) with **0** +mismatches, and 6,291,453 pairs (names ≤2, patterns ≤6) with **108** across +**27** patterns — all one class, all CPython-True/ours-False, all a bracket +whose contents reduce to a bare `!` after `translate` drops a REVERSED range +(`[?-.!]` → `(?s:.)\Z`). Recorded in the header and NOT fixed: matching it means +reproducing `translate`'s rewriting, and ModelOpt emits no bracket expressions +at all, so no reachable input touches it. + +**LOW-1, corrected 2026-08-13.** That count read **20** here and at +`modelopt_mixed_precision.h:253` when the branch went to review. It was wrong, +and wrong in a way the numbers refute on their own: every one of those patterns +translates to `(?s:.)\Z`, which matches a length-1 name and nothing else, so a +pattern can contribute at most the **4** one-character names in the sweep +alphabet `{a,b,.,0}` and 20 patterns cap out at 80 — below the 108 the same +sentence reports. Re-measured independently against CPython 3.12 +`fnmatch.fnmatchcase` on this branch: sweep 1 reproduces `3,183,165 pairs, +0 mismatches`; sweep 2 reproduces `6,291,453 pairs, 108 mismatches` and yields +`27 distinct patterns`, 27 x 4 = 108, direction `{(True, False): 108}`, +`distinct translations: {'(?s:.)\Z': 108}`, `mismatching name lengths: {1: 108}`. +All 27 have the shape `[X-Y!]` with X > Y. `FnMatch`'s behavior is unchanged — +only the recorded number was. + +**Why a plain `ctest` skips the exhaustive arm.** `CHECKPOINT_ROOT` is a `.env` +key and `.env` is not exported by anything; the repo's documented loader is +`set -a; . ./.env; set +a` (`.env.example:8`). There is no CTest-side `.env` +reader, so a shell that has not sourced it skips the arm. The skip banner names +the exact export rather than pretending the arm ran. (This paragraph also +credited `.agents/environment.md` with that loader line and claimed +`hf_snapshot.h` does not apply here; both were wrong — see LOW-3 below.) + +Gate after repair: **26 cases / 167 assertions** curated (was 22 / 137), +**3 cases / 12181 assertions** exhaustive (was 2 / 12145), clean Release +`-Werror`, ASan and UBSan clean, full `ctest` 396/396. + +### W1 land-prep — round-2 review PASS, its LOWs, and the two open dispositions (2026-08-13) + +Round 2 reviewed `e734fe9e` and returned **PASS** with three LOW findings. All +three are repaired on `row/MODEL-NEMOTRON-H-W1-LAND`; each was re-verified +against the file or the measurement rather than taken on report. + +**LOW-1 — the `27` above.** Recorded as `20` in two places. See the correction +paragraph in the block above for the arithmetic and the re-measurement. + +**LOW-2 — `CAPTURE` on a `const char*` prints `1`.** doctest 2.5.2 has no +stringifier for a `const char*` lvalue, and two loops in +`test_modelopt_mixed_precision_checkpoint.cpp` (`:172`, `:248`) each compared +FIVE prefixes through one, so a failure named none of them. Demonstrated RED by +inverting both CHECKs in a scratch copy — `logged: p := 1`, ten times — and +GREEN under the identical inversions after switching both loops to +`const std::string`, which is what `:149` and `:161` already used: +`logged: p := backbone.embeddings`, `... := lm_head`, and so on for all nine +distinct prefixes. + +**LOW-3 — two env vars gated one checkpoint, and neither carried the pin.** +Two parts. + +*(a) A half-true citation.* The test claimed `.env.example` **and** +`.agents/environment.md` document the loader as `set -a; . ./.env; set +a`. +`.env.example:8` does, verbatim; `.agents/environment.md` does not contain the +string at all (`grep -n 'set -a'` exits 1) and instead points at `.env.example` +at `:16`. Corrected to cite `.env.example:8` alone. + +*(b) The pin had no teeth.* `CHECKPOINT_ROOT` (this test, joining the staging +directory name by hand) and `VT_NEMOTRON35_SNAPSHOT` +(`parity::Nemotron35LightningSnapshot()`, landed on main with §5a) resolved the +same NAS `local_dir`, and `kNemotron35LightningNvfP4Revision` could refuse +neither: the accessor's HF-cache spelling is unreachable for a `local_dir` tree, +because there is no `snapshots//` whose NAME carries the revision, and an +env override is deliberately never revision-checked. A re-download of the same +repo name lands a different revision under the identical path — exactly the +substitution `kQwen27NvfP4Revision` exists because of. + +`hf download --local-dir` does record the revision, just not in the path: it +writes a per-revision file manifest at +`/.cache/huggingface/trees/.json`, confirmed present on the NAS +tree for `29f2d174…`. `Nemotron35LightningSnapshot()` is now the SINGLE resolver +for both spellings and gates the `CHECKPOINT_ROOT` path on that manifest. Proven +by construction, the manifest being the only thing that differs between A and B: + +| Run | Setup | Result | +|---|---|---| +| A | staged dir carries only `deadbeef….json` | `EXIT=77`, loud skip | +| B | add `29f2d174….json`, nothing else changed | `EXIT=0`, 3 / 12181 | +| C | `VT_NEMOTRON35_SNAPSHOT` at the real NAS dir | `EXIT=0`, 3 / 12181 | +| D | neither env var set | `EXIT=77`, banner names the export AND the manifest | +| E | `VT_` set to a nonexistent dir, `CHECKPOINT_ROOT` valid | `EXIT=77` — refuses, never falls back | + +`VT_NEMOTRON35_SNAPSHOT` keeps `HfSnapshot`'s documented escape semantics +unchanged and is checked first. That asymmetry is deliberate: naming ONE +directory outright is the deliberate different-checkpoint run the override +exists for, while naming a ROOT is not, so only the root path is revision-gated. + +**Two round-1 dispositions that existed nowhere.** Round 1 returned findings 1-8 +and 7 and 8 were answered verbally only. + +- **Finding 7 — the header lives under `src/`, not `include/`.** + **ACCEPTED, by operator decision.** `check-doc-checkpoint` classifies the + whole `include/vllm/` prefix as user-facing and demands `docs/USAGE.md` move + with it; nothing consumes the resolver yet and it is not on the + `include/vllm.h` ABI, so that edit would have documented nothing. The checker + behavior is tracked as **#515**. **W3 must promote the header to `include/` + when it becomes consumed surface** and pay the public-document obligation that + genuinely applies then. Recorded here so the debt outlives the conversation. +- **Finding 8 — the exhaustive arm is opt-in because `CHECKPOINT_ROOT` is not + exported.** **ANSWERED.** The repo convention is `set -a; . ./.env; set +a` + (`.env.example:8`); no new mechanism was invented, and the skip banner now + names the exact export. LOW-3(b) supersedes the part of this that claimed + `hf_snapshot.h` did not apply — it does, and it is now the resolver. + +**Reported, outside this row's authority to fix.** +`.agents/environment.md:29-30` states that `CHECKPOINT_ROOT` "states an INTENT +and nothing more: no code in the tree reads `CHECKPOINT_ROOT`". The exhaustive +arm reads it — before this branch directly, now through +`parity::Nemotron35LightningSnapshot()` — so that sentence is false as written +and belongs to whoever owns `.agents/environment.md`. + +**Land-prep gate, re-run on the re-merged tree.** Baselines are unchanged, which +is the point: the merge moved the spec and nothing else this row owns. + +| Arm | Result | +|---|---| +| curated `test_modelopt_mixed_precision` | **26 cases / 167 assertions**, `Status: SUCCESS!` | +| exhaustive `test_modelopt_mixed_precision_checkpoint` | **3 cases / 12181 assertions**, `Status: SUCCESS!` (`CHECKPOINT_ROOT=/mnt/nas_share/checkpoints`) | +| clean Release, CUDA off, `-Werror` | 1209/1209 built, **0 warnings** | +| full `ctest` (Release) | **402 tests, 401 passed**; `test_engine_core_proc` failed under `-j` and passed on a serial re-run (the known starvation set); `test_modelopt_mixed_precision_checkpoint` and `test_voxtral_e2e` correctly *Skipped* with no `CHECKPOINT_ROOT` | +| Debug + `VLLM_CPP_SANITIZE=address,undefined`, the four targets this change touches | **4/4 Passed**, including under CI's own `ASAN_OPTIONS=detect_leaks=1:strict_string_checks=1`, `UBSAN_OPTIONS=print_stacktrace=1`, `VT_POOL_BYPASS=1` | +| `scripts/agent-preflight.sh` | green | +| `scripts/check-doc-checkpoint.py` | `OK: public documents match the claims this change makes.` | +| `scripts/check-commit-trailers.py --range origin/main..HEAD` | `OK: commit trailer contract` | + +The whole-tree sanitizer run reports 39 LeakSanitizer failures across unrelated +model/serving binaries. Those are the lane's known pre-existing state, not this +change: `.github/workflows/ci.yml:775-793` marks `sanitize-cpu` +`continue-on-error: true` precisely so "a pre-existing finding cannot block +unrelated work" and cites a live run whose conclusion is `success` with both +sanitizer lanes `failure`. This change cannot reach them either — the only `src/` +file it touches is `modelopt_mixed_precision.h`, which `grep -rln` finds included +by exactly two files, both of them its own tests, so every object in `libvllm.a` +is byte-identical to one built from `origin/main`. + +**The six mutations, re-run against the re-merged tree.** All six RED on the +curated arm; the baseline is `26 | 26 passed | 0 failed` before each. + +| Mutation | Curated result | +|---|---| +| `Parse` back on `ExtractQuantAlgo` | `FAILURE!` 25 passed / 1 failed — and REDs the exhaustive arm too, where the thrown case drops the assertion count to 12152 | +| fixture loaded with plain `nlohmann::json` | `FAILURE!` 163 passed / 4 failed | +| strategies 3+4 return the LAST match | `FAILURE!` 163 passed / 4 failed | +| strategy 2 rebuilds the algo set per candidate | `FAILURE!` 164 passed / 3 failed | +| `e.find(prefix)` → `prefix.find(e)` | `FAILURE!` 163 passed / 4 failed | +| the `experts` branch deleted | `FAILURE!` 163 passed / 4 failed | + +Read `Status:`, not `assertions:` — mutation 1 prints `0 failed` on the +exhaustive arm while failing, because the case THREW and its remaining +assertions were never reached. + ## 5. Gates **Correctness first, always.** No throughput number is recorded by this row diff --git a/src/vllm/model_executor/layers/quantization/modelopt_mixed_precision.h b/src/vllm/model_executor/layers/quantization/modelopt_mixed_precision.h new file mode 100644 index 000000000..c0c89a7bb --- /dev/null +++ b/src/vllm/model_executor/layers/quantization/modelopt_mixed_precision.h @@ -0,0 +1,774 @@ +// ModelOpt `MIXED_PRECISION`: resolving a quantization ALGORITHM PER MODULE. +// +// UPSTREAM (ported FROM, ground-every-impl rule) @ pinned oracle +// 5559679229bc961848b121ccdeaa8fa5d79bec98 (vLLM 0.26.0.dev0): +// vllm/model_executor/layers/quantization/modelopt.py:2279-2410 +// class ModelOptMixedPrecisionConfig, override_quantization_method, +// _from_config (where `quantized_layers` is read and group_size is seeded) +// vllm/model_executor/layers/quantization/modelopt.py:2412-2487 +// _resolve_quant_algo — the FIVE resolution strategies, in order +// vllm/model_executor/layers/quantization/modelopt.py:2489-2505 +// _quantized_layer_prefix_candidates +// vllm/model_executor/layers/quantization/modelopt.py:282-367 +// ModelOptQuantConfigBase.from_config (both config shapes) +// vllm/model_executor/layers/quantization/modelopt.py:145-181 +// ModelOptQuantConfigBase.is_layer_excluded +// vllm/model_executor/layers/quantization/utils/quant_utils.py:510-572 +// is_layer_skipped +// +// WHY THIS EXISTS. Every quantized checkpoint we load so far names ONE scheme +// for the whole model, so `quantization_config` could be read ad-hoc where it +// was needed (`kimi_k3_weights.cpp:171`, `deepseek_v2_weights.cpp:365`) and the +// scheme handed to every layer. `nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B- +// NVFP4` breaks that assumption inside a single file: 5935 of its 5981 +// `quantized_layers` entries are NVFP4 W4A16 with `group_size 16` (routed +// experts, shared experts, `lm_head`), 46 are FP8 W8A8 static (the Mamba2 +// `in_proj`/`out_proj` ONLY), and a 72-entry `ignore` list leaves embeddings, +// conv1d, the MoE gates and the attention tower in bf16. +// +// The repo NAME says NVFP4, and reading it as uniformly NVFP4 is precisely the +// failure a token gate CANNOT SEE: dequantizing an FP8 projection to bf16, or +// treating a bf16 projection as quantized, is still numerically fine, so the +// tokens match, the goldens pass, and we move the wrong bytes forever. Hence +// this header, and hence the refusal policy below. +// +// SCOPE. This is a pure config capability: parse + resolve, no kernels, no +// weights, no forward path. Row MODEL-TEXT-nemotron-h-nemotron-hfor-causal-lm +// W1, issue #517, spec `.agents/specs/nemotron-h-model.md`. +// +// THREE DELIBERATE DIVERGENCES FROM UPSTREAM, each recorded here rather than +// discovered later: +// +// 1. AN UNRESOLVED ALGORITHM IS REFUSED BY NAME. Upstream's consumer +// (`get_quant_method`, modelopt.py:2508-2560) has branches for FP8, NVFP4, +// W4A16_NVFP4 and MXFP8 and falls through to `UnquantizedLinearMethod()` +// for anything else — including `FP8_PB_WO` and +// `FP8_PER_CHANNEL_PER_TOKEN`, which are real entries of its own +// `QUANT_ALGOS` list. Silently dequantizing a quantized layer is invisible +// to a token gate, so `Resolve` throws and names the algorithm instead. +// A prefix that is simply ABSENT from `quantized_layers` is NOT this case: +// upstream leaves it unquantized and so do we, because "not listed" is the +// checkpoint saying bf16, not an unrepresentable scheme. +// +// 2. HEADER-ONLY, AND UNDER `src/` RATHER THAN `include/`. The resolver is +// string and JSON logic with no device or kernel dependency, so it needs no +// translation unit — matching `base_config.h` and +// `compressed_tensors/schemes/nvfp4.h`, which are header-only too. That +// also keeps the change clear of the root `CMakeLists.txt`, which currently +// reds `check-doc-checkpoint` when a source file is added (issue #515). +// +// Its two sibling headers live under `include/vllm/...`; this one does not, +// for a reason worth stating rather than leaving to look like an accident. +// `check-doc-checkpoint` classifies the whole `include/vllm/` prefix as a +// USER-FACING surface and requires `docs/USAGE.md` to move with it — the +// list's own comment calls it "user-facing configuration/build/install +// entrypoints". Nothing consumes this resolver yet: it is not on the +// `include/vllm.h` ABI, no loader calls it, and no command, C API key, +// config key or install step changes because of it, which is exactly what +// AGENTS.md says `docs/USAGE.md` tracks. Putting an internal header there +// would have forced a `docs/USAGE.md` edit that documented nothing. +// `src/vllm/model_executor/models/*.h` is the established precedent for an +// internal header. **W3 should promote this to `include/` when it becomes +// part of the consumed surface**, and pay whatever public-document +// obligation genuinely applies at that point. +// +// 3. `Parse` IS TEMPLATED ON THE JSON TYPE so it accepts +// `nlohmann::ordered_json`. Upstream iterates a Python dict, whose order is +// INSERTION order; plain `nlohmann::json` sorts object keys +// lexicographically, which would silently change which entry the +// group_size seeding (modelopt.py:2360-2372) and the prefix scans +// (:2450, :2458) return. Callers that care about order fidelity parse with +// `ordered_json`; both types compile. +#pragma once + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include + +namespace vllm { +namespace layers { +namespace modelopt { + +// The quantization algorithms the MIXED_PRECISION consumer implements — the +// exact four `get_quant_method` has branches for (modelopt.py:2526-2560). +// Anything else in the checkpoint is refused by name; see divergence 1 above. +enum class QuantAlgo { + kUnquantized, // not listed, or on the `ignore` list: bf16/f16 as loaded + kFp8, // "FP8" — per-tensor weight scale + static input scale + kNvfp4, // "NVFP4" — W4A4 + kW4A16Nvfp4, // "W4A16_NVFP4" — 4-bit weights, bf16/f16 activations + kMxfp8, // "MXFP8" +}; + +// Which of the strategies produced the answer. Upstream returns only the algo +// string; we carry the strategy because a resolver that reaches the right +// answer by the wrong route is a latent defect (a `.experts` container that +// resolves by an accidental substring, say), and because it is the only way a +// test can prove each of the five paths is actually exercised. +enum class Resolution { + kExcluded, // matched the `ignore` list — checked FIRST, wins outright + kDirect, // strategy 1: modelopt.py:2424-2427 + kPacked, // strategy 2: modelopt.py:2429-2447 (packed_modules_mapping) + kPrefix, // strategy 3: modelopt.py:2449-2453 + kExpertsParent, // strategy 4: modelopt.py:2455-2461 (".experts" -> parent) + kFusedShards, // strategy 5: modelopt.py:2463-2486 (qkv / gate_up names) + kUnlisted, // no strategy matched — unquantized, and NOT an error +}; + +inline const char* QuantAlgoName(QuantAlgo a) { + switch (a) { + case QuantAlgo::kFp8: + return "FP8"; + case QuantAlgo::kNvfp4: + return "NVFP4"; + case QuantAlgo::kW4A16Nvfp4: + return "W4A16_NVFP4"; + case QuantAlgo::kMxfp8: + return "MXFP8"; + case QuantAlgo::kUnquantized: + break; + } + return "UNQUANTIZED"; +} + +struct ModuleQuant { + QuantAlgo algo = QuantAlgo::kUnquantized; + Resolution how = Resolution::kUnlisted; + // The NVFP4-family block size. ZERO for FP8/MXFP8/unquantized, which are not + // group-quantized — deliberately not a "default 16" that would read as a real + // group size on a scheme that has none. + int group_size = 0; + + bool Quantized() const { return algo != QuantAlgo::kUnquantized; } +}; + +// vLLM's per-model `packed_modules_mapping`: the fused module name a model +// builds (`qkv_proj`) to the checkpoint shard names it is fused FROM +// (`q_proj`, `k_proj`, `v_proj`). Registered by the model, so it is empty until +// a model supplies one — strategies 2 and the fused arm of the exclusion check +// are inert without it, exactly as upstream. +using PackedModulesMapping = std::map>; + +namespace detail { + +inline std::string Upper(std::string s) { + for (char& c : s) { + if (c >= 'a' && c <= 'z') c = static_cast(c - 'a' + 'A'); + } + return s; +} + +inline bool StartsWith(const std::string& s, const std::string& p) { + return s.size() >= p.size() && s.compare(0, p.size(), p) == 0; +} + +inline bool EndsWith(const std::string& s, const std::string& p) { + return s.size() >= p.size() && s.compare(s.size() - p.size(), p.size(), p) == 0; +} + +// Python `s.rsplit(".", 1)[-1]` / `s.split(".")[-1]`: the whole string when +// there is no separator (which is how the bare "lm_head" key resolves). +inline std::string LastSegment(const std::string& s) { + const std::size_t dot = s.rfind('.'); + return dot == std::string::npos ? s : s.substr(dot + 1); +} + +// Python `s.rsplit(".", 1)[0]`: likewise the whole string when there is no dot. +inline std::string ParentSegment(const std::string& s) { + const std::size_t dot = s.rfind('.'); + return dot == std::string::npos ? s : s.substr(0, dot); +} + +// Python `str.replace(old, new)` — ALL occurrences, which is what +// quant_utils.py:545 relies on. +inline std::string ReplaceAll(std::string s, const std::string& from, + const std::string& to) { + if (from.empty()) return s; + std::size_t at = 0; + while ((at = s.find(from, at)) != std::string::npos) { + s.replace(at, from.size(), to); + at += to.size(); + } + return s; +} + +// One `fnmatch` token against one character; `*np` lands after the token. +inline bool MatchOneToken(char c, const std::string& pat, std::size_t p, + std::size_t* np) { + if (pat[p] == '?') { + *np = p + 1; + return true; + } + if (pat[p] == '[') { + std::size_t i = p + 1; + bool negate = false; + if (i < pat.size() && pat[i] == '!') { + negate = true; + ++i; + } + const std::size_t first = i; + bool matched = false; + while (i < pat.size()) { + if (pat[i] == ']' && i > first) break; + if (i + 2 < pat.size() && pat[i + 1] == '-' && pat[i + 2] != ']') { + if (c >= pat[i] && c <= pat[i + 2]) matched = true; + i += 3; + } else { + if (c == pat[i]) matched = true; + ++i; + } + } + if (i >= pat.size()) { // unterminated class: Python treats '[' as literal + *np = p + 1; + return c == '['; + } + *np = i + 1; + return negate ? !matched : matched; + } + *np = p + 1; + return c == pat[p]; +} + +// Python `fnmatch.fnmatch` semantics (`*`, `?`, `[seq]`, `[!seq]`), as used by +// modelopt.py:177-179. `*` spans separators because the upstream call has no +// FNM_PATHNAME equivalent, and module paths are dot-separated anyway. ModelOpt +// itself only ever emits `module_path*` (the real checkpoint's one wildcard +// entry is `mtp*`); the rest of the syntax is mirrored so a hand-edited or +// future config cannot be silently mis-read. +// +// ONE KNOWN DIVERGENCE CLASS, measured and deliberately left alone. Swept +// differentially against CPython 3.12 `fnmatch.fnmatchcase`: +// +// names <=3 chars over {a,b,.,0} x patterns <=5 over {a,.,?,*,[,],!,-} +// 3,183,165 pairs, 0 mismatches +// names <=2 chars over {a,b,.,0} x patterns <=6 over the same alphabet +// 6,291,453 pairs, 108 mismatches spanning 27 distinct patterns +// +// The 27 is arithmetic, not a tally: every one of the 27 patterns translates to +// `(?s:.)\Z`, which matches a name of length ONE and nothing else, so each +// contributes exactly the 4 one-character names over {a,b,.,0} and 27 x 4 = +// 108. A count of 20 would be impossible on its face (it was in this comment +// until 2026-08-13; the ceiling is 4 per pattern, so 108 needs 27 of them). +// +// All 108 are ONE class and all in the same direction (CPython True, this +// False): a bracket whose contents reduce to a bare `!` once CPython's +// `translate` has DROPPED a reversed range. All 27 have the shape `[X-Y!]` +// with X > Y; `[?-.!]` is the worked example — `?`(0x3f) down to `.`(0x2e) is +// empty, so `stuff` becomes `"!"`, and translate's "negated empty class matches +// any character" rule compiles the whole pattern to `(?s:.)\Z`. Matching that +// means reproducing translate's REWRITING, not its matching: a +// bracket-by-bracket matcher cannot see it. +// +// Not fixed on purpose. It needs a reversed range AND a trailing `!` inside one +// class; ModelOpt emits no bracket expressions at all (`mtp*` is the real +// checkpoint's only wildcard), so nothing in the reachable input space can +// touch it, and the risk of rewriting a matcher to chase a `translate` quirk +// exceeds the risk of leaving it recorded here. +inline bool FnMatch(const std::string& name, const std::string& pat) { + std::size_t n = 0, p = 0, star = std::string::npos, retry = 0; + while (n < name.size()) { + if (p < pat.size() && pat[p] == '*') { + star = p++; + retry = n; + continue; + } + std::size_t next = p; + if (p < pat.size() && MatchOneToken(name[n], pat, p, &next)) { + ++n; + p = next; + continue; + } + if (star != std::string::npos) { + p = star + 1; + n = ++retry; + continue; + } + return false; + } + while (p < pat.size() && pat[p] == '*') ++p; + return p == pat.size(); +} + +} // namespace detail + +// ModelOpt MIXED_PRECISION quantization config: the `quantized_layers` map, the +// `ignore` list, and the resolved config-level group size. +class MixedPrecisionConfig { + public: + // SELECTION, not parsing: modelopt.py:2333-2339 override_quantization_method + // + :245-263 _extract_modelopt_quant_algo. It answers "is this config MINE to + // claim?", so it requires the config to NAME modelopt in `quant_method` + // before its quant_algo is read at all — that is how upstream tells a + // ModelOpt `quantization_config` apart from a compressed-tensors one sitting + // in the same field of the same config.json. `Parse` must NOT share this + // precondition; see the comment on it. + template + static bool IsMixedPrecision(const Json& cfg) { + if (!cfg.is_object()) return false; + return ExtractQuantAlgo(cfg) == "MIXED_PRECISION"; + } + + // modelopt.py:282-367 from_config + :2349-2410 _from_config. + // + // PARSING is deliberately NOT gated on `quant_method`. `from_config` never + // looks at it: it dispatches on the SHAPE (`"quantization" in config`, + // :283-318) and reads `quant_algo` out of whichever shape that was. The + // distinction is not academic — the driver checkpoint's own + // `hf_quant_config.json` + // ($CHECKPOINT_ROOT/nemotron-3.5-lightning-30b-nvfp4, repo + // nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 @29f2d174) has exactly + // two top-level keys, `producer` and `quantization`, and names no + // `quant_method` anywhere in the file. Borrowing the selection hook's + // precondition here refused that file outright while upstream parsed it and + // resolved all 5981 entries. + template + static MixedPrecisionConfig Parse(const Json& cfg) { + if (!cfg.is_object()) { + throw std::invalid_argument( + "modelopt: quantization config must be a JSON object"); + } + if (ShapeQuantAlgo(cfg) != "MIXED_PRECISION") { + throw std::invalid_argument( + "modelopt: not a MIXED_PRECISION config (quant_algo=\"" + + ShapeQuantAlgo(cfg) + "\")"); + } + + // from_config picks BOTH shapes apart: the nested `{"quantization": {...}}` + // of hf_quant_config.json and the flat compressed-tensors-style + // quantization_config of config.json (modelopt.py:283-318). + const bool nested = cfg.contains("quantization") && cfg["quantization"].is_object(); + const Json& section = nested ? cfg["quantization"] : cfg; + + MixedPrecisionConfig out; + + // exclude_modules: "exclude_modules" nested, "ignore" flat. + const char* exclude_key = nested ? "exclude_modules" : "ignore"; + if (section.contains(exclude_key)) { + const Json& ex = section[exclude_key]; + if (!ex.is_array()) { + throw std::invalid_argument(std::string("modelopt: ") + exclude_key + + " must be a list"); + } + for (const auto& e : ex) out.exclude_modules_.push_back(e.template get()); + } + + // kv cache: a plain algo string nested, a `kv_cache_scheme` dict flat + // (modelopt.py:294, :306-314). + if (nested) { + if (section.contains("kv_cache_quant_algo") && + section["kv_cache_quant_algo"].is_string()) { + out.kv_cache_quant_algo_ = + detail::Upper(section["kv_cache_quant_algo"].template get()); + } + } else if (section.contains("kv_cache_scheme") && + section["kv_cache_scheme"].is_object()) { + const Json& kv = section["kv_cache_scheme"]; + const bool is_fp8 = kv.contains("type") && kv["type"].is_string() && + kv["type"].template get() == "float" && + kv.contains("num_bits") && + kv["num_bits"].is_number_integer() && + kv["num_bits"].template get() == 8; + if (is_fp8) out.kv_cache_quant_algo_ = "FP8"; + } + + // quantized_layers lives beside quant_algo in whichever shape this is + // (modelopt.py:2357-2363). + if (!section.contains("quantized_layers") || + !section["quantized_layers"].is_object() || + section["quantized_layers"].empty()) { + throw std::invalid_argument( + "modelopt: MIXED_PRECISION quant_algo requires a non-empty " + "'quantized_layers' mapping in the quantization config"); + } + const Json& layers = section["quantized_layers"]; + out.entries_.reserve(layers.size()); + for (auto it = layers.begin(); it != layers.end(); ++it) { + const Json& info = it.value(); + if (!info.is_object() || !info.contains("quant_algo") || + !info["quant_algo"].is_string()) { + throw std::invalid_argument( + "modelopt: quantized_layers entry \"" + it.key() + + "\" has no string 'quant_algo'"); + } + Entry e; + e.name = it.key(); + e.algo = detail::Upper(info["quant_algo"].template get()); + if (info.contains("group_size") && info["group_size"].is_number_integer()) { + e.group_size = info["group_size"].template get(); + e.has_group_size = true; + } + out.index_.emplace(e.name, out.entries_.size()); + out.entries_.push_back(std::move(e)); + } + + // group_size: an explicit config-level value wins; otherwise it is SEEDED + // from the first NVFP4-family entry, defaulting to 16 (modelopt.py:2365- + // 2377). Upstream then builds ONE nvfp4 config from that single value, so + // it — not the per-entry field — is what every NVFP4 module gets. + if (section.contains("group_size") && !section["group_size"].is_null()) { + const Json& gs = section["group_size"]; + if (gs.is_number_integer()) { + out.group_size_ = gs.template get(); + } else if (gs.is_string()) { + try { + out.group_size_ = std::stoi(gs.template get()); + } catch (const std::exception&) { + throw std::invalid_argument("modelopt: group_size must be an integer"); + } + } else { + throw std::invalid_argument("modelopt: group_size must be an integer"); + } + } else { + out.group_size_ = 16; + for (const Entry& e : out.entries_) { + if (e.algo == "NVFP4" || e.algo == "W4A16_NVFP4") { + out.group_size_ = e.has_group_size ? e.group_size : 16; + break; + } + } + } + return out; + } + + // The owning model's fused-module mapping. Empty until a model registers one, + // which is exactly upstream's state for a model that declares none. + void SetPackedModulesMapping(PackedModulesMapping mapping) { + packed_modules_mapping_ = std::move(mapping); + } + const PackedModulesMapping& packed_modules_mapping() const { + return packed_modules_mapping_; + } + + int group_size() const { return group_size_; } + const std::string& kv_cache_quant_algo() const { return kv_cache_quant_algo_; } + std::size_t num_quantized_layers() const { return entries_.size(); } + const std::vector& exclude_modules() const { + return exclude_modules_; + } + + // What scheme does the module at `prefix` use? + // + // Order matters and mirrors get_quant_method (modelopt.py:2508-2524): the + // `ignore` list is consulted BEFORE `quantized_layers`, so an entry present in + // both is unquantized. Reversing that would quantize a layer the producer + // explicitly excluded. + // + // Throws std::invalid_argument when the shards of one fused module disagree + // (upstream ValueError), and std::runtime_error when the resolved algorithm + // is one this consumer does not implement (divergence 1 above). + ModuleQuant Resolve(const std::string& prefix) const { + if (IsLayerExcluded(prefix)) { + return ModuleQuant{QuantAlgo::kUnquantized, Resolution::kExcluded, 0}; + } + Resolution how = Resolution::kUnlisted; + const std::optional algo = ResolveQuantAlgoString(prefix, &how); + if (!algo.has_value()) { + return ModuleQuant{QuantAlgo::kUnquantized, Resolution::kUnlisted, 0}; + } + const QuantAlgo a = ToQuantAlgo(*algo, prefix); + const int gs = + (a == QuantAlgo::kNvfp4 || a == QuantAlgo::kW4A16Nvfp4) ? group_size_ : 0; + return ModuleQuant{a, how, gs}; + } + + // modelopt.py:145-181. Exact match (with fused unfusing), then the legacy + // substring rule kept for pre-0.39 ModelOpt exports, then wildcards. + bool IsLayerExcluded(const std::string& prefix) const { + if (exclude_modules_.empty()) return false; + if (IsLayerSkipped(prefix)) return true; + + static const std::string kLangPrefix = "language_model."; + for (const std::string& excluded : exclude_modules_) { + if (excluded == prefix) continue; // handled by the exact pass above + if (prefix.find(excluded) != std::string::npos) return true; + // The `language_model.` clause is mirrored verbatim from + // modelopt.py:170-172 but is provably redundant: `removeprefix` returns a + // SUFFIX of `prefix`, so anything found in the stripped string is already + // found in the full one and the line above has returned. It is kept so + // the two implementations read alike and a future upstream edit here is + // easy to follow — it cannot be covered by a test, because no input can + // reach it first. + if (detail::StartsWith(prefix, kLangPrefix) && + prefix.substr(kLangPrefix.size()).find(excluded) != std::string::npos) { + return true; + } + } + for (const std::string& pattern : exclude_modules_) { + if (detail::FnMatch(prefix, pattern)) return true; + } + return false; + } + + private: + struct Entry { + std::string name; + std::string algo; // stored UPPER-cased, as every upstream return does + int group_size = 0; + bool has_group_size = false; + }; + + template + static std::string StringField(const Json& cfg, const char* key) { + if (!cfg.contains(key) || !cfg[key].is_string()) return std::string(); + return cfg[key].template get(); + } + + // The `quant_algo` of whichever config SHAPE this is — the shape dispatch of + // from_config, modelopt.py:283-318, and like it blind to `quant_method`: + // nested `{"quantization": {...}}` for hf_quant_config.json, flat for a + // config.json `quantization_config`. + template + static std::string ShapeQuantAlgo(const Json& cfg) { + if (cfg.contains("quantization")) { + if (!cfg["quantization"].is_object()) return std::string(); + return detail::Upper(StringField(cfg["quantization"], "quant_algo")); + } + return detail::Upper(StringField(cfg, "quant_algo")); + } + + // modelopt.py:245-263 _extract_modelopt_quant_algo — the SELECTION hook, and + // the ONLY place the `quant_method` precondition belongs. + template + static std::string ExtractQuantAlgo(const Json& cfg) { + std::string method = StringField(cfg, "quant_method"); + for (char& c : method) { + if (c >= 'A' && c <= 'Z') c = static_cast(c - 'A' + 'a'); + } + if (!detail::StartsWith(method, "modelopt")) return std::string(); + return ShapeQuantAlgo(cfg); + } + + const Entry* Find(const std::string& name) const { + const auto it = index_.find(name); + return it == index_.end() ? nullptr : &entries_[it->second]; + } + + // modelopt.py:2489-2505 _quantized_layer_prefix_candidates, order-preserving + // and de-duplicated exactly as `dict.fromkeys` does. + static std::vector PrefixCandidates(const std::string& prefix) { + std::vector out; + out.push_back(prefix); + if (detail::EndsWith(prefix, ".lm_head")) out.push_back("lm_head"); + + static const std::string kA = "language_model.model."; + static const std::string kB = "model.language_model."; + if (detail::StartsWith(prefix, kA)) { + out.push_back(kB + prefix.substr(kA.size())); + } else if (detail::StartsWith(prefix, kB)) { + out.push_back(kA + prefix.substr(kB.size())); + } + + std::vector deduped; + for (const std::string& c : out) { + bool seen = false; + for (const std::string& d : deduped) seen = seen || d == c; + if (!seen) deduped.push_back(c); + } + return deduped; + } + + // quant_utils.py:510-572 is_layer_skipped, with skip_with_substr=False. + bool IsLayerSkipped(const std::string& prefix) const { + const std::string proj = detail::LastSegment(prefix); + const auto fused = packed_modules_mapping_.find(proj); + + if (fused != packed_modules_mapping_.end()) { + // A checkpoint may list the FUSED name directly; honour that first + // (quant_utils.py:540-541). + for (const std::string& e : exclude_modules_) { + if (e == prefix) return true; + } + bool have = false; + bool skipped = false; + for (const std::string& shard : fused->second) { + const std::string shard_prefix = detail::ReplaceAll(prefix, proj, shard); + bool shard_skipped = false; + for (const std::string& e : exclude_modules_) { + if (e == shard_prefix) { + shard_skipped = true; + break; + } + } + if (!have) { + have = true; + skipped = shard_skipped; + } else if (shard_skipped != skipped) { + throw std::invalid_argument( + "modelopt: detected some but not all shards of " + prefix + + " are quantized; all shards of a fused layer must have the same " + "precision"); + } + } + if (!have) { + throw std::invalid_argument( + "modelopt: packed_modules_mapping entry for \"" + proj + + "\" is empty, so the exclusion of " + prefix + " is undecidable"); + } + return skipped; + } + + // quant_utils.py:559-565. Note the direction, which is the opposite of + // every other rule here: `prefix in layer_name` — the IGNORE ENTRY must + // contain the PREFIX. ModelOpt lists experts one index at a time while a + // FusedMoE layer is ONE module spanning all of them, so naming any single + // expert child leaves the whole container unquantized. + if (prefix.find("experts") != std::string::npos) { + for (const std::string& e : exclude_modules_) { + // Upstream's `filter(lambda l: "experts" in l, ...)`, mirrored though + // provably redundant: the guard above says `prefix` contains "experts", + // and the next line only returns for an `e` that contains `prefix`, so + // any `e` that could return already contains "experts". Kept so the two + // implementations read alike; no input can make it change the answer. + if (e.find("experts") == std::string::npos) continue; + if (e.find(prefix) != std::string::npos) return true; + } + return false; + } + + for (const std::string& e : exclude_modules_) { + if (e == prefix) return true; + } + return false; + } + + // modelopt.py:2412-2487 _resolve_quant_algo, all five strategies in order. + std::optional ResolveQuantAlgoString(const std::string& prefix, + Resolution* how) const { + const std::vector candidates = PrefixCandidates(prefix); + + // 1. Direct lookup (:2424-2427). + for (const std::string& c : candidates) { + if (const Entry* e = Find(c)) { + *how = Resolution::kDirect; + return e->algo; + } + } + + const std::string proj = detail::LastSegment(prefix); + + // 2. Packed / fused lookup: unfuse via packed_modules_mapping (:2429-2447). + // Note the algo set spans ALL base candidates here — unlike strategy 5, + // which rebuilds it per candidate. Mirrored as written upstream. + const auto fused = packed_modules_mapping_.find(proj); + if (!packed_modules_mapping_.empty() && + fused != packed_modules_mapping_.end()) { + std::set algos; + const std::string base = detail::ParentSegment(prefix); + for (const std::string& bc : PrefixCandidates(base)) { + for (const std::string& shard : fused->second) { + if (const Entry* e = Find(bc + "." + shard)) algos.insert(e->algo); + } + } + if (algos.size() == 1) { + *how = Resolution::kPacked; + return *algos.begin(); + } + if (algos.size() > 1) throw MixedShardError(prefix, algos); + } + + // 3. Prefix lookup, for a parent module such as a routed-expert container + // (:2449-2453). Returns the FIRST child in map order. + for (const std::string& c : candidates) { + const std::string child_prefix = c + "."; + for (const Entry& e : entries_) { + if (detail::StartsWith(e.name, child_prefix)) { + *how = Resolution::kPrefix; + return e.algo; + } + } + } + + // 4. The FusedMoE ".experts" special case (:2455-2461): the layer prefix is + // "...moe.experts" while ModelOpt lists "...moe.up_proj" / "...moe.down_proj". + static const std::string kExperts = ".experts"; + if (detail::EndsWith(prefix, kExperts)) { + const std::string parent_prefix = + prefix.substr(0, prefix.size() - kExperts.size()) + "."; + for (const Entry& e : entries_) { + if (detail::StartsWith(e.name, parent_prefix)) { + *how = Resolution::kExpertsParent; + return e.algo; + } + } + } + + // 5. Fused-projection fallback for configs that list shard names where vLLM + // uses a packed name, with no packed_modules_mapping registered (:2463-2486). + static const std::map> kFusedShards = { + {"qkv_proj", {"q_proj", "k_proj", "v_proj"}}, + {"gate_up_proj", {"gate_proj", "up_proj"}}, + }; + const auto shards = kFusedShards.find(proj); + if (shards != kFusedShards.end()) { + for (const std::string& c : candidates) { + const std::string parent_prefix = detail::ParentSegment(c) + "."; + std::set algos; + for (const std::string& shard : shards->second) { + if (const Entry* e = Find(parent_prefix + shard)) algos.insert(e->algo); + } + if (algos.size() == 1) { + *how = Resolution::kFusedShards; + return *algos.begin(); + } + if (algos.size() > 1) throw MixedShardError(prefix, algos); + } + } + + return std::nullopt; + } + + static std::invalid_argument MixedShardError(const std::string& prefix, + const std::set& algos) { + std::string joined; + for (const std::string& a : algos) { + if (!joined.empty()) joined += ", "; + joined += a; + } + return std::invalid_argument("modelopt: mixed quant_algo within fused layer " + + prefix + ": {" + joined + + "}. All shards must use the same quantization."); + } + + // Divergence 1: refuse by name rather than fall through to unquantized. + static QuantAlgo ToQuantAlgo(const std::string& algo, const std::string& prefix) { + if (algo == "FP8") return QuantAlgo::kFp8; + if (algo == "NVFP4") return QuantAlgo::kNvfp4; + if (algo == "W4A16_NVFP4") return QuantAlgo::kW4A16Nvfp4; + if (algo == "MXFP8") return QuantAlgo::kMxfp8; + + // These ARE ModelOpt algorithms (modelopt.py:105-120) — they are simply not + // implemented by this consumer, and the distinction is worth saying out loud + // in the message so a reader knows whether to port a scheme or distrust the + // checkpoint. + const bool known = algo == "FP8_PER_CHANNEL_PER_TOKEN" || algo == "FP8_PB_WO" || + algo == "MIXED_PRECISION"; + throw std::runtime_error( + std::string("modelopt MIXED_PRECISION: ") + + (known ? "quant_algo \"" + algo + + "\" is a recognized ModelOpt algorithm that is not " + "implemented here" + : "unknown quant_algo \"" + algo + "\"") + + " for module \"" + prefix + + "\". Refusing rather than loading it unquantized: a silent dequantization " + "is numerically correct and therefore invisible to a token gate."); + } + + std::vector entries_; // map order preserved + std::unordered_map index_; + std::vector exclude_modules_; + PackedModulesMapping packed_modules_mapping_; + std::string kv_cache_quant_algo_; + int group_size_ = 16; +}; + +} // namespace modelopt +} // namespace layers +} // namespace vllm diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt index 77d4fda9a..d980d5eb3 100644 --- a/tests/CMakeLists.txt +++ b/tests/CMakeLists.txt @@ -55,6 +55,31 @@ vllm_cpp_add_test(test_nvfp4_dequant vllm/test_nvfp4_dequant.cpp) vllm_cpp_add_test(test_awq_gptq_dequant vllm/test_awq_gptq_dequant.cpp) vllm_cpp_add_test(test_mxfp4_dequant vllm/test_mxfp4_dequant.cpp) vllm_cpp_add_test(test_ct_nvfp4_emulation vllm/test_ct_nvfp4_emulation.cpp) +# MODEL-TEXT-nemotron-h W1 (#517): the ModelOpt MIXED_PRECISION per-module +# quant-algo resolver. Two arms, deliberately two BINARIES: the curated fixture +# arm is always-on, and the exhaustive real-config arm exits 77 (Skipped) when +# the 30B checkpoint is not staged. Folding them into one target would let the +# skip swallow the always-on gate as well. +# +# The resolver header is INTERNAL (src/, not include/) until a loader consumes +# it, so both targets take ${CMAKE_SOURCE_DIR}/src on the include path — the +# same arrangement test_linear_method and test_model_registry already use. +vllm_cpp_add_test(test_modelopt_mixed_precision + vllm/model_executor/layers/quantization/test_modelopt_mixed_precision.cpp) +target_include_directories(test_modelopt_mixed_precision PRIVATE + ${CMAKE_SOURCE_DIR}/src) +target_compile_definitions(test_modelopt_mixed_precision PRIVATE + MODELOPT_MIXED_FIXTURE_DIR="${CMAKE_SOURCE_DIR}/tests/fixtures/modelopt_mixed_precision") +vllm_cpp_add_test(test_modelopt_mixed_precision_checkpoint + vllm/model_executor/layers/quantization/test_modelopt_mixed_precision_checkpoint.cpp) +# tests/parity as well as src/: the exhaustive arm resolves its checkpoint +# through the SINGLE pinned parity::Nemotron35LightningSnapshot() rather than +# joining $CHECKPOINT_ROOT by hand, so that one resolver owns the revision pin +# for both env spellings (#517 LOW-3) -- the same arrangement test_qwen36_weights +# uses for the 35B pin. +target_include_directories(test_modelopt_mixed_precision_checkpoint PRIVATE + ${CMAKE_SOURCE_DIR}/src + ${CMAKE_SOURCE_DIR}/tests/parity) vllm_cpp_add_test(test_gguf_dequant vllm/test_gguf_dequant.cpp) vllm_cpp_add_test(test_gguf_nvfp4 vllm/test_gguf_nvfp4.cpp) # gguf_nvfp4_goldens.inc lives next to the test source. diff --git a/tests/fixtures/modelopt_mixed_precision/curated_config.json b/tests/fixtures/modelopt_mixed_precision/curated_config.json new file mode 100644 index 000000000..92f058ee3 --- /dev/null +++ b/tests/fixtures/modelopt_mixed_precision/curated_config.json @@ -0,0 +1,107 @@ +{ + "_derivation": { + "purpose": "Curated ModelOpt MIXED_PRECISION fixture for the per-module quant-algo resolver (row MODEL-TEXT-nemotron-h-nemotron-hfor-causal-lm W1, issue #517).", + "derived_from": { + "checkpoint": "nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4", + "revision": "29f2d1746d8f41e316523194b19018707749b1b1", + "file": "config.json -> quantization_config", + "real_shape": "5981 quantized_layers (5935 W4A16_NVFP4 group_size 16, 46 FP8), 72 ignore entries, kv_cache_scheme fp8" + }, + "real_entries_copied_verbatim": [ + "producer / quant_method / quant_algo / kv_cache_scheme are the real values", + "backbone.layers.0.mixer.in_proj -> FP8 (real, mamba in_proj)", + "backbone.layers.0.mixer.out_proj -> FP8 (real, mamba out_proj)", + "backbone.layers.1.mixer.experts.{0,1}.{up,down}_proj -> W4A16_NVFP4 g16 (real; the real map has 128 experts per MoE layer, 2 are kept here)", + "backbone.layers.1.mixer.shared_experts.{up,down}_proj -> W4A16_NVFP4 g16 (real)", + "lm_head -> W4A16_NVFP4 g16 (real, and really a BARE key with no 'model.' prefix)", + "ignore: backbone.embeddings, backbone.layers.0.mixer.conv1d, backbone.layers.1.mixer.gate, backbone.layers.{5,12}.mixer.{q,k,v,o}_proj, mtp* (all real, verbatim)" + ], + "synthetic_entries_and_why": [ + "The real checkpoint resolves ALL 5981 entries by the DIRECT strategy alone (verified: strategy histogram {direct: 5981}). It therefore cannot exercise the packed/fused, .experts-special-case, or fused_projection_shards strategies, nor a shard disagreement, nor an unknown algo. Those five paths exist in the pinned upstream (modelopt.py:2412-2487) and each gets a synthetic 'synthetic.*' entry below.", + "synthetic.layers.0.moe.{up,down}_proj -> the '.experts' special case: ModelOpt lists '...moe.up_proj' while a FusedMoE layer's prefix is '...moe.experts' (upstream comment at modelopt.py:2455-2457).", + "synthetic.layers.1.self_attn.{q,k,v}_proj (all FP8) -> packed_modules_mapping strategy 2 agreement.", + "synthetic.layers.2.self_attn.{q,k,v}_proj (q,v FP8 / k W4A16_NVFP4) -> packed strategy 2 DISAGREEMENT, which upstream RAISES on.", + "synthetic.layers.3.mlp.{gate,up}_proj (both FP8) -> fused_projection_shards fallback (strategy 5) agreement, reached with NO packed mapping registered.", + "synthetic.layers.4.mlp.{gate,up}_proj (gate FP8 / up W4A16_NVFP4) -> fused_projection_shards DISAGREEMENT, which upstream RAISES on.", + "synthetic.layers.5.mixer.in_proj -> quant_algo 'AWQ_LITE', an algorithm ModelOpt never emits: the resolver must REFUSE IT BY NAME rather than fall through to a supported path or to unquantized.", + "synthetic.layers.6.mixer.in_proj -> quant_algo 'FP8_PB_WO', a REAL ModelOpt algo name (upstream QUANT_ALGOS, modelopt.py:105-120) that the MIXED_PRECISION consumer has no branch for. Upstream silently returns UnquantizedLinearMethod for it; we refuse by name (spec stop condition: a silent fallback is invisible to a token gate).", + "language_model.model.layers.0.mlp.down_proj AND model.language_model.layers.9.mlp.up_proj -> the language_model/model prefix SWAP in _quantized_layer_prefix_candidates (modelopt.py:2491-2505), one entry per DIRECTION. One direction alone leaves the other branch untested (my own mutation run proved it).", + "ignore: legacy_substr.mixer -> the pre-ModelOpt-0.39 SUBSTRING exclusion rule (modelopt.py:165-174), which no exact match and no wildcard can reach.", + "ignore: synthetic.layers.7.self_attn.q_proj ONLY (k/v absent) -> is_layer_skipped's partial-fused-exclusion RAISE (quant_utils.py:549-556).", + "synthetic.layers.8.moe.{a,b}_proj (a FP8 / b W4A16_NVFP4) -> strategies 3 and 4 return the FIRST matching child in map order (modelopt.py:2450, :2458). A parent whose children AGREE cannot tell first from last: synthetic.layers.2.self_attn is q/k/v = FP8/W4A16/FP8, so its first and its last child are both FP8 and 'return the last match' survives it. These two disagree AND sort the same way under insertion and lexicographic order (a < b), so they isolate first-vs-last from the ordered_json question that synthetic.layers.2.self_attn pins.", + "language_model.model.layers.5.self_attn.q_proj (FP8) AND model.language_model.layers.5.self_attn.k_proj (W4A16_NVFP4) -> the strategy-2 / strategy-5 ASYMMETRY (modelopt.py:2429-2447 vs :2463-2486). Strategy 2 accumulates ONE algo set across ALL base prefix candidates; strategy 5 rebuilds it per candidate. Every other fused fixture entry has a single prefix candidate, where the two are indistinguishable. Split across the two language_model spellings, the union is {FP8, W4A16_NVFP4} and RAISES, while a per-candidate rebuild would return FP8 from the first spelling and never see the second." + ] + }, + "architectures": ["NemotronHForCausalLM"], + "model_type": "nemotron_h", + "quantization_config": { + "producer": "modelopt 0.44.0rc5", + "quant_method": "modelopt", + "quant_algo": "MIXED_PRECISION", + "kv_cache_scheme": { + "dynamic": false, + "num_bits": 8, + "type": "float" + }, + "ignore": [ + "backbone.embeddings", + "backbone.layers.0.mixer.conv1d", + "backbone.layers.1.mixer.gate", + "backbone.layers.5.mixer.q_proj", + "backbone.layers.5.mixer.k_proj", + "backbone.layers.5.mixer.v_proj", + "backbone.layers.5.mixer.o_proj", + "backbone.layers.12.mixer.q_proj", + "backbone.layers.12.mixer.k_proj", + "backbone.layers.12.mixer.v_proj", + "backbone.layers.12.mixer.o_proj", + "mtp*", + "synthetic.layers.7.self_attn.q_proj", + "legacy_substr.mixer" + ], + "quantized_layers": { + "backbone.layers.0.mixer.in_proj": { "quant_algo": "FP8" }, + "backbone.layers.0.mixer.out_proj": { "quant_algo": "FP8" }, + "backbone.layers.1.mixer.experts.0.up_proj": { "quant_algo": "W4A16_NVFP4", "group_size": 16 }, + "backbone.layers.1.mixer.experts.0.down_proj": { "quant_algo": "W4A16_NVFP4", "group_size": 16 }, + "backbone.layers.1.mixer.experts.1.up_proj": { "quant_algo": "W4A16_NVFP4", "group_size": 16 }, + "backbone.layers.1.mixer.experts.1.down_proj": { "quant_algo": "W4A16_NVFP4", "group_size": 16 }, + "backbone.layers.1.mixer.shared_experts.up_proj": { "quant_algo": "W4A16_NVFP4", "group_size": 16 }, + "backbone.layers.1.mixer.shared_experts.down_proj": { "quant_algo": "W4A16_NVFP4", "group_size": 16 }, + "lm_head": { "quant_algo": "W4A16_NVFP4", "group_size": 16 }, + + "synthetic.layers.0.moe.up_proj": { "quant_algo": "W4A16_NVFP4", "group_size": 16 }, + "synthetic.layers.0.moe.down_proj": { "quant_algo": "W4A16_NVFP4", "group_size": 16 }, + + "synthetic.layers.1.self_attn.q_proj": { "quant_algo": "FP8" }, + "synthetic.layers.1.self_attn.k_proj": { "quant_algo": "FP8" }, + "synthetic.layers.1.self_attn.v_proj": { "quant_algo": "FP8" }, + + "synthetic.layers.2.self_attn.q_proj": { "quant_algo": "FP8" }, + "synthetic.layers.2.self_attn.k_proj": { "quant_algo": "W4A16_NVFP4", "group_size": 16 }, + "synthetic.layers.2.self_attn.v_proj": { "quant_algo": "FP8" }, + + "synthetic.layers.3.mlp.gate_proj": { "quant_algo": "FP8" }, + "synthetic.layers.3.mlp.up_proj": { "quant_algo": "FP8" }, + + "synthetic.layers.4.mlp.gate_proj": { "quant_algo": "FP8" }, + "synthetic.layers.4.mlp.up_proj": { "quant_algo": "W4A16_NVFP4", "group_size": 16 }, + + "synthetic.layers.5.mixer.in_proj": { "quant_algo": "AWQ_LITE" }, + "synthetic.layers.6.mixer.in_proj": { "quant_algo": "FP8_PB_WO" }, + + "synthetic.layers.7.self_attn.q_proj": { "quant_algo": "FP8" }, + "synthetic.layers.7.self_attn.k_proj": { "quant_algo": "FP8" }, + "synthetic.layers.7.self_attn.v_proj": { "quant_algo": "FP8" }, + + "language_model.model.layers.0.mlp.down_proj": { "quant_algo": "FP8" }, + "model.language_model.layers.9.mlp.up_proj": { "quant_algo": "FP8" }, + + "synthetic.layers.8.moe.a_proj": { "quant_algo": "FP8" }, + "synthetic.layers.8.moe.b_proj": { "quant_algo": "W4A16_NVFP4", "group_size": 16 }, + + "language_model.model.layers.5.self_attn.q_proj": { "quant_algo": "FP8" }, + "model.language_model.layers.5.self_attn.k_proj": { "quant_algo": "W4A16_NVFP4", "group_size": 16 } + } + } +} diff --git a/tests/parity/hf_snapshot.h b/tests/parity/hf_snapshot.h index 82f076b8b..c5478bc73 100644 --- a/tests/parity/hf_snapshot.h +++ b/tests/parity/hf_snapshot.h @@ -41,6 +41,13 @@ inline constexpr const char* kQwen27NvfP4Revision = inline constexpr const char* kNemotron35LightningNvfP4Revision = "29f2d1746d8f41e316523194b19018707749b1b1"; +// The directory this checkpoint is staged under inside `$CHECKPOINT_ROOT`. It +// is a `hf download --local-dir` tree, not an HF cache repo, so the revision +// does not appear in the PATH the way `snapshots//` does -- see +// Nemotron35LightningSnapshot below for where it does appear. +inline constexpr const char* kNemotron35LightningLocalDirName = + "nemotron-3.5-lightning-30b-nvfp4"; + // Snapshot directory for `` at `revision`, or "" when it is not cached // (the caller then emits its loud SKIP). `env_override`, when set and non-empty, // names an explicit snapshot directory for a deliberate different-checkpoint @@ -65,14 +72,59 @@ inline std::string HfSnapshot(const char* repo_dir, const char* revision, return snap.string(); } -// The Nemotron-3.5-Lightning gate model (#517). Unlike the Qwen pins above, -// this one is NOT in the HF cache: it is staged on the NAS as a `local_dir` -// snapshot at `$CHECKPOINT_ROOT/nemotron-3.5-lightning-30b-nvfp4`, so there is -// no `models--org--name/snapshots/` layout to resolve. The env override is -// therefore the ONLY reachable path, and the cache spelling below exists so the -// revision still names what the golden belongs to. Absent env var => "" => the -// caller emits its loud SKIP, which is the intended behavior off the gate host. +// The Nemotron-3.5-Lightning gate model (#517), and the ONE resolver for it. +// +// Unlike the Qwen pins above, this one is NOT in the HF cache: it is staged on +// the NAS as a `hf download --local-dir` tree at +// `$CHECKPOINT_ROOT/nemotron-3.5-lightning-30b-nvfp4`, so there is no +// `models--org--name/snapshots/` directory whose NAME carries the revision. +// +// LOW-3 (#517). Two env vars used to reach this same checkpoint -- +// `VT_NEMOTRON35_SNAPSHOT` here and `CHECKPOINT_ROOT` in +// test_modelopt_mixed_precision_checkpoint.cpp -- and NEITHER enforced the +// revision: the cache spelling above is unreachable for a `local_dir` tree, and +// an env override is deliberately not revision-checked. So the pin named the +// revision the goldens belong to and could not refuse a different one, which is +// the exact failure `kQwen27NvfP4Revision` exists because of. Both spellings now +// resolve HERE, and the `local_dir` layout does record its revision, just not in +// the path: `hf download --local-dir` writes a per-revision file manifest at +// `/.cache/huggingface/trees/.json`. That is what is checked. +// +// Resolution order, in the order the code checks it, and why: +// +// 1. `VT_NEMOTRON35_SNAPSHOT`, when set and non-empty -> the explicit-directory +// escape `HfSnapshot` documents, with its semantics UNCHANGED, including +// that a set-but-wrong override refuses rather than falling back. First, so +// that setting it OVERRIDES `CHECKPOINT_ROOT` rather than racing it. +// 2. Otherwise `CHECKPOINT_ROOT` -> `/`, +// and the revision manifest MUST be present. This is the DEFAULT path every +// gate takes, so it is the one that has to carry the pin: a re-download of +// the same repo name lands a different revision under the identical path, +// and a gate that cannot tell would substitute it silently. Missing +// manifest => "" => the caller's loud skip, never a substitution. +// 3. Otherwise the ordinary HF cache layout, for a host that fetched it that +// way. +// +// Only (2) is revision-gated, deliberately: naming ONE directory outright is +// the deliberate different-checkpoint run the override exists for, while naming +// a ROOT is not. +// +// Absent both env vars => "" => the caller emits its loud SKIP, which is the +// intended behavior off the gate host. inline std::string Nemotron35LightningSnapshot() { + namespace fs = std::filesystem; + std::error_code ec; + const char* over = std::getenv("VT_NEMOTRON35_SNAPSHOT"); + const char* root = std::getenv("CHECKPOINT_ROOT"); + if ((over == nullptr || *over == '\0') && root != nullptr && *root != '\0') { + const fs::path dir = fs::path(root) / kNemotron35LightningLocalDirName; + const fs::path tree = dir / ".cache/huggingface/trees" / + (std::string(kNemotron35LightningNvfP4Revision) + + ".json"); + if (!fs::exists(dir / "config.json", ec)) return ""; + if (!fs::exists(tree, ec)) return ""; + return dir.string(); + } return HfSnapshot("models--nvidia--NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4", kNemotron35LightningNvfP4Revision, "VT_NEMOTRON35_SNAPSHOT"); diff --git a/tests/vllm/model_executor/layers/quantization/test_modelopt_mixed_precision.cpp b/tests/vllm/model_executor/layers/quantization/test_modelopt_mixed_precision.cpp new file mode 100644 index 000000000..30c4f354d --- /dev/null +++ b/tests/vllm/model_executor/layers/quantization/test_modelopt_mixed_precision.cpp @@ -0,0 +1,557 @@ +// CPU unit tests for the ModelOpt MIXED_PRECISION per-module quant-algo +// resolver (modelopt_mixed_precision.h). No weights, no GPU, no oracle +// process: the whole capability is config parsing plus string resolution, so +// the always-on gate runs anywhere. +// +// Row MODEL-TEXT-nemotron-h-nemotron-hfor-causal-lm W1, issue #517, spec +// .agents/specs/nemotron-h-model.md. +// +// UPSTREAM (ported FROM) @ 5559679229bc961848b121ccdeaa8fa5d79bec98: +// vllm/model_executor/layers/quantization/modelopt.py:2279-2410 +// ModelOptMixedPrecisionConfig + _from_config +// vllm/model_executor/layers/quantization/modelopt.py:2412-2487 +// _resolve_quant_algo — the FIVE strategies, in order +// vllm/model_executor/layers/quantization/modelopt.py:2491-2505 +// _quantized_layer_prefix_candidates +// vllm/model_executor/layers/quantization/modelopt.py:145-181 +// ModelOptQuantConfigBase.is_layer_excluded +// vllm/model_executor/layers/quantization/utils/quant_utils.py:510-572 +// is_layer_skipped +// +// Every expectation below was cross-checked against a line-by-line Python +// transcription of those upstream functions, executed on the REAL 1.3 MB +// config.json of nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 @29f2d174. +#include + +#include +#include +#include + +#include + +#include "vllm/model_executor/layers/quantization/modelopt_mixed_precision.h" + +using vllm::layers::modelopt::MixedPrecisionConfig; +using vllm::layers::modelopt::ModuleQuant; +using vllm::layers::modelopt::PackedModulesMapping; +using vllm::layers::modelopt::QuantAlgo; +using vllm::layers::modelopt::Resolution; + +namespace { + +std::string FixturePath() { + return std::string(MODELOPT_MIXED_FIXTURE_DIR) + "/curated_config.json"; +} + +// ordered_json, not json: upstream iterates a Python dict, whose order is +// INSERTION order. nlohmann::json sorts object keys lexicographically, which +// would silently change which entry "the first NVFP4-family entry" (group_size +// seeding, modelopt.py:2360-2372) and the prefix scans (:2450, :2458) return. +nlohmann::ordered_json LoadFixture() { + std::ifstream f(FixturePath()); + REQUIRE_MESSAGE(f.good(), "cannot open fixture: ", FixturePath()); + nlohmann::ordered_json doc = nlohmann::ordered_json::parse(f); + return doc.at("quantization_config"); +} + +MixedPrecisionConfig Curated() { + return MixedPrecisionConfig::Parse(LoadFixture()); +} + +// vLLM registers packed_modules_mapping per MODEL. Strategy 2 is dead without +// one, so the packed cases install the standard attention/MLP mapping. +PackedModulesMapping StandardPacked() { + return PackedModulesMapping{ + {"qkv_proj", {"q_proj", "k_proj", "v_proj"}}, + {"gate_up_proj", {"gate_proj", "up_proj"}}, + }; +} + +} // namespace + +// --- detection + parse (modelopt.py:2330-2410) ----------------------------- + +TEST_CASE("modelopt-mixed: MIXED_PRECISION is detected, other algos are not") { + const nlohmann::ordered_json qc = LoadFixture(); + CHECK(MixedPrecisionConfig::IsMixedPrecision(qc)); + + nlohmann::ordered_json plain = qc; + plain["quant_algo"] = "NVFP4"; + CHECK_FALSE(MixedPrecisionConfig::IsMixedPrecision(plain)); + + nlohmann::ordered_json other_vendor = qc; + other_vendor["quant_method"] = "compressed-tensors"; + CHECK_FALSE(MixedPrecisionConfig::IsMixedPrecision(other_vendor)); + + // modelopt.py:256 lower-cases and uses startswith("modelopt"). + nlohmann::ordered_json cased = qc; + cased["quant_method"] = "ModelOpt"; + CHECK(MixedPrecisionConfig::IsMixedPrecision(cased)); + + // modelopt.py:2340-2347 also accepts the legacy nested shape. Built with the + // REAL top-level key set of an hf_quant_config.json — {producer, + // quantization} — which names NO quant_method anywhere in the file. + nlohmann::ordered_json nested; + nested["producer"] = nlohmann::ordered_json::object(); + nested["producer"]["name"] = "modelopt"; + nested["quantization"] = nlohmann::ordered_json::object(); + nested["quantization"]["quant_algo"] = "MIXED_PRECISION"; + nested["quantization"]["quantized_layers"] = qc.at("quantized_layers"); + + // PARSING mirrors from_config (modelopt.py:282-367), which dispatches on the + // SHAPE and never reads quant_method. Gating Parse on quant_method — the + // precondition that belongs to the SELECTION hook :245-263 — refused the + // driver checkpoint's own hf_quant_config.json outright. + CHECK(MixedPrecisionConfig::Parse(nested).num_quantized_layers() == + qc.at("quantized_layers").size()); + // DETECTION does keep it: _extract_modelopt_quant_algo returns None for a + // config that does not name modelopt, so the override hook must not claim it. + CHECK_FALSE(MixedPrecisionConfig::IsMixedPrecision(nested)); + nested["quant_method"] = "modelopt"; + CHECK(MixedPrecisionConfig::IsMixedPrecision(nested)); + CHECK(MixedPrecisionConfig::Parse(nested).num_quantized_layers() == + qc.at("quantized_layers").size()); +} + +TEST_CASE("modelopt-mixed: parse reads the fixture's shape") { + const MixedPrecisionConfig c = Curated(); + CHECK(c.num_quantized_layers() == 32); + CHECK(c.exclude_modules().size() == 14); + // modelopt.py:2360-2372: no top-level group_size, so it is SEEDED from the + // first NVFP4-family entry. + CHECK(c.group_size() == 16); + // kv_cache_scheme {type:"float", num_bits:8} -> "FP8" (modelopt.py:2306-2314) + CHECK(c.kv_cache_quant_algo() == "FP8"); +} + +TEST_CASE("modelopt-mixed: an empty quantized_layers map is refused") { + nlohmann::ordered_json qc = LoadFixture(); + qc["quantized_layers"] = nlohmann::ordered_json::object(); + CHECK_THROWS_AS(MixedPrecisionConfig::Parse(qc), std::invalid_argument); + qc.erase("quantized_layers"); + CHECK_THROWS_AS(MixedPrecisionConfig::Parse(qc), std::invalid_argument); +} + +// --- strategy 1: direct lookup (modelopt.py:2424-2427) --------------------- + +TEST_CASE("modelopt-mixed: strategy 1 direct lookup, both real algos") { + const MixedPrecisionConfig c = Curated(); + + const ModuleQuant in_proj = c.Resolve("backbone.layers.0.mixer.in_proj"); + CHECK(in_proj.algo == QuantAlgo::kFp8); + CHECK(in_proj.how == Resolution::kDirect); + CHECK(in_proj.group_size == 0); // FP8 is not group-quantized + + const ModuleQuant expert = c.Resolve("backbone.layers.1.mixer.experts.0.up_proj"); + CHECK(expert.algo == QuantAlgo::kW4A16Nvfp4); + CHECK(expert.how == Resolution::kDirect); + CHECK(expert.group_size == 16); + + // Not listed anywhere and not ignored -> unquantized, NOT an error. + const ModuleQuant absent = c.Resolve("backbone.layers.3.mixer.nonexistent"); + CHECK(absent.algo == QuantAlgo::kUnquantized); + CHECK(absent.how == Resolution::kUnlisted); + CHECK_FALSE(absent.Quantized()); +} + +TEST_CASE("modelopt-mixed: prefix candidates — bare lm_head and the lm swap") { + const MixedPrecisionConfig c = Curated(); + + // The real map stores a BARE "lm_head"; our loader's prefix is "model.lm_head". + CHECK(c.Resolve("lm_head").algo == QuantAlgo::kW4A16Nvfp4); + const ModuleQuant head = c.Resolve("model.lm_head"); + CHECK(head.algo == QuantAlgo::kW4A16Nvfp4); + CHECK(head.how == Resolution::kDirect); + CHECK(head.group_size == 16); + // The suffix rule is ".lm_head", so a name merely CONTAINING it must miss. + CHECK(c.Resolve("model.not_lm_head").algo == QuantAlgo::kUnquantized); + + // modelopt.py:2496-2503, both directions of the language_model swap. Each + // direction needs its OWN map entry: with only one, whichever branch is not + // exercised can be deleted with the suite still green (my mutation run + // proved exactly that, so the second entry is here because of it). + CHECK(c.Resolve("language_model.model.layers.0.mlp.down_proj").algo == + QuantAlgo::kFp8); // direct hit, no swap needed + CHECK(c.Resolve("model.language_model.layers.0.mlp.down_proj").how == + Resolution::kDirect); // model.language_model -> language_model.model + CHECK(c.Resolve("model.language_model.layers.9.mlp.up_proj").algo == + QuantAlgo::kFp8); // direct hit the other way round + CHECK(c.Resolve("language_model.model.layers.9.mlp.up_proj").how == + Resolution::kDirect); // language_model.model -> model.language_model + CHECK(c.Resolve("language_model.model.layers.9.mlp.up_proj").algo == + QuantAlgo::kFp8); +} + +// --- strategy 2: packed / fused lookup (modelopt.py:2429-2447) ------------- + +TEST_CASE("modelopt-mixed: strategy 2 packed lookup unfuses qkv_proj") { + MixedPrecisionConfig c = Curated(); + c.SetPackedModulesMapping(StandardPacked()); + + const ModuleQuant qkv = c.Resolve("synthetic.layers.1.self_attn.qkv_proj"); + CHECK(qkv.algo == QuantAlgo::kFp8); + CHECK(qkv.how == Resolution::kPacked); + + // Without the mapping registered, strategy 2 cannot fire; strategies 3 and 4 + // do not apply to qkv_proj either, so strategy 5 catches it instead. + const MixedPrecisionConfig unmapped = Curated(); + const ModuleQuant fallback = + unmapped.Resolve("synthetic.layers.1.self_attn.qkv_proj"); + CHECK(fallback.algo == QuantAlgo::kFp8); + CHECK(fallback.how == Resolution::kFusedShards); +} + +TEST_CASE("modelopt-mixed: strategy 2 RAISES when fused shards disagree") { + MixedPrecisionConfig c = Curated(); + c.SetPackedModulesMapping(StandardPacked()); + + // q,v are FP8 and k is W4A16_NVFP4. Upstream raises ValueError rather than + // picking one (modelopt.py:2444-2447). Returning ANY value here is the + // silent-wrong-bytes failure a token gate cannot see. + CHECK_THROWS_AS(c.Resolve("synthetic.layers.2.self_attn.qkv_proj"), + std::invalid_argument); + CHECK_THROWS_WITH_AS(c.Resolve("synthetic.layers.2.self_attn.qkv_proj"), + doctest::Contains("synthetic.layers.2.self_attn.qkv_proj"), + std::invalid_argument); +} + +TEST_CASE("modelopt-mixed: strategy 2's algo set spans ALL base candidates") { + MixedPrecisionConfig c = Curated(); + c.SetPackedModulesMapping(StandardPacked()); + + // Strategy 2 (modelopt.py:2429-2447) builds ONE algo set over every base + // prefix candidate and decides once. Strategy 5 (:2463-2486) rebuilds the set + // per candidate and returns on the first candidate that yields exactly one. + // The code carries a comment saying so, and every other fused entry in this + // fixture has a SINGLE prefix candidate — where the two are indistinguishable, + // so flattening strategy 2 into strategy 5's shape survives them all. + // + // Here the shards are split across the two `language_model` spellings + // (:2496-2503): q_proj=FP8 under "language_model.model.", k_proj=W4A16_NVFP4 + // under "model.language_model.". The union is {FP8, W4A16_NVFP4} and must + // RAISE. Rebuilding per candidate would see {FP8} on the first spelling, + // return it, and never look at the second — a fused layer half loaded as 4-bit + // and half as 8-bit, silently. + CHECK_THROWS_AS(c.Resolve("language_model.model.layers.5.self_attn.qkv_proj"), + std::invalid_argument); + CHECK_THROWS_WITH_AS( + c.Resolve("language_model.model.layers.5.self_attn.qkv_proj"), + doctest::Contains("W4A16_NVFP4"), std::invalid_argument); + // ...and from the other spelling, where the candidate ORDER is reversed. + CHECK_THROWS_AS(c.Resolve("model.language_model.layers.5.self_attn.qkv_proj"), + std::invalid_argument); +} + +// --- strategy 3: prefix lookup (modelopt.py:2449-2453) --------------------- + +TEST_CASE("modelopt-mixed: strategy 3 resolves a parent module by prefix") { + const MixedPrecisionConfig c = Curated(); + + // The routed-expert container: children are "...experts..". + const ModuleQuant experts = c.Resolve("backbone.layers.1.mixer.experts"); + CHECK(experts.algo == QuantAlgo::kW4A16Nvfp4); + CHECK(experts.how == Resolution::kPrefix); + CHECK(experts.group_size == 16); + + const ModuleQuant shared = c.Resolve("backbone.layers.1.mixer.shared_experts"); + CHECK(shared.algo == QuantAlgo::kW4A16Nvfp4); + CHECK(shared.how == Resolution::kPrefix); + + // The scan is on `prefix + "."`, so a bare string prefix must NOT match. + CHECK(c.Resolve("backbone.layers.1.mixer.experts.").algo == + QuantAlgo::kUnquantized); + CHECK(c.Resolve("backbone.layers.1.mixer.expert").algo == + QuantAlgo::kUnquantized); +} + +TEST_CASE("modelopt-mixed: strategy 3 returns the FIRST child, not the last") { + const MixedPrecisionConfig c = Curated(); + + // modelopt.py:2449-2453 iterates `quantized_layers` and returns on the first + // key that startswith(candidate + "."). Every OTHER parent in this fixture + // has children that agree, or whose first and last child agree + // (synthetic.layers.2.self_attn is q/k/v = FP8/W4A16/FP8), so "return the + // LAST match" survives them. synthetic.layers.8.moe is a_proj=FP8 then + // b_proj=W4A16_NVFP4: first and last differ. + const ModuleQuant moe = c.Resolve("synthetic.layers.8.moe"); + CHECK(moe.algo == QuantAlgo::kFp8); + CHECK(moe.how == Resolution::kPrefix); + CHECK(moe.group_size == 0); + + // Strategy 4 (:2455-2461) scans the same way and owes the same guarantee. + const ModuleQuant experts = c.Resolve("synthetic.layers.8.moe.experts"); + CHECK(experts.algo == QuantAlgo::kFp8); + CHECK(experts.how == Resolution::kExpertsParent); + CHECK(experts.group_size == 0); +} + +TEST_CASE("modelopt-mixed: INSERTION order, not lexicographic, decides the scans") { + // Divergence 3 in the header: `Parse` is templated so callers can hand it + // `ordered_json`. Upstream iterates a Python dict, i.e. INSERTION order; + // plain `nlohmann::json` sorts object keys lexicographically. That is not a + // stylistic preference — it changes the ANSWER, and this case is what pins + // it. synthetic.layers.2.self_attn is inserted q_proj(FP8), k_proj(W4A16), + // v_proj(FP8); sorted, k_proj comes FIRST. So a plain-`json` load flips both + // the strategy-3 and the strategy-4 result from FP8 to W4A16_NVFP4 — a wrong + // scheme on a real module, which is exactly the invisible-to-a-token-gate + // failure this row exists for. + const MixedPrecisionConfig c = Curated(); + + const ModuleQuant parent = c.Resolve("synthetic.layers.2.self_attn"); + CHECK(parent.algo == QuantAlgo::kFp8); + CHECK(parent.how == Resolution::kPrefix); + CHECK(parent.group_size == 0); + + const ModuleQuant experts = c.Resolve("synthetic.layers.2.self_attn.experts"); + CHECK(experts.algo == QuantAlgo::kFp8); + CHECK(experts.how == Resolution::kExpertsParent); + CHECK(experts.group_size == 0); +} + +// --- strategy 4: the ".experts" special case (modelopt.py:2455-2461) ------- + +TEST_CASE("modelopt-mixed: strategy 4 maps a FusedMoE .experts prefix onto its parent") { + const MixedPrecisionConfig c = Curated(); + + // ModelOpt lists "synthetic.layers.0.moe.up_proj"; a FusedMoE layer's prefix + // is "synthetic.layers.0.moe.experts", which strategies 1-3 all miss. + const ModuleQuant moe = c.Resolve("synthetic.layers.0.moe.experts"); + CHECK(moe.algo == QuantAlgo::kW4A16Nvfp4); + CHECK(moe.how == Resolution::kExpertsParent); + CHECK(moe.group_size == 16); + + // The special case is keyed on the ".experts" SUFFIX only. + CHECK(c.Resolve("synthetic.layers.0.moe.expertsx").algo == + QuantAlgo::kUnquantized); +} + +// --- strategy 5: fused_projection_shards fallback (modelopt.py:2463-2486) -- + +TEST_CASE("modelopt-mixed: strategy 5 falls back to gate/up shard names") { + const MixedPrecisionConfig c = Curated(); // no packed mapping registered + + const ModuleQuant gate_up = c.Resolve("synthetic.layers.3.mlp.gate_up_proj"); + CHECK(gate_up.algo == QuantAlgo::kFp8); + CHECK(gate_up.how == Resolution::kFusedShards); + + // Only qkv_proj and gate_up_proj are in the fallback table. + CHECK(c.Resolve("synthetic.layers.3.mlp.fc_proj").algo == + QuantAlgo::kUnquantized); +} + +TEST_CASE("modelopt-mixed: strategy 5 RAISES when gate/up disagree") { + const MixedPrecisionConfig c = Curated(); + CHECK_THROWS_AS(c.Resolve("synthetic.layers.4.mlp.gate_up_proj"), + std::invalid_argument); +} + +// --- the ignore list (modelopt.py:145-181, quant_utils.py:510-572) --------- + +TEST_CASE("modelopt-mixed: ignore-list entries resolve to unquantized") { + const MixedPrecisionConfig c = Curated(); + + for (const char* p : {"backbone.embeddings", "backbone.layers.0.mixer.conv1d", + "backbone.layers.1.mixer.gate", + "backbone.layers.12.mixer.q_proj", + "backbone.layers.12.mixer.o_proj"}) { + CAPTURE(p); + const ModuleQuant m = c.Resolve(p); + CHECK(m.algo == QuantAlgo::kUnquantized); + CHECK(m.how == Resolution::kExcluded); + CHECK(c.IsLayerExcluded(p)); + } + + // "mtp*" is a real wildcard entry — the whole MTP head is unquantized. + CHECK(c.IsLayerExcluded("mtp*")); + CHECK(c.IsLayerExcluded("mtp.layers.0.mixer.up_proj")); + CHECK(c.Resolve("mtp.layers.0.mixer.up_proj").how == Resolution::kExcluded); + CHECK_FALSE(c.IsLayerExcluded("backbone.layers.1.mixer.experts.0.up_proj")); +} + +TEST_CASE("modelopt-mixed: an ignored expert CHILD excludes its CONTAINER") { + nlohmann::ordered_json qc = LoadFixture(); + qc["ignore"] = nlohmann::ordered_json::array( + {"backbone.layers.1.mixer.experts.0.up_proj"}); + const MixedPrecisionConfig c = MixedPrecisionConfig::Parse(qc); + + // quant_utils.py:559-565 gives a prefix containing "experts" its own rule, + // and the direction is the surprising one: `prefix in layer_name` — the + // IGNORE ENTRY must contain the PREFIX, not the other way round. ModelOpt + // lists experts per index, while a FusedMoE layer is ONE module covering all + // of them, so naming a single expert child unquantizes the whole container. + // + // Nothing else can reach this branch: the exact pass wants equality, the + // legacy substring pass tests `prefix.find(entry)` (the OPPOSITE direction, + // which is npos here), and the wildcard pass has no wildcard to match. So + // without this case both inverting the test to `prefix.find(entry)` and + // deleting the branch outright leave the suite green. + CHECK(c.IsLayerExcluded("backbone.layers.1.mixer.experts")); + CHECK(c.Resolve("backbone.layers.1.mixer.experts").how == Resolution::kExcluded); + CHECK(c.Resolve("backbone.layers.1.mixer.experts").algo == + QuantAlgo::kUnquantized); + + // The named child itself, and every deeper path the entry contains. + CHECK(c.IsLayerExcluded("backbone.layers.1.mixer.experts.0.up_proj")); + CHECK(c.IsLayerExcluded("backbone.layers.1.mixer.experts.0")); + + // ...but NOT a sibling the entry does not contain: expert 1 stays quantized, + // and so does a differently named container in the same layer. + CHECK_FALSE(c.IsLayerExcluded("backbone.layers.1.mixer.experts.1.up_proj")); + CHECK(c.Resolve("backbone.layers.1.mixer.experts.1.up_proj").algo == + QuantAlgo::kW4A16Nvfp4); + CHECK_FALSE(c.IsLayerExcluded("backbone.layers.1.mixer.shared_experts")); + + // A prefix WITHOUT "experts" never enters the branch, so the same entry does + // not exclude it even though it is a strict prefix of that entry. + CHECK_FALSE(c.IsLayerExcluded("backbone.layers.1.mixer")); +} + +TEST_CASE("modelopt-mixed: the legacy SUBSTRING exclusion rule still applies") { + const MixedPrecisionConfig c = Curated(); + // modelopt.py:165-174 keeps a substring rule for pre-0.39 ModelOpt exports, + // where "ignore" carried a bare module name rather than a full path. Neither + // the exact pass nor the wildcard pass can reach it, so without a case here + // the whole rule deletes clean. + CHECK(c.IsLayerExcluded("deep.legacy_substr.mixer.q_proj")); + CHECK(c.Resolve("deep.legacy_substr.mixer.q_proj").how == Resolution::kExcluded); + CHECK(c.IsLayerExcluded("language_model.deep.legacy_substr.mixer.o_proj")); + // It is a SUBSTRING rule, not a segment rule: "legacy_substr.mixerX" still + // contains "legacy_substr.mixer" and so is excluded, whereas breaking the + // match earlier is not. Upstream's bluntness here is deliberate and mirrored. + CHECK(c.IsLayerExcluded("deep.legacy_substr.mixerX.q_proj")); + CHECK_FALSE(c.IsLayerExcluded("deep.legacy_substrX.mixer.q_proj")); +} + +TEST_CASE("modelopt-mixed: exclusion WINS over a quantized_layers entry") { + const MixedPrecisionConfig c = Curated(); + // synthetic.layers.7.self_attn.q_proj is in BOTH maps. get_quant_method + // (modelopt.py:2515-2522) tests exclusion BEFORE _resolve_quant_algo, so the + // ignore list wins. Reversing that order would quantize an excluded layer. + const ModuleQuant m = c.Resolve("synthetic.layers.7.self_attn.q_proj"); + CHECK(m.algo == QuantAlgo::kUnquantized); + CHECK(m.how == Resolution::kExcluded); + // ...while its unexcluded siblings still resolve. + CHECK(c.Resolve("synthetic.layers.7.self_attn.k_proj").algo == QuantAlgo::kFp8); +} + +TEST_CASE("modelopt-mixed: a PARTIALLY excluded fused layer RAISES") { + MixedPrecisionConfig c = Curated(); + c.SetPackedModulesMapping(StandardPacked()); + // q_proj is ignored, k/v are not (quant_utils.py:549-556). + CHECK_THROWS_AS(c.Resolve("synthetic.layers.7.self_attn.qkv_proj"), + std::invalid_argument); + + // A FULLY excluded fused layer is excluded, not an error. + const ModuleQuant all_ignored = c.Resolve("backbone.layers.12.mixer.qkv_proj"); + CHECK(all_ignored.how == Resolution::kExcluded); +} + +TEST_CASE("modelopt-mixed: the ignore matcher is fnmatch, not startswith") { + nlohmann::ordered_json qc = LoadFixture(); + qc["ignore"] = nlohmann::ordered_json::array( + {"a.mid*.tail", "b.?.gate", "c.[0-9].gate", "d.[!0-9].gate", "e.literal["}); + const MixedPrecisionConfig c = MixedPrecisionConfig::Parse(qc); + + CHECK(c.IsLayerExcluded("a.mid.tail")); // '*' matches empty + CHECK(c.IsLayerExcluded("a.midXY.z.tail")); // ...and spans separators + CHECK_FALSE(c.IsLayerExcluded("a.mid.tail.x")); // anchored at both ends + CHECK(c.IsLayerExcluded("b.7.gate")); + CHECK_FALSE(c.IsLayerExcluded("b.77.gate")); // '?' is exactly one char + CHECK(c.IsLayerExcluded("c.4.gate")); + CHECK_FALSE(c.IsLayerExcluded("c.x.gate")); + CHECK(c.IsLayerExcluded("d.x.gate")); // negated class + CHECK_FALSE(c.IsLayerExcluded("d.4.gate")); + CHECK(c.IsLayerExcluded("e.literal[")); // unterminated class is literal +} + +TEST_CASE("modelopt-mixed: an empty ignore list excludes nothing") { + nlohmann::ordered_json qc = LoadFixture(); + qc["ignore"] = nlohmann::ordered_json::array(); + const MixedPrecisionConfig c = MixedPrecisionConfig::Parse(qc); + CHECK_FALSE(c.IsLayerExcluded("backbone.embeddings")); + CHECK(c.Resolve("synthetic.layers.7.self_attn.q_proj").algo == QuantAlgo::kFp8); +} + +// --- refusal by name (spec stop condition) --------------------------------- + +TEST_CASE("modelopt-mixed: an UNKNOWN quant_algo is refused BY NAME") { + const MixedPrecisionConfig c = Curated(); + CHECK_THROWS_WITH_AS(c.Resolve("synthetic.layers.5.mixer.in_proj"), + doctest::Contains("AWQ_LITE"), std::runtime_error); + CHECK_THROWS_WITH_AS(c.Resolve("synthetic.layers.5.mixer.in_proj"), + doctest::Contains("synthetic.layers.5.mixer.in_proj"), + std::runtime_error); +} + +TEST_CASE("modelopt-mixed: a KNOWN but unimplemented algo is refused BY NAME") { + const MixedPrecisionConfig c = Curated(); + // FP8_PB_WO is a real ModelOpt algo (modelopt.py:105-120) that the + // MIXED_PRECISION consumer has no branch for. Upstream silently hands it an + // UnquantizedLinearMethod; we refuse, because dequantizing a quantized layer + // is numerically fine and therefore INVISIBLE to a token gate. + CHECK_THROWS_WITH_AS(c.Resolve("synthetic.layers.6.mixer.in_proj"), + doctest::Contains("FP8_PB_WO"), std::runtime_error); +} + +TEST_CASE("modelopt-mixed: refusal survives every resolution strategy") { + nlohmann::ordered_json qc = LoadFixture(); + auto& ql = qc["quantized_layers"]; + ql["refuse.layers.0.moe.up_proj"] = {{"quant_algo", "AWQ_LITE"}}; + ql["refuse.layers.0.moe.down_proj"] = {{"quant_algo", "AWQ_LITE"}}; + ql["refuse.layers.1.mlp.gate_proj"] = {{"quant_algo", "AWQ_LITE"}}; + ql["refuse.layers.1.mlp.up_proj"] = {{"quant_algo", "AWQ_LITE"}}; + ql["refuse.layers.2.self_attn.q_proj"] = {{"quant_algo", "AWQ_LITE"}}; + ql["refuse.layers.2.self_attn.k_proj"] = {{"quant_algo", "AWQ_LITE"}}; + ql["refuse.layers.2.self_attn.v_proj"] = {{"quant_algo", "AWQ_LITE"}}; + MixedPrecisionConfig c = MixedPrecisionConfig::Parse(qc); + c.SetPackedModulesMapping(StandardPacked()); + + // strategy 4 (.experts), strategy 5 (gate_up), strategy 2 (packed), + // strategy 3 (prefix). None of them may launder an unknown algo into a + // supported one or into "unquantized". + CHECK_THROWS_AS(c.Resolve("refuse.layers.0.moe.experts"), std::runtime_error); + CHECK_THROWS_AS(c.Resolve("refuse.layers.1.mlp.gate_up_proj"), std::runtime_error); + CHECK_THROWS_AS(c.Resolve("refuse.layers.2.self_attn.qkv_proj"), std::runtime_error); + CHECK_THROWS_AS(c.Resolve("refuse.layers.0.moe"), std::runtime_error); +} + +// --- algo mapping is exhaustive and case-normalised ------------------------ + +TEST_CASE("modelopt-mixed: every implemented algo maps, lower-case included") { + nlohmann::ordered_json qc = LoadFixture(); + auto& ql = qc["quantized_layers"]; + ql["algo.fp8"] = {{"quant_algo", "FP8"}}; + ql["algo.nvfp4"] = {{"quant_algo", "NVFP4"}}; + ql["algo.w4a16"] = {{"quant_algo", "W4A16_NVFP4"}}; + ql["algo.mxfp8"] = {{"quant_algo", "MXFP8"}}; + ql["algo.lowercase"] = {{"quant_algo", "w4a16_nvfp4"}}; // modelopt.py:2427 .upper() + const MixedPrecisionConfig c = MixedPrecisionConfig::Parse(qc); + + CHECK(c.Resolve("algo.fp8").algo == QuantAlgo::kFp8); + CHECK(c.Resolve("algo.nvfp4").algo == QuantAlgo::kNvfp4); + CHECK(c.Resolve("algo.w4a16").algo == QuantAlgo::kW4A16Nvfp4); + CHECK(c.Resolve("algo.mxfp8").algo == QuantAlgo::kMxfp8); + CHECK(c.Resolve("algo.lowercase").algo == QuantAlgo::kW4A16Nvfp4); + + // The NVFP4 family carries the group size; FP8/MXFP8 do not. + CHECK(c.Resolve("algo.nvfp4").group_size == 16); + CHECK(c.Resolve("algo.w4a16").group_size == 16); + CHECK(c.Resolve("algo.fp8").group_size == 0); + CHECK(c.Resolve("algo.mxfp8").group_size == 0); + + CHECK(std::string(QuantAlgoName(QuantAlgo::kW4A16Nvfp4)) == "W4A16_NVFP4"); + CHECK(std::string(QuantAlgoName(QuantAlgo::kFp8)) == "FP8"); + CHECK(std::string(QuantAlgoName(QuantAlgo::kUnquantized)) == "UNQUANTIZED"); +} + +TEST_CASE("modelopt-mixed: an explicit top-level group_size overrides seeding") { + nlohmann::ordered_json qc = LoadFixture(); + qc["group_size"] = 32; + const MixedPrecisionConfig c = MixedPrecisionConfig::Parse(qc); + CHECK(c.group_size() == 32); + // modelopt.py builds ONE nvfp4 config from the config-level group_size, so + // that is the value every NVFP4 module gets — the per-entry field only SEEDS + // it when the top level is absent. Mirroring that polarity matters: picking + // the per-entry value instead would be an invention. + CHECK(c.Resolve("backbone.layers.1.mixer.experts.0.up_proj").group_size == 32); +} diff --git a/tests/vllm/model_executor/layers/quantization/test_modelopt_mixed_precision_checkpoint.cpp b/tests/vllm/model_executor/layers/quantization/test_modelopt_mixed_precision_checkpoint.cpp new file mode 100644 index 000000000..1f6709ef1 --- /dev/null +++ b/tests/vllm/model_executor/layers/quantization/test_modelopt_mixed_precision_checkpoint.cpp @@ -0,0 +1,274 @@ +// EXHAUSTIVE arm of the ModelOpt MIXED_PRECISION resolver gate: the REAL +// 1.3 MB config.json of nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4, +// all 5981 quantized_layers entries and all 72 ignore entries. +// +// Weights are NOT needed — only config.json is read — but the file lives with +// the 20 GiB checkpoint on shared storage, so this arm is opt-in and SKIPS +// LOUDLY (CTest SKIP_RETURN_CODE 77) when the checkpoint is not staged. It is +// a separate binary from the always-on curated gate precisely so that skipping +// here can never mask that one. +// +// Row MODEL-TEXT-nemotron-h-nemotron-hfor-causal-lm W1, issue #517. +// Upstream anchors: see test_modelopt_mixed_precision.cpp. +#include + +#include +#include +#include +#include +#include +#include + +#include + +#include "hf_snapshot.h" +#include "vllm/model_executor/layers/quantization/modelopt_mixed_precision.h" + +using vllm::layers::modelopt::MixedPrecisionConfig; +using vllm::layers::modelopt::ModuleQuant; +using vllm::layers::modelopt::QuantAlgo; +using vllm::layers::modelopt::Resolution; + +namespace { + +// A gate that CANNOT RUN must never report success. Returning early out of a +// TEST_CASE prints "0 passed | 0 failed" + "Status: SUCCESS!" and exits 0, +// which is indistinguishable in a log from a real pass (issue #463). Exiting +// 77 makes CTest report **Skipped** and stops any `&&` chain. +[[noreturn]] void SkipGate(const std::string& why) { + std::fprintf(stderr, + "\n*** GATE NOT RUN — SKIPPED (exit 77), this is NOT a pass ***\n" + "*** test_modelopt_mixed_precision_checkpoint: %s\n\n", + why.c_str()); + std::fflush(stderr); + std::exit(77); +} + +// `/`, where the snapshot is resolved by the SINGLE pinned +// resolver `parity::Nemotron35LightningSnapshot()` (tests/parity/hf_snapshot.h). +// +// LOW-3 (#517). This arm used to read `CHECKPOINT_ROOT` itself and join the +// staging directory name by hand, which meant two env vars — +// `VT_NEMOTRON35_SNAPSHOT` there, `CHECKPOINT_ROOT` here — reached one +// checkpoint and neither refused a revision the goldens were not captured +// against. Both spellings now go through that resolver, which gates the staged +// `local_dir` on its own `.cache/huggingface/trees/.json` manifest, so +// a re-download landing a different revision under the identical path SKIPS +// here rather than being silently substituted. +// +// HOW TO MAKE THIS ARM RUN. `CHECKPOINT_ROOT` is a `.env` key, and `.env` is not +// exported into a login shell by anything: `.env.example:8` documents the loader +// verbatim as `set -a; . ./.env; set +a`, and `.agents/environment.md:16` points +// at that file. (An earlier version of this comment credited the loader line to +// `.agents/environment.md` as well; it is not there — LOW-3.) There is no +// CTest-side mechanism that reads `.env`, so a plain `ctest` from a shell that +// has not sourced it SKIPS this arm, which is correct behavior and not a pass — +// the banner below names the exact export. +std::string CheckpointFile(const char* filename) { + const std::string snapshot = parity::Nemotron35LightningSnapshot(); + if (snapshot.empty()) { + SkipGate( + std::string("no pinned Nemotron-3.5-Lightning snapshot, so ") + + filename + + " cannot be located.\n" + "*** To RUN this arm, load the repo env first — `.env.example:8`\n" + "*** documents exactly this:\n" + "*** set -a; . ./.env; set +a\n" + "*** then re-run ctest. Equivalently, export it for one run:\n" + "*** CHECKPOINT_ROOT= ctest -R " + "test_modelopt_mixed_precision_checkpoint\n" + "*** The arm needs only\n" + "*** $CHECKPOINT_ROOT/" + + parity::kNemotron35LightningLocalDirName + + "/{config,hf_quant_config}.json — no weights, no GPU.\n" + "*** A staged directory that IS present skips too unless it carries\n" + "*** .cache/huggingface/trees/" + + parity::kNemotron35LightningNvfP4Revision + + ".json,\n" + "*** the revision the committed goldens were captured against."); + } + const std::filesystem::path p = std::filesystem::path(snapshot) / filename; + std::error_code ec; + if (!std::filesystem::exists(p, ec)) { + SkipGate("not staged: " + p.string()); + } + return p.string(); +} + +nlohmann::ordered_json LoadJson(const char* filename) { + const std::string path = CheckpointFile(filename); + std::ifstream f(path); + REQUIRE_MESSAGE(f.good(), "cannot open: ", path); + return nlohmann::ordered_json::parse(f); +} + +// config.json -> quantization_config: the FLAT (compressed-tensors style) +// shape, `{"quant_method": "modelopt", "quant_algo": "MIXED_PRECISION", ...}`. +nlohmann::ordered_json LoadQuantizationConfig() { + nlohmann::ordered_json doc = LoadJson("config.json"); + REQUIRE_MESSAGE(doc.contains("quantization_config"), + "no quantization_config in config.json"); + return doc.at("quantization_config"); +} + +} // namespace + +TEST_CASE("modelopt-mixed[ckpt]: every real entry resolves, histogram exact") { + const nlohmann::ordered_json qc = LoadQuantizationConfig(); + + // Assert the FIXTURE is the checkpoint this gate claims to cover before + // asserting anything about it (oracle-identity discipline: a repo silently + // re-quantized under the same name has cost this project a campaign). + REQUIRE(MixedPrecisionConfig::IsMixedPrecision(qc)); + const MixedPrecisionConfig c = MixedPrecisionConfig::Parse(qc); + REQUIRE(c.num_quantized_layers() == 5981); + REQUIRE(c.exclude_modules().size() == 72); + CHECK(c.group_size() == 16); + CHECK(c.kv_cache_quant_algo() == "FP8"); + + std::map histogram; + const auto& ql = qc.at("quantized_layers"); + for (auto it = ql.begin(); it != ql.end(); ++it) { + const std::string name = it.key(); + const ModuleQuant m = c.Resolve(name); + CAPTURE(name); + // A listed entry must resolve to a quantized algorithm. "Unquantized" here + // would mean the loader silently dequantizes a quantized tensor — correct + // numerics, wrong bytes, invisible to a token gate. + REQUIRE(m.Quantized()); + CHECK(m.how == Resolution::kDirect); + histogram[QuantAlgoName(m.algo)] += 1; + } + + CHECK(histogram.size() == 2); + CHECK(histogram["W4A16_NVFP4"] == 5935); + CHECK(histogram["FP8"] == 46); + + // Every one of the 72 ignore entries is excluded, and therefore unquantized. + for (const auto& entry : qc.at("ignore")) { + const std::string name = entry.get(); + CAPTURE(name); + CHECK(c.IsLayerExcluded(name)); + CHECK(c.Resolve(name).algo == QuantAlgo::kUnquantized); + } +} + +TEST_CASE("modelopt-mixed[ckpt]: the module prefixes the loader will actually ask for") { + const MixedPrecisionConfig c = MixedPrecisionConfig::Parse(LoadQuantizationConfig()); + + // Routed + shared experts resolve through the PREFIX strategy: the layer + // module is "...mixer.experts", the map lists "...mixer.experts..". + for (int layer : {1, 3, 6, 51}) { + const std::string p = + "backbone.layers." + std::to_string(layer) + ".mixer.experts"; + CAPTURE(p); + const ModuleQuant m = c.Resolve(p); + CHECK(m.algo == QuantAlgo::kW4A16Nvfp4); + CHECK(m.how == Resolution::kPrefix); + CHECK(m.group_size == 16); + } + + // Mamba projections are FP8 — the polarity trap this row exists to catch: + // the repo NAME says NVFP4 and 5935 of 5981 entries are, but these 46 are not. + for (int layer : {0, 2, 50}) { + const std::string p = + "backbone.layers." + std::to_string(layer) + ".mixer.in_proj"; + CAPTURE(p); + CHECK(c.Resolve(p).algo == QuantAlgo::kFp8); + CHECK(c.Resolve(p).group_size == 0); + } + + // Attention (layers 5/12/19/26/33/42), conv1d, gates and embeddings are bf16. + // + // `const std::string`, NOT `const char*`, and that is load-bearing: doctest + // 2.5.2 has no stringifier for a `const char*` VARIABLE, so `CAPTURE(p)` on + // one prints `logged: p := 1` and a failure here would name none of the five + // prefixes (LOW-2). The two loops above already spell it `std::string`. + for (const std::string p : {"backbone.layers.5.mixer.q_proj", + "backbone.layers.42.mixer.o_proj", + "backbone.layers.0.mixer.conv1d", + "backbone.layers.1.mixer.gate", + "backbone.embeddings"}) { + CAPTURE(p); + CHECK(c.Resolve(p).how == Resolution::kExcluded); + } + + // lm_head is a BARE key in the map; the loader's prefix carries no "model.". + CHECK(c.Resolve("lm_head").algo == QuantAlgo::kW4A16Nvfp4); + CHECK(c.Resolve("model.lm_head").algo == QuantAlgo::kW4A16Nvfp4); + + // The whole MTP head is covered by the "mtp*" wildcard. + CHECK(c.IsLayerExcluded("mtp.layers.0.mixer.up_proj")); + CHECK(c.IsLayerExcluded("mtp.layers.0.eh_proj")); +} + +// The checkpoint ships its quantization config TWICE: once flat inside +// config.json's `quantization_config`, and once in the standalone +// `hf_quant_config.json` that ModelOpt actually writes and that +// `get_config_filenames()` (modelopt.py:265-267) names. Only the first shape +// was gated, and the second is the one upstream's own file list points at. +// +// Its top-level key set is exactly {"producer", "quantization"} — there is NO +// `quant_method` anywhere in the file. `from_config` (modelopt.py:282-367) +// does not want one: it dispatches on the SHAPE and reads `quant_algo` out of +// the nested section. Borrowing the SELECTION hook's `quant_method` +// precondition (`_extract_modelopt_quant_algo`, :245-263, used only by +// `override_quantization_method`) made `Parse` REFUSE this file outright while +// a verbatim upstream transcription resolved all 5981 entries from it. +TEST_CASE("modelopt-mixed[ckpt]: the standalone hf_quant_config.json parses") { + const nlohmann::ordered_json hq = LoadJson("hf_quant_config.json"); + + // Assert the SHAPE this case exists for, so a re-quantized publish that + // changed it fails here rather than quietly retargeting the case. + REQUIRE(hq.is_object()); + REQUIRE(hq.contains("quantization")); + REQUIRE_FALSE(hq.contains("quant_method")); + REQUIRE_FALSE(hq.at("quantization").contains("quant_method")); + CHECK(hq.at("quantization").at("quant_algo") == "MIXED_PRECISION"); + + // DETECTION still refuses it, and that is upstream's behavior, not a bug: + // `_extract_modelopt_quant_algo` returns None for a config naming no + // quantizer. The override hook is for `config.json`'s `quantization_config`, + // where several vendors share one field and the name is what tells them apart. + CHECK_FALSE(MixedPrecisionConfig::IsMixedPrecision(hq)); + + // PARSING must accept it — this is the file the loader will be handed. + const MixedPrecisionConfig c = MixedPrecisionConfig::Parse(hq); + CHECK(c.num_quantized_layers() == 5981); + CHECK(c.exclude_modules().size() == 72); + CHECK(c.group_size() == 16); + // The nested shape spells kv cache as a plain algo STRING + // (`kv_cache_quant_algo`), where the flat shape uses a `kv_cache_scheme` + // dict. Both must land on the same answer. + CHECK(c.kv_cache_quant_algo() == "FP8"); + + // ABSOLUTE expectations first. Comparing the two shapes to each other proves + // consistency, not correctness — both arms call the same resolver, so a + // wrong answer agrees with itself perfectly. + CHECK(c.Resolve("backbone.layers.0.mixer.in_proj").algo == QuantAlgo::kFp8); + CHECK(c.Resolve("backbone.layers.0.mixer.in_proj").group_size == 0); + CHECK(c.Resolve("backbone.layers.1.mixer.experts").algo == + QuantAlgo::kW4A16Nvfp4); + CHECK(c.Resolve("backbone.layers.1.mixer.experts").how == Resolution::kPrefix); + CHECK(c.Resolve("backbone.layers.1.mixer.experts").group_size == 16); + CHECK(c.Resolve("backbone.layers.5.mixer.q_proj").how == Resolution::kExcluded); + CHECK(c.Resolve("lm_head").algo == QuantAlgo::kW4A16Nvfp4); + CHECK(c.IsLayerExcluded("mtp.layers.0.eh_proj")); + + // ...and only THEN that the flat shape agrees module for module. + const MixedPrecisionConfig flat = + MixedPrecisionConfig::Parse(LoadQuantizationConfig()); + // `const std::string` for the same reason as the loop above: a `const char*` + // capture prints `1` and this loop compares FIVE prefixes (LOW-2). + for (const std::string p : {"backbone.layers.0.mixer.in_proj", + "backbone.layers.1.mixer.experts", + "backbone.layers.5.mixer.q_proj", "lm_head", + "mtp.layers.0.eh_proj"}) { + CAPTURE(p); + const ModuleQuant a = c.Resolve(p); + const ModuleQuant b = flat.Resolve(p); + CHECK(a.algo == b.algo); + CHECK(a.how == b.how); + CHECK(a.group_size == b.group_size); + } +}