27B dense bring-up: W4A4 loader routing + dense SwiGLU forward (CPU)

Continues the 27B (Qwen3_5ForConditionalGeneration, text_config qwen3_5_text)
bring-up — §5 steps 2-3 of qwen27b-w4a4-notes.md, CPU-only (no GPU touched).

Loader (qwen3_5_dense.h / qwen3_5_dense_weights.cpp), mirroring LoadQwen3_5Moe:
- Qwen3_5DenseWeights = embed + final_norm + bf16 lm_head + per-layer
  Qwen3_5DenseLayerWeights (reused GdnLayerWeights/FullAttnLayerWeights + new
  DenseMlpWeights gate/up/down).
- IsQwen27QuantizedLinear routes each Linear bf16 vs W4A4 per the §3.6 ignore
  list; MaterializeCtNvfp4Bf16Transposed dequants W4A4 -> bf16 (Matmul-B) via
  the existing DequantCtNvfp4WeightToF32 (CT weight_packed/weight_scale/
  weight_global_scale names). Activation-quant dropped on this bf16 path
  (step-6a fast path).

Dense forward (qwen3_5.cpp), mirroring Qwen3_5Model::ForwardDense:
- Qwen3_5DenseModel::ForwardDense reuses the 35B GdnBlock/FullAttnBlock/norm
  helpers verbatim; DenseMlpBlock = down(silu(gate(x))*up(x)) replaces MoeBlock.
  Text path only; ViT/merger/MTP deferred.

CPU tests (test_qwen27_dense_forward.cpp, 4 cases / 280 assertions): loader
routing, W4A4 materialization vs the hand-computed CT block, finite+deterministic
forward, and MLP-perturbation-moves-output. Full CPU ctest suite 83/83 green;
clean -Werror rebuild. test_qwen27_paged_engine stays SKIPPING (kW4A4ForwardReady
false — flips at the GPU oracle+GEMM step). Notes §5 + porting-inventory §9.7
updated.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UJyFKcK62CcR3imhgbiBnW

Author: mudler

.agents/porting-inventory.md

@@ -275,7 +275,21 @@ Examples: `examples/cli` ✅ (C-API client), `examples/server` ✅ (OpenAI serve
    → C ABI callback registries, added per tier.
 5. **GGUF as a first-class input** (upstream treats it as one loader among many;
    for us it is a gate, including NVFP4 GGUF extension types).
-6. **Extension platforms** (T2): Apple Metal and Vulkan backends — upstream has
+7. **Explicit owned-tensor model weights** (no `nn.Module`/`AutoWeightsLoader`).
+   The dense 27B (`Qwen3_5ForConditionalGeneration`, text_config `qwen3_5_text`)
+   loads through `LoadQwen3_5Dense` into `Qwen3_5DenseWeights` — mirroring the
+   35B's `LoadQwen3_5Moe` — with the MoE block replaced by a `DenseMlpWeights`
+   SwiGLU MLP. Two recorded remaps: (a) the compressed-tensors NVFP4 W4A4 tensor
+   NAMES (`weight_packed`/`weight_scale`/`weight_global_scale`/`input_global_scale`)
+   vs the 35B modelopt names (`weight`/`weight_scale`/`weight_scale_2`/`input_scale`);
+   (b) on the CPU correctness path every W4A4 Linear is MATERIALIZED to bf16 at
+   load (`MaterializeCtNvfp4Bf16Transposed` → `DequantCtNvfp4WeightToF32`) so the
+   existing bf16 forward carries it — the fp4-resident tensor-core GEMM reuse is
+   the later GPU step (qwen27b-w4a4-notes.md §5 steps 6-7), not a permanent
+   deviation. Per-Linear bf16-vs-W4A4 routing is `IsQwen27QuantizedLinear`
+   (encodes the checkpoint `ignore` list, §3.6). Text path only; the ViT/merger
+   and MTP head are deferred stubs.
+8. **Extension platforms** (T2): Apple Metal and Vulkan backends — upstream has
    no equivalent under `vllm/platforms/`; we add them through the mirrored
    Platform/AttentionBackend/vt-op seams so they behave as vLLM platforms
    would. Intel is NOT a deviation (upstream `platforms/xpu.py` is ported

.agents/qwen27b-w4a4-notes.md

@@ -10,11 +10,20 @@ pinned upstream checkout `/home/mudler/_git/vllm` @ `e24d1b24`. Cites are
 Checkpoint (read-only on dgx):
 `~/.cache/huggingface/hub/models--unsloth--Qwen3.6-27B-NVFP4`.
 
-STATUS (2026-07-04): CPU-first correctness scaffolding only. Delivered: the CPU
-W4A4 dequant + activation-quant reference (`nvfp4_emulation.h`, unit-tested) and
-the skipping greedy-parity gate scaffold (`test_qwen27_paged_engine.cpp`). The
-27B forward + the GB10 W4A4 GPU GEMM are NOT yet written — §5 is the ordered
-plan, GPU-gated steps marked.
+STATUS (2026-07-04): CPU-first correctness path LANDED (§5 steps 1-4). Delivered:
+the CPU W4A4 dequant + activation-quant reference (`nvfp4_emulation.h`,
+unit-tested); the dense loader (`LoadQwen3_5Dense` → `Qwen3_5DenseWeights`,
+`qwen3_5_dense.h`/`qwen3_5_dense_weights.cpp`) that routes each Linear bf16 vs
+W4A4-materialized-to-bf16 by name (`IsQwen27QuantizedLinear` +
+`MaterializeCtNvfp4Bf16Transposed`); the dense TEXT forward
+(`Qwen3_5DenseModel::ForwardDense` in `qwen3_5.cpp`, reusing the 35B GDN +
+gated-attn + norm helpers with the MoE block swapped for the dense SwiGLU MLP);
+and CPU unit tests (`test_qwen27_dense_forward.cpp`: routing + materialization +
+finite/deterministic forward + MLP-wired, 4 cases / 280 assertions, CPU-green).
+The full CPU ctest suite (83 targets) stays green. What is NOT done and is GPU-gated: the
+pip-vLLM oracle greedy golden capture (step 5), the W4A4 matmul kernel wiring
+(step 6), and flipping `kW4A4ForwardReady` to close the gate (step 7). §5 is the
+ordered plan, GPU steps marked.
 
 ---
 
@@ -250,22 +259,37 @@ Legend: **[CPU]** doable on the dev box now; **[GPU]** needs the free GB10.
    the pinned emulation math (`test_ct_nvfp4_emulation.cpp`, 6 cases / 81
    assertions, CPU-green). This is the CPU-truth the GPU GEMM validates against.
 
-2. **[CPU] Config + loader plumbing.** Recognize
-   `Qwen3_5ForConditionalGeneration` (dense; `is_moe=false`), read `text_config`
-   (§1), the compressed-tensors quant block + the `ignore` list, and
-   `partial_rotary_factor`/`mrope_section`/`output_gate_type`/`attn_output_gate`
-   from `HfConfig.raw`. Add a dense-MLP + W4A4 weight-loader path (mirror the
-   existing modelopt loader in `qwen3_5_weights.cpp`): route bf16 vs W4A4 by name
-   (§3.6); for the CPU reference forward, materialize each W4A4 linear to bf16
-   via `DequantCtNvfp4WeightToF32` (as the 35B loader dequants modelopt).
-   Mirror upstream names so the port stays mechanical; record the tensor-name
-   remap in porting-inventory §9.
-
-3. **[CPU] Dense forward assembly.** New `Qwen3_5Model::ForwardDense`-style path
-   for the dense arch: reuse the 35B GDN + full-attn + norm components, swap the
-   MoE block for the dense SwiGLU MLP (§2), wire the quantized linears. Validate
-   per-layer activations + greedy on CPU where feasible (full 64-layer CPU
-   greedy is slow — the authoritative gate is GPU, step 6).
+2. **[CPU] ✅ DONE — Config + loader plumbing.** `LoadHfConfig` already
+   recognizes `Qwen3_5ForConditionalGeneration` via the `text_config`/
+   `qwen3_5_text` resolution (partial_rotary_factor default 0.25 → rotary_dim 64;
+   `num_experts` reads 0 = dense). `LoadQwen3_5Dense`
+   (`qwen3_5_dense.h`/`qwen3_5_dense_weights.cpp`) mirrors `LoadQwen3_5Moe`:
+   `Qwen3_5DenseWeights` = embed + final_norm + bf16 `lm_head` + per-layer
+   `Qwen3_5DenseLayerWeights` (reused `GdnLayerWeights`/`FullAttnLayerWeights` +
+   new `DenseMlpWeights`). Each Linear is routed bf16 vs W4A4-materialized-to-bf16
+   by `IsQwen27QuantizedLinear` (encodes the §3.6 `ignore` list) and materialized
+   via `MaterializeCtNvfp4Bf16Transposed` → `DequantCtNvfp4WeightToF32` (reads the
+   CT `weight_packed`/`weight_scale`/`weight_global_scale` names; the on-disk
+   `input_global_scale` activation divisor is IGNORED on this bf16-activation path
+   — the step-6a fast path). Tensor-name remap recorded in porting-inventory §9.7.
+
+3. **[CPU] ✅ DONE — Dense forward assembly.** `Qwen3_5DenseModel::ForwardDense`
+   (in `qwen3_5.cpp`, so it reuses the file-local `GdnBlock`/`FullAttnBlock`/norm/
+   `DBuf`/matmul helpers verbatim) mirrors `Qwen3_5Model::ForwardDense`: embed →
+   N `RunDenseLayer` (input_layernorm → GDN|full-attn → post_attn_layernorm →
+   `DenseMlpBlock`) → final RMSNorm → bf16 `lm_head`. `DenseMlpBlock` =
+   `down( silu(gate(x)) * up(x) )` (the shared-expert silu-mul, no router/gate).
+   Text-only: the three mRoPE streams coincide so partial NeoX RoPE degenerates to
+   1-D over `positions` (§2). CPU-validated on a synthetic small hybrid model
+   (finite + deterministic logits; MLP-perturbation moves the output). The ViT +
+   image/video merger + MTP head are DEFERRED (text-first, §0.1) — no stub code,
+   the loader simply does not request `model.visual.*`/`mtp.*`.
+   ⚠ REMAINING before step 5/7 flips the gate: (a) a paged `Qwen3_5DenseModel`
+   forward + a runner route (the 35B rides `Qwen3_5Model::Forward` + the runner;
+   the dense arch needs the analogous paged path OR the runner extended to the
+   dense weights) — the dense ForwardDense here is the single-sequence reference,
+   as the 35B's ForwardDense was before its paged refactor; (b) mrope_section
+   handling for genuine multimodal positions (inert for text).
 
 4. **[CPU] ✅ DONE — greedy-parity gate scaffold.**
    `test_qwen27_paged_engine.cpp` resolves the `unsloth/Qwen3.6-27B-NVFP4`

CMakeLists.txt

@@ -67,6 +67,7 @@ add_library(vllm STATIC
   src/vllm/model_executor/model_loader/nvfp4_dequant.cpp
   src/vllm/model_executor/layers/quantization/compressed_tensors/nvfp4_emulation.cpp
   src/vllm/model_executor/models/qwen3_5_weights.cpp
+  src/vllm/model_executor/models/qwen3_5_dense_weights.cpp
   src/vllm/model_executor/models/qwen3_5_gguf_weights.cpp
   src/vllm/model_executor/models/qwen3_5.cpp
   src/vllm/model_executor/models/registry.cpp

include/vllm/model_executor/models/qwen3_5_dense.h

@@ -0,0 +1,108 @@
+// vllm.cpp original — the DENSE Qwen3.6-27B text gate
+// (unsloth/Qwen3.6-27B-NVFP4, arch Qwen3_5ForConditionalGeneration, text_config
+// model_type "qwen3_5_text"). See .agents/qwen27b-w4a4-notes.md.
+//
+// The 27B shares the 35B hybrid backbone WHOLESALE (GDN linear-attention + gated
+// full-attention + Gemma (1+w) RMSNorm + mRoPE->NeoX); it REUSES the 35B's
+// GdnLayerWeights / FullAttnLayerWeights sub-structs and the GdnBlock /
+// FullAttnBlock forward helpers verbatim. The ONLY structural change is the
+// per-layer sparse-MoE block being replaced by a DENSE SwiGLU MLP (gate/up/down,
+// intermediate 17408): down( silu(gate(x)) * up(x) ). See notes §2.
+//
+// Quant: compressed-tensors NVFP4 W4A4 (notes §3). For the CPU correctness path
+// each quantized Linear is MATERIALIZED to bf16 at load via the CT weight-dequant
+// reference (DequantCtNvfp4WeightToF32, multiply by 1/weight_global_scale) and
+// the existing bf16 forward carries it — the true GB10 fp4xfp4 GEMM is a later,
+// GPU-gated step (notes §5 steps 5-7). The activation-quant round-trip is dropped
+// on this correctness path (bf16 activations), matching the notes' §5 step-6a
+// FAST PATH; that is a tiny numeric deviation vs true W4A4, validated later vs
+// the pip-vLLM oracle golden.
+//
+// Which Linears are quantized (notes §3.6): QUANTIZED (W4A4) = every dense-MLP
+// gate/up/down_proj, every self_attn q/k/v/o_proj, and the GDN linear_attn
+// out_proj. NOT quantized (bf16 on disk) = the GDN in_proj_{qkv,z,a,b}, conv1d,
+// A_log, dt_bias, all norms, embed_tokens, lm_head, mtp.*, and visual.*.
+#pragma once
+
+#include <cstdint>
+#include <string>
+#include <vector>
+
+#include "vllm/model_executor/models/qwen3_5_weights.h"  // OwnedTensor, Gdn/FullAttn weights, TensorResolver
+#include "vllm/transformers_utils/hf_config.h"
+#include "vt/device.h"
+#include "vt/tensor.h"
+
+namespace vllm {
+
+// Dense SwiGLU MLP (replaces the 35B MoE block). Projections in Matmul-B layout
+// [in, out]; W4A4-materialized to bf16 at load. down( silu(gate(x)) * up(x) ).
+struct DenseMlpWeights {
+  OwnedTensor gate_proj;  // bf16 [H, I]
+  OwnedTensor up_proj;    // bf16 [H, I]
+  OwnedTensor down_proj;  // bf16 [I, H]
+};
+
+// One dense decoder layer: input/post norms + one attention variant + dense MLP.
+struct Qwen3_5DenseLayerWeights {
+  bool is_linear_attention = false;
+  OwnedTensor input_layernorm;           // bf16 [H]
+  OwnedTensor post_attention_layernorm;  // bf16 [H]
+  GdnLayerWeights gdn;                    // valid iff is_linear_attention
+  FullAttnLayerWeights attn;             // valid iff !is_linear_attention
+  DenseMlpWeights mlp;                   // every layer has a dense MLP
+};
+
+// Whole dense-model text weights. lm_head is bf16 (unquantized in the 27B).
+struct Qwen3_5DenseWeights {
+  OwnedTensor embed_tokens;  // bf16 [vocab, H]  (NOT transposed; embed lookup)
+  OwnedTensor final_norm;    // bf16 [H]
+  OwnedTensor lm_head;       // bf16 [H, vocab]  (unquantized -> Matmul-B layout)
+  std::vector<Qwen3_5DenseLayerWeights> layers;
+};
+
+// True iff the projection named `name` is a W4A4-quantized Linear in the 27B
+// (notes §3.6). `name` is the module path WITHOUT the trailing ".weight*" (e.g.
+// "model.language_model.layers.0.mlp.gate_proj"). Encodes the checkpoint's
+// config.json `ignore` list: the quantized set is the dense-MLP {gate,up,down}
+// proj, the self_attn {q,k,v,o} proj, and the GDN linear_attn out_proj; every
+// other Linear (GDN in_proj_*, lm_head, mtp.*, visual.*, ...) is bf16.
+bool IsQwen27QuantizedLinear(const std::string& name);
+
+// Materialize one compressed-tensors NVFP4 W4A4 Linear to an owned bf16 tensor
+// in Matmul-B layout [in, out]. Reads `<proj>.weight_packed` (U8 [out, in/2]),
+// `<proj>.weight_scale` (F8_E4M3 [out, in/16]) and `<proj>.weight_global_scale`
+// (F32 scalar divisor); dequants to f32 via DequantCtNvfp4WeightToF32 (which
+// reciprocates the global scale), rounds to bf16, and transposes. Exposed for
+// unit testing. The `<proj>.input_global_scale` (activation divisor) is ignored
+// on this bf16-activation correctness path (notes §3.4 / §5 step-6a).
+OwnedTensor MaterializeCtNvfp4Bf16Transposed(const TensorResolver& get,
+                                             const std::string& proj);
+
+// Load one dense decoder layer. `layer_type` is "linear_attention" or
+// "full_attention". Prefix is "model.language_model.layers.{layer_idx}.". Routes
+// each Linear to bf16 vs W4A4-materialized-to-bf16 per IsQwen27QuantizedLinear.
+Qwen3_5DenseLayerWeights LoadQwen3_5DenseLayer(const TensorResolver& get,
+                                               const std::string& layer_type,
+                                               int64_t layer_idx);
+
+// Full dense-model load across the given shards. Uses config.num_hidden_layers
+// and config.layer_types. Text path only — the vision tower (model.visual.*),
+// the MTP head (mtp.*) and the image/video merger are DEFERRED (notes §0.1).
+Qwen3_5DenseWeights LoadQwen3_5Dense(const std::vector<SafetensorsFile>& shards,
+                                     const HfConfig& config);
+
+// Dense single-sequence reference forward (text path). Mirrors
+// Qwen3_5Model::ForwardDense but runs the dense SwiGLU MLP in place of the MoE
+// block; reuses the 35B GDN + gated-attention + norm forward helpers. Returns
+// logits [T, vocab] f32 (T = token_ids.size()). CPU or CUDA per `queue`.
+class Qwen3_5DenseModel {
+ public:
+  static std::vector<float> ForwardDense(const std::vector<int32_t>& token_ids,
+                                         const std::vector<int32_t>& positions,
+                                         const Qwen3_5DenseWeights& weights,
+                                         const HfConfig& config,
+                                         vt::Queue& queue);
+};
+
+}  // namespace vllm

src/vllm/model_executor/models/qwen3_5.cpp

@@ -7,6 +7,8 @@
 // .agents/moe-semantics.md (§1-§6 MoE block + activated-expert gather).
 #include "vllm/model_executor/models/qwen3_5.h"
 
+#include "vllm/model_executor/models/qwen3_5_dense.h"
+
 #include <algorithm>
 #include <cmath>
 #include <cstdlib>
@@ -1089,6 +1091,46 @@ void RunLayer(Dev d, const Qwen3_5MoeLayerWeights& layer, const HfConfig& cfg,
   hidden = MoeBlock(d, layer.moe, cfg, dh2.t(), T);
 }
 
+// --- Dense SwiGLU MLP block (the 27B's replacement for the MoE block; notes
+// §2). down( silu(gate(x)) * up(x) ), intermediate = cfg.intermediate_size.
+// Mirrors the shared-expert silu-mul MLP (no router, no expert gather, no output
+// gate). h [T,H] bf16 (device) -> DBuf [T,H] bf16 (device). Reused by the dense
+// forward below; the gate/up/down weights are W4A4-materialized-to-bf16 at load.
+DBuf DenseMlpBlock(Dev d, const DenseMlpWeights& w, const HfConfig& cfg,
+                   const Tensor& dh, int64_t T) {
+  const int64_t I = cfg.intermediate_size;
+  DBuf gate = MatmulF32D(d, dh, w.gate_proj);  // [T,I] f32
+  DBuf up = MatmulF32D(d, dh, w.up_proj);      // [T,I] f32
+  DBuf act(d, DType::kBF16, {T, I});
+  vt::MoeSiluMul(d.q, act.t(), gate.t(), up.t());  // silu(gate)*up -> bf16
+  return MatmulBf16D(d, act.t(), w.down_proj);     // [T,H] bf16
+}
+
+// One dense decoder layer (notes §2). Same residual/norm thread as RunLayer, but
+// the MoE block is swapped for the dense SwiGLU MLP; the GDN / full-attention
+// blocks are the 35B helpers reused verbatim. `hidden` (bf16 [T,H]) is the delta;
+// `res` (f32 [T,H]) the accumulator.
+void RunDenseLayer(Dev d, const Qwen3_5DenseLayerWeights& layer,
+                   const HfConfig& cfg, DBuf& hidden, DBuf& res,
+                   const std::vector<int32_t>& positions, int64_t T) {
+  const int64_t H = cfg.hidden_size;
+  const float eps = static_cast<float>(cfg.rms_norm_eps);
+
+  Tensor dw_in = ResidentWeight(d, layer.input_layernorm, {H});
+  DBuf dhn(d, DType::kBF16, {T, H});
+  vt::RmsNorm(d.q, dhn.t(), hidden.t(), dw_in, vt::RmsNormArgs{eps, true}, &res.t());
+
+  DBuf attn = layer.is_linear_attention
+                  ? GdnBlock(d, layer.gdn, cfg, dhn.t(), T)
+                  : FullAttnBlock(d, layer.attn, cfg, dhn.t(), positions, T);
+
+  Tensor dw_post = ResidentWeight(d, layer.post_attention_layernorm, {H});
+  DBuf dh2(d, DType::kBF16, {T, H});
+  vt::RmsNorm(d.q, dh2.t(), attn.t(), dw_post, vt::RmsNormArgs{eps, true}, &res.t());
+
+  hidden = DenseMlpBlock(d, layer.mlp, cfg, dh2.t(), T);
+}
+
 // Batched PAGED decoder layer (M1.8 Task 3). Same residual/norm/MoE thread as
 // RunLayer, but the attention block reads/writes the paged KV cache
 // (full-attn: attn_kv) or the persistent GDN mamba state (GDN: gdn_state).
@@ -1299,6 +1341,49 @@ std::vector<float> Qwen3_5Model::ForwardDense(const std::vector<int32_t>& token_
   return logits;
 }
 
+std::vector<float> Qwen3_5DenseModel::ForwardDense(
+    const std::vector<int32_t>& token_ids, const std::vector<int32_t>& positions,
+    const Qwen3_5DenseWeights& weights, const HfConfig& config,
+    vt::Queue& queue) {
+  const int64_t T = static_cast<int64_t>(token_ids.size());
+  const int64_t H = config.hidden_size;
+  const int64_t vocab = config.vocab_size;
+  VT_CHECK(T > 0, "qwen3_5 dense forward: empty token_ids");
+  VT_CHECK(static_cast<int64_t>(positions.size()) == T,
+           "qwen3_5 dense forward: positions length must equal token count");
+  VT_CHECK(static_cast<int64_t>(weights.layers.size()) == config.num_hidden_layers,
+           "qwen3_5 dense forward: weights.layers size must equal num_hidden_layers");
+  Dev d{vt::GetBackend(queue.device.type), queue};
+  const float eps = static_cast<float>(config.rms_norm_eps);
+
+  // Embed: hidden = embed_tokens[token_ids] (bf16, device-resident). res = 0.
+  // For a TEXT-only step the three mRoPE position streams are identical, so the
+  // partial NeoX RoPE in FullAttnBlock degenerates to 1-D RoPE over `positions`
+  // (notes §2). The vision tower / image-video merger are DEFERRED.
+  Tensor dtab = ResidentWeight(d, weights.embed_tokens, {vocab, H});
+  DBuf dids(d, DType::kI32, {T}, token_ids.data());
+  DBuf hidden(d, DType::kBF16, {T, H});
+  vt::Embedding(d.q, hidden.t(), dtab, dids.t());
+
+  DBuf res(d, DType::kF32, {T, H});
+  res.Zero(d);
+
+  for (int64_t l = 0; l < config.num_hidden_layers; ++l)
+    RunDenseLayer(d, weights.layers[static_cast<size_t>(l)], config, hidden, res,
+                  positions, T);
+
+  // Final RMSNorm over the fused stream (res += hidden; norm), then lm_head.
+  Tensor dfn = ResidentWeight(d, weights.final_norm, {H});
+  DBuf dnorm(d, DType::kBF16, {T, H});
+  vt::RmsNorm(d.q, dnorm.t(), hidden.t(), dfn, vt::RmsNormArgs{eps, true}, &res.t());
+
+  // lm_head is unquantized bf16 in the 27B (notes §3.6): the one host Download.
+  DBuf dlogits = MatmulF32D(d, dnorm.t(), weights.lm_head);
+  std::vector<float> logits(static_cast<size_t>(T) * vocab);
+  dlogits.Download(d, logits.data());
+  return logits;
+}
+
 std::vector<float> Qwen3_5ReplayLayer(const Qwen3_5MoeLayerWeights& layer,
                                       const HfConfig& config,
                                       const std::vector<float>& hidden_in,