27B: paged dense forward + runner route (M0.8 paged dense path)

Adds the batched/paged 27B dense text forward and wires the runner to it,
so the dense (Qwen3.6-27B W4A4) arch runs through the same PAGED engine as
the 35B MoE model. CPU-only; the W4A4 GPU GEMM + oracle golden stay GPU-gated.

- Qwen3_5DenseModel::Forward (qwen3_5.cpp): batched/paged dense forward with
  the same signature/structure as Qwen3_5Model::Forward (paged KV cache for
  full-attn, batched GDN recurrent state for GDN, f32 residual thread). Reuses
  the 35B GdnBlockPaged/FullAttnBlockPaged + paged machinery VERBATIM via a new
  RunDenseLayerPaged (copy of RunLayerPaged with DenseMlpBlock in place of
  MoeBlock). Single-seq ForwardDense kept as the paged==dense reference anchor.
- GPUModelRunner: Qwen3_5DenseWeights constructor overload + a {moe,dense}
  weights pointer pair (was a single MoE reference); execute_model routes to the
  dense forward when dense_weights_ is set. The MoE fp4 decode-graph fast path
  stays inert on the dense arch. initialize_kv_cache unchanged (config-driven).
- test_qwen27_paged_forward.cpp: 27B analogue of the 35B paged anchor test
  (paged==dense full-prefill, multi-block, decode-via-KV-cache, GDN-state
  zeroing), all within tolerance. Full CPU ctest 86/86 green; clean -Werror.
- Notes §5 step 3b checked off; deviations recorded in porting-inventory §9.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UJyFKcK62CcR3imhgbiBnW

Author: mudler

.agents/porting-inventory.md

@@ -131,7 +131,7 @@ implementation) is ported at T0.
 
 | Family | Marquee members | Needs | Tier |
 |---|---|---|---|
-| **Qwen3.5/3.6 hybrid (incl. MoE)** | `Qwen3_5ForConditionalGeneration` (27B dense-hybrid, **VL multimodal** wrapper — vision_config present), `Qwen3_5MoeForConditionalGeneration` / `qwen35moe` (35B-A3B) | GDN layers ×3 : 1 gated full-attn (qk-norm, partial RoPE 64d, output gate), MoE 256e top-8 + shared expert (35B) / **dense SwiGLU MLP** (27B), GemmaRMSNorm-style `(1+w)` — ✅ `25326fc` (35B forward correctness-grade, **safetensors**; 16/16 greedy on GB10 = M0 exit; GGUF k-quant load M0.10). **27B (co-equal gate):** CPU-first scaffolding started — arch/quant surveyed (`.agents/qwen27b-w4a4-notes.md`: dense hybrid, W4A4, **and a VL wrapper — text path first, ViT deferred**), CPU W4A4 emulation reference + skipping greedy-parity gate landed; dense forward + W4A4 GPU GEMM pending (GPU-gated). serving M1–M3 | **T0 (the gate)** |
+| **Qwen3.5/3.6 hybrid (incl. MoE)** | `Qwen3_5ForConditionalGeneration` (27B dense-hybrid, **VL multimodal** wrapper — vision_config present), `Qwen3_5MoeForConditionalGeneration` / `qwen35moe` (35B-A3B) | GDN layers ×3 : 1 gated full-attn (qk-norm, partial RoPE 64d, output gate), MoE 256e top-8 + shared expert (35B) / **dense SwiGLU MLP** (27B), GemmaRMSNorm-style `(1+w)` — ✅ `25326fc` (35B forward correctness-grade, **safetensors**; 16/16 greedy on GB10 = M0 exit; GGUF k-quant load M0.10). **27B (co-equal gate):** CPU-first scaffolding started — arch/quant surveyed (`.agents/qwen27b-w4a4-notes.md`: dense hybrid, W4A4, **and a VL wrapper — text path first, ViT deferred**), CPU W4A4 emulation reference + skipping greedy-parity gate landed; dense loader + single-seq `ForwardDense` + batched PAGED `Qwen3_5DenseModel::Forward` (paged==dense CPU-anchored) + `GPUModelRunner` dense route all landed CPU-green; W4A4 GPU GEMM + oracle golden pending (GPU-gated). serving M1–M3 | **T0 (the gate)** |
 | Dense decoders | Llama 3.x, Qwen2/3, Mistral, Gemma 2/3, Phi | GQA + RoPE + SwiGLU + RMSNorm (subset of T0 layer set) | T1 |
 | MoE decoders | Mixtral, Qwen3-MoE (30B-A3B), GLM-4-MoE, OLMoE | FusedMoE 🚧 `65788b3` (correctness-grade eager; grouped-GEMM perf M2.2) | T1 |
 | Qwen3-Next | `Qwen3NextForCausalLM` | same stack, interleaved-GQA weight layout | T1 |
@@ -288,7 +288,18 @@ Examples: `examples/cli` ✅ (C-API client), `examples/server` ✅ (OpenAI serve
    the later GPU step (qwen27b-w4a4-notes.md §5 steps 6-7), not a permanent
    deviation. Per-Linear bf16-vs-W4A4 routing is `IsQwen27QuantizedLinear`
    (encodes the checkpoint `ignore` list, §3.6). Text path only; the ViT/merger
-   and MTP head are deferred stubs.
+   and MTP head are deferred stubs. The batched PAGED 27B forward is
+   `Qwen3_5DenseModel::Forward` — same signature/structure as the 35B
+   `Qwen3_5Model::Forward`, reusing the file-local `GdnBlockPaged`/
+   `FullAttnBlockPaged`/paged machinery VERBATIM via `RunDenseLayerPaged` (the
+   only delta vs `RunLayerPaged` is `DenseMlpBlock` in place of `MoeBlock`).
+   Runner deviation: `GPUModelRunner` now carries EITHER arch via a
+   `{moe,dense}_weights_` pointer pair (was a single `Qwen3_5MoeWeights&`
+   reference) + a `Qwen3_5DenseWeights` constructor overload; `execute_model`
+   dispatches to the dense forward when `dense_weights_` is set. `initialize_kv_cache`
+   is unchanged (config-driven; same hybrid backbone). Full LoadedEngine dense
+   dispatch (arch-select in `model_loader.cpp`) is a small follow-up, gated behind
+   the W4A4 GPU GEMM (qwen27b-w4a4-notes.md §5 step 6).
 8. **Extension platforms** (T2): Apple Metal and Vulkan backends — upstream has
    no equivalent under `vllm/platforms/`; we add them through the mirrored
    Platform/AttentionBackend/vt-op seams so they behave as vLLM platforms

.agents/qwen27b-w4a4-notes.md

@@ -10,20 +10,29 @@ pinned upstream checkout `/home/mudler/_git/vllm` @ `e24d1b24`. Cites are
 Checkpoint (read-only on dgx):
 `~/.cache/huggingface/hub/models--unsloth--Qwen3.6-27B-NVFP4`.
 
-STATUS (2026-07-04): CPU-first correctness path LANDED (§5 steps 1-4). Delivered:
+STATUS (2026-07-04): CPU-first correctness path LANDED (§5 steps 1-4b). Delivered:
 the CPU W4A4 dequant + activation-quant reference (`nvfp4_emulation.h`,
 unit-tested); the dense loader (`LoadQwen3_5Dense` → `Qwen3_5DenseWeights`,
 `qwen3_5_dense.h`/`qwen3_5_dense_weights.cpp`) that routes each Linear bf16 vs
 W4A4-materialized-to-bf16 by name (`IsQwen27QuantizedLinear` +
 `MaterializeCtNvfp4Bf16Transposed`); the dense TEXT forward
 (`Qwen3_5DenseModel::ForwardDense` in `qwen3_5.cpp`, reusing the 35B GDN +
 gated-attn + norm helpers with the MoE block swapped for the dense SwiGLU MLP);
-and CPU unit tests (`test_qwen27_dense_forward.cpp`: routing + materialization +
-finite/deterministic forward + MLP-wired, 4 cases / 280 assertions, CPU-green).
-The full CPU ctest suite (83 targets) stays green. What is NOT done and is GPU-gated: the
-pip-vLLM oracle greedy golden capture (step 5), the W4A4 matmul kernel wiring
-(step 6), and flipping `kW4A4ForwardReady` to close the gate (step 7). §5 is the
-ordered plan, GPU steps marked.
+the batched PAGED dense forward (`Qwen3_5DenseModel::Forward`, same signature/
+structure as the 35B `Qwen3_5Model::Forward` — paged KV cache for full-attn,
+batched GDN state for GDN, via a new `RunDenseLayerPaged` reusing the 35B
+`GdnBlockPaged`/`FullAttnBlockPaged` VERBATIM) + the runner route (a dense-arch
+`GPUModelRunner` constructor overload holding `Qwen3_5DenseWeights` and routing
+`execute_model` to the dense forward; the MoE reference member became a
+`{moe,dense}_weights_` pointer pair); and CPU unit tests
+(`test_qwen27_dense_forward.cpp`: routing + materialization + finite/deterministic
+forward + MLP-wired, 4 cases / 280 assertions; `test_qwen27_paged_forward.cpp`:
+the paged==dense anchor + multi-block + decode-via-cache + GDN-state-zeroing, 4
+cases / 8 assertions, CPU-green). The full CPU ctest suite (86 targets) stays
+green. What is NOT done and is GPU-gated: the pip-vLLM oracle greedy golden
+capture (step 5), the W4A4 matmul kernel wiring (step 6), and flipping
+`kW4A4ForwardReady` to close the gate (step 7). §5 is the ordered plan, GPU steps
+marked.
 
 ---
 
@@ -284,17 +293,43 @@ Legend: **[CPU]** doable on the dev box now; **[GPU]** needs the free GB10.
    (finite + deterministic logits; MLP-perturbation moves the output). The ViT +
    image/video merger + MTP head are DEFERRED (text-first, §0.1) — no stub code,
    the loader simply does not request `model.visual.*`/`mtp.*`.
-   ⚠ REMAINING before step 5/7 flips the gate: (a) a paged `Qwen3_5DenseModel`
-   forward + a runner route (the 35B rides `Qwen3_5Model::Forward` + the runner;
-   the dense arch needs the analogous paged path OR the runner extended to the
-   dense weights) — the dense ForwardDense here is the single-sequence reference,
-   as the 35B's ForwardDense was before its paged refactor; (b) mrope_section
-   handling for genuine multimodal positions (inert for text).
+   `ForwardDense` is retained as the single-sequence parity reference (the
+   paged==dense anchor), exactly as the 35B's `ForwardDense`.
+   ⚠ REMAINING before step 7 flips the gate (all GPU-gated now): (a) the W4A4
+   matmul kernel (step 6); (b) mrope_section handling for genuine multimodal
+   positions (inert for text).
+
+3b. **[CPU] ✅ DONE — Paged dense path + runner wiring.**
+   `Qwen3_5DenseModel::Forward` (in `qwen3_5.cpp`) is the batched/paged 27B text
+   forward with the SAME signature/structure as `Qwen3_5Model::Forward`: paged KV
+   cache (`PagedKvCache`) for the full-attn layers + batched GDN recurrent state
+   (`GdnStateCache`) for the GDN layers + the f32 residual thread, per-layer
+   `fa_idx`/`gdn_idx` indexing identical to the 35B. It reuses the 35B
+   `GdnBlockPaged`/`FullAttnBlockPaged` + paged machinery VERBATIM via a new
+   `RunDenseLayerPaged` (a copy of `RunLayerPaged` with `DenseMlpBlock` in place
+   of `MoeBlock` and `Qwen3_5DenseLayerWeights` in place of the MoE weights). No
+   dense CUDA-graph driver (the MoE `Qwen3_5DecodeGraph` is an fp4/CUDA decode
+   optimization; the dense path runs eager, as its GPU GEMM is step 6). Runner
+   route: `GPUModelRunner` gained a `Qwen3_5DenseWeights` constructor overload;
+   its MoE reference member became a `{moe,dense}_weights_` pointer pair, and
+   `execute_model` routes to `Qwen3_5DenseModel::Forward` when `dense_weights_` is
+   set (the MoE-only fp4 decode-graph fast path stays inert on the dense arch).
+   `initialize_kv_cache` is unchanged (config-driven; same hybrid backbone).
+   CPU-validated by `test_qwen27_paged_forward.cpp` (the 27B analogue of the 35B
+   `test_qwen35_paged_forward.cpp`): paged==dense full-prefill, multi-block
+   (block_size<T non-contiguous), decode-via-KV-cache, and GDN-state-zeroing on a
+   garbage-seeded mamba block — all within tolerance (`max|diff|` 0 on the
+   zeroing/mixed-batch gate). Deviations recorded in porting-inventory §9.
+   Full LoadedEngine dense loading (config-arch dispatch in `model_loader.cpp` →
+   `LoadQwen3_5Dense` + a dense `LoadedEngine`/executor path) is NOT wired here —
+   the executor/engine stack is MoE-typed; that end-to-end plumbing is a small
+   follow-up once the GPU GEMM (step 6) makes a full 27B run meaningful.
 
 4. **[CPU] ✅ DONE — greedy-parity gate scaffold.**
    `test_qwen27_paged_engine.cpp` resolves the `unsloth/Qwen3.6-27B-NVFP4`
    snapshot and SKIPS (checkpoint-gated + `kW4A4ForwardReady=false`). Compiles
-   + links on CPU; flip the flag when steps 3+6 land.
+   + links on CPU; flip the flag when step 6 lands (the paged forward is now
+   ready — step 3b).
 
 5. **[GPU] Capture the pip-vLLM oracle greedy golden** (AGENTS.md STANDING
    DIRECTIVE; gates.md §PROTOCOL). Run pip-vLLM (`~/venvs/vllm-oracle`) on the

include/vllm/model_executor/models/qwen3_5_dense.h

@@ -28,6 +28,7 @@
 #include <string>
 #include <vector>
 
+#include "vllm/model_executor/models/qwen3_5.h"  // PagedKvCache, GdnStateCache + v1 attention metadata
 #include "vllm/model_executor/models/qwen3_5_weights.h"  // OwnedTensor, Gdn/FullAttn weights, TensorResolver
 #include "vllm/transformers_utils/hf_config.h"
 #include "vt/device.h"
@@ -98,6 +99,26 @@ Qwen3_5DenseWeights LoadQwen3_5Dense(const std::vector<SafetensorsFile>& shards,
 // logits [T, vocab] f32 (T = token_ids.size()). CPU or CUDA per `queue`.
 class Qwen3_5DenseModel {
  public:
+  // Batched PAGED dense forward — the 27B analogue of Qwen3_5Model::Forward.
+  // Same signature/structure (paged KV cache for the full-attn layers, batched
+  // GDN recurrent state for the GDN layers, the f32 residual thread), reusing the
+  // 35B GDN/FullAttn paged machinery VERBATIM with the dense SwiGLU MLP
+  // (RunDenseLayerPaged) in place of the MoE block. One PagedKvCache per full-attn
+  // layer + one GdnStateCache per GDN layer, in layer order. Returns
+  // [num_actual_tokens, vocab] f32 logits (lm_head applied). Runs on `queue`'s
+  // device. See qwen3_5.h::Qwen3_5Model::Forward for the metadata contract.
+  static std::vector<float> Forward(const std::vector<int32_t>& token_ids,
+                                    const std::vector<int32_t>& positions,
+                                    const v1::CommonAttentionMetadata& attn_meta,
+                                    const v1::GDNAttentionMetadata& gdn_meta,
+                                    const std::vector<PagedKvCache>& attn_kv,
+                                    const std::vector<GdnStateCache>& gdn_state,
+                                    const Qwen3_5DenseWeights& weights,
+                                    const HfConfig& config, vt::Queue& queue);
+
+  // Dense single-sequence reference forward (M0.9 anchor). Runs the whole model
+  // for a single non-paged sequence and returns logits [T, vocab] f32 (T =
+  // token_ids.size()). Retained as the paged==dense parity reference.
   static std::vector<float> ForwardDense(const std::vector<int32_t>& token_ids,
                                          const std::vector<int32_t>& positions,
                                          const Qwen3_5DenseWeights& weights,

include/vllm/v1/worker/gpu/runner.h

@@ -60,6 +60,7 @@
 #include <vector>
 
 #include "vllm/model_executor/models/qwen3_5.h"
+#include "vllm/model_executor/models/qwen3_5_dense.h"
 #include "vllm/model_executor/models/qwen3_5_weights.h"
 #include "vllm/transformers_utils/hf_config.h"
 #include "vllm/v1/attention/backend.h"
@@ -125,6 +126,17 @@ class GPUModelRunner final : public ModelRunnerBase {
                  int max_num_reqs, int max_model_len,
                  int max_num_batched_tokens);
 
+  // DENSE-arch overload (27B, Qwen3_5ForConditionalGeneration / num_experts==0).
+  // Identical to the MoE constructor except the model runs through the dense
+  // weights + the paged dense forward (Qwen3_5DenseModel::Forward). The KV-cache
+  // layout is config-driven (same GDN + full-attn hybrid backbone), so
+  // initialize_kv_cache is unchanged. `config` and `weights` must outlive the
+  // runner. The MoE-only fp4 decode-graph fast path stays inert on this arch.
+  GPUModelRunner(const HfConfig& config, const Qwen3_5DenseWeights& weights,
+                 const KVCacheConfig& kv_cache_config, vt::Queue queue,
+                 int max_num_reqs, int max_model_len,
+                 int max_num_batched_tokens);
+
   // ModelRunnerBase (the MRV2 execute_model / sample_tokens split).
   std::optional<ModelRunnerOutput> execute_model(
       const SchedulerOutput& scheduler_output) override;
@@ -158,7 +170,11 @@ class GPUModelRunner final : public ModelRunnerBase {
                                           int* num_cols) const;
 
   const HfConfig& config_;
-  const Qwen3_5MoeWeights& weights_;
+  // Exactly one of {moe_weights_, dense_weights_} is non-null, selecting the MoE
+  // (35B) or dense (27B) forward. Held by pointer (not reference) so the single
+  // runner class carries either arch; both are borrowed (must outlive the runner).
+  const Qwen3_5MoeWeights* moe_weights_ = nullptr;
+  const Qwen3_5DenseWeights* dense_weights_ = nullptr;
   vt::Queue queue_;
   InputBatch input_batch_;
   Sampler sampler_;

src/vllm/model_executor/models/qwen3_5.cpp

@@ -1165,6 +1165,44 @@ void RunLayerPaged(Dev d, const Qwen3_5MoeLayerWeights& layer, const HfConfig& c
   hidden = MoeBlock(d, layer.moe, cfg, dh2.t(), T);
 }
 
+// Batched PAGED dense decoder layer (27B; notes §5). Identical residual/norm
+// thread + paged attention wiring as RunLayerPaged, but the MoE block is swapped
+// for the dense SwiGLU MLP (DenseMlpBlock) and the layer carries dense weights.
+// The GDN / full-attn paged blocks are the 35B helpers reused VERBATIM. Exactly
+// one of {attn_kv, gdn_state} is non-null (per layer type).
+void RunDenseLayerPaged(Dev d, const Qwen3_5DenseLayerWeights& layer,
+                        const HfConfig& cfg, DBuf& hidden, DBuf& res,
+                        const std::vector<int32_t>& positions,
+                        const CommonAttentionMetadata& attn_meta,
+                        const GDNAttentionMetadata& gdn_meta,
+                        const PagedKvCache* attn_kv,
+                        const GdnStateCache* gdn_state, int64_t T) {
+  const int64_t H = cfg.hidden_size;
+  const float eps = static_cast<float>(cfg.rms_norm_eps);
+
+  Tensor dw_in = ResidentWeight(d, layer.input_layernorm, {H});
+  DBuf dhn(d, DType::kBF16, {T, H});
+  vt::RmsNorm(d.q, dhn.t(), hidden.t(), dw_in, vt::RmsNormArgs{eps, true}, &res.t());
+
+  DBuf attn = [&] {
+    if (layer.is_linear_attention) {
+      VT_CHECK(gdn_state != nullptr,
+               "paged dense layer: GDN layer needs a GdnStateCache");
+      return GdnBlockPaged(d, layer.gdn, cfg, dhn.t(), gdn_meta, *gdn_state, T);
+    }
+    VT_CHECK(attn_kv != nullptr,
+             "paged dense layer: full-attn layer needs a PagedKvCache");
+    return FullAttnBlockPaged(d, layer.attn, cfg, dhn.t(), positions, attn_meta,
+                              *attn_kv, T);
+  }();
+
+  Tensor dw_post = ResidentWeight(d, layer.post_attention_layernorm, {H});
+  DBuf dh2(d, DType::kBF16, {T, H});
+  vt::RmsNorm(d.q, dh2.t(), attn.t(), dw_post, vt::RmsNormArgs{eps, true}, &res.t());
+
+  hidden = DenseMlpBlock(d, layer.mlp, cfg, dh2.t(), T);
+}
+
 }  // namespace
 
 // Embed: hidden[T,H] bf16 = embed_tokens[token_ids] (device-resident table).
@@ -1384,6 +1422,74 @@ std::vector<float> Qwen3_5DenseModel::ForwardDense(
   return logits;
 }
 
+std::vector<float> Qwen3_5DenseModel::Forward(
+    const std::vector<int32_t>& token_ids, const std::vector<int32_t>& positions,
+    const CommonAttentionMetadata& attn_meta, const GDNAttentionMetadata& gdn_meta,
+    const std::vector<PagedKvCache>& attn_kv,
+    const std::vector<GdnStateCache>& gdn_state,
+    const Qwen3_5DenseWeights& weights, const HfConfig& config,
+    vt::Queue& queue) {
+  // Same shape/count contract as Qwen3_5Model::Forward (CheckPagedForward), over
+  // the dense weights. One PagedKvCache per full-attn layer, one GdnStateCache
+  // per GDN layer, in layer order.
+  const int64_t T = static_cast<int64_t>(token_ids.size());
+  const int64_t H = config.hidden_size;
+  const int64_t vocab = config.vocab_size;
+  VT_CHECK(T > 0, "qwen3_5 dense paged forward: empty token_ids");
+  VT_CHECK(static_cast<int64_t>(positions.size()) == T,
+           "qwen3_5 dense paged forward: positions length must equal token count");
+  VT_CHECK(static_cast<int64_t>(weights.layers.size()) == config.num_hidden_layers,
+           "qwen3_5 dense paged forward: weights.layers size must equal "
+           "num_hidden_layers");
+  VT_CHECK(attn_meta.num_actual_tokens == T,
+           "qwen3_5 dense paged forward: attn_meta.num_actual_tokens must equal T");
+  int64_t n_full = 0, n_gdn = 0;
+  for (const auto& l : weights.layers) (l.is_linear_attention ? n_gdn : n_full) += 1;
+  VT_CHECK(static_cast<int64_t>(attn_kv.size()) == n_full,
+           "qwen3_5 dense paged forward: attn_kv count must equal full-attn layers");
+  VT_CHECK(static_cast<int64_t>(gdn_state.size()) == n_gdn,
+           "qwen3_5 dense paged forward: gdn_state count must equal GDN layers");
+
+  Dev d{vt::GetBackend(queue.device.type), queue};
+  const float eps = static_cast<float>(config.rms_norm_eps);
+
+  // Embed: hidden = embed_tokens[token_ids] (bf16, device-resident). res = 0.
+  // Text-only: the three mRoPE streams coincide so partial NeoX RoPE degenerates
+  // to 1-D RoPE over `positions` (notes §2); the vision tower is DEFERRED.
+  Tensor dtab = ResidentWeight(d, weights.embed_tokens, {vocab, H});
+  DBuf dids(d, DType::kI32, {T}, token_ids.data());
+  DBuf hidden(d, DType::kBF16, {T, H});
+  vt::Embedding(d.q, hidden.t(), dtab, dids.t());
+
+  DBuf res(d, DType::kF32, {T, H});
+  res.Zero(d);
+
+  // N paged decoder layers: full-attn layers read/write attn_kv[fa_idx], GDN
+  // layers the persistent gdn_state[gdn_idx] (same layer-order indexing as the
+  // 35B paged forward).
+  int64_t fa_idx = 0, gdn_idx = 0;
+  for (int64_t l = 0; l < config.num_hidden_layers; ++l) {
+    const Qwen3_5DenseLayerWeights& layer = weights.layers[static_cast<size_t>(l)];
+    const PagedKvCache* kv =
+        layer.is_linear_attention ? nullptr : &attn_kv[static_cast<size_t>(fa_idx++)];
+    const GdnStateCache* gs =
+        layer.is_linear_attention ? &gdn_state[static_cast<size_t>(gdn_idx++)] : nullptr;
+    RunDenseLayerPaged(d, layer, config, hidden, res, positions, attn_meta,
+                       gdn_meta, kv, gs, T);
+  }
+
+  // Final RMSNorm over the fused stream (res += hidden; norm), then lm_head.
+  Tensor dfn = ResidentWeight(d, weights.final_norm, {H});
+  DBuf dnorm(d, DType::kBF16, {T, H});
+  vt::RmsNorm(d.q, dnorm.t(), hidden.t(), dfn, vt::RmsNormArgs{eps, true}, &res.t());
+
+  // lm_head is unquantized bf16 in the 27B (notes §3.6): the one host Download.
+  DBuf dlogits = MatmulF32D(d, dnorm.t(), weights.lm_head);
+  std::vector<float> logits(static_cast<size_t>(T) * vocab);
+  dlogits.Download(d, logits.data());
+  return logits;
+}
+
 std::vector<float> Qwen3_5ReplayLayer(const Qwen3_5MoeLayerWeights& layer,
                                       const HfConfig& config,
                                       const std::vector<float>& hidden_in,