Skip to content

feat(qwen36): CUDA VRAM expert tier — heat-based placement across GPUs via the shared CUDA backend - #713

Draft
kreuzzelg wants to merge 8 commits into
JustVugg:devfrom
kreuzzelg:qwen36-cuda-tier
Draft

feat(qwen36): CUDA VRAM expert tier — heat-based placement across GPUs via the shared CUDA backend#713
kreuzzelg wants to merge 8 commits into
JustVugg:devfrom
kreuzzelg:qwen36-cuda-tier

Conversation

@kreuzzelg

@kreuzzelg kreuzzelg commented Jul 30, 2026 •
edited
Loading

Copy link
Copy Markdown

Depends on: #712 (qwen36 engine) — this branch includes those commits; only the tier commit is new here.

Summary

Applies colibri's placement concept — route → place → overlap → learn — to
Qwen3.6 one level up from the GLM disk tier: RAM → VRAM. All experts stay in RAM; routing heat promotes the hot ones into DEVICE_LOCAL VRAM across one or more GPUs, computed through the existing backend_cuda.cu
expert-group API (no new backend, unlike the Vulkan approach in #602).

  • one home device per expert (eid % n_gpus), no duplicates
  • tier.h LFRU semantics + hysteresis; parallel warmstart fills the budget before token 1; HEAT_FILE persists heat across runs ("the more you run, the hotter the right experts get")
  • async expert groups on all devices; VRAM misses fall back to the CPU int8 path, overlapped with the in-flight groups (as is the shared expert)
  • warmstart frees the RAM int8 copies of VRAM residents (rematerialized from packed int4 on LFRU eviction) → 35B int4 fits in ~29 GB RSS
  • default build stays CPU-only (inline stubs); CUDA=1 compiles the tier

Try it

Pre-converted self-contained int4 container (all numbers above were measured
with it): https://huggingface.co/Kreuzzelg/qwen36-35b-a3b-colibri-i4

Limitations

  • int4 containers only (int8 containers keep the CPU path)
  • requires cap == n_experts (full RAM residency)
  • single-stream decode (no batching yet)

Thanks on Claude Code

@kreuzzelg kreuzzelg mentioned this pull request Jul 30, 2026
Closed
5 tasks
@terrizoaguimor terrizoaguimor mentioned this pull request Jul 31, 2026
Open
5 tasks
@JustVugg JustVugg added cuda Backend CUDA/NVIDIA model-support Supporto a nuovi modelli labels Jul 31, 2026
This was referenced Aug 1, 2026
Open
Merged
kreuzzelg and others added 8 commits August 3, 2026 21:16
@kreuzzelg @minne100 @CniHam
feat(qwen36): add Qwen3.6-35B-A3B engine (CPU): hybrid Gated Attentio…
0e5807b 
…n + Gated DeltaNet + streaming MoE

Self-contained engine for Qwen3.6-35B-A3B (35B total / 3B active, 256 experts
per layer), split out of PR JustVugg#602 per maintainer guidance (engine first, GPU
backends as follow-ups; no shared GLM/Inkling files touched):

- c/qwen36.c: 40-layer hybrid forward (10x Gated Attention with partial RoPE
  + per-head q/k RMSNorm; 30x Gated DeltaNet recurrent linear attention with
  carried conv ring + per-head state), streaming MoE with per-layer LRU expert
  cache, shared expert, router lookahead prefetch (PILOT), resident-expert
  pinning (COLIBRI_RESIDENT), per-phase timers (COLI_TIMERS=1).
- CPU performance work: optional per-row int8 quantization of the large dense
  matrices (COLI_DENSE_I8, on by default, f32 originals freed), hand-written
  AVX2/FMA int8 GEMV, DeltaNet recurrence parallelized per value head.
  Decode on a 12-core Zen2 + int4 container: ~2.5 tok/s CPU-only
  (vs 0.35 tok/s scalar baseline).
- c/tools/convert_qwen36.py: HF -> container converter (true int4 packing,
  per-row scales, --selftest). Containers now run out of the box: config.json
  is written flat (VL checkpoints nest dims under text_config) and
  tokenizer.json is bundled.
- tools/make_qwen36_tiny.py + make_qwen36_oracle.py: weight-free tiny model
  and oracle for engine validation without the 70 GB checkpoint.
- docs/qwen36-phase01.md, qwen36-phase02.md: design notes.

Numerics: logits cosine vs f32 reference 0.9992 with COLI_DENSE_I8=1,
bit-identical with COLI_DENSE_I8=0.

Co-authored-by: minne100 <xufenglu@hotmail.com>
@minne100 @CniHam
feat(qwen36): make engine drivable by coli + SERVE=1 gateway protocol
72dd8a9 
Addresses the main blocker on JustVugg#712: the engine built clean and touched no
existing files, but coli could not reach it.

c/coli:
- model_arch(): return 'qwen' for any model_type containing 'qwen'
- engine_for(): resolve 'qwen36' binary
- need_model(): qwen36 build target
- chat banner model_id: qwen36-colibri

c/qwen36.c (SERVE=1 mode):
- implements the colibri gateway wire protocol (READY + STAT + SUBMIT/
  CANCEL + DATA + DONE + PROF), mirroring kimi_k3.c / inkling.c so
  'coli chat' / 'coli web' / 'coli serve' can drive the engine
- READY/STAT handshake, SUBMIT payload -> encode_text -> step() prefill +
  per-token step() decode, stream decoded bytes as DATA frames, stop on
  EOS (Qwen3 151645, Q36_EOS override), DONE/STAT on completion
- temperature + top-p sampler (ported from kimi_k3.c, qsort O(V log V))
- Windows binary-mode fix inlined (JustVugg#748 hang: CRT rewrites n->rn and the
  gateway never matches READY). Uses _setmode(_fileno(stdout/stdin),
  _O_BINARY) before the sentinel; switch to compat.h's
  coli_serve_binary_mode() once JustVugg#749 lands (helper not in base yet).
@kreuzzelg @CniHam
feat(qwen36): group-scaled int4 (gs64) containers — converter flag + …
85ad00a 
…CPU read path

convert_qwen36.py --gs N writes one f32 scale per N input elements per row
(scales [O, ceil(I/gs)], expert_gs in qwen36_meta.json) instead of one per
row. The engine reads expert_gs and dispatches expert GEMVs through
matmul_qe: per-row scales unchanged, grouped scales via matmul_q_gs.

Same trade as GLM's gs64 containers: ~9% more container bytes for scales
that track short rows (moe_intermediate_size=512) much more closely.
Validated on the tiny fixture: grouped beats per-row on cosine to the int8
anchor. Full-model A/B on the published containers is in the PR thread.
@kreuzzelg @CniHam
docs(qwen36): user-facing page — prebuilt containers as the easy path
0e9b3c7 
Mirrors docs/inkling.md's structure: pre-converted containers first
(gs64 recommended, per-row available), converter as the DIY route,
coli chat as the primary interface now that the engine speaks the serve
protocol. The container recommendation follows the GLM JustVugg#455 experience;
the per-row-vs-gs64 measurement backing it is in the PR thread.
@kreuzzelg @CniHam
fix(qwen36): serve mode must not require an argv prompt file
4891f5c 
main loaded the prompt (argv[3], default ref.json) before the SERVE
branch, so an engine launched the way coli launches it — SERVE=1 and no
prompt argument — exited 1 on the missing ref.json before ever emitting
READY. Serve gets its prompts over the wire; skip the argv prompt load
in serve mode. Found by running the serve handshake end-to-end.
@kreuzzelg @CniHam
fix(qwen36): serve EOS ids from the tokenizer, not the 151k-vocab con…
c562cc2 
…stant

serve_one compared against a hardcoded 151645 — <|im_end|> in the older
151k-token Qwen vocabs. Qwen3.6's vocab is 248320 tokens with the specials
at 248044+ (<|endoftext|> 248044, <|im_end|> 248046), so EOS never matched
and every serve turn ran to max_tok; in the worst case 151645 is an
ordinary BPE piece here and could truncate a completion mid-sentence.
Resolve <|im_end|> and <|endoftext|> from the tokenizer's added_tokens
(same spirit as kimi_k3's config-driven eos list); Q36_EOS still
overrides. Found by the serve-path termination check the A/B comment
called for.
@kreuzzelg @CniHam
feat(qwen36): qwen chat-template family in the gateway
853391c 
openai_server.py knew glm/inkling/kimi only, so a qwen model fell to the
GLM renderer and the engine never saw its native frames. render_chat_qwen
mirrors Qwen3.6's chat_template byte for byte in the text-only subset
(inkling's approach): <|im_start|>role\n...<|im_end|>\n frames and a
generation prompt that OPENS the mandatory <think> block — the model was
never trained on a bare 'assistant\n' state, and greedy argmax there is
an EOS special (measured: gen=0 on the very first token). Thinking
disabled pre-closes the block, exactly like the official jinja.

Arch detection, --arch choice, renderer dispatch and the single-KV-slot
guard follow the kimi pattern. Spawn needs nothing new: argv is (cap),
bits defaults to 4, tokenizer comes from SNAP.
@kreuzzelg @CniHam
feat(qwen36): CUDA VRAM expert tier — heat-based placement across GPU…
1c53042 
…s via the shared CUDA backend

Promote hot experts into DEVICE_LOCAL VRAM (one home device per expert,
eid % n_gpus) and compute them through the existing backend_cuda.cu
expert-group API; no new backend. Routing heat + tier.h LFRU semantics decide
placement, a parallel warmstart fills the budget before the first token
(persisted across runs via HEAT_FILE), uploads run on a background thread,
and VRAM misses fall back to the overlapped CPU int8 path — placement never
changes routing or precision.

- c/qwen36_tier.{c,h}: tier implementation; inline stubs keep the default
  build CPU-only, CUDA=1 compiles the tier against the shared backend.
- c/qwen36.c: moe() integration (issue -> CPU misses + shared expert
  overlapped -> take), warmstart that loads all experts to RAM and frees the
  int8 copies of VRAM residents (rematerialized from packed int4 on
  eviction), tier telemetry.
- docs/qwen36-cuda-tier.md: design + measured results.

Measured (12-core Zen2, RTX 3070 + Quadro RTX 4000, 35B int4, 200 tokens):
11.3 tok/s decode with two 8 GB GPUs (100% VRAM hit warm, 29 GB RSS),

Rebased onto dev with JustVugg#762 merged (the async grouped-int4 dispatch this
tier rides on) and extended for gs64 containers: qt_init takes expert_gs,
experts upload as fmt=4 with grouped scales via coli_cuda_tensor_upload_g
when expert_gs>0, CPU-miss fallbacks dispatch through matmul_qe.
@kreuzzelg

kreuzzelg commented Aug 3, 2026

Copy link
Copy Markdown
Author

Rebased onto dev with #762 merged — the async grouped-int4 dispatch this tier rides on — and onto the current #712 branch (serve + gs64), extended for gs64 containers: qt_init takes expert_gs, experts upload as fmt=4 with grouped scales via coli_cuda_tensor_upload_g, CPU-miss fallbacks dispatch through the same grouped GEMV the engine uses.

Verified on a single RTX 3070: output bit-identical to the CPU run on the gs64 container (greedy, 200 tokens), 11.94 tok/s with 3731/10240 experts resident (6.15 GB VRAM), and the serve path rides the tier unchanged (tier init runs before serve_loop).

Staying draft until #712 lands; after that this is one commit on top of it.

@JustVugg

JustVugg commented Aug 4, 2026

Copy link
Copy Markdown
Owner

Same situation as #712 — see my note there. #165 has landed on dev; your render_chat_qwen and the new render_chat_v4 collide textually in c/openai_server.py without contesting anything, but the hunks interleave both function bodies, so the resolution belongs with you rather than with me.

Your CI had never run either — it was stuck in action_required. It is running now, so check the results before you rebase.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

cuda Backend CUDA/NVIDIA model-support Supporto a nuovi modelli

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@kreuzzelg @JustVugg @minne100