feat(qwen36): Qwen3.6-35B-A3B engine (CPU): hybrid Gated Attention + Gated DeltaNet + streaming MoE - #712
feat(qwen36): Qwen3.6-35B-A3B engine (CPU): hybrid Gated Attention + Gated DeltaNet + streaming MoE#712kreuzzelg wants to merge 7 commits into
Conversation
|
Can you add Qwen 27B support? It would be awesome to use since it's pretty much entirely out of reach for my system right now. 32 GB ram, 12 GB 4070 super, 12700k. 27B support with your method would make it usable and fit entirely in VRAM i think |
|
Hi jazir555,
Thanks for the suggestion! Unfortunately I can't add dense-27B support to colibri the way you're hoping, and here's the honest reason why.
colibri's whole trick is expert streaming ― for MoE models we only keep the active experts (plus shared experts + attention) resident and stream the rest from disk on demand. That's what lets a 30B/3B-active model run on a 16 GB laptop. Your 12 GB 4070 Super is exactly the kind of machine this is built for if the model is MoE.
A dense 27B (e.g. Qwen3-32B / Qwen2.5-32B) has no experts to stream ― every parameter is needed for every token. Even in int4 that's ~16 GB, which already exceeds your 12 GB VRAM before you add KV cache and CUDA overhead. The only way to run it would be CPU offload + paging/swap, which is slow and is precisely what llama.cpp / KTransformers already do very well. Building dense-27B support into colibri would just be a worse llama.cpp, so it's not worth doing.
If you actually meant Qwen3-30B-A3B (the MoE: 30.5B total / 3.3B active), that's a completely different story ― it fits your 12 GB card comfortably (active weights are only ~1.65 GB in int4, and pinning the hot experts keeps the whole working set in VRAM). I'd happily add that. Just say the word.
Cheers
…________________________________
[https://avatars.githubusercontent.com/u/31750063?s=20&v=4]jazir555 left a comment (JustVugg/colibri#712)<#712 (comment)>
Can you add Qwen 27B support? It would be awesome to use since it's pretty much entirely out of reach for my system right now. 32 GB ram, 12 GB 4070 super, 12700k. 27B support with your method would make it usable and fit entirely in VRAM i think
―
Reply to this email directly, view it on GitHub<#712?email_source=notifications&email_token=AAX3ZWIUIO6UAHNLEONKX3L5HUCVJA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTKMJUG42DIMRSGQ42M4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJLDGN5XXIZLSL5RWY2LDNM#issuecomment-5147442249>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AAX3ZWJILVQV3ASEJ542MO35HUCVJAVCNFSNUABGKJSXA33TNF2G64TZHMYTEOBWGA4DAMZZG45US43TOVSTWNJQGE3TMOJXGMYTPILWAI>.
Triage notifications, keep track of coding agent tasks and review pull requests on the go with GitHub Mobile for iOS<https://github.com/notifications/mobile/ios/AAX3ZWNFKWKIGW236QZGRJD5HUCVJA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTKMJUG42DIMRSGQ42M4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJKTGN5XXIZLSL5UW64Y> and Android<https://github.com/notifications/mobile/android/AAX3ZWOUE3MO3VRVY7KI47T5HUCVJA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTKMJUG42DIMRSGQ42M4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJLTGN5XXIZLSL5QW4ZDSN5UWI>. Download it today!
You are receiving this because you were mentioned.Message ID: ***@***.***>
|
…wrong
The site said colibri runs GLM-5.2. It has run four families since v1.3.0, and
two of the model cards contradicted the project outright:
Inkling 975B MoE - Planned -> Live (docs/inkling.md ships; runs on 25 GB)
Kimi K2 1T MoE - Planned -> Kimi K3, 2.8T MoE, Live
Telling visitors that Inkling and Kimi are on the roadmap, while the README
front page says both run today, is the kind of contradiction someone finds in
thirty seconds.
Hero rewritten rather than merely widened. The old line worked because it put
two incompatible things next to each other -- an enormous model, your machine.
Replacing that with a range ('744B to 2.8T') informs and stops landing; the
contradiction was the message. It now reads:
These models do not fit in your machine. They run in it anyway.
Same rhetorical shape as the copy further down the page ('Weights are not state
to hold. They are data to stage.'), so the page speaks with one voice. The
subtitle now also explains WHY it is possible -- a MoE token touches a small
fraction of the weights -- which was missing entirely and is what turns an
unbelievable claim into an understandable one. The numbers move there, where
they serve the reader who wants detail instead of the one who is skimming.
DeepSeek and Qwen3 deliberately stay 'Planned': JustVugg#165 and JustVugg#712 are not merged,
and the site should not promise what the code does not do.
Text only. No CSS, structure or script changes -- the sole markup edits are the
two cards' buttons becoming real links now that both models are runnable.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
@minne100 already exists #544 |
|
Reviewed this properly, and I owe you both a compliment and a correction of something I said internally about it. The file shape is exactly right. But it cannot be driven by
So What it takes1. # model_arch(), c/coli:160
if "qwen" in model_type: return "qwen" # config.json says model_type: qwen3_5_moe
# engine_for(), c/coli:176
name = "inkling" if arch == "inkling" else "kimi_k3" # -> add "qwen36"
2. Serve mode in the engine. 3. One thing that will bite you on Windows, which we only found today. Call Two questionsWhich container should people use? Qwen3.6-35B-A3B upstream is bf16, so it needs converting — Per-row int4 or gs64? Your container is per-row. On GLM, per-row int4 was the root cause of #455's think-mode loops and never-terminating generations, and the group-scaled gs64 container fixed those controlled A/Bs — that is why the GLM README steers people away from the per-row mirrors. With Related#729 (@minne100) is a Vulkan MoE GEMV backend for integrated/AMD GPUs, explicitly written as the GPU half for this engine and currently a draft blocked on this PR landing. Worth the two of you talking — there is also a question there about a vendored 549 KB No rush from us, and none of this is a criticism of the engine — 2,009 lines that build clean and leave every existing file untouched is a good PR. It just needs the front end before people can actually reach it. |
Addresses the main blocker on JustVugg#712: the engine built clean and touched no existing files, but coli could not reach it. c/coli: - model_arch(): return 'qwen' for any model_type containing 'qwen' - engine_for(): resolve 'qwen36' binary - need_model(): qwen36 build target - chat banner model_id: qwen36-colibri c/qwen36.c (SERVE=1 mode): - implements the colibri gateway wire protocol (READY + STAT + SUBMIT/ CANCEL + DATA + DONE + PROF), mirroring kimi_k3.c / inkling.c so 'coli chat' / 'coli web' / 'coli serve' can drive the engine - READY/STAT handshake, SUBMIT payload -> encode_text -> step() prefill + per-token step() decode, stream decoded bytes as DATA frames, stop on EOS (Qwen3 151645, Q36_EOS override), DONE/STAT on completion - temperature + top-p sampler (ported from kimi_k3.c, qsort O(V log V)) - Windows binary-mode fix inlined (JustVugg#748 hang: CRT rewrites n->rn and the gateway never matches READY). Uses _setmode(_fileno(stdout/stdin), _O_BINARY) before the sentinel; switch to compat.h's coli_serve_binary_mode() once JustVugg#749 lands (helper not in base yet).
|
@maintainer — pushed the 1.
|
|
@JustVugg — ran the per-row vs gs64 A/B you asked for. Short version: gs64 is measurably more accurate on Qwen (≈44 % less quantization error), but per-row does not reproduce the #455 loop symptom here — so switching is a quality improvement, not a bug fix. Details below; both containers built from the same bf16 base, same engine, same prompts, only the expert quant differs. 1. Accuracy — first-token logit distribution vs an int8 anchorint8 (per-row) container as the near-lossless reference; both int4 containers compared against it. 4 prompts (MoE-explain, a multi-step math word problem, a primes/twin-primes list, a self-critique task).
gs64 wins on cosine on all 4 prompts and on KL on 3 of 4 (p4 is a wash on cosine and slightly worse on KL — honest outlier, not cherry-picked out). Mean cosine error to the anchor drops 0.01223 → 0.00687, i.e. ~44 % less. So your intuition that per-row loses more with 2. Behavior — the #455 symptom does not reproduce on QwenSame 4 prompts, greedy decode, 512 tokens, both containers. I checked for the degenerate-repetition / non-termination pattern from #455 (top-4-gram repeat count, unique-token ratio):
Both generate healthy, non-repetitive text on all 4 prompts. Whatever made per-row int4 loop on GLM think-mode, it isn't firing on Qwen3.6-35B-A3B here. (Caveat: this engine's 3. RecommendationMake gs64 the default converter output — it's a free accuracy win (44 % lower logit error, no measured downside except ~10 % larger container: 20 → 22 GB for the 35B). But since per-row is not broken on Qwen, I'd suggest keeping per-row selectable ( Implementation is already on the branch ( Containers (both self-contained, produced with this converter):
Raw numbers + harness are in the branch; happy to paste the full per-prompt Thanks to Claude :-) |
|
Hi @minne100 — thanks for turning the One thing we'd like to keep from #712 so it isn't lost when this one closes — all small and self-contained, on top of the base engine:
Whatever's least work for you: we can (a) hand you these as cherry-pickable commits / a patch to fold into your PR, or (b) submit them as one small follow-up PR stacked on yours. We slightly lean toward (b) since the maintainer asked for small reviewable PRs, but your call — you own the engine head. Once your PR is the engine base, we'll close #712 and rebase our #713 (CUDA VRAM expert tier) onto your branch so it stacks cleanly on top. #713 stays our PR (dedicated-NVIDIA path), your Vulkan #729 stays yours (iGPU/AMD) — complementary, as discussed. On the containers: no ownership issue from our side — we published the two under the Just say which option (a/b) you prefer and I'll get our three items over to you. |
|
@minne100 — to save you the wait on whichever option you pick, I've prepared our three items as clean single-purpose commits on top of your Branch:
Each builds and passes tiny-model validation on your branch (per-row unchanged, grouped path verified, dense-int8 cosine 0.9992 / bit-exact with COLI_DENSE_I8=0). git remote add kreuzzelg https://github.com/kreuzzelg/colibri
git fetch kreuzzelg
git cherry-pick c1677f6 ca9cca2 94495ec # or pick a subsetIf you'd rather I open them as a follow-up PR on top of your fresh PR (option b), just say so. |
|
Heads-up on a gs64 + GPU-tier gap I hit while benchmarking, so nobody trips over it: gs64 containers don't run on the CUDA VRAM tier yet — gs64 is CPU-path only right now. The two pieces were built on separate branches and haven't met:
So a gs64 container loaded by the tier binary is rejected at load (
The good news: |
|
Follow-up on the gs64 note above: the reason gs64 containers were CPU-only so far is a backend gap, not an engine limitation — the async CUDA expert-group path ( Once #762 lands, the gs64 containers ( |
|
how about cherry-picking bcae842 onto this branch rather than starting over with the original PR that was dead for a week until you revived it here? the same might happen again - it appears the original author has way less time to work on this than you do. |
|
@LionelColaso — the pre-converted container is up: Recap from the verification above: KAT-Coder-V2.5-Dev shares the Qwen3.6-35B-A3B architecture field for field, so the qwen36 engine runs it with zero code changes. I published the gs64 (group-scaled int4) variant — on the sibling Which commits you need (until the PRs land):
One caveat worth knowing: there are two KAT-Coder-V2.5 versions. The paper's scores are for the commercial flagship (e.g. SWE-Bench Pro 65.2, Terminal-Bench 2.1 60.7); the open weights — and this container — are KAT-Coder-V2.5-Dev (35B/A3B, text-only, Apache-2.0), which reports considerably lower agentic-coding scores (SWE-Bench Pro 45.96, Terminal-Bench 2.1 41.02, but SWE-Bench Verified 69.40). The model card has the full comparison table, so nobody benchmarks the Dev container against flagship expectations. Also still true: it's a Since you seem to follow the KAT releases: do you know anything more about the commercial flagship V2.5 — parameter count, architecture, whether it stays closed or open weights are planned? Kwaipilot doesn't disclose specs in the |
…n + Gated DeltaNet + streaming MoE Self-contained engine for Qwen3.6-35B-A3B (35B total / 3B active, 256 experts per layer), split out of PR JustVugg#602 per maintainer guidance (engine first, GPU backends as follow-ups; no shared GLM/Inkling files touched): - c/qwen36.c: 40-layer hybrid forward (10x Gated Attention with partial RoPE + per-head q/k RMSNorm; 30x Gated DeltaNet recurrent linear attention with carried conv ring + per-head state), streaming MoE with per-layer LRU expert cache, shared expert, router lookahead prefetch (PILOT), resident-expert pinning (COLIBRI_RESIDENT), per-phase timers (COLI_TIMERS=1). - CPU performance work: optional per-row int8 quantization of the large dense matrices (COLI_DENSE_I8, on by default, f32 originals freed), hand-written AVX2/FMA int8 GEMV, DeltaNet recurrence parallelized per value head. Decode on a 12-core Zen2 + int4 container: ~2.5 tok/s CPU-only (vs 0.35 tok/s scalar baseline). - c/tools/convert_qwen36.py: HF -> container converter (true int4 packing, per-row scales, --selftest). Containers now run out of the box: config.json is written flat (VL checkpoints nest dims under text_config) and tokenizer.json is bundled. - tools/make_qwen36_tiny.py + make_qwen36_oracle.py: weight-free tiny model and oracle for engine validation without the 70 GB checkpoint. - docs/qwen36-phase01.md, qwen36-phase02.md: design notes. Numerics: logits cosine vs f32 reference 0.9992 with COLI_DENSE_I8=1, bit-identical with COLI_DENSE_I8=0. Co-authored-by: minne100 <xufenglu@hotmail.com>
Addresses the main blocker on JustVugg#712: the engine built clean and touched no existing files, but coli could not reach it. c/coli: - model_arch(): return 'qwen' for any model_type containing 'qwen' - engine_for(): resolve 'qwen36' binary - need_model(): qwen36 build target - chat banner model_id: qwen36-colibri c/qwen36.c (SERVE=1 mode): - implements the colibri gateway wire protocol (READY + STAT + SUBMIT/ CANCEL + DATA + DONE + PROF), mirroring kimi_k3.c / inkling.c so 'coli chat' / 'coli web' / 'coli serve' can drive the engine - READY/STAT handshake, SUBMIT payload -> encode_text -> step() prefill + per-token step() decode, stream decoded bytes as DATA frames, stop on EOS (Qwen3 151645, Q36_EOS override), DONE/STAT on completion - temperature + top-p sampler (ported from kimi_k3.c, qsort O(V log V)) - Windows binary-mode fix inlined (JustVugg#748 hang: CRT rewrites n->rn and the gateway never matches READY). Uses _setmode(_fileno(stdout/stdin), _O_BINARY) before the sentinel; switch to compat.h's coli_serve_binary_mode() once JustVugg#749 lands (helper not in base yet).
…CPU read path convert_qwen36.py --gs N writes one f32 scale per N input elements per row (scales [O, ceil(I/gs)], expert_gs in qwen36_meta.json) instead of one per row. The engine reads expert_gs and dispatches expert GEMVs through matmul_qe: per-row scales unchanged, grouped scales via matmul_q_gs. Same trade as GLM's gs64 containers: ~9% more container bytes for scales that track short rows (moe_intermediate_size=512) much more closely. Validated on the tiny fixture: grouped beats per-row on cosine to the int8 anchor. Full-model A/B on the published containers is in the PR thread.
Mirrors docs/inkling.md's structure: pre-converted containers first (gs64 recommended, per-row available), converter as the DIY route, coli chat as the primary interface now that the engine speaks the serve protocol. The container recommendation follows the GLM JustVugg#455 experience; the per-row-vs-gs64 measurement backing it is in the PR thread.
main loaded the prompt (argv[3], default ref.json) before the SERVE branch, so an engine launched the way coli launches it — SERVE=1 and no prompt argument — exited 1 on the missing ref.json before ever emitting READY. Serve gets its prompts over the wire; skip the argv prompt load in serve mode. Found by running the serve handshake end-to-end.
…stant serve_one compared against a hardcoded 151645 — <|im_end|> in the older 151k-token Qwen vocabs. Qwen3.6's vocab is 248320 tokens with the specials at 248044+ (<|endoftext|> 248044, <|im_end|> 248046), so EOS never matched and every serve turn ran to max_tok; in the worst case 151645 is an ordinary BPE piece here and could truncate a completion mid-sentence. Resolve <|im_end|> and <|endoftext|> from the tokenizer's added_tokens (same spirit as kimi_k3's config-driven eos list); Q36_EOS still overrides. Found by the serve-path termination check the A/B comment called for.
openai_server.py knew glm/inkling/kimi only, so a qwen model fell to the GLM renderer and the engine never saw its native frames. render_chat_qwen mirrors Qwen3.6's chat_template byte for byte in the text-only subset (inkling's approach): <|im_start|>role\n...<|im_end|>\n frames and a generation prompt that OPENS the mandatory <think> block — the model was never trained on a bare 'assistant\n' state, and greedy argmax there is an EOS special (measured: gen=0 on the very first token). Thinking disabled pre-closes the block, exactly like the official jinja. Arch detection, --arch choice, renderer dispatch and the single-KV-slot guard follow the kimi pattern. Spawn needs nothing new: argv is (cap), bits defaults to 4, tokenizer comes from SNAP.
16b945a to
853391c
Compare
|
Front-end wiring is in — done the way @rofl0r suggested: @minne100's bcae842 cherry-picked with authorship preserved (coli's Driving the handshake end-to-end then found three more gaps, all fixed on the branch:
Verified: READY/STAT → SUBMIT → ACCEPT → DATA → DONE transcript on the real container, and — closing the caveat from the A/B comment above — natural termination on the serve path: with the model's own sampling ( On your two questions: Branch is 7 commits: engine, serve protocol (@minne100), gs64, docs, and the three fixes above. |
6 participants
Summary
The idea of this and next PR is to optimize loading experters (for MoE model) from RAM to VRAM when the VRAM is limited, so that for example the "Qwen3.6:35b" model can run at a single GPU with 8 GB VRAM with acceptable performance.
Measured (Threadripper 3945WX 12C, RTX 3070 8 GB + Quadro RTX 4000 8 GB, 200-token decode)
Relationship to #602: the engine core (hybrid forward, container format, converter, tiny/oracle validation) originates from @minne100's PR; this PR retargets it to
dev, adds substantial CPU performance work and converter fixes, and drops the Vulkan/serve parts per review guidance.Self-contained engine for Qwen3.6-35B-A3B (35B total / 3B active, 256 experts/layer, Apache-2.0). This is the "engine first" slice requested in the #602 review (thanks for the guidance!): no shared GLM/Inkling files are touched, no GPU backend is included — those come as separate follow-ups.
What's included
c/qwen36.c— 40-layer hybrid forward: 10x Gated Attention (GQA, partial RoPE rope_dim=64, per-head q/k RMSNorm, output gate) + 30x Gated DeltaNet (causal depthwise conv ring + recurrent gated delta rule), streaming MoE with per-layer LRU expert cache + shared expert, PILOT router-lookahead prefetch, COLIBRI_RESIDENT pinning, per-phase timers (COLI_TIMERS=1).c/tools/convert_qwen36.py— HF→container converter (true int4, per-row scales, --selftest). Containers now run out of the box: flat config.json (VL checkpoints nest dims under text_config) and bundled tokenizer.json.c/tools/make_qwen36_tiny.py,make_qwen36_oracle.py— weight-free validation without the 70 GB checkpoint.docs/qwen36-phase01.md,qwen36-phase02.md.Validation
--selftestPASS (int4 pack/unpack exact)0.9992 vs f32 reference (COLI_DENSE_I8=0 → bit-exact)
Try it
A pre-converted, self-contained int4 container (tokenizer + flat config bundled, produced with this PR's converter) is available: https://huggingface.co/Kreuzzelg/qwen36-35b-a3b-colibri-i4
Notes
Thanks on Claude Code.