New int4-g64 container is ~12% more resource-hungry than per-row int4; possible VRAM-crash correlation (unconfirmed)

[woolcoxm]

New int4-g64 model container is ~12% more resource-hungry than the old per-row int4; possible link to VRAM crash

Filing this to capture a real resource regression (measured) and a possible related
stability symptom (unconfirmed mechanism — see bottom). Honest about what I know vs
what I'm guessing.

Environment

• CPU: Intel Core Ultra 9 185H (16c / 22t)
• RAM: 31.7 GB total
• GPU: NVIDIA RTX 5070 Ti, 16 GB VRAM (14.4 GB free at idle)
• Disk: NVMe, ~2.8 GB/s
• OS: Windows (win32, Git Bash)
• Engine: c/glm.c at fa821a1, built gcc -march=x86-64-v3 (AVX2, no AVX-512)

What changed between the two models

These are the same GLM-5.2 (744B MoE, 78 layers, 256 experts, 8 active/token,
hidden 6144). The difference is the int4 container only:

	old glm52_i4	new glm52_i4_g64	delta
int4 scale scheme	per-row (fmt=2)	grouped, gs=64 (fmt=4)	—
one expert on disk	18.9 MB	21.2 MB	+12.3%
resident dense weights	~9.9 GB	10.87 GB	+9.8%
cold bytes read / token	~11.3 GB	12.7 GB	+12.3%
MTP draft heads	int4 (0% acceptance)	int8 (correct)	better

The +12% is the grouped scale array: a scale every 64 elements instead of one per row.
That's a deliberate quality trade (better per-expert numeric range) — which is why we
switched — but it costs resources on every dimension.

Measured gains and losses (same machine, CPU-only, greedy, cold-ish cache)

Honest numbers, not cherry-picked. CPU-only because CUDA now hard-crashes this
machine (see below), so the GPU path is unavailable for both models.

metric	old glm52_i4	new glm52_i4_g64
CPU decode, stock config	0.19 tok/s	0.05 tok/s (config cascade — see note)
CPU decode, properly tuned	0.27 tok/s	0.17 tok/s
expert cache hit rate	1.2–10%	26.6%
experts loaded / token	~575	600 (was 2424 before fix)
prefill (10 tokens)	—	~37s (21s I/O + 12.6s matmul)
MTP draft acceptance (CPU)	33%	0–8% cold
GPU decode (when CUDA worked)	0.26–0.30 tok/s	n/a — crashes

Note on the 0.05: not the model being slow. The auto RAM budget (≈22 GB) starved the
expert cache to cap=1 (one slot/layer), forcing 2,424 expert disk-loads per token.
Setting RAM_GB=28 → cap=3 → 0.17 tok/s. Same model, 3.4×, zero code change.

Things I am NOT claiming

• I could not find any 0.86 tok/s figure for the old model in the repo. The measured
  old-model GPU speed on this exact GPU is 0.30 tok/s. The 0.86 may be warmer-cache
  or a different run; either way that GPU path is dead now for both models.
• The repo's headline 0.66 tok/s came from EXPERT_BUDGET=4, which the maintainers
  later quarantined as garbage-producing (commit 35f90b9, "budget=4 → decode is
  literal noise"). Not counting it.

The stability symptom — unconfirmed mechanism

⚠️ I want to be precise here: I have not reproduced the crash in a controlled way,
and the code I read sizes VRAM off the actual model footprint
(expert_bytes_probe reads real safetensors byte counts; g_cuda_dense_projected
accumulates qt_bytes per dense tensor — both reflect the larger g64 model, not the
old one). So I am not asserting a sizing bug. I'm reporting a correlation.

What happens: when loading experts into VRAM on the new g64 model, the computer
crashes (hard, not a clean exit). It did not do this before. My current hypothesis,
not proof: because the new container needs ~12% more bytes per expert and ~1 GB more
resident dense, the VRAM/RAM tiers are tighter, and a budget that was comfortable on
the old model now overshoots — but I have not confirmed the engine is mis-accounting
the larger size. It may be a pure resource-pressure crash, or something else entirely.

What I'd want before calling it a bug:

• a clean repro (exact COLI_CUDA=1 CUDA_EXPERT_GB=... line that crashes),
• the [CUDA] hot expert tier: ... VRAM X GB (total budget Y GB) line from the log
  right before the crash,
• confirmation it does NOT crash with a lower CUDA_EXPERT_GB.

Quality

We switched for accuracy — the old model had a repetition problem. By that measure
the new container is the win. I did not rigorously quality-benchmark either model
(this session was speed-focused), and the old model directory has been deleted, so I
have no head-to-head accuracy data. Flagging it as open, not resolved.

What actually moves the needle (both models)

This is a 744B model on consumer hardware; "fast without optimization" was never on
the table (the README's own disk ceiling is ~0.05–0.1 tok/s). To make the new model
genuinely usable:

1. RAM is the only knob today — push cap from 3 toward 8 and hit rate climbs.
2. Interactive use warms the PIN set across turns.
3. int2 for cold experts halves the ~10.5 GB/token expert I/O while keeping the
   hot PIN set at int4. The engine already has the matmul_i2 kernel + converter
   plumbing; it needs a converter fix, an int2 exactness test, and a quality
   measurement. That's the real path to recover the speed the lost GPU took away —
   and it would apply to the old model too.

Working CPU-only recipe (no CUDA)

cd c
SNAP=../glm52_i4_g64 RAM_GB=28 CACHE_ROUTE=1 DRAFT=0 ./glm.exe 75
# → ~0.17 tok/s, ~27 GB peak RSS, cap=3, MTP off (0% acceptance cold)

[woolcoxm]

the new model is killing my disk i/o i dont have enough resources to run it functionally without seeing regressions.

the only thing i can suggest and where i was going in the first place is cold experts at int 2 the rest a combo of int3/4 then the hot experts int 8, this would reduce the disk i/o substantially, as well as improve resource management(less resources used on unimportant things, more resources for the important things)

it would improve disk i/o it would improve the tok/s incredibly, currently 90% of the program is waiting on disk i/o, i have an nvme ssd, i can only imaging what it would be like on an hdd or an ssd. this would be a big win for low resource users while also keeping support for the users with higher end systems that can run gs64/128

if this is a direction you are interested in going please post back and let me know, i have the coli_usage and everything for the quantizations.

[woolcoxm]

Update: research on mixed-precision int2 for cold experts

I dug into the literature on the int2-cold-experts idea from my earlier comment, and into GLM-5.2's actual routing distribution. Short version: the approach is well-supported in recent research, GLM-5.2's routing skew favors it, but there's one engine-specific caveat worth flagging before anyone commits to it.

The architecture is established, not speculative

Mixed-precision-by-expert-heat is an active 2024–2025 research line, and the recent papers all validate the hot/high-precision + cold/2-bit structure:

• MxMoE (ICML 2025, arXiv:2505.05799) — per-expert bit allocation by sensitivity; 2.4 lower WikiText-2 PPL than GPTQ at 2.25-bit, up to 3.4× speedup.
• DynaExq (arXiv:2511.15015, Nov 2025) — runtime hot/cold precision controller; demoting rarely-activated experts to int2 degrades PPL smoothly, not as a cliff.
• MoPEQ (ICCV 2025) — explicitly argues mixed-precision is "particularly well-suited for MoE" due to selective activation.
• MoQE (NeurIPS 2023) — foundational: 2-bit MoE experts can match dense-model accuracy; low-bit and sparsity are complementary.
• GEMQ (arXiv:2605.23078, 2025) — global expert-level bit-width allocation by importance.

The mechanism is simple: an MoE layer's output is the routing-weighted sum of active experts, so a cold expert's quantization error is attenuated by its small gate weight. Hot experts carry the loss.

GLM-5.2's routing favors it

I parsed the .coli_usage traces from this model:

• Whole-model Gini coefficient 0.667 (0 = uniform, 1 = all traffic to one expert) — more concentrated than the models in the published experiments, so the cold tail is lower-leverage.
• Top 32/256 experts carry 17.7% of routing (random would be 12.5%); the bottom-quartile experts fire ≤9 times.
• Median expert fires only 25 times across the trace.

By the concentration logic in those papers, GLM-5.2 is a better candidate for cold-int2 than the models they tested.

The I/O win is the real prize

The motivation here isn't accuracy — it's that ~90% of my runtime is disk I/O. Per-token cold expert read:

all int4-g64 (current)	all int2	cold-only int2 (mixed)
12.7 GB/token	5.7 GB/token	~6–7 GB/token

On my NVMe that's the difference between a ~5 s/token I/O floor and ~2 s/token. Halving expert size (21.2 → 9.5 MB) also doubles effective RAM-cache capacity at the same RAM budget, so cap effectively goes up for free. The literature consistently reports 2–3.4× speedups.

Two corrections / caveats

1. No int3 in this engine. I'd referenced "int3" earlier — the engine formats are 0=F32, 1=INT8, 2=INT4-per-row, 3=INT2, 4=INT4-grouped. The realistic middle/hot tier is int4, not int3.

2. pack_int2 is naive RTN, not GPTQ. This is the important one. Every positive int2 result above uses advanced PTQ (GPTQ/AWQ/AutoRound — Hessian-based error compensation). Colibri's pack_int2 (c/glm.c:1061) is plain round-to-nearest — one absmax scale per row, no calibration. The literature is unambiguous that naive RTN at 2-bit is the catastrophic-degradation regime for uniform quantization (the practitioner-observed "quadratic cliff around ~2.7 bpw"; pure int2 is 2.0 bpw). The MoE cold-expert attenuation is what makes it plausible, not guaranteed. So the honest status is: the approach is sound, but whether colibri's specific quantizer is good enough at 2-bit on cold GLM experts needs an actual perplexity/output measurement before committing — and if naive RTN int2 comes back degraded, the fallback is adding a GPTQ-style error-compensation pass to pack_int2.

Still my read that this is the most promising path to recover what the lost GPU took away, especially for low-resource users. Happy to help with the int2 quality measurement harness if there's interest.

[woolcoxm]

Resolved — root cause found and fixed

The crash-correlation hypothesis in this issue is now confirmed and resolved:

• "Is the engine mis-accounting the larger g64 footprint?" → No. Audited every VRAM-sizing path (row_bytes, qt_bytes, tensor_upload, tensor_bytes, tensor_free, placement budget) — the +12% grouped-scale overhead is correctly counted as O*ng everywhere. The sizing is sound.
• Actual root cause: the two MLA attention absorb kernels in backend_cuda.cu kept the per-row scale semantic (wscale[row]) from fmt=2; PR cuda+engine: full fmt=4 (grouped int4 gs=64) support + diagnostic harness #298 added fmt=4 per-group scales to quant_matmul but missed them. The g64 kv_b is fmt=4 (ng=8) → wrong scale stride → GPU fault → 0x116 VIDEO_TDR_FAILURE reboot.
• Fix: PR cuda+engine: full fmt=4 (grouped int4 gs=64) support + diagnostic harness #298 commit 22a224e (absorb_scale() helper + per-group indexing in both kernels + gs/ng threaded through 6 launch sites). Validated; full benchmark no longer crashes.

Full writeup: #298 (comment)
Benchmark data + follow-ups tracked in #334.

Closing this as answered — feel free to reopen if the symptom reappears on a different config.

[mohamedmastouri2000-boop]

Public g64 container is up: https://huggingface.co/mastouri/GLM-5.2-colibri-int4-g64-with-int8-mtp -- the fmt=4 CUDA config from this issue (COLI_CUDA=1 COLI_GPU=0 CUDA_DENSE=1 CUDA_EXPERT_GB=4) runs on RTX 5080 / sm_120 with no TDR: coherent output, 81.6% expert hit, 109 experts resident in VRAM. Full numbers and the build/usage notes are on PR #298.

[woolcoxm]

thanks a ton!!!!

just one ask of you, can you run the benchmarks? ive been running into issues with memory leaks, it seems i get 4-5 tests done then my computer runs out of memory, not sure if its something i did or what but the testing harness is killing my pc, ive been patching it all day and cant figure it out,

[bokiko]

First task-level quality numbers for a grouped GLM-5.2 container — grouped scales recover what per-row loses (same host/day/binary pair)

Answering the benchmark ask above, since our rig could run it. Setup: converted the FP8 checkpoint to an all-fmt=4 g128 container (convert_fp8_to_int4.py --ebits 4 --group-size 128, MTP head int8) — note this container needs #375 to decode at all (the CPU absorb path predated fmt=4 kv_b; the mixed-precision recipe never exposed it). hellaswag n=40, prefix-fixed harness (#194), 9955HX Zen 5, engine cead3d8+#375, both containers evaluated the same day on the same host with the same binary:

container	disk	acc	acc_norm
per-row int4	357 GB	77.5%	80.0%
fmt=4 g128	369 GB (+3.3%)	82.5%	87.5%

+7.5pp acc_norm for +3.3% disk. Direction consistent with @ZacharyZcR's ablation grid in #225 (per-row scales are the dominant quality cost), now at full GLM-5.2 744B scale on a task metric.

Honest caveats, per house rules: n=40 means the gap is 3 questions — the direction is solid, the magnitude has real error bars. Happy to run n=200 and the rest of the suite (arc_challenge, mmlu) on this container while it's warm, if that's useful for the "should g-scaled become the default container" question.

One datapoint for the OP's size concern: g128 costs +3.3% over per-row (vs +12% measured for g64) — group size is a real lever on the overhead side too.

[mohamedmastouri2000-boop]

@woolcoxm — ran the quality harness on the #306 box (Core Ultra 9 285K / RTX 5080 / 128 GB). Three things: the numbers, your OOM, and a perf caveat.

1. arc_challenge, g64 vs per-row int4

tools/eval_glm.py (log-likelihood scoring, GLM prefix on), --limit 10, GPU tier + RAM_GB=60:

container	acc	acc_norm
g64 (fmt=4, gs=64)	70.0%	70.0%
per-row int4 (fmt=2)	70.0%	60.0%

Raw acc is identical; acc_norm is +10pp for g64 — but n=10, so that's a single question, firmly within noise. Fair read: g64 preserves quality, and is at least as good as per-row here — no regression from the grouped-scale format. Not a statistically final result; see the caveat below for why n is small.

2. Your OOM-after-4-5-tests is the RAM_GB cap, not a leak

I read run_score() — it's leak-clean: buffers (x, lo, row, ids, KV) are allocated once before the loop, reused every request, freed once at the end. Nothing grows per-request.

What grows is the expert LRU cache, up to RAM_GB. You mentioned forcing RAM_GB=28 — on a 32 GB box that's 28 GB of experts + ~11 GB resident dense + OS + browser, which overcommits right as the cache warms (about the point the [score N req] log first ticks — hence "4-5 tests"). I watched exactly this live here: RSS climbs to the RAM_GB ceiling and parks (52 GB at RAM_GB=60). On 32 GB, try RAM_GB=12-14 — it'll fit alongside dense+OS and stop the OOM (at the cost of a lower hit rate).

3. Perf caveat: log-likelihood scoring is disk+compute bound here

Why n is only 10: the g64 container is ~390 GB, and a 60 GB cache holds only ~15% of the experts. Scoring routes very diverse experts across arc/hellaswag/mmlu, so cache hit rate was 8-14% — most of the 8 experts/token/layer stream from NVMe and get computed on CPU int4 rows. Measured cost: ~55-65 s per forward, ~40 min per container even at n=10. So a full-size multi-task run (n≥40 × 3 tasks × 2 containers) is many hours on a single consumer box — impractical, not a harness bug.

A high-RAM host, a RAM-disk for the container, or higher CUDA_EXPERT_GB residency would push the hit rate up and make large-n tractable. If someone has a box like that (or wants to point me at a faster scoring path), I'm happy to run the full hellaswag/arc/mmlu sweep and post a proper table.

[woolcoxm]

i should learn to read :)

thanks for the infor!!

it was definitely that flag, the cache kept building till i ooms.

[JustVugg]

@mohamedmastouri2000-boop — heads-up on the published container, checked before downloading it here: the experts in mastouri/GLM-5.2-colibri-int4-g64-with-int8-mtp are int8, not int4-g64. Verified via HTTP range-reads of the safetensors headers on three shards across the depth: layers 10, 45 and 75, gate/up/down_proj all at 1.00 byte/param (int4 would be 0.50), which also explains the 429 GB size (an actual g64 container should be ~230 GB).

This smells exactly like the #383/#355 trap family that just got closed — ebits resolving to 8 and the grouped branch silently gating off (bits <= 4), so the conversion emitted per-row int8 while the invocation asked for g64. The fixed converter on dev now prints a [PLAN] line that would have caught this at second 1:

[PLAN] mode: main model | ... | experts 8-bit ... | PER-ROW (grouped branch needs bits<=4; ebits=8 disables it)

Worth checking your conversion logs and, if it's confirmed, re-running the main pass on current dev (your CUDA/quality results on this container are still valid as int8 results — but anyone downloading 429 GB expecting the g64 size/quality point is getting something else). Happy to re-verify the headers of a re-upload the same way — it's a 2-second range-read, no download needed.