Quality benchmark (the one you asked for): int4 GLM-5.2 scores 62.5% mean acc_norm — but the scoring protocol is a confound; here's the experiment that would settle it

[dnnspaul]

You flagged this as "the single most valuable thing a faster machine can contribute," so here it is: a full ./coli bench run on the int4 container.

Headline: the int4 model scores ~62.5% mean acc_norm, well below the 85–95% the README anticipates for full-precision GLM-5.2. The gap is larger than the sampling error — but I cannot attribute it to quantization yet, and I don't think anyone should until the control below is run.

Result

task                  n     acc  acc_norm
hellaswag            40   60.0%     67.5%
arc_challenge        40   62.5%     60.0%
mmlu                 40   60.0%     60.0%

MEAN acc_norm: 62.5% across 3 tasks

With Wilson 95% intervals (correct for n=40; the normal approximation is not):

task	acc_norm	95% CI
hellaswag	67.5%	[52.0%, 79.9%]
arc_challenge	60.0%	[44.6%, 73.7%]
mmlu	60.0%	[44.6%, 73.7%]
pooled (n=120)	62.5%	[53.6%, 70.6%]

Random baseline is 25% (4 options), so the model is clearly working — it is not broken, just well short of published figures. The pooled upper bound (70.6%) still sits ~15 points under the low end of your 85–95% expectation, and the deficit is consistent across three independent tasks, which is harder to explain as noise.

Why I am NOT concluding "int4 costs 25 points"

Two confounds have to die first, and neither can be killed with this run alone:

1. Scoring method vs published numbers. This harness does 0-shot log-likelihood scoring (EleutherAI style). Published frontier-model scores of 85–95% are typically few-shot and/or CoT-prompted. GLM-5.2 is a reasoning model, and reasoning models are known to underperform their headline scores badly under raw log-likelihood MC scoring, because you never let them think. Some — possibly most — of this gap may be the scoring protocol, not the quantization. Notably REFERENCE in tools/eval_glm.py is still unfilled, so the harness itself has no calibrated comparison point.

2. n=40 is small. ±14pp per task. I'd want --limit 200+ before quoting any single task's number.

The decisive experiment isolates quantization from everything else: run this same harness, on the same machine, against a model small enough to evaluate at BOTH fp16 and int4. OLMoE (which colibrì already supports, tools/convert_olmoe.py) is ideal — a few GB, so both precisions fit. The delta between fp16-OLMoE and int4-OLMoE under this exact harness is the quantization cost, with the scoring protocol held constant and cancelling out. If that delta is ~2 points, then int4 is fine and the GLM gap above is a scoring artifact. If it's ~20 points, your int4 container is genuinely lossy and mixed/grouped-scale quantization becomes the priority.

I'm happy to run that A/B — say the word and I'll do it. I'd rather hand you a clean causal answer than a scary number with a confound in it.

Run details (for the record)

• Runtime: 4h30m for 480 forwards (40 questions × 3 tasks × answer options)
• 75.3 TB read from disk, sustained ~4.8 GB/s
• Config: CPU reference path — COLI_CUDA=0 MTP=0 IDOT=1, full expert routing (no --topp), RAM_GB=65, OMP tuning on
• Deliberately not using MTP, the CUDA tier, or --topp: per Greedy decoding is not reproducible: MTP (3/5 prompts) and the CUDA expert tier (2/5) both change the output #100 all three can flip near-tie tokens, and none of them belong anywhere near an accuracy measurement
• Machine: Ryzen 7 9800X3D · 70 GB RAM (WSL2) · Samsung 9100 PRO PCIe 5.0 (10.51 GB/s O_DIRECT) · RTX 5090 (unused here)
• Model: mateogrgic/GLM-5.2-colibri-int4-with-int8-mtp
• colibrì f38e74e

Two small harness papercuts, easy fixes:

• ./coli bench needs tokenizers and datasets, which aren't mentioned in the README's quick-start pip line (torch safetensors huggingface_hub numpy). It dies with ModuleNotFoundError after downloading the datasets.
• It prints nothing for hours — the engine gets all 480 requests in one subprocess.run and only reports at the end. On a slow disk that's a day of silence with no way to tell progress from a hang. A per-question counter to stderr would help a lot. (I resorted to reading the request file's offset out of /proc/<pid>/fdinfo to get a progress percentage.)

Relation to #100

This may be the same story from the other end. In #100 we found the int4 model sits close enough to argmax ties that any rounding change (batched MTP verification, the CUDA tier, even swapping the kernel family via IDOT=0) flips tokens, occasionally into degenerate text. A model with healthy logit margins would not be that fragile. A genuine int4 quality deficit and that tie-fragility would be two symptoms of one cause: the int4 container leaving too little margin. The OLMoE A/B above would test that directly.

[dnnspaul]

Follow-up — I think there may be a sharper experiment than the OLMoE A/B, entirely inside GLM-5.2, and it might point at something fixable.

My run above used the default engine invocation (./glm 64), and the banner reports:

== GLM C engine (glm_moe_dsa), cache=8 experts/layer | experts@8-bit dense@8-bit | idot: avx512-vnni ==

So the compute path is 8-bit, even though the container on disk is int4 — the engine dequantizes int4 → int8 to feed the integer-dot kernels. But eval_glm.py --bits forwards positional args to the engine (cmd = [a.glm, str(a.cap)] + a.bits.split()), and ./glm 64 16 16 gives 16-bit compute — the mode your tiny-oracle self-test uses to hit token-exact 32/32.

That splits the 62.5% into two very different diagnoses:

• --bits "16 16" scores ~the same (~62.5%) → the loss is in the int4 container itself. Quantization is the ceiling, and mixed/grouped-scale quantization is the fix. (Or it's the scoring-protocol confound from my post above.)
• --bits "16 16" scores materially higher → the int4 weights are fine and the default 8-bit compute path is throwing away accuracy. That would be a bug-shaped problem, not a quantization ceiling — and it would also explain Greedy decoding is not reproducible: MTP (3/5 prompts) and the CUDA expert tier (2/5) both change the output #100 elegantly: an 8-bit compute path with thin margins is exactly what makes near-tie tokens flip whenever the rounding changes (batched MTP, CUDA kernels, IDOT=0).

Same model, same harness, same machine, one variable. It cleanly separates "int4 is lossy" from "the int4→int8 compute path is lossy," which the OLMoE A/B cannot do.

I have the machine and I'm happy to run it (~5-6 h; 16-bit matmul is slower). Say the word, or if you'd rather I ran the OLMoE fp16-vs-int4 control instead — or both — just tell me which order. Also happy to run --limit 200 on any of them to shrink the ±14pp error bars, now that we know a full 40×3 sweep costs ~4.5 h here rather than a day.

[ZacharyZcR]

Thank you very much; we will investigate this further based on this issue.

[JustVugg]

This is the benchmark I asked for, and you did the hard part that most people skip: you refused to over-conclude. Reporting 62.5% and immediately naming the two confounds that could account for most of it — that's the difference between a scary headline and a usable measurement. Thank you.

You're right on both confounds, and I'd weight the first heavily: 0-shot log-likelihood MC scoring badly underserves a reasoning model. GLM-5.2 is trained to think before answering; raw log-likelihood never lets it, so a 15-25 point gap vs few-shot/CoT published numbers is expected even at full precision. The unfilled REFERENCE in eval_glm.py is the tell — there's no calibrated fp16 anchor, so 62.5% has nothing honest to be compared against yet. And n=40 (±14pp) can't carry a strong claim.

The OLMoE fp16-vs-int4 A/B is exactly the decisive experiment — please run it. It's the one design that cancels the scoring protocol: same harness, same machine, same small model, only the precision changes, so the delta is the quantization cost, confound-free. That's worth far more than a larger-n GLM run, because it answers the actual question. --limit 200 on OLMoE (small enough that both precisions fit and it's fast) would give tight intervals too. If the delta is ~2 points, int4 is fine and the GLM gap is scoring; if it's ~20, the int4 container is genuinely lossy and grouped/mixed-scale quantization moves from 'roadmap' to 'priority.' Say the word — actually you already did, so: yes, go, and I'll be watching for it.

Your #100 connection is the sharp part and I think you're right: a genuine int4 deficit and the near-tie fragility would be two faces of one cause — the container leaving too little logit margin. Healthy margins don't flip tokens when the kernel family changes. Grouped-scale / error-feedback quantization exists precisely to widen that margin (measured ~5× lower output error at equal bits on real GLM experts), so it would fix both symptoms at once — the accuracy floor and the reproducibility fragility. Your OLMoE A/B is the single measurement that tells us how much of that lever we need.

Both papercuts are real and I'm fixing them:

1. README pip line — adding tokenizers datasets to the quick-start (fixing now; sorry you hit the ModuleNotFoundError after a dataset download).
2. Silent bench — a per-question progress counter to stderr is the right fix; reading /proc/<pid>/fdinfo to get a percentage is heroic and shouldn't be necessary. Noting it as a small harness fix.

4h30m and 75.3 TB read for this — that's real generosity. You've handed the project its first quality datapoint and the experiment design that makes it conclusive. The OLMoE number is the one I most want to see next.

[dnnspaul]

Starting on it — but first, the OLMoE A/B as designed cannot run, and I think there's a better experiment anyway. Two findings from digging into the code:

1. Blocker: olmoe.c has no scoring mode

eval_glm.py drives the engine through the SCORE protocol (glm.c:1881 run_score(), dispatched at glm.c:2734). olmoe.c is 392 lines and its only env knob is SNAP — there is no SCORE, no run_score. So --glm ./olmoe can't score anything; the harness would need run_score ported into the OLMoE engine first.

Happy to write that port and send it as a PR if you want it. But I don't think we need it, because:

2. A cleaner design: fake-quant A/B in transformers, using your exact quantizer

The confound we're killing is the scoring protocol. We can kill it far more completely: take OLMoE fp16, pass its weights through colibrì's own quantize→dequantize, and evaluate both in the same transformers harness. Same model, same tokenizer, same scoring code, same machine — the only difference is your quantization scheme. No engine port, no kernel differences, no tokenizer differences, nothing left to argue about. The delta is the quantization cost of your int4 container, isolated.

I'm replicating quant_int4 from tools/convert_fp8_to_int4.py:39 exactly (per-row absmax, symmetric nibble [-8,7], s = amax/qmax), applied to the same tensor classes the converter hits (attn/mlp/shared/expert/embed/lm_head), leaving norms/router/bias in f32 — same as convert_fp8_to_int4.py:12-14.

3. Two ablations that could turn this into an actionable fix

Reading the converter, two things jumped out that the A/B can test directly:

(a) lm_head is quantized to int4. Per convert_fp8_to_int4.py:12, lm_head and embed go through the same per-row int4 path as everything else. Quantizing the output head injects error straight into the logits — and thin logit margins are precisely what makes near-ties flip whenever rounding changes, which is the mechanism you diagnosed in #100. The head is a rounding error away from the argmax by construction. Most quantization recipes keep lm_head at 8-bit or fp16 for exactly this reason, and it's a tiny fraction of parameters — so if this is the culprit, it's a cheap fix with almost no size cost.

(b) scales are per-row, not grouped. amax = |w|.max(axis=1) — one scale for an entire input row (thousands of elements). A single outlier anywhere in the row inflates the scale and crushes the resolution of every other weight in it. That's the classic failure mode grouped scales fix.

So I'm running four configs, same harness, same questions:

config	what it isolates
A fp16 baseline	reference
B colibrì int4, faithful (per-row, incl. lm_head/embed)	the true cost of the current container
C = B, but lm_head+embed kept fp16	is the output head the culprit?
D = B, but grouped scales (group=128)	how much would grouping recover?

A − B answers your question (is int4 lossy, confound-free). C and D tell you which lever to pull — and whether the #100 tie-fragility and the accuracy floor really are one bug, as you suspected. If C alone recovers most of the gap, you get most of the win by leaving one matrix un-quantized.

OLMoE is small enough to run all four at --limit 200+ on the 5090, so the error bars will be tight, unlike my ±14pp GLM numbers.

Meanwhile: the 16-bit compute run is in flight

Separately, the --bits "16 16" GLM run from my earlier comment is ~50% done (ETA ~2 h). Recall the engine computes at 8-bit by default (experts@8-bit dense@8-bit in the banner) even though the container is int4 — so that run tells us whether the default compute path costs accuracy on top of the container. I'll post it when it lands.

Will report all of it here.

[ZacharyZcR]

Claiming the decisive experiment — the 6×5090 full-residency rig from #94 is the right machine for it, and this is queued next on it.

Plan, matching your design:

1. OLMoE control: same harness, same machine, fp16 vs int4 (tools/convert_olmoe.py). The delta isolates quantization cost with the scoring protocol cancelling out.
2. GLM int4 at n=200/task on hellaswag/arc_challenge/mmlu: at full residency each log-likelihood item is a short teacher-forced forward, so 600 items is hours, not days — enough to shrink your ±14pp to ~±6pp.
3. Everything with the discipline from the experiment record: DRAFT=0, fixed binary, and (per Greedy decoding is not reproducible: MTP (3/5 prompts) and the CUDA expert tier (2/5) both change the output #100) IDOT=0 COLI_CUDA=0 exact-mode spot-checks on a subset to bound any kernel-order effect on the scores themselves.

One caveat worth stating up front: if the OLMoE delta comes back small (~2pp), the remaining GLM gap is protocol — and the right follow-up is a CoT/few-shot harness rather than more MC log-likelihood. If it comes back large, that's the strongest possible argument for the grouped-scale/error-feedback quant work (and #81's rotation idea) jumping the priority queue.

Will report both tables in this thread.

[dnnspaul]

@ZacharyZcR — great, your rig is much better suited to the GLM half than mine. Two things before you burn cycles, then a proposed split so we don't duplicate.

⚠️ Your step 1 will not run as written: olmoe.c has no scoring mode

eval_glm.py drives the engine through the SCORE protocol — glm.c:1881 run_score(), dispatched at glm.c:2734. olmoe.c is 392 lines and its only getenv is SNAP. There is no SCORE, no run_score. So --glm ./olmoe cannot score anything; the harness would need run_score ported into the OLMoE engine first. (Also worth noting: tools/convert_olmoe.py emits row-wise int8 for experts, not the per-row int4 nibble container convert_fp8_to_int4.py builds for GLM — so even with a port, an OLMoE int4-vs-fp16 delta wouldn't be measuring the same quantizer GLM actually ships.)

A design that avoids the port entirely — and I'm already running it

Rather than compare two engines, fake-quantize inside one: take fp16 OLMoE, push its weights through colibrì's exact quantizer (quant_int4 from convert_fp8_to_int4.py:39 — per-row absmax, symmetric nibble [-8,7]), dequantize, and score in transformers. Same model, same tokenizer, same harness, same machine — the only variable is whether colibrì's quantizer touched the weights. No engine port, no kernel differences, no tokenizer differences. The delta is the quantization cost.

I mirrored eval_glm.py's scoring exactly (acc = argmax Σ logprob; acc_norm = argmax Σ logprob / character length of the continuation — note it's characters, not tokens), reading the same bench/*.jsonl with the same seed/shuffle.

Two gotchas that cost me, so they don't cost you:

• transformers fuses OLMoE's experts into 3D tensors (mlp.experts.gate_up_proj = [64, 2048, 2048]), even though the checkpoint stores 3,072 separate 2D matrices. A naive if p.ndim == 2 filter silently skips every expert and leaves ~85% of the model in fp16 while looking like it worked. I hit exactly this and only caught it because the tensor count (66) was implausible. Print your quantized-param coverage — it must be ~100%.
• The input dim to reduce over is dim=1 for nn.Linear [out, in] and dim=1 for fused experts [E, in, out] — coincidentally the same axis, but for different reasons. Get it wrong and you're quantizing across output channels.

I'm running 4 configs at n=200/task (±7pp), not 2:

config	isolates
fp16	reference
int4 per-row (incl. lm_head+embed)	true cost of the shipped container
int4 per-row, lm_head+embed left fp16	is the output head the culprit?
int4 grouped (128)	how much would grouped scales recover?

The last two are the point: A − B diagnoses, but C and D tell @JustVugg which lever to pull. Note convert_fp8_to_int4.py:12 quantizes lm_head to int4 — that injects error straight into the logits, which is exactly the mechanism behind the near-tie token flips in #100. If C recovers most of the gap, the fix is one un-quantized matrix, ~0.2% of parameters.

Proposed split

• You (6×5090, full residency): GLM int4 at n=200/task — you can do in hours what took me 4h30m at n=40, and it's the number that actually matters. Your IDOT=0 COLI_CUDA=0 exact-mode spot-check is a good call.
• Me: the OLMoE quantizer ablation above (running now, results here shortly), plus the --bits "16 16" GLM run already in flight (~60% done) — that one tests whether the engine's default 8-bit compute path costs accuracy on top of the int4 container, which is a separate question from the container itself.

Happy to hand over my ablation script if useful — say the word and I'll paste it or PR it into tools/.

And your caveat is exactly right: small OLMoE delta ⇒ the GLM gap is protocol, and the follow-up is a CoT/few-shot harness, not more MC log-likelihood. That distinction is the whole ballgame, and it's why I refused to headline the 62.5% as a quantization result.

[ZacharyZcR]

Split accepted — you take the OLMoE 4-config, we take GLM int4 at n=200/task, which is now running on the 6×5090 rig (single engine load, all ~2,400 log-likelihood requests in one SCORE pass, full residency, DRAFT=0). Table lands in this thread when it finishes.

And a concrete thank-you for gotcha #1: I had independently built the same fake-quant control before reading your comment — with exactly the p.ndim == 2 filter you warned about. It would have skipped every fused expert, quantized ~15% of the model, and reported a reassuringly tiny delta: a false acquittal, published with confidence. Your tensor-count sanity check killed it before it ran. That's peer review working exactly as intended.

Your 4-config design is strictly better than the 2-config anyway — C (fp16 head) is the actionable one: if it recovers most of the gap, the fix costs ~0.2% of parameters, and it lines up suspiciously well with the near-tie logit flips in #100. Looking forward to your table.

[dnnspaul]

--bits "16 16" result: negative — the 8-bit compute path is NOT the culprit

My hypothesis from earlier in this thread is refuted. Same model, same harness, same machine, same 40 questions/task; only the engine's compute precision changed.

compute precision	hellaswag	arc_challenge	mmlu	mean acc_norm
8-bit (./glm 64, the default)	67.5%	60.0%	60.0%	62.5%
16-bit (./glm 64 16 16)	67.5%	62.5%	60.0%	63.3%

+0.8pp, which at n=40 is literally one ARC question flipping. Nothing. Dequantizing int4 → int8 for the integer-dot kernels costs no measurable accuracy, so the default compute path is fine and there is no cheap win hiding there.

(Run cost, for the record: 4h47m, COLI_CUDA=0 MTP=0, full expert routing. Interestingly 16-bit was not slower end-to-end despite doubling cache entry size — this box is disk-bound, and the extra matmul work hides under the I/O.)

This matters because it narrows the field to two suspects. The 62.5% is now explained by either the int4 container or the scoring protocol — and nothing else. The compute path is eliminated.

Which makes the OLMoE fake-quant ablation the whole ballgame, exactly as @ZacharyZcR said. It's running now; results shortly.

[dnnspaul]

The int4 container IS genuinely lossy — and grouped scales fix most of it

Confound-free A/B, n=200/task (600 questions). fp16 OLMoE vs the same model with colibrì's exact quantizer applied in-place (quant_int4, convert_fp8_to_int4.py:39) — same weights, same tokenizer, same harness, same machine, same questions. The only variable is the quantizer. Coverage verified at 100.0% of 6.92B params (see gotcha below).

config	hellaswag	arc_challenge	mmlu	mean acc_norm	Δ vs fp16
fp16 (reference)	77.0%	47.0%	47.0%	57.0%	—
int4 per-row — as shipped	74.0%	41.0%	31.5%	48.8%	−8.2pp
int4 per-row, lm_head+embed left fp16	73.5%	40.5%	37.5%	50.5%	−6.5pp
int4 + grouped scales (group=128)	78.5%	45.5%	38.0%	54.0%	−3.0pp

1. The container costs ~8pp. This is not the scoring protocol.

The protocol confound is gone by construction — fp16 and int4 are scored by the identical harness on the identical questions. −8.2pp is the quantizer. (~2.8σ on a conservative unpaired SE of 2.9pp; the true paired interval is tighter still, since all configs see the same 600 items.)

So your int4 container is genuinely lossy, and part of the GLM 62.5% gap in the OP is real quantization damage rather than log-likelihood-vs-CoT artifact. Both effects are present; this bounds the quantization half.

2. The damage concentrates on the hardest task, and it's brutal

MMLU: 47.0% → 31.5%. Random is 25%. Per-row int4 takes MMLU most of the way to chance, while HellaSwag (easy, high-margin) barely moves (−3.0pp). That's the signature of a quantizer eating the small logit margins that hard questions depend on — and it's the same margin-erosion that makes near-tie tokens flip in #100. The two symptoms really are one cause, as you suspected.

It also means a headline "mean" number understates the harm: the quantization is nearly free on easy tasks and near-catastrophic on hard ones.

3. My lm_head theory was mostly WRONG

Leaving lm_head+embed in fp16 recovers only 1.7pp of the 8.2. It helps MMLU (31.5% → 37.5%), so the head isn't innocent — but it is not the main culprit. I proposed this as the likely cheap fix earlier in the thread; the data says no. Don't spend the roadmap slot on it.

4. Grouped scales are the lever — they recover ~63% of the loss

group=128 takes it from −8.2pp to −3.0pp, and at −3.0pp (1.0σ) it is not statistically distinguishable from fp16 on this sample. HellaSwag actually lands above fp16 (78.5 vs 77.0), which is noise, and usefully calibrates the noise floor at ~±3pp.

The mechanism is exactly what you'd expect from the code: amax = |w|.max(axis=1) puts one scale on an entire input row (2048+ elements for OLMoE; far more for GLM). A single outlier anywhere in that row inflates the scale and crushes the resolution of every other weight in it. Grouping bounds the blast radius of each outlier to 128 weights.

Cost is small and known: one fp32 scale per 128 weights ≈ +0.25 bits/weight (~+6% container size at int4 — GLM's ~370 GB → ~392 GB). For recovering ~5pp of accuracy and, plausibly, the #100 reproducibility fragility, that looks like a very good trade.

Recommendation: grouped-scale int4 moves from "roadmap" to "priority." Exactly the call @ZacharyZcR framed. I'm running the combined config (grouped and fp16 head) now to see whether they stack — will post.

---

Caveats, stated plainly

• OLMoE is a proxy for GLM-5.2. Same quantizer, different model (7B dense-ish MoE vs 744B). The mechanism transfers; the magnitude may not. GLM's rows are much wider, which if anything makes per-row scales worse, not better.
• This measures the quantization scheme, not colibrì's engine/kernels — deliberately, to remove them as confounds. Fake-quant (quantize→dequantize→fp16 matmul) isolates weight error only.
• n=200/task. Noise floor ~±3pp, calibrated by the HellaSwag over-shoot above.

Reproducer

Script: quantizer replicated verbatim from convert_fp8_to_int4.py:39, scoring mirrored from eval_glm.py (including acc_norm normalising by character length, not tokens). Happy to PR it into tools/ as quant_ablation.py if you want it in-tree — it's a fast, engine-free way to A/B any future quantization scheme (grouped, error-feedback, rotation per #81) before committing to a 370 GB conversion.

One trap for anyone reproducing this (@ZacharyZcR): transformers fuses OLMoE's experts into 3D tensors (mlp.experts.gate_up_proj = [64, 2048, 2048]) even though the checkpoint stores 3,072 separate 2D matrices. A if p.ndim == 2 filter silently skips every expert, leaving ~85% of the model in fp16 while appearing to work. My first run did exactly that and produced a plausible, entirely fictional number. Print quantized-param coverage and assert it's ~100%.