GLM-5.2 int4 think mode: stochastic reasoning loops / EOS starvation on stock v1.0.0; #307's TEMP=0.9 NUCLEUS=0.95 stopgap makes it severe

[lukascechovic]

Environment

• colibrì v1.0.0, commit 72874f3, fully stock (coli chat — no HTTP layer; also confirmed via stock coli serve)
• GLM-5.2 744B MoE, int4 checkpoint in the pre-fix per-row-scale format (the format Root cause of incoherent output: per-row int4 scales — fix with group-scaled quantization (fmt=4) #225 identified as an incoherence root cause)
• Linux, CPU-only (no ROCm/CUDA), 122 GB RAM, --ram 80 --topp 0.7, DIRECT=1 PIPE_WORKERS=2 PILOT_REAL=1
• .coli_kv deleted before each run, KVSAVE=0 (except the KV probe below)
• Prompt: 2+2 = ?, THINK=1

Repro

rm -f "$MODEL/.coli_kv"
printf '2+2 = ?\n' | env DIRECT=1 PIPE_WORKERS=2 PILOT_REAL=1 THINK=1 KVSAVE=0 \
  TEMP=0.9 NUCLEUS=0.95 \
  coli chat --no-attach --model "$MODEL" --ram 80 --topp 0.7

Observed

With #307's suggested TEMP=0.9 NUCLEUS=0.95 (severe, reproduced 1/1): the model hallucinated a different problem and never terminated — killed after 30 min / ~1050 tokens. Excerpt:

2 × 1 = 2
Simplify 4(???). What is the value of ?+?+?* ?
...
Okay, I think we're going to to the theater. (Ha ha!) But I'm going to the movie. (Ha, ha!)
...
So, what is the denominator? Is there a second "denominator"? ...

Stray </think> tokens appear mid-text; no EOS ever sampled.

With defaults (0.7/0.90) (stochastic): one coli serve sample was clean; the identical repeat produced a 10× "Wait, …" self-correction spiral before finally answering. A stock coli chat sample never closed its <think> block (answered inside reasoning, then EOS).

KV resume amplifies it: with KV saving enabled, two fresh-server requests were clean, but after a server restart resumed the saved .coli_kv (29 MB), the same request ran into a "Wait" loop to the max_tokens cap with no post-think answer — and the cap-hit was reported as finish_reason:"stop" instead of "length".

Notes / suspected causes

1. Our checkpoint is the pre-fix per-row-scale int4 (Root cause of incoherent output: per-row int4 scales — fix with group-scaled quantization (fmt=4) #225). The failure worsening as nucleus opens (0.90 → 0.95) fits quantization-tail noise being sampled back in — i.e., [Bug]: python coli chat doesn't end #307's stopgap points the wrong way for pre-fix checkpoints; the tightened 0.90 default is what keeps it mostly coherent.
2. No loop/repetition protection exists in generation ([Bug]: python coli chat doesn't end #307), so once the reasoning channel starts self-correcting ("Wait, …") nothing bounds it.
3. Two side-bugs observable on stock coli serve:
   • finish_reason:"stop" reported on max_tokens cap-hit (should be "length").
   • <think> stripping is string-based; when the model never emits </think>, the whole reasoning block lands in content (and when it does, the literal </think> can leak).

Full SSE captures, chat transcripts, and the metrics script available on request.

References

• Root cause of incoherent output: per-row int4 scales — fix with group-scaled quantization (fmt=4) #225 (per-row scale incoherence — our checkpoint predates the fix)
• [Bug]: python coli chat doesn't end #307 (sampling stopgap / absence of loop protection)
• Downstream capture set: lukascechovic/llm-server#109, lukascechovic/llm-server#110

Ask

1. Could a gs=64 / fmt=4 group-scaled int4 container be published (e.g. the test container built for cuda+engine: full fmt=4 (grouped int4 gs=64) support + diagnostic harness #298)? Both circulating HF checkpoints (jlnsrk/GLM-5.2-colibri-int4, mateogrgic/GLM-5.2-colibri-int4-with-int8-mtp) are still the pre-fix per-row-scale format with no warning on their cards — downstream users who skip the 756 GB FP8 conversion currently can't get the Root cause of incoherent output: per-row int4 scales — fix with group-scaled quantization (fmt=4) #225 fix. We'd gladly rerun our full capture matrix on it and report back.
2. Consider some form of loop/repetition protection in generation ([Bug]: python coli chat doesn't end #307) — with pre-fix checkpoints the reasoning channel can self-correct unboundedly.

[ZacharyZcR]

One datapoint that may bear on the quantization half of this: your checkpoint is the pre-fix per-row-scale int4 format, and we measured that scheme's cost directly today on OLMoE-1B-7B (n=200 × hellaswag/arc/mmlu, #453):

scheme	MEAN acc_norm
fp16	58.0%
int4 per-row (your checkpoint's scheme)	48.7% (−9.3pp)
int3-g64	50.5% (−7.5pp)
int3-g64-e8-rot / int3-iq3-rot	51.5% (−6.5pp)

Per-row scales erode exactly the small logit margins that decide near-ties — which is the mechanism behind #225, and plausibly why an EOS that should win by a hair loses it and the model wanders. Note the ordering: a 3-bit grouped container beats the shipped 4-bit per-row one by 1.8–2.8pp. If a grouped-scale (fmt=4) GLM checkpoint is available to you, it would be a clean A/B against this repro — same prompt, same flags, only the container changing. That would separate "sampling/stop-set bug" from "quantization noise in the stopping decision", which the current data can't distinguish.

(Not claiming it's the whole story — THINK=1 loops at TEMP=0.9 NUCLEUS=0.95 sound like they have a sampling component too.)

[JustVugg]

Thanks for the exceptionally thorough repro @lukascechovic. Two things stand out, and they point at the same root cause:

1. Your container is the pre-fix per-row-scale int4 — the exact format Root cause of incoherent output: per-row int4 scales — fix with group-scaled quantization (fmt=4) #225 identified as an incoherence source (one F32 scale per output row, ~48× coarser than the FP8 source's 128×128 block scales, ~26% expert-output error). Under THINK=1 that quantization noise is amplified: reasoning is a long autoregressive chain, so small per-token logit errors compound into the loops, the hallucinated sub-problems, and the stray mid-text </think> you're seeing. The model's control-token handling (including sampling EOS) is one of the first things per-row int4 degrades.

2. TEMP=0.9 NUCLEUS=0.95 (the [Bug]: python coli chat doesn't end #307 stopgap) makes it worse, not better — adding sampling entropy on top of an already-noisy quantized distribution is what turns a self-correction into a 30-min spiral. On a quantized model, greedy (TEMP=0) is far more stable for reasoning.

Recommended path, in order of impact:

• Re-convert (or download) a group-scaled int4 container (--group-size 64, fmt=4) — Root cause of incoherent output: per-row int4 scales — fix with group-scaled quantization (fmt=4) #225's measured cure for exactly this incoherence. It keeps int4's size while restoring near-FP8 coherence, and it's what makes think-mode terminate reliably.
• Meanwhile, run TEMP=0 (drop the 0.9/0.95 stopgap) for reasoning.
• int8 experts (--ebits 8) are the belt-and-suspenders option if you have the disk/bandwidth.

This isn't a stock-v1.0.0 engine bug so much as the per-row-int4 quality cliff surfacing hardest in the longest generations. Keeping the issue open to track: (a) making the converter default to group scales, and (b) a repetition/EOS-starvation guard for degraded containers. Could you retry with a gs64 container at TEMP=0 and report back?

[lukascechovic]

Update: we ran the A/B @ZacharyZcR proposed — a public fmt=4 container appeared, and group scales eliminate the failure. 5/5 cells clean.

@mohamedmastouri2000-boop's mastouri/GLM-5.2-colibri-int4-g64-with-int8-mtp (gs=64, official converter from FP8, int8 MTP head) is exactly the checkpoint this issue's ask #1 requested — thank you for publishing it. We reran our full capture matrix from the report above: same engine (stock v1.0.0, 72874f3), same prompts, same flags, only the container changed from per-row-scale int4 to gs=64.

Cell	per-row (report above)	gs=64
coli chat defaults, THINK=1	never closed <think>	clean 2+2=4, 5 tok, EOS, 53 s
coli serve SSE, defaults	stochastic (1 clean / 1 "Wait"-spiral)	clean, 8 tok, stop, 21 s
variance repeat	the coin-flip cell	clean, 9 s
TEMP=0.9 NUCLEUS=0.95 (#307 stopgap)	hallucinated a different problem, stray </think>, no EOS, killed at 30 min / ~1050 tok	clean 2+2=4, 5 tok, EOS, 20 s
KV-resume probe (save → restart → resume)	resumed .coli_kv degraded to cap-hit "Wait" loop	clean stop after verified restart + 3.2 MB .coli_kv resume

So the "sampling/stop-set bug vs quantization noise in the stopping decision" question separates cleanly: it was the per-row scales. On gs=64 even the widened nucleus (0.95) that made per-row catastrophically worse is benign — consistent with the #453 ablation ordering (per-row int4 −9.3pp, below int3-g64) and the near-tie-EOS mechanism @ZacharyZcR described.

Remaining suggestions from our report, updated:

1. publish a gs=64 container — done by mastouri; it would help downstream users if the two per-row HF cards (jlnsrk/…, mateogrgic/…) gained a warning pointing to it (per-row remains the format most people will find first).
2. Loop/repetition protection ([Bug]: python coli chat doesn't end #307) is now demoted from "critical" to "defense-in-depth" for post-fix checkpoints, based on this data.

Full captures (SSE streams, chat transcripts, KV probe) preserved; happy to share. Decode on our 6-core box is unchanged (~0.25–0.78 tok/s, cold cache) — quality was the only variable, as intended. Next we'll test the int8 MTP head for speculative-decode lift.

[lukascechovic]

Final report: all asks answered — resolved on our side, fine to close

@JustVugg — closing the loop on your three recommendations, in order:

1. gs64 container — done, and it's the cure: 5/5 capture-matrix cells clean (comment above), same engine/prompts/flags, only the container changed.

2. TEMP=0 retry on gs64 — clean. THINK=1 TEMP=0, same trivial prompt that looped on per-row: 2+2=4, 5 tokens, EOS, 9.5 s decode. Also ran four greedy 250-token generations over the serve path (temperature 0): all coherent, terminate cleanly. So greedy is confirmed stable on gs64 — but note the stronger result from the matrix: on gs64 even the widened TEMP=0.9 NUCLEUS=0.95 stopgap that was catastrophic on per-row is benign. On a fixed container the sampling knobs stop being safety-critical.

3. --ebits 8 — can't test here (int8 experts roughly double the 429 GB container; our models drive can't hold both), and after 1+2 it's no longer motivated for this failure: gs64 at 4-bit already restores reliable termination.

One new datapoint from the same box, since it bears on your item (b) — we also measured the int8 MTP head (writeup): it works as designed (39% acceptance — bottom of the card's 39–59% range — 2.17 tokens/forward, the #163 guard never fired), but on a 6-core compute-bound box it halves decode: an S=4 verify forward costs ~4.2× an S=1 forward, so speculation can't break even at any draft depth, and the 4-token routing union drops expert hit rate 90%→79%. Might be worth a docs note (or an auto-off heuristic) for CPU-streaming users: MTP pays only where verify forwards are bandwidth-bound.

From our side the issue is resolved: root cause confirmed (per-row scales), fix verified (gs64), stopgap demoted, greedy confirmed. Happy for this to close whenever you like — your two tracking items (converter defaulting to group scales; EOS-starvation guard for degraded containers) may deserve their own issues so this repro thread can archive cleanly. Thanks to you, @ZacharyZcR, and @mohamedmastouri2000-boop — the ablation, the mechanism, and the container between them turned a stochastic 30-minute spiral into a closed root-cause in a day.

[JustVugg]

big up for @ZacharyZcR and @mohamedmastouri2000-boop !!

[fcorrea]

Data point corroborating this — hit the same class of failure on a different trigger pattern.

Setup: mateogrgic/GLM-5.2-colibri-int4-with-int8-mtp checkpoint (so likely the pre-fix per-row-scale int4 format mentioned above), CPU-only coli serve (Ryzen 7 5800X, AVX2 only, 128GB DDR4-3600), --ram 90, DIRECT=1 URING=1 PIPE_WORKERS=16 DRAFT=0, default sampling (temp ~0.6–0.7, no explicit top_p/top_k — did not use the TEMP=0.9/NUCLEUS=0.95 stopgap from #307).

Erratic, same as reported here: one long generation (~2100 words / ~2700 tokens, a technical planning task) completed cleanly and hit a natural stop. A second generation on the same server/session, same default settings, collapsed mid-response — not into repeated words/phrases, but into a run of dozens of meaningless (0.4) (0.2) (0.8) (1.0) (0.4)... decimal fragments, triggered right as the model was in the middle of listing several numeric parameter values (physics thresholds) in sequence. Never recovered; killed manually well before the token cap.

Two things this adds:

1. The trigger doesn't have to be "sky is blue"-style trivia — dense numeric-value enumeration mid-response seems to be another reliable trigger for the same failure class, distinct from whatever <think>-mode self-correction loop is described above.
2. Confirms there's genuinely no mitigation available via the API today — openai_server.py rejects both frequency_penalty and presence_penalty outright ("Token penalties are not supported yet."), so there's no client-side lever to reach for here.

+1 on both asks: a republished group-scaled int4 (gs=64) container, and some form of loop/repetition guard in generation — this makes the engine risky for any unattended long-form generation task right now, since there's no way to detect or bound the failure from the client side.