does this transfer to other models?

[cktang88]

Curious if you've done tests to see if this transfers to other OSS or closed source models.

[lucaspirola]

I cannot answer that, but what I can tell is that it is not enough to make the 3-bits unsloth quantizations to behave properly.

[andthattoo]

Not yet in a systematic way. The main benchmark results in the repo are on Qwen3.6-35B-A3B GGUF Q4_K_M, plus some follow-up tool/grammar compatibility tests on Qwen3.6-27B-FP8 via vLLM.

My expectation is that transfer will be model- and quantization-dependent. The grammar can enforce structure, but it does not create capability. If the model’s raw distribution is far from the constrained format, decoding can slow down or quality can degrade. That may explain why very low-bit quants, like some 3-bit variants, behave poorly.

The next experiment for me:

• Qwen3.6 at different quant levels
• smaller Qwen models

So short answer: promising on the tested Qwen3.6 27B setup, but not yet proven as a universal transfer result.

[noonghunna]

Cross-rig replication data point: Qwen3.6-27B-AutoRound-INT4 + vLLM.

We're noonghunna/club-3090 — running Qwen3.6-27B on RTX 3090s via vLLM. Ported your structured-CoT idea from the llama.cpp + Qwen3-Coder reference to a vLLM compose, ran the full HumanEval+ + LCB v6 eval, and the technique transfers cleanly across this quant + engine substitution. Numbers below for cktang88's "does it transfer" question and to give @andthattoo a second cross-quant data point.

Setup:

• Model: Qwen3.6-27B-AutoRound-INT4 (group_size=128, Marlin path). Different quant from your FP8 reference; both INT4 quants tend to have larger raw-distribution drift from the constrained format (lucaspirola's 3-bit observation), so this is a meaningful test.
• Engine: vLLM 0.19.2rc1.dev205+g07351e088 + Sandermage's Genesis v7.64 patches + our local sidecars
• KV: TurboQuant 3-bit, MTP K=3, max-model-len 185K, mem-util 0.975
• Grammar: GBNF as in your repo, ported to vLLM via extra_body={"structured_outputs": {"grammar": ...}} (the legacy guided_grammar field is silently dropped on dev205+ — caveat for vLLM users coming from older pins)
• Required vLLM flag: --structured-outputs-config.enable_in_reasoning true. Without it, grammar enforcement doesn't reach the <think> block and FSM mode reduces to FREE (token counts identical between the two — easy to miss).

Headline results (n=full bench, single 3090 single-stream):

	HumanEval+	LCB v6
FREE pass@1	88.4%	42%
FSM pass@1	92.7%	66%
Δ	+4.3pp	+24pp
Compression (FREE think tokens / FSM think tokens)	30.7×	26.2×

For comparison to your README numbers (from memory: Qwen3.6-35B-A3B GGUF Q4_K_M, 22.4× HE+ / 43.3× LCB v6 compression, FSM pass@1 within ~2pp of FREE on HE+, larger gain on LCB v6): same shape of result. The compression ratio differs slightly (30.7× vs 22.4× on HE+; 26.2× vs 43.3× on LCB v6) but the directional finding holds and the FSM pass@1 lands within 2pp of yours on both. Cross-rig + cross-quant + cross-engine replication is more meaningful than any single-rig number.

Honest caveat we want to flag (it's a real one):

Our FREE baseline ran with max_tokens=4096, which is enough for FSM (compressed think + answer fits comfortably) but truncates ~half of LCB v6's FREE think traces mid-sentence. So part of our +24pp on LCB v6 is "FSM rescued FREE from a max_tokens trap" rather than "FSM made the model smarter." Re-running FREE at higher max_tokens (we tested 8192) shows ~half of those rescued problems converge with their FSM counterpart given enough tokens. The +4.3pp on HE+ is real (HE+ traces fit in 4096 reliably either way). On LCB v6 the honest delta is closer to ~12-15pp once max_tokens is matched; the structural compression argument still holds though — bounded thinking gives you predictable cost as a guarantee, which is the actual operational value.

Implementation:

• Compose: docker-compose.bounded-thinking.yml. Identical to our long-text 185K + 0.975 compose plus --structured-outputs-config.enable_in_reasoning true — single-flag opt-in.
• Public writeup with full method + bench detail: docs/STRUCTURED_COT.md.
• Bench harness: ported your eval scripts from llama.cpp shape to vLLM's OpenAI-compatible endpoint. Diff is small enough we can upstream a vllm-flavored eval harness if useful.

Net answer to cktang88's question: transfers cleanly from llama.cpp + Qwen3.6-35B-A3B GGUF Q4_K_M (your reference) → vLLM + Qwen3.6-27B-AutoRound-INT4. Compression magnitude varies (30.7× / 26.2× here vs 22.4× / 43.3× on your stack) but the FSM pass@1 lands within 2pp of yours and the structural-bounded-thinking value is preserved. So at least Qwen3.6 family transfers across the quant + engine axis. Whether it transfers to other model families (Llama, Mistral, etc.) is the next question — we haven't tested.

Happy to share the eval harness, raw per-problem data, or run additional probes if useful. Thanks for shipping the original — the cleanly-bounded grammar made the port easy.

[andthattoo]

This is fantastic, thank you for running such a careful replication.

I had not realized guided_grammar could silently no-op on newer dev pins, and --structured-outputs-config.enable_in_reasoning true should be called out in the README. I’ll add a vLLM note so others don’t accidentally benchmark FREE while thinking they’re enforcing the grammar.

Some of the gain is clearly “bounded thinking avoids truncation,” but that is still a real operational advantage. The HE+ result and the matched-token LCB estimate are the more important apples-to-apples signal.

Big picture, this is a really valuable data point: Qwen3.6 transfers across llama.cpp/GGUF -> vLLM/AutoRound-INT4, which is much stronger than a single-stack result. I’d definitely be interested in the harness diff / raw per-problem data if you’re willing to share it, and I’m happy to link your writeup from the repo.

Thanks again for doing this. Independent replication is the best possible kind of issue comment :)

[noonghunna]

Thanks for the warm response! Glad the replication was useful.

Quick notes on what you asked for:

Harness diff is genuinely tiny — one line in fsm_vs_free_eval.py and one grammar file variant:

1. Line 711 of fsm_vs_free_eval.py:

   # was:
   extra_body={"grammar": grammar}
   # now:
   extra_body={"structured_outputs": {"grammar": grammar}}

   The legacy guided_grammar field returns 200 OK on dev205+ but the mask never applies (silent no-op). Tip-off: FREE and FSM produce identical token counts. Clean wrap behind a vLLM-version check would be the friendlier upstream fix.

2. grammars/fsm_grammar_no_open.gbnf: variant that drops the leading "<think>\n" literal from your fsm_grammar.gbnf. Qwen3.6's chat template auto-prefixes it on this stack, so the literal in the grammar would double up and the FSM would refuse the model's first token.

Raw per-problem data: 4 run dirs (full-humaneval-2026-04-30, full-lcb-v6-2026-04-30, plus a parallel-GPU re-run for LCB v6 reproducibility). Each has results.jsonl (per-problem verdicts + token counts), per_problem.md (human-readable), summary.json, and run.log. Easiest delivery would be a PR to your repo adding them under runs/external-replications/club-3090/ — or a GitHub gist if that's lighter weight. Whichever you prefer.

Writeup link: docs/STRUCTURED_COT.md on our master is the public version — same headline numbers as the issue comment, with the truncation caveat called out openly under "Honest caveats" so future readers get the same context you flagged.

On the truncation caveat: fully agree it explains a meaningful chunk of the +Δpp. We benched at max_tokens=4096 because that's our shipped default for the bounded-thinking compose; re-running both modes at max_tokens=8192 would give a cleaner pure-reasoning signal. If we get a chance to do that on the v0.20 + Genesis v7.65 substrate (we just migrated this morning, would re-establish the baseline), I'll comment back here with the matched-tokens numbers — that's the apples-to-apples signal you flagged as more important.

Thanks again — happy to keep the cross-stack data flowing.