The current version of transformers-CFG does not support QWen and versions of Llama-3.2 and above

[neverleave111]

I'm glad to see such a good work, but it seems that the current version of transformers-CFG does not support QWen and versions of Llama-3.2 and above. Do you have any good methods that can be compatible with the new version? Please feel free to share

[Hobeom]

Hi @neverleave111,

I ran into the same transformers-CFG compatibility issue. In my fork (Communicative-PARTNR), I bypassed this limitation by using a remote vLLM serving setup instead of the local HuggingFace inference mode.

With vLLM's OpenAI-compatible API server, you can serve newer models like GPT-4o (via OpenAI API), Llama-3.3, and Qwen without depending on transformers-CFG for constrained decoding. vLLM natively supports guided decoding (e.g., via outlines or lm-format-enforcer), which covers the grammar-constrained generation that transformers-CFG was handling.

Basically:

1. Spin up a vLLM server with your desired model (e.g., meta-llama/Llama-3.3-70B-Instruct)
2. Point the planner's LLM config to the remote endpoint using an OpenAI-compatible inference mode
3. Use vLLM's built-in guided decoding instead of transformers-CFG

This approach also makes it easy to experiment with different OSS models without reinstalling or worrying about local transformers version conflicts.

Hope this helps!

[neverleave111]

Hi @neverleave111,

I ran into the same transformers-CFG compatibility issue. In my fork (Communicative-PARTNR), I bypassed this limitation by using a remote vLLM serving setup instead of the local HuggingFace inference mode.

With vLLM's OpenAI-compatible API server, you can serve newer models like GPT-4o (via OpenAI API), Llama-3.3, and Qwen without depending on transformers-CFG for constrained decoding. vLLM natively supports guided decoding (e.g., via outlines or lm-format-enforcer), which covers the grammar-constrained generation that transformers-CFG was handling.

Basically:

1. Spin up a vLLM server with your desired model (e.g., meta-llama/Llama-3.3-70B-Instruct)
2. Point the planner's LLM config to the remote endpoint using an OpenAI-compatible inference mode
3. Use vLLM's built-in guided decoding instead of transformers-CFG

This approach also makes it easy to experiment with different OSS models without reinstalling or worrying about local transformers version conflicts.

Hope this helps!

Thank you for your reply. In fact, upgrading transformers CFG can be compatible with Llama-3.2 and above versions, while for QWen, the source code needs to be modified, and a reasonable solution has not been found yet

[Hobeom]

I've confirmed that the vLLM approach works for Qwen models — tested with Qwen/Qwen3.5-4B. The key is that the original openai_chat.py uses AzureOpenAI hardcoded, so I'm working off my fork (HoBeom/Communicative-PARTNR) which switches to the standard OpenAI client (compatible with any OpenAI-compatible server including vLLM).

Changes needed vs. the original code

openai_chat.py — two additions on top of switching AzureOpenAI → OpenAI:

1. enable_thinking / thinking_budget passed as extra_body to suppress or cap Qwen3's <think> phase
2. Fallback to msg.reasoning when msg.content is None (happens when the <end_act> stop token fires inside the <think> block)

habitat_llm/conf/llm/qwen3_5_4b_vllm.yaml (new file):

llm:
  _target_  : habitat_llm.llm.OpenAIChat
  _partial_ : True

keep_message_history: False
enable_thinking     : False   # see pitfall #1 below
thinking_budget     : null

generation_params:
  model      : Qwen/Qwen3.5-4b
  max_tokens : 1536
  temperature: 0
  stop       : "<end_act>"
  request_timeout: 120

Serve via vLLM

vllm serve Qwen/Qwen3.5-4b \
  --port 8000 \
  --reasoning-parser qwen3 \
  --language-model-only \
  --gpu-memory-utilization 0.85

--language-model-only : Qwen3.5 is a VLM — this disables the vision encoder
--reasoning-parser qwen3 : strips <think>…</think> tokens from output

Pitfall specific to Qwen3.5-4B reasoning models

Disable thinking (enable_thinking: False)
With thinking enabled, the <end_act> stop token fires inside the <think> block (because the system prompt's format description itself contains <end_act>), so message.content comes back None before any action is ever generated.

Note: I only tested with the 4B model, which showed poor zero-shot performance on PARTNR. Larger Qwen3.x models may handle the format better without additional prompt tuning.

---

The implementation changes and verification notes above were prepared with assistance from Claude Code and validated through automated local tests on my fork.