TensorRT-LLM PyTorch backend Qwen3.5-VL parity issue: HF/vLLM greedy output matches, TRT-LLM repeats to max tokens

[jiuzhuanzhuan]

System Info

Environment

• GPU: NVIDIA GeForce RTX 4090
• TensorRT-LLM: 1.3.0rc21
• Backend: --backend pytorch
• Python: 3.10
• CUDA driver/runtime from nvidia-smi: Driver 595.58.03, CUDA 13.2
• PyTorch in venv: 2.11.0+cu130
• Transformers in venv: 5.5.4
• Model architecture in config.json: Qwen3_5ForConditionalGeneration

Startup warnings observed:

Skipping import of cpp extensions due to incompatible torch version 2.11.0+cu130 for torchao version 0.15.0+cu130
transformers version 5.5.4 is incompatible with nvidia-modelopt

Who can help?

No response

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

Summary

I am seeing a deterministic generation parity issue when serving a Qwen3.5-VL checkpoint with TensorRT-LLM PyTorch backend.

For the same checkpoint, same prompt, same 4 images, same chat template, and greedy decoding:

• Hugging Face transformers offline output matches vLLM exactly.
• TensorRT-LLM PyTorch backend produces different text and, for some samples, repeats a phrase until max_tokens with finish_reason="length".

This does not appear to be a missing stop-token issue anymore: generation_config.json is present, EOS IDs are configured, and the failing sample still repeats.

Model / generation config

The model is a local fine-tuned Qwen3.5-VL checkpoint derived from a Qwen3.5 model family checkpoint.

The model did not originally have generation_config.json; I added:

{
  "eos_token_id": [248044, 248046],
  "pad_token_id": 248044,
  "bos_token_id": null
}

These token IDs are from the model tokenizer:

• 248044: <|endoftext|>
• 248046: <|im_end|>

Adding this config fixed some service startup / stop handling issues, but it did not fix the parity issue below.

TensorRT-LLM startup command

Important: PATH must include the virtualenv bin directory, otherwise FlashInfer JIT may fail to find ninja.

PATH=<TLLM_VENV>/bin:$PATH \
CUDA_VISIBLE_DEVICES=0 \
<TLLM_VENV>/bin/trtllm-serve serve \
  <LOCAL_QWEN3_5_VL_CHECKPOINT> \
  --host 0.0.0.0 \
  --port 8005 \
  --backend pytorch \
  --tp_size 1 \
  --max_batch_size 16 \
  --max_num_tokens 8192 \
  --max_seq_len 4096 \
  --kv_cache_free_gpu_memory_fraction 0.8 \
  --trust_remote_code \
  --config <CONFIG_YAML>

<CONFIG_YAML>:

max_input_len: 2048

The override is confirmed in TensorRT-LLM logs:

max_input_len=2048 max_seq_len=4096 ...
attn_backend='TRTLLM'
reasoning_parser=None

Reproduction request

The request is OpenAI-compatible chat completion with:

• One system message:

  你是专业的监控画面理解和安防事件分析助手。
  

• One user message containing:
  • the long Chinese instruction prompt
  • four image frames
  • before each image, a text part like:

    Frame 1: index=0, t=0.000s
    

• temperature=0.0
• max_tokens=1200
• chat_template_kwargs={"enable_thinking": false}

The exact per-image frame indices for the failing sample:

Frame 1: index=0,   t=0.000s
Frame 2: index=75,  t=0.000s
Frame 3: index=149, t=0.000s
Frame 4: index=224, t=0.000s

The same request shape is used for HF offline, vLLM, and TensorRT-LLM.

Expected behavior

Expected result

Hugging Face offline generation and vLLM both return exactly:

灰衣男子坐在桌前，身体前倾，桌上堆满杂物。

HF offline details:

input_ids shape: [1, 1178]
image_grid_thw:
  [[1, 28, 28],
   [1, 28, 28],
   [1, 28, 28],
   [1, 28, 28]]
new_tokens: 17

vLLM details from the same benchmark:

prompt_tokens: 1178
completion_tokens: 17
finish_reason: stop
output: 灰衣男子坐在桌前，身体前倾，桌上堆满杂物。

actual behavior

Actual TensorRT-LLM result

TensorRT-LLM PyTorch backend returns:

prompt_tokens: 1182
completion_tokens: 1200
finish_reason: length

Output begins:

灰衣男子坐在桌前，身体前倾，身体前倾，身体前倾，身体前倾，身体前倾，身体前倾，身体前倾，身体前倾，身体前倾，身体前倾，...

So TensorRT-LLM both:

1. reports prompt_tokens=1182 while HF/vLLM have 1178;
2. diverges from the first generated sequence and repeats until max_tokens.

The +4 prompt-token difference is suspicious because there are exactly 4 images.

additional notes

Additional observations

When I omit the per-frame text parts (Frame N: index=..., t=...) and send only prompt + images, TensorRT-LLM sometimes stops normally. But that is not the benchmark input and does not match the HF/vLLM reproduction.

With the exact benchmark input, increasing max_input_len from the default internal 1024 to 2048 does not change the bad output. So this does not appear to be a simple input truncation issue.

I also tested:

• no --reasoning_parser
• no --tool_parser
• generation_config.json with correct EOS IDs
• explicit stop strings / stop token IDs in the request
• max_input_len: 2048

The parity issue remains.

Suspected area

This looks like a TensorRT-LLM PyTorch backend Qwen3.5-VL multimodal prompt / embedding alignment issue rather than a sampler or stop-token issue.

The prompt-token delta is exactly +1 per image compared to HF/vLLM. I noticed Qwen3VL-specific multimodal expansion logic in:

tensorrt_llm/_torch/models/modeling_qwen3vl.py

Relevant function:

def _expand_prompt_token_ids_for_mm_handoff(...):
    ...
    has_leading_special = (
        read_pos > 0 and int(input_ids[read_pos - 1].item()) == vision_start_token_id
    )
    run_start = write_pos - 1 if has_leading_special else write_pos
    prompt_mm_length = mm_token_num + int(has_leading_special)
    ...

This may be unrelated, but it is consistent with the observed +4 token count for 4 images.

Other relevant paths:

tensorrt_llm/_torch/models/modeling_qwen3vl.py
tensorrt_llm/_torch/models/modeling_qwen3_5.py
tensorrt_llm/_torch/models/modeling_multimodal_mixin.py
tensorrt_llm/_torch/models/modeling_multimodal_utils.py

Why I believe this is not a model/checkpoint issue

For the exact same sample and exact same constructed input:

HF transformers output == vLLM output
TensorRT-LLM PyTorch backend output != HF/vLLM output

HF offline result:

{
  "new_tokens": 17,
  "text": "灰衣男子坐在桌前，身体前倾，桌上堆满杂物。"
}

TensorRT-LLM result:

{
  "prompt_tokens": 1182,
  "completion_tokens": 1200,
  "finish_reason": "length",
  "text_prefix": "灰衣男子坐在桌前，身体前倾，身体前倾，身体前倾..."
}

Request

Could you confirm whether Qwen3.5-VL / Qwen3_5ForConditionalGeneration is expected to be parity-compatible in TensorRT-LLM PyTorch backend 1.3.0rc21?

If yes, could you advise which debug hook or internal flag should be used to compare:

1. expanded prompt token IDs,
2. multimodal token positions,
3. vision embeddings after Qwen3VisionModel,
4. prefill logits for the first generated token

against Hugging Face?

I can provide the local sample frames and full request payload if needed.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and checked the documentation and examples for answers to frequently asked questions.