During Qwen3.6-35B A3B NVPF4 Quantization IndexError: tuple index out of range error when Tensor RTT is used during the process
docker run --rm -it --gpus all --ipc=host --ulimit memlock=-1 --ulimit stack=67108864 -v "./output_models:/workspace/output_models" -v "$HOME/.cache/huggingface:/root/.cache/huggi
ngface" -e HF_TOKEN=$HF_TOKEN nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc20 bash -c "
git clone --depth 1 --branch main https://github.com/NVIDIA/Model-Optimizer.git /app/Model-Optimizer && \
cd /app/Model-Optimizer/ && \
pip install -e '.' && \
cd /app/Model-Optimizer/examples/hf_ptq && \
export ROOT_SAVE_PATH='/workspace/output_models' && \
scripts/huggingface_example.sh --model 'Qwen/Qwen3.6-35B-A3B' --quant nvfp4 --tasks quant --tp 1
"
Loading safetensors weights in parallel: 100%|██████████| 3/3 [00:02<00:00, 1.18it/s]
[07/07/2026-03:31:36] [TRT-LLM] [E] [executor] Failed to initialize executor on rank 0: tuple index out of range
[07/07/2026-03:31:36] [TRT-LLM] [E] [executor] Traceback (most recent call last):
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/executor/worker.py", line 302, in worker_main
worker: GenerationExecutorWorker = worker_cls(
^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/executor/worker.py", line 67, in __init__
self.setup_engine()
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/executor/base_worker.py", line 286, in setup_engine
self.engine = _create_py_executor(
^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/executor/base_worker.py", line 257, in _create_py_executor
_executor = create_executor(**args)
^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/_torch/pyexecutor/py_executor_creator.py", line 516, in create_py_executor
model_engine = PyTorchModelEngine(
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/_torch/pyexecutor/model_engine.py", line 341, in __init__
self.model, moe_load_balancer = self.model_loader.load(
^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/_torch/pyexecutor/model_loader.py", line 519, in load
self._call_load_weights(model.load_weights, weights,
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/_torch/pyexecutor/model_loader.py", line 1135, in _call_load_weights
load_method(weights, **kargs)
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/_torch/models/modeling_qwen3_next.py", line 980, in load_weights
new_weights = weight_mapper.preprocess_weights(weights)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/_torch/models/checkpoints/hf/qwen3_5_weight_mapper.py", line 327, in preprocess_weights
packed_weights = self._pack_split_projections(normalized_weights)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/_torch/models/checkpoints/hf/qwen3_5_weight_mapper.py", line 258, in _pack_split_projections
q_tensor, k_tensor, v_tensor = split_packed_qkv(
^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/_torch/models/checkpoints/hf/qwen3_5_weight_mapper.py", line 169, in _split_qkv_tensor
assert tensor.shape[0] == expected_total, (
~~~~~~~~~~~~^^^
IndexError: tuple index out of range
[07/07/2026-03:31:36] [TRT-LLM] [E] [executor] Executor worker initialization error: Traceback (most recent call last):
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/executor/worker.py", line 302, in worker_main
worker: GenerationExecutorWorker = worker_cls(
^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/executor/worker.py", line 67, in __init__
self.setup_engine()
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/executor/base_worker.py", line 286, in setup_engine
self.engine = _create_py_executor(
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/executor/base_worker.py", line 257, in _create_py_executor
_executor = create_executor(**args)
^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/_torch/pyexecutor/py_executor_creator.py", line 516, in create_py_executor
model_engine = PyTorchModelEngine(
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/_torch/pyexecutor/model_engine.py", line 341, in __init__
self.model, moe_load_balancer = self.model_loader.load(
^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/_torch/pyexecutor/model_loader.py", line 519, in load
self._call_load_weights(model.load_weights, weights,
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/_torch/pyexecutor/model_loader.py", line 1135, in _call_load_weights
load_method(weights, **kargs)
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/_torch/models/modeling_qwen3_next.py", line 980, in load_weights
new_weights = weight_mapper.preprocess_weights(weights)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/_torch/models/checkpoints/hf/qwen3_5_weight_mapper.py", line 327, in preprocess_weights
packed_weights = self._pack_split_projections(normalized_weights)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/_torch/models/checkpoints/hf/qwen3_5_weight_mapper.py", line 258, in _pack_split_projections
q_tensor, k_tensor, v_tensor = split_packed_qkv(
^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/_torch/models/checkpoints/hf/qwen3_5_weight_mapper.py", line 169, in _split_qkv_tensor
assert tensor.shape[0] == expected_total, (
~~~~~~~~~~~~^^^
IndexError: tuple index out of range
IndexError: tuple index out of range
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/app/Model-Optimizer/examples/hf_ptq/run_tensorrt_llm.py", line 96, in <module>
run(args)
File "/app/Model-Optimizer/examples/hf_ptq/run_tensorrt_llm.py", line 71, in run
llm = LLM(
^^^^
File "/app/Model-Optimizer/modelopt/deploy/llm/generate.py", line 148, in __init__
super().__init__(
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/llmapi/llm.py", line 1877, in __init__
super().__init__(model, tokenizer, tokenizer_mode, skip_tokenizer_init,
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/llmapi/llm.py", line 1717, in __init__
super().__init__(model,
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/llmapi/llm.py", line 349, in __init__
self._build_model()
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/llmapi/llm.py", line 1824, in _build_model
self._executor = self._executor_cls.create(
^^^^^^^^^^^^^^^^^^^^^^^^^^ 22:42:35 [196/2078]
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/executor/executor.py", line 639, in create
return GenerationExecutor._create_ipc_executor(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/executor/executor.py", line 510, in _create_ipc_executor
return GenerationExecutorProxy(
^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/executor/proxy.py", line 150, in __init__
self._start_executor_workers(worker_kwargs)
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/executor/proxy.py", line 437, in _start_executor_workers
raise RuntimeError(
RuntimeError: Executor worker returned error
docker run --rm -it --gpus all --ipc=host --ulimit memlock=-1 --ulimit stack=67108864 -v "./output_models:/workspace/output_models" -v "$HOME/.cache/huggingface:/root/.cache/huggi
ngface" -e HF_TOKEN=$HF_TOKEN nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc20 bash -c "
git clone --depth 1 --branch main https://github.com/NVIDIA/Model-Optimizer.git /app/Model-Optimizer && \
cd /app/Model-Optimizer/ && \
pip install -e '.' && \
cd /app/Model-Optimizer/examples/hf_ptq && \
export ROOT_SAVE_PATH='/workspace/output_models' && \
scripts/huggingface_example.sh --model 'Qwen/Qwen3.6-35B-A3B' --quant nvfp4 --tasks quant --tp 1
"
It also "hangs" when tensorRT LLM loads Qwen3.6 model, so i'm not 100% sure this is on TensorRT-LLM or Model Optimizer where i also opened an issue NVIDIA/Model-Optimizer#1933
SNAPSHOT_HASH=$(ls "$HOME/.cache/huggingface/hub/models--Qwen--Qwen3.6-35B-A3B/snapshots/" | head -1)
docker run \
-e HF_TOKEN=$HF_TOKEN \
-v "$HOME/.cache/huggingface/:/root/.cache/huggingface/" \
--rm -it --ulimit memlock=-1 --ulimit stack=67108864 \
--gpus=all --ipc=host --network host \
nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc20 \
bash -c "
python examples/llm-api/quickstart_advanced.py \
--model_dir /root/.cache/huggingface/hub/models--Qwen--Qwen3.6-35B-A3B/snapshots/${SNAPSHOT_HASH}/ \
--prompt 'Colombia is known because' \
--max_tokens 64
"
=============
== PyTorch ==
=============
NVIDIA Release 26.02 (build 269443100)
PyTorch Version 2.11.0a0+eb65b36
Container image Copyright (c) 2025, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
Copyright (c) 2014-2024 Facebook Inc.
Copyright (c) 2011-2014 Idiap Research Institute (Ronan Collobert)
Copyright (c) 2012-2014 Deepmind Technologies (Koray Kavukcuoglu)
Copyright (c) 2011-2012 NEC Laboratories America (Koray Kavukcuoglu)
Copyright (c) 2011-2013 NYU (Clement Farabet)
Copyright (c) 2006-2010 NEC Laboratories America (Ronan Collobert, Leon Bottou, Iain Melvin, Jason Weston)
Copyright (c) 2006 Idiap Research Institute (Samy Bengio)
Copyright (c) 2001-2004 Idiap Research Institute (Ronan Collobert, Samy Bengio, Johnny Mariethoz)
Copyright (c) 2015 Google Inc.
Copyright (c) 2015 Yangqing Jia
Copyright (c) 2013-2016 The Caffe contributors
All rights reserved.
Various files include modifications (c) NVIDIA CORPORATION & AFFILIATES. All rights reserved.
GOVERNING TERMS: The software and materials are governed by the NVIDIA Software License Agreement
(found at https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-software-license-agreement/)
and the Product-Specific Terms for NVIDIA AI Products
(found at https://www.nvidia.com/en-us/agreements/enterprise-software/product-specific-terms-for-ai-products/).
NOTE: CUDA Forward Compatibility mode ENABLED.
Using CUDA 13.1 driver version 590.48.01 with kernel driver version 580.159.03.
See https://docs.nvidia.com/deploy/cuda-compatibility/ for details.
Skipping import of cpp extensions due to incompatible torch version 2.11.0a0+eb65b36914.nv26.02 for torchao version 0.15.0 Please see https://github.com/pytorch/ao/issues/2919 for more info
/usr/local/lib/python3.12/dist-packages/modelopt/torch/__init__.py:36: UserWarning: transformers version 5.5.4 is incompatible with nvidia-modelopt and may cause issues. Please install recommended version with `pip install nvidia-modelopt[hf]` if working with HF models.
_warnings.warn(
[TensorRT-LLM] TensorRT LLM version: 1.3.0rc20
[07/07/2026-16:28:39] [TRT-LLM] [I] [llmapi] Using LLM with PyTorch backend
[07/07/2026-16:28:39] [TRT-LLM] [W] [llmapi] Using default gpus_per_node: 1
[07/07/2026-16:28:39] [TRT-LLM] [I] [llmapi] neither checkpoint_format nor checkpoint_loader were provided, checkpoint_format will be set to HF.
`torch_dtype` is deprecated! Use `dtype` instead!
rank 0 using MpiPoolSession to spawn MPI processes
Skipping import of cpp extensions due to incompatible torch version 2.11.0a0+eb65b36914.nv26.02 for torchao version 0.15.0 Please see https://github.com/pytorch/ao/issues/2919 for more info
Multiple distributions found for package optimum. Picked distribution: optimum
Multiple distributions found for package modelopt. Picked distribution: nvidia-modelopt
/usr/local/lib/python3.12/dist-packages/modelopt/torch/__init__.py:36: UserWarning: transformers version 5.5.4 is incompatible with nvidia-modelopt and may cause issues. Please install recommended version with `pip
install nvidia-modelopt[hf]` if working with HF models.
_warnings.warn(
[TensorRT-LLM] TensorRT LLM version: 1.3.0rc20
[TRT-LLM] [I] [runtime][RANK 0] Refreshed the MPI local session
/usr/local/lib/python3.12/dist-packages/tensorrt_llm/serve/openai_protocol.py:145: UserWarning: Field name "schema" in "ResponseFormat" shadows an attribute in parent "OpenAIBaseModel"
class ResponseFormat(OpenAIBaseModel):
`torch_dtype` is deprecated! Use `dtype` instead!
Loading safetensors weights in parallel: 100%|██████████| 26/26 [00:00<00:00, 86.42it/s]
Loading weights serially: 0it [00:00, ?it/s]
Loading weights concurrently: 100%|██████████| 1178/1178 [01:04<00:00, 18.27it/s]
Model init total -- 70.47s
[TRT-LLM] [I] [batchmgr][RANK 0] Max KV cache blocks per sequence: 1003 [window size=-2147483647], tokens per block=32, primary blocks=2173, secondary blocks=0, max sequence length=32096
[TRT-LLM] [I] [batchmgr][RANK 0] Max KV cache blocks per sequence: 1003 [window size=32096], tokens per block=32, primary blocks=1003, secondary blocks=0, max sequence length=32096
--------------------------------------------------------------------------
Child job 2 terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
System Info
Describe the bug
During Qwen3.6-35B A3B NVPF4 Quantization IndexError: tuple index out of range error when Tensor RTT is used during the process
Steps/Code to reproduce bug
docker run --rm -it --gpus all --ipc=host --ulimit memlock=-1 --ulimit stack=67108864 -v "./output_models:/workspace/output_models" -v "$HOME/.cache/huggingface:/root/.cache/huggi
ngface" -e HF_TOKEN=$HF_TOKEN nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc20 bash -c "
git clone --depth 1 --branch main https://github.com/NVIDIA/Model-Optimizer.git /app/Model-Optimizer && \
cd /app/Model-Optimizer/ && \
pip install -e '.' && \
cd /app/Model-Optimizer/examples/hf_ptq && \
export ROOT_SAVE_PATH='/workspace/output_models' && \
scripts/huggingface_example.sh --model 'Qwen/Qwen3.6-35B-A3B' --quant nvfp4 --tasks quant --tp 1
"
Expected behavior
Working quantized model output
System information
Who can help?
No response
Information
Tasks
examplesfolder (such as GLUE/SQuAD, ...)Reproduction
Expected behavior
Quantization done and model exported
actual behavior
IndexError: tuple index out of rangeerror from tensorRT LLMadditional notes
It also "hangs" when tensorRT LLM loads Qwen3.6 model, so i'm not 100% sure this is on TensorRT-LLM or Model Optimizer where i also opened an issue NVIDIA/Model-Optimizer#1933
Before submitting a new issue...