Skip to content

[Feature]: Expose SmoothQuant alpha for int8_sq quantization in quantize.pyΒ #16151

Description

@BillRenCN

πŸš€ The feature, motivation and pitch

ModelOpt's SmoothQuant calibration defaults to alpha=1.0 and TensorRT-LLM's
examples/quantization/quantize.py exposes no way to change it β€” the only override in the
tree is the hardcoded Gemma special case in quantize_by_modelopt.py
({"method": "smoothquant", "alpha": 0.5}). The SmoothQuant paper recommends 0.5–0.9
depending on the model.

In our sweeps on Qwen3-0.6B/1.7B (11 alphas Γ— 2 calib sets, real TRT engines, GSM8K-1319),
alpha=1.0 was never optimal and cost up to 20 GSM8K points vs the best alpha in the same
setting (worst case: Qwen3-0.6B, 15.39% at alpha=1.0 vs 35.41% at alpha=0.85, against a
36.32% bf16 baseline).

Proposal: add --smoothquant_alpha (float, default None = current behavior) to quantize.py
and thread it to quantize_and_export, where it sets the same algorithm dict the Gemma
special case already uses. Backward compatible; ModelOpt already pydantic-validates the
field (bounds [0, 1]). Happy to submit the PR (change is ready).

Alternatives

ModelOpt's hf_ptq.py has a --recipe mechanism, so an alternative is to contribute a recipe
YAML on the Model-Optimizer side instead. But TRT-LLM's quantize.py has no recipe mechanism,
so exposing the flag directly is the minimal, self-contained fix for this CLI. The two
approaches don't conflict.

Additional context

Full sweep (11 alphas Γ— 2 calib sets Γ— 2 Qwen3 sizes):

Model Calib alpha=1.0 (current) best alpha GSM8K cost of default
Qwen3-1.7B wikitext 39.80% 57.09% @ Ξ±=0.75 βˆ’17.3 pts
Qwen3-1.7B gsm8k-train 61.87% 63.76% @ Ξ±=0.85 βˆ’1.9 pts
Qwen3-0.6B wikitext 21.76% 31.54% @ Ξ±=0.75 βˆ’9.8 pts
Qwen3-0.6B gsm8k-train 15.39% 35.41% @ Ξ±=0.85 βˆ’20.0 pts

The PR is ready: +121/βˆ’1, includes a GPU-free unit test
(tests/unittest/trt/quantization/test_smoothquant_alpha.py). No behavior change when the
flag is unset.

Before submitting a new issue...

  • Make sure you already searched for relevant issues, and checked the documentation and examples for answers to frequently asked questions.

Metadata

Metadata

Assignees

Labels

Model optimization<NV>Model-specific performance optimizations and tuningfeature requestNew feature or request. This includes new model, dtype, functionality support

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions