System Info
N/A — documentation issue, not environment-specific. Traced against main branch, tensorrt_llm/llmapi/llm_args.py (and _torch/model_config.py for the resolution logic).
Who can help?
@juney-nvidia
Information
Tasks
Reproduction
from tensorrt_llm import LLM
llm = LLM(model="") # dtype defaults to "auto"
print(llm.args.dtype) # prints "auto" — the resolved dtype is never
# surfaced here, and the docstring doesn't say
# what it resolves to
Expected behavior
The dtype and KvCacheConfig.dtype docstrings should state which config field is read, the fallback order, and any hardware-dependent override — so users can predict the resolved value without reading llm_args.py / model_config.py source.
actual behavior
LLM.dtype docstring: "The data type to use for the model. Defaults to auto." — no resolution detail.
KvCacheConfig.dtype docstring: "Use 'auto' to follow checkpoint metadata, otherwise force the specified dtype." — doesn't say what "checkpoint metadata" means.
Neither explains the actual chain, which I traced from source:
Model dtype: reads dtype/deprecated torch_dtype from HF config.json; falls back to nested text_config.dtype for composite/VLM configs; defaults to bfloat16 if nothing found; overridden to float16 on GPUs with compute capability < 8.0.
KV cache dtype: "auto" leaves quant_config.kv_cache_quant_algo untouched (inherits checkpoint's own quant metadata); "fp8"/"nvfp4" override it explicitly. Applies at LLM-construction time, including via trtllm-serve --extra_llm_api_options.
additional notes
Happy to send a PR with the exact docstring wording if useful — flagging the gap first since I don't have hardware to verify the pre-Ampere fallback path end-to-end (traced from code, not measured).
Related: #8843, #8125
Before submitting a new issue...
System Info
N/A — documentation issue, not environment-specific. Traced against main branch, tensorrt_llm/llmapi/llm_args.py (and _torch/model_config.py for the resolution logic).
Who can help?
@juney-nvidia
Information
Tasks
examplesfolder (such as GLUE/SQuAD, ...)Reproduction
from tensorrt_llm import LLM
llm = LLM(model="") # dtype defaults to "auto"
print(llm.args.dtype) # prints "auto" — the resolved dtype is never
# surfaced here, and the docstring doesn't say
# what it resolves to
Expected behavior
The dtype and KvCacheConfig.dtype docstrings should state which config field is read, the fallback order, and any hardware-dependent override — so users can predict the resolved value without reading llm_args.py / model_config.py source.
actual behavior
LLM.dtype docstring: "The data type to use for the model. Defaults to auto." — no resolution detail.
KvCacheConfig.dtype docstring: "Use 'auto' to follow checkpoint metadata, otherwise force the specified dtype." — doesn't say what "checkpoint metadata" means.
Neither explains the actual chain, which I traced from source:
Model dtype: reads dtype/deprecated torch_dtype from HF config.json; falls back to nested text_config.dtype for composite/VLM configs; defaults to bfloat16 if nothing found; overridden to float16 on GPUs with compute capability < 8.0.
KV cache dtype: "auto" leaves quant_config.kv_cache_quant_algo untouched (inherits checkpoint's own quant metadata); "fp8"/"nvfp4" override it explicitly. Applies at LLM-construction time, including via trtllm-serve --extra_llm_api_options.
additional notes
Happy to send a PR with the exact docstring wording if useful — flagging the gap first since I don't have hardware to verify the pre-Ampere fallback path end-to-end (traced from code, not measured).
Related: #8843, #8125
Before submitting a new issue...