Skip to content
4 changes: 4 additions & 0 deletions docs/source/features/sampling.md
Original file line number Diff line number Diff line change
Expand Up @@ -115,6 +115,10 @@ llm.generate(["Hello, my name is",
* Top-P decay is not supported in combination with beam search or with speculative decoding
modes that route draft tokens through the Torch Sampler; such requests are rejected.

* If `no_repeat_ngram_size = n` is specified, any token that would recreate an `n`-gram already
present in the sequence (prompt included) is excluded from sampling. `None` or `0` disables
the restriction.

### Performance

The Torch Sampler leverages the optimized sampling kernels provided by
Expand Down
7 changes: 7 additions & 0 deletions tensorrt_llm/_torch/pyexecutor/llm_request.py
Original file line number Diff line number Diff line change
Expand Up @@ -1201,6 +1201,13 @@ def executor_request_to_llm_request(
list(word) for word in executor_request.bad_words
] if executor_request.bad_words else None

# No-repeat-ngram size for the TorchSampler path, normalized once here so
# the per-step sampler gate is a plain attribute read. The C++
# SamplingConfig rejects negative values up front and 0 means disabled
# (same convention as the C++ banRepeatNgram kernel), so falsy == off.
ngram_size = executor_request.sampling_config.no_repeat_ngram_size
llm_request.py_no_repeat_ngram_size = ngram_size if ngram_size else None

llm_request.py_original_end_id = getattr(executor_request,
"py_original_end_id",
llm_request.py_end_id)
Expand Down
Loading
Loading