[Bug]: Bad Words Missing Space-Prefixed Token Variants for BPE Tokenizers (e.g., GPT-2, Qwen)

[chfeng-cs]

System Info

CPU architecture: x86_64

CPU: Intel Xeon Gold 6462C, 8 vCPUs (4 cores × 2 threads), max 3900 MHz

GPU name: NVIDIA L20
GPU memory size: 48G

TensorRT-LLM branch or tag: main
TensorRT-LLM commit: 28acbad
PyTorch: 2.10.0+cu130
CUDA: 13.0.88
cuDNN: 9.15.0
NVIDIA driver version: 550.163.01

OS: Ubuntu 24.04.3 LTS

Who can help?

No response

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

from tensorrt_llm.llmapi import TransformersTokenizer
from tensorrt_llm.sampling_params import SamplingParams

tokenizer = TransformersTokenizer.from_pretrained("gpt2")
bad_word = "London"

unprefixed_ids = tokenizer.encode(bad_word, add_special_tokens=False)
prefixed_ids   = tokenizer.encode(f" {bad_word}", add_special_tokens=False)
print(f'"London"  -> {unprefixed_ids}')   # e.g. [23421]
print(f'" London" -> {prefixed_ids}')     # e.g. [3995]
# GPT-2 produces different ids; TinyLlama would not trigger this bug

params = SamplingParams(bad=bad_word)
params._setup(tokenizer, hf_model_config=None,
              generation_config=None, add_special_tokens=False)

bad_words = params._get_bad_words()
print(f"Recorded bad word token sequences: {bad_words}")

assert unprefixed_ids in bad_words, "Missing unprefixed form"
assert prefixed_ids   in bad_words, "Missing space-prefixed form"  # FAILS before fix

Expected behavior

SamplingParams.bad should block a bad word regardless of where it appears in the generated sequence. For prefix-space/BPE tokenizers (e.g., GPT-2, Qwen), a word like "London" tokenizes differently depending on its position:

• Beginning of text: "London" → [23421]
• Middle of text: " London" → [3995]

Both token sequences should be recorded in _bad_word_ids so the word is blocked in all positions.

actual behavior

Only the unprefixed form is encoded:

# Current implementation
self._bad_word_ids = [_encode(tokenizer, s, add_special_tokens) for s in strs]

This records only [23421] for "London". The space-prefixed variant [3995] is never stored, so the bad word is silently skipped when it appears in the middle of generated text.

For tokenizers like TinyLlama where both forms collapse to the same token ids, the bug has no visible effect. It only manifests with prefix-space/BPE tokenizers such as GPT-2 and Qwen-family models.

additional notes

• Affected tokenizer families: GPT-2 (BPE), Qwen2/Qwen2.5/Qwen3, DeepSeek-R1-Distill-Qwen, and any tokenizer where space-prefixed forms produce different token ids. TinyLlama is not affected, which is why the existing test_generate_with_bad_words does not catch this bug.
• Scope: Fix is scoped to SamplingParams.bad only; stop tokenization is left unchanged.
• No regression for unaffected tokenizers: Deduplication ensures identical sequences are stored only once, so behavior is preserved for tokenizers like TinyLlama.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and checked the documentation and examples for answers to frequently asked questions.

[smqd19]

This issue seems to stem from how the BPE tokenizer handles bad words with and without preceding spaces. In our experience with similar tokenization challenges across various models, here are some points to consider:

• Tokenization Method: Make sure that the tokenizer is consistently applied to both the input text and the bad words list. Sometimes, minor variations in whitespace can lead to discrepancies. You can preprocess your bad words list to ensure it matches the tokenizer's expectations.

• Mapping Checks: After you generate the token IDs, verify that the mapping between the space-prefixed and unprefixed variants is handled correctly in your bad words filtering process.

• Debugging: Use print statements or logging to inspect the exact token IDs being generated. This can reveal whether the issue lies in the encoding or how the bad words are being checked.

• Library Versions: Confirm you’re using compatible versions of TensorRT-LLM and PyTorch. Sometimes, bugs can appear in specific combinations, so a quick check on the release notes might help.

If the problem persists, consider isolating the tokenizer's behavior in a minimal script to see if the issue replicates outside your main codebase.