Skip to content

[https://nvbugs/6507080][fix] Override TokenizerBase.__repr__ to return f"{self.__class__.__name__}()"… #16839

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
trtllm-agent wants to merge 2 commits into
base: main
Choose a base branch
from
+6 −1
Open

[https://nvbugs/6507080][fix] Override TokenizerBase.__repr__ to return f"{self.__class__.__name__}()"… #16839

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
5f397d6
[nvbugs/6507080][fix] Add safe __repr__ to TokenizerBase
HandongLi-01 Jul 24, 2026
30b981f
[nvbugs/6507080][chore] Remove stale waiver after fix
HandongLi-01 Jul 24, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions tensorrt_llm/tokenizer/tokenizer.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,12 @@
class TokenizerBase(PreTrainedTokenizerBase):
''' This is a protocol for the tokenizer. Users can implement their own tokenizer by inheriting this class. '''

def __repr__(self) -> str:
# PreTrainedTokenizerBase.__repr__ reads properties (e.g.
# added_tokens_decoder) that TokenizerBase subclasses are not
# required to implement, so fall back to a class-name-only repr.
return f"{self.__class__.__name__}()"


def _reconstruct_transformers_tokenizer(inner_bytes: bytes):
'''Reconstruct a TransformersTokenizer from cloudpickle-serialized bytes.'''
Expand Down
1 change: 0 additions & 1 deletion tests/integration/test_lists/waives.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -376,7 +376,6 @@ unittest/executor/test_rpc_proxy.py SKIP (https://nvbugs/5605741)
unittest/executor/test_rpc_worker.py SKIP (https://nvbugs/5605741)
unittest/llmapi/test_additional_model_outputs.py -m "gpu2" SKIP (https://nvbugs/6428091)
unittest/llmapi/test_additional_model_outputs.py::test_additional_model_outputs_integration_pp2 SKIP (https://nvbugs/6427411)
unittest/llmapi/test_llm.py::test_llm_with_customized_tokenizer SKIP (https://nvbugs/6507080)
unittest/llmapi/test_llm.py::test_openai_completion_list_prompt_stream_reuses_stream_metadata SKIP (https://nvbugs/6507081)
unittest/llmapi/test_llm.py::test_tokenizer_decode_incrementally[/scratch.trt_llm_data/llm-models/falcon-7b-instruct-False-0.95-HF] SKIP (https://nvbugs/6507082)
unittest/llmapi/test_llm.py::test_tokenizer_decode_incrementally[/scratch.trt_llm_data/llm-models/falcon-7b-instruct-False-0.95-TRTLLM] SKIP (https://nvbugs/6507082)
Expand Down
Loading