[https://nvbugs/6507080][fix] Override TokenizerBase.__repr__ to return f"{self.__class__.__name__}()"…

[trtllm-agent]

Summary

• Root cause: TokenizerBase inherits transformers __repr__, which reads the abstract added_tokens_decoder property and raises NotImplementedError on user protocol subclasses
• Fix: Override TokenizerBase.__repr__ to return f"{self.__class__.__name__}()" so pydantic/logging never touches transformers-internal state on custom tokenizers
• Automated fix generated by repair-bot

Test plan

• Verify fix on the same GPU type as the original failure
• Check for regressions in related tests

Links

• Bug: https://nvbugs/6507080

Dev Engineer Review

• Updated TokenizerBase in tensorrt_llm/tokenizer/tokenizer.py to override __repr__, returning f"{self.__class__.__name__}()".
• Prevents inherited PreTrainedTokenizerBase.__repr__ from touching tokenizer properties (e.g., added_tokens_decoder) that TokenizerBase subclasses may not implement, avoiding NotImplementedError during pydantic representation and startup/logging.
• Change is minimal, API-consistent, and has negligible performance impact.

QA Engineer Review

• Test-list changes: tests/integration/test_lists/waives.txt
  • Removed the waivers entry for unittest/llmapi/test_llm.py::test_llm_with_customized_tokenizer (no entries added).
  • Verdict: needs follow-up

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: f3323517-52b0-4e48-8eee-70e23e131369

📥 Commits

Reviewing files that changed from the base of the PR and between e0af668 and 30b981f.

📒 Files selected for processing (2)

• tensorrt_llm/tokenizer/tokenizer.py
• tests/integration/test_lists/waives.txt

💤 Files with no reviewable changes (1)

• tests/integration/test_lists/waives.txt

---

Walkthrough

Adds a fallback TokenizerBase.__repr__ implementation returning ClassName() and removes the waiver for the customized tokenizer integration test.

Changes

Tokenizer representation and test enablement

Layer / File(s)	Summary
TokenizerBase representation fallback and waiver removal tensorrt_llm/tokenizer/tokenizer.py, tests/integration/test_lists/waives.txt	Adds a class-name-only __repr__ implementation and removes the customized tokenizer test from the waiver list.

Estimated code review effort: 1 (Trivial) | ~2 minutes

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title is specific, concise, and accurately describes the main fix to TokenizerBase.repr.
Description check	✅ Passed	The description explains the bug, fix, test plan, and bug link, with only the template's checklist section omitted.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}