Skip to content

[https://nvbugs/6157892][fix] Mistral format refactor#15123

Open
evezhier wants to merge 19 commits into
NVIDIA:mainfrom
evezhier:fix/mistral-format-refactor
Open

[https://nvbugs/6157892][fix] Mistral format refactor#15123
evezhier wants to merge 19 commits into
NVIDIA:mainfrom
evezhier:fix/mistral-format-refactor

Conversation

@evezhier

@evezhier evezhier commented Jun 8, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Summary by CodeRabbit

  • New Features

    • Added --custom_tokenizer CLI option to specify custom or built-in tokenizers

  • Bug Fixes

    • Fixed Mistral chat template support
    • Improved Mistral model configuration loading and robustness
    • Enhanced quantization configuration detection and handling
    • Improved RoPE parameter configuration fallback logic

  • Improvements

    • Refined model type resolution for multimodal serving scenarios

Description

Mistral format refactor for transformers v5.

  • HF checkpoint loading supported for Ministral3 models
  • v5 PretrainedConfig null rope parameters guarded
  • Both Mistral and HF processors supported. Each checkpoint format uses the corresponding processor.
  • Native tokenizer for text-only models is exposed via --custom_tokenizer, as in deepseekv3.2

Fixes also [https://nvbugs/6162588], [https://nvbugs/6162121]

Test Coverage

Existing Mistral tests should pass.

PR Checklist

Please review the following before submitting your PR:

  • PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.

  • PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.

  • Test cases are provided for new code paths (see test instructions)

  • If PR introduces API changes, an appropriate PR label is added - either api-compatible or api-breaking. For api-breaking, include BREAKING in the PR title.

  • Any new dependencies have been scanned for license and vulnerabilities

  • CODEOWNERS updated if ownership changes

  • Documentation updated as needed

  • Update tava architecture diagram if there is a significant design change in PR.

  • The reviewers assigned automatically/manually are appropriate for the PR.

  • Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

@evezhier evezhier changed the title [https://nvbugs/6157892] fix/mistral format refactor [https://nvbugs/6157892][BREAKING] fix/mistral format refactor Jun 9, 2026
@evezhier evezhier changed the title [https://nvbugs/6157892][BREAKING] fix/mistral format refactor [https://nvbugs/6157892][BREAKING][fix] Mistral format refactor Jun 9, 2026
@evezhier evezhier changed the title [https://nvbugs/6157892][BREAKING][fix] Mistral format refactor [https://nvbugs/6157892][fix][BREAKING] Mistral format refactor Jun 9, 2026
@evezhier
evezhier force-pushed the fix/mistral-format-refactor branch from 5f16ab7 to bedc948 Compare June 9, 2026 07:48
@evezhier evezhier changed the title [https://nvbugs/6157892][fix][BREAKING] Mistral format refactor [https://nvbugs/6157892][fix] [BREAKING] Mistral format refactor Jun 9, 2026
@evezhier

evezhier commented Jun 9, 2026

Copy link
Copy Markdown
Collaborator Author

/bot run --disable-fail-fast

@tensorrt-cicd

tensorrt-cicd commented Jun 9, 2026

Copy link
Copy Markdown
Collaborator

PR_Github #53004 [ run ] triggered by Bot. Commit: bedc948 Link to invocation

@evezhier
evezhier force-pushed the fix/mistral-format-refactor branch from bedc948 to 5884020 Compare June 9, 2026 07:58
@evezhier

evezhier commented Jun 9, 2026

Copy link
Copy Markdown
Collaborator Author

/bot run --disable-fail-fast

@tensorrt-cicd

tensorrt-cicd commented Jun 9, 2026

Copy link
Copy Markdown
Collaborator

PR_Github #53008 [ run ] triggered by Bot. Commit: 5884020 Link to invocation

@tensorrt-cicd

tensorrt-cicd commented Jun 9, 2026

Copy link
Copy Markdown
Collaborator

PR_Github #53004 [ run ] completed with state ABORTED. Commit: bedc948

Link to invocation

@evezhier
evezhier force-pushed the fix/mistral-format-refactor branch from 5884020 to fe7e247 Compare June 9, 2026 08:12
@evezhier

evezhier commented Jun 9, 2026

Copy link
Copy Markdown
Collaborator Author

/bot run --disable-fail-fast

@tensorrt-cicd

tensorrt-cicd commented Jun 9, 2026

Copy link
Copy Markdown
Collaborator

PR_Github #53017 [ run ] triggered by Bot. Commit: fe7e247 Link to invocation

@tensorrt-cicd

tensorrt-cicd commented Jun 9, 2026

Copy link
Copy Markdown
Collaborator

PR_Github #53008 [ run ] completed with state ABORTED. Commit: 5884020

Link to invocation

@tensorrt-cicd

tensorrt-cicd commented Jun 9, 2026

Copy link
Copy Markdown
Collaborator

PR_Github #53017 [ run ] completed with state SUCCESS. Commit: fe7e247
/LLM/main/L0_MergeRequest_PR pipeline #42241 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

@evezhier
evezhier force-pushed the fix/mistral-format-refactor branch from 6b33be9 to b6f16a8 Compare June 9, 2026 18:37
@evezhier

evezhier commented Jun 9, 2026

Copy link
Copy Markdown
Collaborator Author

/bot run --disable-fail-fast

@tensorrt-cicd

tensorrt-cicd commented Jun 9, 2026

Copy link
Copy Markdown
Collaborator

PR_Github #53124 [ run ] triggered by Bot. Commit: b6f16a8 Link to invocation

@tensorrt-cicd

tensorrt-cicd commented Jun 9, 2026

Copy link
Copy Markdown
Collaborator

PR_Github #53124 [ run ] completed with state FAILURE. Commit: b6f16a8
/LLM/main/L0_MergeRequest_PR pipeline #42329 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

@evezhier

evezhier commented Jun 9, 2026

Copy link
Copy Markdown
Collaborator Author

/bot run --disable-fail-fast

@tensorrt-cicd

tensorrt-cicd commented Jun 9, 2026

Copy link
Copy Markdown
Collaborator

PR_Github #53142 [ run ] triggered by Bot. Commit: 12009ef Link to invocation

@tensorrt-cicd

tensorrt-cicd commented Jun 10, 2026

Copy link
Copy Markdown
Collaborator

PR_Github #53142 [ run ] completed with state FAILURE. Commit: 12009ef
/LLM/main/L0_MergeRequest_PR pipeline #42346 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

@evezhier

evezhier commented Jun 10, 2026

Copy link
Copy Markdown
Collaborator Author

/bot run --disable-fail-fast

@tensorrt-cicd

tensorrt-cicd commented Jun 10, 2026

Copy link
Copy Markdown
Collaborator

PR_Github #53321 [ run ] triggered by Bot. Commit: 46e8964 Link to invocation

@tensorrt-cicd

tensorrt-cicd commented Jun 11, 2026

Copy link
Copy Markdown
Collaborator

PR_Github #53321 [ run ] completed with state FAILURE. Commit: 46e8964
/LLM/main/L0_MergeRequest_PR pipeline #42507 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

@evezhier
evezhier force-pushed the fix/mistral-format-refactor branch from 23ac419 to 6d03eba Compare June 11, 2026 08:55
@evezhier

evezhier commented Jun 11, 2026

Copy link
Copy Markdown
Collaborator Author

/bot run --disable-fail-fast

@evezhier evezhier changed the title [https://nvbugs/6157892][fix] [BREAKING] Mistral format refactor [https://nvbugs/6157892][fix] Mistral format refactor Jun 11, 2026
@tensorrt-cicd

tensorrt-cicd commented Jul 20, 2026

Copy link
Copy Markdown
Collaborator

PR_Github #60313 [ run ] triggered by Bot. Commit: 15c98ae Link to invocation

@tensorrt-cicd

tensorrt-cicd commented Jul 20, 2026

Copy link
Copy Markdown
Collaborator

PR_Github #60313 [ run ] completed with state SUCCESS. Commit: 15c98ae
/LLM/main/L0_MergeRequest_PR pipeline #48658 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

@evezhier

evezhier commented Jul 23, 2026

Copy link
Copy Markdown
Collaborator Author

/bot run

@tensorrt-cicd

tensorrt-cicd commented Jul 23, 2026

Copy link
Copy Markdown
Collaborator

PR_Github #61384 [ run ] triggered by Bot. Commit: 2911b7f Link to invocation

@tensorrt-cicd

tensorrt-cicd commented Jul 23, 2026

Copy link
Copy Markdown
Collaborator

PR_Github #61384 [ run ] completed with state FAILURE. Commit: 2911b7f
/LLM/main/L0_MergeRequest_PR pipeline #49611 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

evezhier added 19 commits July 24, 2026 09:39
@evezhier
add HF preprocessor
6139ebc 
Signed-off-by: Olya Kozlova <okozlova@nvidia.com>
@evezhier
allow HF processor for Mistral checkpoints
3e2ca5d 
Signed-off-by: Olya Kozlova <okozlova@nvidia.com>
@evezhier
small fixes
92be6c2 
Signed-off-by: Olya Kozlova <okozlova@nvidia.com>
@evezhier
text-only path fix
790bd0a 
Signed-off-by: Olya Kozlova <okozlova@nvidia.com>
@evezhier
text-only Mistral models fix
ba1895b 
Signed-off-by: Olya Kozlova <okozlova@nvidia.com>
@evezhier
unify naming
8fb00d1 
Signed-off-by: Olya Kozlova <okozlova@nvidia.com>
@evezhier
Mistral processor disambiguation in tests
1aad308 
Signed-off-by: Olya Kozlova <okozlova@nvidia.com>
@evezhier
remove circular import
429f3e4 
Signed-off-by: Olya Kozlova <okozlova@nvidia.com>
@evezhier
revert arbitrary processor selection
8807d8a 
Signed-off-by: Olya Kozlova <okozlova@nvidia.com>
@evezhier
model_type fix in test
7e89e28 
Signed-off-by: Olya Kozlova <okozlova@nvidia.com>
@evezhier
remove stale base processor
a91743c 
Signed-off-by: Olya Kozlova <okozlova@nvidia.com>
@evezhier
review fixes
6cd35b8 
Signed-off-by: Olya Kozlova <okozlova@nvidia.com>
@evezhier
unwaive
2bc73bd 
Signed-off-by: Olya Kozlova <okozlova@nvidia.com>
@evezhier
hardcode mistral_common for native checkpoints + comments
cf9217e 
Signed-off-by: Olya Kozlova <okozlova@nvidia.com>
@evezhier
review
3d282e8 
Signed-off-by: Olya Kozlova <okozlova@nvidia.com>
@evezhier
review
2a302b0 
Signed-off-by: Olya Kozlova <okozlova@nvidia.com>
@evezhier
rebase issue
483eaf8 
Signed-off-by: Olya Kozlova <okozlova@nvidia.com>
@evezhier
argparse fix
017bbb3 
Signed-off-by: Olya Kozlova <okozlova@nvidia.com>
@evezhier
eagle fix
98508b4 
Signed-off-by: Olya Kozlova <okozlova@nvidia.com>
@evezhier

evezhier commented Jul 24, 2026

Copy link
Copy Markdown
Collaborator Author

/bot run

@tensorrt-cicd

tensorrt-cicd commented Jul 24, 2026

Copy link
Copy Markdown
Collaborator

PR_Github #61535 [ run ] triggered by Bot. Commit: 98508b4 Link to invocation

@tensorrt-cicd

tensorrt-cicd commented Jul 24, 2026

Copy link
Copy Markdown
Collaborator

PR_Github #61535 [ run ] completed with state SUCCESS. Commit: 98508b4
/LLM/main/L0_MergeRequest_PR pipeline #49747 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

@evezhier

evezhier commented Jul 24, 2026

Copy link
Copy Markdown
Collaborator Author

/bot run

@tensorrt-cicd

tensorrt-cicd commented Jul 24, 2026

Copy link
Copy Markdown
Collaborator

PR_Github #61588 [ run ] triggered by Bot. Commit: 98508b4 Link to invocation

@tensorrt-cicd

tensorrt-cicd commented Jul 24, 2026

Copy link
Copy Markdown
Collaborator

PR_Github #61588 [ run ] completed with state FAILURE. Commit: 98508b4
/LLM/main/L0_MergeRequest_PR pipeline #49799 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@coderabbitai coderabbitai[bot] coderabbitai[bot] left review comments
@hchings hchings hchings approved these changes
@Funatiq Funatiq Funatiq approved these changes
@jieli-matrix jieli-matrix jieli-matrix approved these changes
@2ez4bz 2ez4bz 2ez4bz approved these changes
@moraxu moraxu Awaiting requested review from moraxu moraxu was automatically assigned from NVIDIA/trt-llm-multimodal-devs
@Wanli-Jiang Wanli-Jiang Awaiting requested review from Wanli-Jiang Wanli-Jiang was automatically assigned from NVIDIA/trt-llm-multimodal-devs
@syuoni syuoni Awaiting requested review from syuoni syuoni was automatically assigned from NVIDIA/trt-llm-llmapi-devs
@jdebache jdebache Awaiting requested review from jdebache jdebache was automatically assigned from NVIDIA/trt-llm-torch-models-mistral-devs
@QiJune QiJune Awaiting requested review from QiJune QiJune is a code owner automatically assigned from NVIDIA/trt-llm-torch-attention-devs
@venkywonka venkywonka Awaiting requested review from venkywonka venkywonka is a code owner automatically assigned from NVIDIA/trt-llm-doc-owners
@lori-ren lori-ren Awaiting requested review from lori-ren lori-ren is a code owner automatically assigned from NVIDIA/trt-llm-runtime-devs
@brnguyen2 brnguyen2 Awaiting requested review from brnguyen2 brnguyen2 is a code owner automatically assigned from NVIDIA/trt-llm-devs

At least 1 approving review is required to merge this pull request.

Assignees

@evezhier evezhier

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@evezhier @tensorrt-cicd @hchings @Funatiq @jieli-matrix @2ez4bz