[TRTLLM-13694][perf] Refresh recipe configs from IBDB#16809
Open
Mgluhovskoi wants to merge 2 commits into
Open
[TRTLLM-13694][perf] Refresh recipe configs from IBDB#16809Mgluhovskoi wants to merge 2 commits into
Mgluhovskoi wants to merge 2 commits into
Conversation
Signed-off-by: Maxim Gluhovskoi <mgluhovskoi@nvidia.com>
Contributor
WalkthroughThe PR expands DeepSeek-R1-0528 and gpt-oss-120b configuration coverage, updates serving parameters for B200 and H200 variants, and adds validated TRT-LLM commit/version metadata to configuration database records. ChangesModel configuration database
Estimated code review effort: 3 (Moderate) | ~25 minutes Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (1)
examples/configs/database/deepseek-ai/DeepSeek-R1-0528/H200/1k1k_tp8_conc32.yaml (1)
1-4: 🚀 Performance & Scalability | 🔵 Trivial | ⚡ Quick winAlign CUDA graph and runtime batch limits.
Every recipe sets
cuda_graph_config.max_batch_sizeto 128 while the runtimemax_batch_sizeis lower. With padding enabled, this can create unused graph sizes and unnecessary capture/memory overhead. The TensorRT-LLM v1.3.0rc14 guidance recommends matching these limits. (nvidia.github.io)
examples/configs/database/deepseek-ai/DeepSeek-R1-0528/H200/1k1k_tp8_conc32.yaml#L1-L4: set the CUDA graph maximum to 32.examples/configs/database/deepseek-ai/DeepSeek-R1-0528/H200/1k1k_tp8_conc4.yaml#L1-L4: set it to 4.examples/configs/database/deepseek-ai/DeepSeek-R1-0528/H200/1k1k_tp8_conc64.yaml#L1-L4: set it to 64.examples/configs/database/deepseek-ai/DeepSeek-R1-0528/H200/1k1k_tp8_conc8.yaml#L1-L4: set it to 8.examples/configs/database/deepseek-ai/DeepSeek-R1-0528/H200/8k1k_tp8_conc128.yaml#L1-L4: set it to 16.examples/configs/database/deepseek-ai/DeepSeek-R1-0528/H200/8k1k_tp8_conc16.yaml#L1-L4: set it to 16.examples/configs/database/deepseek-ai/DeepSeek-R1-0528/H200/8k1k_tp8_conc256.yaml#L1-L4: set it to 32.examples/configs/database/deepseek-ai/DeepSeek-R1-0528/H200/8k1k_tp8_conc32.yaml#L1-L4: set it to 32.examples/configs/database/deepseek-ai/DeepSeek-R1-0528/H200/8k1k_tp8_conc4.yaml#L1-L4: set it to 4.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@examples/configs/database/deepseek-ai/DeepSeek-R1-0528/H200/1k1k_tp8_conc32.yaml` around lines 1 - 4, Align cuda_graph_config.max_batch_size with max_batch_size in every affected recipe: set it to 32 in examples/configs/database/deepseek-ai/DeepSeek-R1-0528/H200/1k1k_tp8_conc32.yaml and 8k1k_tp8_conc32.yaml, 4 in 1k1k_tp8_conc4.yaml and 8k1k_tp8_conc4.yaml, 64 in 1k1k_tp8_conc64.yaml, 8 in 1k1k_tp8_conc8.yaml, 16 in 8k1k_tp8_conc128.yaml and 8k1k_tp8_conc16.yaml, and 32 in 8k1k_tp8_conc256.yaml. Keep enable_padding unchanged.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@examples/configs/database/lookup.yaml`:
- Around line 57-58: Add the repository-standard NVIDIA copyright header at the
beginning of the modified database lookup configuration, using 2026 as the
copyright year. Keep the existing validated_trtllm_commit and
validated_trtllm_version entries unchanged.
In `@examples/configs/database/openai/gpt-oss-120b/B200/8k1k_tp1_conc256.yaml`:
- Line 1: Prepend the repository-standard NVIDIA copyright header to each recipe
file: examples/configs/database/openai/gpt-oss-120b/B200/8k1k_tp1_conc256.yaml,
examples/configs/database/openai/gpt-oss-120b/B200/8k1k_tp1_conc32.yaml,
examples/configs/database/openai/gpt-oss-120b/B200/8k1k_tp1_conc4.yaml,
examples/configs/database/openai/gpt-oss-120b/B200/8k1k_tp1_conc64.yaml,
examples/configs/database/openai/gpt-oss-120b/B200/8k1k_tp1_conc8.yaml,
examples/configs/database/openai/gpt-oss-120b/H200/1k1k_tp1_conc64.yaml,
examples/configs/database/openai/gpt-oss-120b/H200/1k1k_tp1_conc8.yaml,
examples/configs/database/openai/gpt-oss-120b/H200/1k1k_tp2_conc32.yaml,
examples/configs/database/openai/gpt-oss-120b/H200/1k1k_tp2_conc4.yaml,
examples/configs/database/openai/gpt-oss-120b/H200/1k1k_tp4_conc16.yaml, and
examples/configs/database/openai/gpt-oss-120b/H200/1k1k_tp4_conc32.yaml, while
preserving each recipe's existing YAML content.
---
Nitpick comments:
In
`@examples/configs/database/deepseek-ai/DeepSeek-R1-0528/H200/1k1k_tp8_conc32.yaml`:
- Around line 1-4: Align cuda_graph_config.max_batch_size with max_batch_size in
every affected recipe: set it to 32 in
examples/configs/database/deepseek-ai/DeepSeek-R1-0528/H200/1k1k_tp8_conc32.yaml
and 8k1k_tp8_conc32.yaml, 4 in 1k1k_tp8_conc4.yaml and 8k1k_tp8_conc4.yaml, 64
in 1k1k_tp8_conc64.yaml, 8 in 1k1k_tp8_conc8.yaml, 16 in 8k1k_tp8_conc128.yaml
and 8k1k_tp8_conc16.yaml, and 32 in 8k1k_tp8_conc256.yaml. Keep enable_padding
unchanged.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 8701a11c-e7c4-4b9b-a238-87481545d24f
📒 Files selected for processing (50)
docs/source/_static/config_db.jsonexamples/configs/database/deepseek-ai/DeepSeek-R1-0528/B200/1k1k_tp4_conc16.yamlexamples/configs/database/deepseek-ai/DeepSeek-R1-0528/B200/1k1k_tp4_conc8.yamlexamples/configs/database/deepseek-ai/DeepSeek-R1-0528/B200/1k1k_tp8_conc128.yamlexamples/configs/database/deepseek-ai/DeepSeek-R1-0528/B200/1k1k_tp8_conc64.yamlexamples/configs/database/deepseek-ai/DeepSeek-R1-0528/B200/8k1k_tp4_conc16.yamlexamples/configs/database/deepseek-ai/DeepSeek-R1-0528/B200/8k1k_tp4_conc32.yamlexamples/configs/database/deepseek-ai/DeepSeek-R1-0528/B200/8k1k_tp4_conc8.yamlexamples/configs/database/deepseek-ai/DeepSeek-R1-0528/B200/8k1k_tp8_conc128.yamlexamples/configs/database/deepseek-ai/DeepSeek-R1-0528/B200/8k1k_tp8_conc16.yamlexamples/configs/database/deepseek-ai/DeepSeek-R1-0528/B200/8k1k_tp8_conc256.yamlexamples/configs/database/deepseek-ai/DeepSeek-R1-0528/B200/8k1k_tp8_conc32.yamlexamples/configs/database/deepseek-ai/DeepSeek-R1-0528/B200/8k1k_tp8_conc64.yamlexamples/configs/database/deepseek-ai/DeepSeek-R1-0528/H200/1k1k_tp8_conc128.yamlexamples/configs/database/deepseek-ai/DeepSeek-R1-0528/H200/1k1k_tp8_conc16.yamlexamples/configs/database/deepseek-ai/DeepSeek-R1-0528/H200/1k1k_tp8_conc256.yamlexamples/configs/database/deepseek-ai/DeepSeek-R1-0528/H200/1k1k_tp8_conc32.yamlexamples/configs/database/deepseek-ai/DeepSeek-R1-0528/H200/1k1k_tp8_conc4.yamlexamples/configs/database/deepseek-ai/DeepSeek-R1-0528/H200/1k1k_tp8_conc64.yamlexamples/configs/database/deepseek-ai/DeepSeek-R1-0528/H200/1k1k_tp8_conc8.yamlexamples/configs/database/deepseek-ai/DeepSeek-R1-0528/H200/8k1k_tp8_conc128.yamlexamples/configs/database/deepseek-ai/DeepSeek-R1-0528/H200/8k1k_tp8_conc16.yamlexamples/configs/database/deepseek-ai/DeepSeek-R1-0528/H200/8k1k_tp8_conc256.yamlexamples/configs/database/deepseek-ai/DeepSeek-R1-0528/H200/8k1k_tp8_conc32.yamlexamples/configs/database/deepseek-ai/DeepSeek-R1-0528/H200/8k1k_tp8_conc4.yamlexamples/configs/database/deepseek-ai/DeepSeek-R1-0528/H200/8k1k_tp8_conc64.yamlexamples/configs/database/deepseek-ai/DeepSeek-R1-0528/H200/8k1k_tp8_conc8.yamlexamples/configs/database/lookup.yamlexamples/configs/database/openai/gpt-oss-120b/B200/1k1k_tp1_conc256.yamlexamples/configs/database/openai/gpt-oss-120b/B200/1k1k_tp2_conc256.yamlexamples/configs/database/openai/gpt-oss-120b/B200/8k1k_tp1_conc128.yamlexamples/configs/database/openai/gpt-oss-120b/B200/8k1k_tp1_conc16.yamlexamples/configs/database/openai/gpt-oss-120b/B200/8k1k_tp1_conc256.yamlexamples/configs/database/openai/gpt-oss-120b/B200/8k1k_tp1_conc32.yamlexamples/configs/database/openai/gpt-oss-120b/B200/8k1k_tp1_conc4.yamlexamples/configs/database/openai/gpt-oss-120b/B200/8k1k_tp1_conc64.yamlexamples/configs/database/openai/gpt-oss-120b/B200/8k1k_tp1_conc8.yamlexamples/configs/database/openai/gpt-oss-120b/H200/1k1k_tp1_conc64.yamlexamples/configs/database/openai/gpt-oss-120b/H200/1k1k_tp1_conc8.yamlexamples/configs/database/openai/gpt-oss-120b/H200/1k1k_tp2_conc32.yamlexamples/configs/database/openai/gpt-oss-120b/H200/1k1k_tp2_conc4.yamlexamples/configs/database/openai/gpt-oss-120b/H200/1k1k_tp4_conc16.yamlexamples/configs/database/openai/gpt-oss-120b/H200/1k1k_tp4_conc32.yamlexamples/configs/database/openai/gpt-oss-120b/H200/1k1k_tp4_conc4.yamlexamples/configs/database/openai/gpt-oss-120b/H200/1k1k_tp4_conc8.yamlexamples/configs/database/openai/gpt-oss-120b/H200/1k1k_tp8_conc4.yamlexamples/configs/database/openai/gpt-oss-120b/H200/1k1k_tp8_conc8.yamlexamples/configs/database/openai/gpt-oss-120b/H200/8k1k_tp1_conc4.yamlexamples/configs/database/openai/gpt-oss-120b/H200/8k1k_tp4_conc64.yamlexamples/configs/database/openai/gpt-oss-120b/H200/8k1k_tp8_conc8.yaml
Contributor
Author
|
/bot run |
Contributor
Author
|
Regarding the CodeRabbit nitpick about making — auto-IBDB bot |
Collaborator
|
PR_Github #61389 [ run ] triggered by Bot. Commit: |
Collaborator
|
PR_Github #61389 [ run ] completed with state
|
Contributor
Author
|
/bot run |
Collaborator
|
PR_Github #61396 [ run ] triggered by Bot. Commit: |
Collaborator
|
PR_Github #61396 [ run ] completed with state
|
Contributor
Author
|
/bot run |
Collaborator
|
PR_Github #61408 [ run ] triggered by Bot. Commit: |
Collaborator
|
PR_Github #61408 [ run ] completed with state
|
Contributor
Author
|
/bot run |
Collaborator
|
PR_Github #61418 [ run ] triggered by Bot. Commit: |
Collaborator
|
PR_Github #61418 [ run ] completed with state
|
Contributor
Author
|
/bot run |
Collaborator
|
PR_Github #61519 [ run ] triggered by Bot. Commit: |
Collaborator
|
PR_Github #61519 [ run ] completed with state
|
Contributor
Author
|
/bot run |
Collaborator
|
PR_Github #61595 [ run ] triggered by Bot. Commit: |
Collaborator
|
PR_Github #61595 [ run ] completed with state
|
Contributor
Author
|
/bot run |
Collaborator
|
PR_Github #61625 [ run ] triggered by Bot. Commit: |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Dev Engineer Review
config_db.jsonwith the new configurations and validation metadata.QA Engineer Review
No test changes.
Description
Refresh the scoped TensorRT-LLM recipe configurations from the latest validated IBDB Pareto results.
The sync covered 65 usable mapped workloads: 24 new entries, 24 updated entries, and 17 entries already current. This PR adds 24 recipe YAML files, refreshes 24 existing recipe YAML files, records exact TensorRT-LLM v1.3.0rc14 provenance in
examples/configs/database/lookup.yaml, and regeneratesdocs/source/_static/config_db.json.Public scope:
nvidia/DeepSeek-R1-0528-FP4-v2on B200 NVLdeepseek-ai/DeepSeek-R1-0528on B200 NVL and H200 SXMopenai/gpt-oss-120bon B200 NVL and H200 SXMNo API or dependency changes are included.
Test Coverage
git diff --check: passed.PR Checklist
PR description clearly explains what and why.
PR follows the TRT-LLM coding guidelines to the best of my knowledge.
Existing config database tests cover the changed paths; no new code path is introduced.
No API change is introduced.
No new dependency is introduced.
CODEOWNERS changes are not required.
The generated customer-facing config database is updated.
No architecture change is introduced.
Reviewers can be assigned by the repository's normal automation.
Please check this after reviewing the above items as appropriate for this PR.
GitHub Bot Help
To see a list of available CI bot commands, please comment
/bot help.