[None][test] Add per-cluster GPU-bucketed QA functional test-db YAMLs#16825
Open
jieli-matrix wants to merge 3 commits into
Open
[None][test] Add per-cluster GPU-bucketed QA functional test-db YAMLs#16825jieli-matrix wants to merge 3 commits into
jieli-matrix wants to merge 3 commits into
Conversation
Add per-cluster QA functional test-db YAMLs
(qa_{h100,b200,gb200,b300,gb300}.yml) that bucket llm_function_core.txt by
GPU count, mirroring the L0 test-db condition/ranges/wildcards schema, so a
follow-up pipeline change can request right-sized SLURM allocations instead
of feeding one flat list into whole-node jobs. Each condition block is one
GPU-count bucket (ranges.system_gpu_count).
This first commit is the BASELINE: every core test is bucketed per cluster
with NO skip removal applied -- L0-covered, arch/device/unconditional,
memory-gated and waived tests are all present. The next two commits remove
the tests the SLURM pipeline already drops at run time (L0-covered, then
arch-incompatible), so each removal's effect on the per-machine list is
reviewable in isolation. Waived and memory-gated tests are never removed;
they stay in the list and surface as SKIPPED with a reason at run time.
Signed-off-by: Jie Li <lijie@nvidia.com>
Regenerate the per-cluster QA test-db YAMLs with the L0 removal applied. The SLURM pipeline's excludeL0Tests renders each cluster's L0 contexts (e.g. H100 -> l0_h100 + l0_dgx_h100) with an empty match -- the entire L0 context files -- and subtracts them from the QA list, so QA never re-runs on a machine what L0 already covers there. The union of node ids in those L0 files is dropped. The diff from the baseline commit shows exactly which tests each cluster stops running because L0 covers them there (h100 315, b200 240, gb200 97, b300 56, gb300 39). Arch/device/unconditional skips are still KEPT in this commit; the next commit removes them. Signed-off-by: Jie Li <lijie@nvidia.com>
Regenerate the per-cluster QA test-db YAMLs with the arch removal applied (the committed final state). SM-based arch markers (skip_pre_blackwell, skip_post_blackwell, skip_no_hopper, skip_no_sm120, ...), including per-pytest.param marks, are evaluated against each cluster's compute capability (h100=90, b200/gb200=100, b300/gb300=103) and tests that would skip are dropped, along with the handful of device-string (skip_device_not_contain / skip_device_contain_gb200) and unconditional @pytest.mark.skip cases decided by the same static pass. So the YAML shows the real per-machine set instead of deferring to a runtime skip. The diff from the previous commit shows the arch/device/unconditional drops per cluster (h100 302, b200 78, gb200 68, b300 85, gb300 85). Memory-gated (skip_less_device_memory) and waived tests are intentionally KEPT: they vary by HBM or carry an nvbug reason and stay visible as SKIPPED at run time. NVLS / feature-probe markers can't be resolved statically and are also left in place. This matches trt_jenkins runtime behavior. Signed-off-by: Jie Li <lijie@nvidia.com>
Contributor
WalkthroughAdds GPU- and GPU-count-specific QA test selection matrices for B200, B300, GB200, GB300, and H100 systems. Each configuration targets Ubuntu hosts and defines suites for 1, 2, 4, and 8 GPUs. ChangesGPU-specific QA test matrices
Estimated code review effort: 3 (Moderate) | ~25 minutes Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@tests/integration/test_lists/qa/test-db/qa_b200.yml`:
- Around line 8-11: Update wildcards.gpu in
tests/integration/test_lists/qa/test-db/qa_b200.yml lines 8-11 to match B200
GPUs without also matching GB200, and update the corresponding wildcards.gpu
entry in tests/integration/test_lists/qa/test-db/qa_b300.yml lines 8-11 to match
B300 GPUs without matching GB300; preserve the existing Linux distribution
wildcard.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: e82bef84-48fd-46f6-84e1-f0dcf2a39fd6
📒 Files selected for processing (5)
tests/integration/test_lists/qa/test-db/qa_b200.ymltests/integration/test_lists/qa/test-db/qa_b300.ymltests/integration/test_lists/qa/test-db/qa_gb200.ymltests/integration/test_lists/qa/test-db/qa_gb300.ymltests/integration/test_lists/qa/test-db/qa_h100.yml
Comment on lines
+8
to
+11
| wildcards: | ||
| gpu: | ||
| - '*b200*' | ||
| linux_distribution_name: ubuntu* |
Contributor
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Find how the `gpu` mako value is produced from stage names and what literals are used.
rg -nP -C3 '\bgpu\s*=\s*' jenkins/ | rg -i 'b200|b300|gb200|gb300' || true
# List gpu wildcard usage across existing test-db to see the established convention.
fd -e yml -e yaml . tests/integration/test_lists | xargs rg -nP "gpu:" -A2 | rg -i 'b200|b300|h100' | head -50Repository: NVIDIA/TensorRT-LLM
Length of output: 4899
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== relevant list files =="
git ls-files 'tests/integration/test_lists/qa/test-db/qa_b200.yml' \
'tests/integration/test_lists/qa/test-db/qa_b300.yml' \
'tests/integration/test_lists/test-db/*gb300*' \
'tests/integration/test_lists/test-db/*b200*' \
'tests/integration/test_lists/test-db/*b300*'
echo
echo "== selector/search code for gpu wildcards =="
rg -n --hidden -S "fnmatch|wildcards\.gpu|gpu:" tests jenkins .github -g '!**/node_modules/**' || true
echo
echo "== locate stage-name production references =="
rg -n --hidden -S "gb200|gb300|b200|b300|gpu.*stage|stage.*gpu|mako" jenkins tests .github || trueRepository: NVIDIA/TensorRT-LLM
Length of output: 50376
Tighten the gpu globs to exclude GB200/GB300. wildcards.gpu is matched with case-insensitive fnmatch against the stage-derived mako.gpu, so *b200* also catches gb200 and *b300* catches gb300.
tests/integration/test_lists/qa/test-db/qa_b200.yml#L8-L11tests/integration/test_lists/qa/test-db/qa_b300.yml#L8-L11
📍 Affects 2 files
tests/integration/test_lists/qa/test-db/qa_b200.yml#L8-L11(this comment)tests/integration/test_lists/qa/test-db/qa_b300.yml#L8-L11
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@tests/integration/test_lists/qa/test-db/qa_b200.yml` around lines 8 - 11,
Update wildcards.gpu in tests/integration/test_lists/qa/test-db/qa_b200.yml
lines 8-11 to match B200 GPUs without also matching GB200, and update the
corresponding wildcards.gpu entry in
tests/integration/test_lists/qa/test-db/qa_b300.yml lines 8-11 to match B300
GPUs without matching GB300; preserve the existing Linux distribution wildcard.
Collaborator
Author
There was a problem hiding this comment.
good point, so how to fix?
Contributor
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== QA selector implementation and wildcard semantics =="
rg -n -C 5 -i 'fnmatch|wildcards|mako\.gpu|system_gpu_count' tests jenkins .github \
-g '!**/node_modules/**' | head -400 || true
echo
echo "== B200/B300/GB200/GB300 stage and GPU-value references =="
rg -n -i -C 4 'gb200|gb300|b200|b300' jenkins tests .github \
-g '!tests/integration/test_lists/qa/test-db/qa_b200.yml' \
-g '!tests/integration/test_lists/qa/test-db/qa_b300.yml' | head -500 || true
echo
echo "== Existing GPU wildcard conventions =="
fd -e yml -e yaml . tests/integration/test_lists \
--exec rg -n -i -C 2 'gpu:|b200|b300|gb200|gb300' {} \; | head -400 || trueLength of output: 100373
@jieli-matrix, replace the broad substring globs with the exact stage-derived GPU values. The matcher treats entries under wildcards.gpu as OR patterns, so list both the standard and DGX B-series names:
# qa_b200.yml — apply in each of its 1/2/4/8-GPU condition blocks
wildcards:
gpu:
- b200
- dgx_b200
linux_distribution_name: ubuntu*# qa_b300.yml — apply in each of its 1/2/4/8-GPU condition blocks
wildcards:
gpu:
- b300
- dgx_b300
linux_distribution_name: ubuntu*This is safer than attempting a negative glob: the stage parser produces values such as dgx_b200, b300, dgx_b300, gb200, and gb300; exact patterns retain B200/B300 coverage and no longer select GB200/GB300.
Test coverage summary: this PR changes QA test-list data only. Update all four condition blocks in each affected YAML consistently; no test-function change is needed. Coverage verdict: needs follow-up, since no CBTS coverage report was available to verify the resulting pipeline selection.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds per-cluster QA functional test-db YAMLs under tests/integration/test_lists/qa/test-db/ — qa_{h100,b200,gb200,b300,gb300}.yml — that bucket llm_function_core.txt by GPU count, mirroring the L0 test-db condition/ranges/wildcards schema. Each condition block is one system_gpu_count bucket, so a follow-up pipeline change can request right-sized SLURM allocations instead of feeding one flat list into whole-node jobs (where a 1-GPU test still grabs a full node).
Why
QA feeds one flat ~955-test list into jobs that allocate a whole node and shard only by test count. Bucketing by GPU count lets the pipeline size allocations to actual need. This PR delivers the checked-in lists; pipeline consumption is a separate change (qa-testdb-pipeline-wiring).
Commits — reviewable skip progression
The three commits peel off, one at a time, the tests the SLURM pipeline already drops at run time, so each removal's effect on the per-machine list is a clean isolated diff:
GPU count for each remaining test is resolved statically from skip_less_device / skip_less_mpi_world_size markers, tp/pp/cp params, and each test's get_device_count() skip guard, rounded up to {1,2,4,8,16}.
Dev Engineer Review
qa_h100.yml,qa_b200.yml,qa_gb200.yml,qa_b300.yml, andqa_gb300.yml.QA Engineer Review
tests/integration/test_lists/qa/test-db/.llm_function_coretest selections for H100, B200, GB200, B300, and GB300 clusters.