Add large-tensor convolution fuzzer#401
Open
msalasooNV wants to merge 6 commits into
Open
Conversation
Contributor
📝 WalkthroughWalkthroughAdds a configurable large-tensor cuDNN convolution fuzzer covering forward, data-gradient, and filter-gradient operations, with deterministic PyTorch references, engine filtering, workspace poisoning, repro payloads, regeneration, and L0/L1 pytest tiers. ChangesLarge-tensor convolution regression testing
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant Pytest
participant Runner as _run_single_config
participant Reference as PyTorch reference
participant CuDNN as cudnn.pygraph
participant GPU as CUDA tensors
Pytest->>Runner: execute LargeTensorConfig
Runner->>Reference: compute float32 convolution reference
Runner->>GPU: transfer initialized inputs
Runner->>CuDNN: build and execute convolution graph
CuDNN->>GPU: write convolution result
Runner->>Runner: compare result with reference
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@test/python/test_conv_large_tensor_fuzzer.py`:
- Around line 452-476: Update _tolerances to derive accum from
_effective_reduction_size(cfg) rather than always using cfg.c multiplied by
filter_spatial, while preserving the existing tolerance formulas and bounds.
Ensure the effective size is capped by the applicable sparse nonzero count, and
extend the related tests to validate tolerance policies for sparse FPROP, DGRAD,
and WGRAD cases.
- Around line 376-390: Update _estimate_work_flops so the DGRAD branch
calculates work from output_spatial, matching the convolution contribution count
used by FPROP/WGRAD, while retaining the existing batch, channel, filter, and
spatial factors needed for C*R*S coverage. Remove the input_spatial factor from
DGRAD to avoid overestimating valid cases.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: d3a1713a-cb6c-4828-bdea-d66440134550
📒 Files selected for processing (1)
test/python/test_conv_large_tensor_fuzzer.py
msalasooNV
force-pushed
the
msalasoo/conv-large-tensor-fuzzer
branch
from
July 24, 2026 03:01
f0a3001 to
0d489d8
Compare
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@test/python/test_conv_large_tensor_fuzzer.py`:
- Around line 1419-1431: Mark the test_conv_large_tensor_repro test with an
appropriate L0–L4 pytest level marker, consistent with its fast-skip behavior
when repro configuration environment variables are unset, so it participates in
marker-filtered runs.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: d12ebe82-e3b3-4eed-95ef-20b7d23fb6b6
📒 Files selected for processing (1)
test/python/test_conv_large_tensor_fuzzer.py
Comment on lines
+1419
to
+1431
| def test_conv_large_tensor_repro(cudnn_handle): | ||
| cfg = _load_repro_config() | ||
| if cfg is None: | ||
| pytest.skip(f"Set {_REPRO_CONFIG_ENV} or {_REPRO_FILE_ENV} to run an exact configuration") | ||
|
|
||
| ok, msg = _run_single_config(cfg, cudnn_handle) | ||
| if ok: | ||
| return | ||
| if msg.startswith("insufficient_memory"): | ||
| pytest.skip(f"Insufficient GPU memory: {msg}" f"{_format_repro_context(cfg, message=msg)}") | ||
| if msg.startswith("not_supported"): | ||
| pytest.skip(f"Graph not supported on this arch: {msg}") | ||
| pytest.fail(f"cuDNN execution error: {msg}" f"{_format_repro_context(cfg, message=msg)}") |
Contributor
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Add an L0–L4 marker to test_conv_large_tensor_repro.
Unlike the L0/L1 entry points, this collected test carries no level marker, so it is excluded from marker-filtered runs (e.g. -m L0). Mark it with an appropriate level (it fast-skips when the repro env vars are unset).
Proposed fix
+@pytest.mark.L0
def test_conv_large_tensor_repro(cudnn_handle):
cfg = _load_repro_config()As per coding guidelines: "Mark every new Python test with a level from L0 through L4".
📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| def test_conv_large_tensor_repro(cudnn_handle): | |
| cfg = _load_repro_config() | |
| if cfg is None: | |
| pytest.skip(f"Set {_REPRO_CONFIG_ENV} or {_REPRO_FILE_ENV} to run an exact configuration") | |
| ok, msg = _run_single_config(cfg, cudnn_handle) | |
| if ok: | |
| return | |
| if msg.startswith("insufficient_memory"): | |
| pytest.skip(f"Insufficient GPU memory: {msg}" f"{_format_repro_context(cfg, message=msg)}") | |
| if msg.startswith("not_supported"): | |
| pytest.skip(f"Graph not supported on this arch: {msg}") | |
| pytest.fail(f"cuDNN execution error: {msg}" f"{_format_repro_context(cfg, message=msg)}") | |
| `@pytest.mark.L0` | |
| def test_conv_large_tensor_repro(cudnn_handle): | |
| cfg = _load_repro_config() | |
| if cfg is None: | |
| pytest.skip(f"Set {_REPRO_CONFIG_ENV} or {_REPRO_FILE_ENV} to run an exact configuration") | |
| ok, msg = _run_single_config(cfg, cudnn_handle) | |
| if ok: | |
| return | |
| if msg.startswith("insufficient_memory"): | |
| pytest.skip(f"Insufficient GPU memory: {msg}" f"{_format_repro_context(cfg, message=msg)}") | |
| if msg.startswith("not_supported"): | |
| pytest.skip(f"Graph not supported on this arch: {msg}") | |
| pytest.fail(f"cuDNN execution error: {msg}" f"{_format_repro_context(cfg, message=msg)}") |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@test/python/test_conv_large_tensor_fuzzer.py` around lines 1419 - 1431, Mark
the test_conv_large_tensor_repro test with an appropriate L0–L4 pytest level
marker, consistent with its fast-skip behavior when repro configuration
environment variables are unset, so it participates in marker-filtered runs.
Source: Coding guidelines
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Affected area
Summary
Adds
test/python/test_conv_large_tensor_fuzzer.py, a seeded Python large-tensorconvolution fuzzer for FPROP, DGRAD, and WGRAD. The test generates configurations
with large tensors, offsets, and reductions, compares cuDNN execution against a
PyTorch float32 reference, and skips cases whose generated graph has no supported
engine configuration on the current architecture.
FPROP and WGRAD include
C*R*S > 2^27boundary coverage. DGRAD still exerciseslarge tensors and filters, but uses a conservative runtime-work estimate rather than
claiming the same reduction boundary.
The fuzzer includes configurable L0/L1 testcase counts, a conservative per-case
runtime-work cap to reject unusually expensive generated problems, exact configuration
JSON emission for failures, and data policies for very large reductions that keep
reference comparisons stable while still exercising large tensor shapes.
Integer-policy comparisons use fixed dtype-specific bounds, while dense-random
comparisons scale with each operation's effective reduction depth. Developer
diagnostics can restrict generation to one operation and optionally select one or more
public graph engine indices for that operation.
Why
Large tensor convolution shapes can expose backend and frontend issues that are not
covered by smaller deterministic tests. This adds reproducible fuzz coverage for those
cases without wiring a new CI job in this repo.
The generator rejects configs above a conservative runtime-work estimate, uses
deterministic seeds, emits copy-pasteable repro context, and treats unsupported
generated graphs as skips instead of hard failures. GPU allocation failures are reported
separately as insufficient-memory skips with configuration and lifecycle-phase context.
Related issues
None.
API and compatibility impact
None. This adds a Python test file only and does not change public frontend APIs
or compatibility behavior.
Testing
Full-sweep validation used:
618 passed,22 skipped1567.26s (0:26:07)618 passed,22 skipped2772.69s (0:46:12)623 passed,17 skipped1289.71s (0:21:29)621 passed,19 skipped1457.25s (0:24:17)618 passed,22 skipped1747.12s (0:29:07)618 passed,22 skipped3335.29s (0:55:35)The pass/skip counts can differ across pytest worker counts because the fuzzer
budgets tensor memory per worker at collection time. Different worker counts can
therefore accept different generated configs from the same top-level seeds, so
these rows are runtime validation for each worker mode rather than exact
testcase-for-testcase comparisons.
Developer configuration is exposed through these environment variables:
CUDNN_FUZZ_NUM_TESTS_L048.CUDNN_FUZZ_NUM_TESTS_L164.CUDNN_FUZZ_WORK_BUDGET_FLOPS1e14.CUDNN_FUZZ_ENGINE_OPfprop,dgrad, orwgrad; aliasesfp,dg, andwgare accepted. It can be used without an engine index.CUDNN_FUZZ_GRAPH_ENGINE_INDICESCUDNN_FUZZ_ENGINE_OP; each index is validated against the generated graph and one is selected deterministically per testcase.CUDNN_FUZZ_REGEN_ON_UNSUPPORTED1,true,yes, andon; false values are0,false,no,off, or empty. Other values fail during collection.CUDNN_FUZZ_REGEN_ATTEMPTS50and effective minimum1.CUDNN_FUZZ_REPRO_CONFIGtest_conv_large_tensor_repro. Takes precedence overCUDNN_FUZZ_REPRO_FILE.CUDNN_FUZZ_REPRO_FILEtest_conv_large_tensor_repro.Standard pytest controls remain available, including
-m "L0 or L1",-n <workers>, and-o addopts= -k test_conv_large_tensor_repro.Focused repro validation:
CUDNN_FUZZ_REPRO_CONFIG, file-basedCUDNN_FUZZ_REPRO_FILE, andinline-over-file precedence passed on both architectures.
-m "L0 or L1"marker runs.architectures.
Focused engine-filter and argument validation:
architectures.
0executed successfully for FPROP, DGRAD, andWGRAD on both architectures.
0,1FPROP engine list executed successfully on botharchitectures.
operations/counts/work budgets/regeneration limits, regeneration without
indices, and malformed JSON all produced the expected diagnostics.
Summary by CodeRabbit