Skip to content

Commit 5d07ce0

Browse files
Cadence tests should retry (#19313)
With these giant 500+ op tests we often will get a flakey 1/500 failure. Just adding retries to make this a little less noisy. Failure is something like FAILED backends/cadence/aot/tests/test_replace_ops_passes.py::TestReplaceOpsPasses::test_replace_conv2d_with_linear - AssertionError: Pass validation failed for pass ReplaceTrivialConvWithLinear. Output tensor 0 differs by max 1.525879e-05. Expected rtol=2e-05, atol=1e-06. Original output: tensor([[[[ 6.5604]],
1 parent 165ac2e commit 5d07ce0

1 file changed

Lines changed: 4 additions & 4 deletions

File tree

.github/workflows/_test_cadence.yml

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -45,9 +45,9 @@ jobs:
4545
4646
./install_requirements.sh > /dev/null
4747
pip install -e . --no-build-isolation > /dev/null
48-
pip install beartype later pyre_extensions pytest-xdist
48+
pip install beartype later pyre_extensions pytest-rerunfailures==15.1 pytest-xdist
4949
50-
python -m pytest backends/cadence/aot/tests/ -v -n auto
50+
python -m pytest backends/cadence/aot/tests/ -v -n auto --reruns 2 --reruns-delay 1
5151
5252
test-ops:
5353
uses: pytorch/test-infra/.github/workflows/linux_job_v2.yml@main
@@ -66,12 +66,12 @@ jobs:
6666
6767
./install_requirements.sh > /dev/null
6868
pip install -e . --no-build-isolation > /dev/null
69-
pip install beartype later pyre_extensions pytest-xdist
69+
pip install beartype later pyre_extensions pytest-rerunfailures==15.1 pytest-xdist
7070
7171
# Use the pre-built runner from the build job
7272
mkdir -p cmake-out/backends/cadence
7373
cp "${RUNNER_ARTIFACT_DIR}/cadence_runner" cmake-out/backends/cadence/cadence_runner
7474
chmod +x cmake-out/backends/cadence/cadence_runner
7575
7676
export PYTHONPATH="${PYTHONPATH:-}:$(pwd)/backends/cadence/utils/FACTO"
77-
python -m pytest examples/cadence/operators/ -v -n auto
77+
python -m pytest examples/cadence/operators/ -v -n auto --reruns 2 --reruns-delay 1

0 commit comments

Comments
 (0)