Fix flaky test_pdf_argpercentile_gaussian with an absolute tolerance#187
Merged
Conversation
test_pdf_argpercentile_gaussian compares a percentile read off the discretized grid `x` against the analytic Gaussian percentile. The grid value has an absolute error of up to one grid spacing (10 * std / (num - 1) ~ 1e-4). For the random NormalUncertainScalarArray mean, when a distribution sample makes the expected percentile fall near zero, that error exceeds rtol * |expected| and the relative-only np.allclose intermittently fails. Add atol=1e-2 (well above the ~1e-4 grid error, far below the order-1 scale) so near-zero samples are compared on an absolute basis. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #187 +/- ##
=======================================
Coverage 97.24% 97.24%
=======================================
Files 105 105
Lines 14210 14210
=======================================
Hits 13819 13819
Misses 391 391
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
test_pdf_argpercentile_gaussianintermittently fails in CI (e.g. on the runs for #186), for the randommean = na.NormalUncertainScalarArray(0, width=1)parametrization.Root cause
The test reads the 25th percentile off a discretized grid
x = linspace(mean - 5*std, mean + 5*std, num=100001)and compares it to the analytic Gaussian percentilemean + std*sqrt(2)*erfinv(2p - 1).The grid value carries an absolute error of up to one grid spacing,
10 * std / (num - 1) ≈ 1e-4(confirmed empirically: worst observed error1.0e-4). The comparison usednp.allclose(..., rtol=1e-1)with no absolute tolerance. For the random uncertainmean, whenever a distribution sample lands such that the expected percentile falls near zero (|expected| ≲ 1e-3), the bounded ~1e-4grid error exceedsrtol * |expected|and the assertion fails. This is rare per run but shows up across many CI runs.Fix
Add
atol=1e-2to thenp.allclosecall. This is ~100× the worst-case grid error yet far below the order-1 scale of the values, so it absorbs the near-zero discretization error without weakening the test for normal-magnitude values (wherertol=1e-1still governs).Demonstration of the failure mode and the fix:
Tests
pytest named_arrays/tests/test_pdf.py::test_pdf_argpercentile_gaussianpasses (16 cases).🤖 Generated with Claude Code