Skip to content

Fix flaky test_pdf_argpercentile_gaussian with an absolute tolerance#187

Merged
roytsmart merged 2 commits into
mainfrom
fix-flaky-pdf-argpercentile
Jun 12, 2026
Merged

Fix flaky test_pdf_argpercentile_gaussian with an absolute tolerance#187
roytsmart merged 2 commits into
mainfrom
fix-flaky-pdf-argpercentile

Conversation

@roytsmart

Copy link
Copy Markdown
Contributor

Summary

test_pdf_argpercentile_gaussian intermittently fails in CI (e.g. on the runs for #186), for the random mean = na.NormalUncertainScalarArray(0, width=1) parametrization.

Root cause

The test reads the 25th percentile off a discretized grid x = linspace(mean - 5*std, mean + 5*std, num=100001) and compares it to the analytic Gaussian percentile mean + std*sqrt(2)*erfinv(2p - 1).

The grid value carries an absolute error of up to one grid spacing, 10 * std / (num - 1) ≈ 1e-4 (confirmed empirically: worst observed error 1.0e-4). The comparison used np.allclose(..., rtol=1e-1) with no absolute tolerance. For the random uncertain mean, whenever a distribution sample lands such that the expected percentile falls near zero (|expected| ≲ 1e-3), the bounded ~1e-4 grid error exceeds rtol * |expected| and the assertion fails. This is rare per run but shows up across many CI runs.

Fix

Add atol=1e-2 to the np.allclose call. This is ~100× the worst-case grid error yet far below the order-1 scale of the values, so it absorbs the near-zero discretization error without weakening the test for normal-magnitude values (where rtol=1e-1 still governs).

Demonstration of the failure mode and the fix:

expected = 5e-4; got = expected + 1e-4   # one grid cell of error near zero
np.allclose(got, expected, rtol=1e-1)              # False  (old)
np.allclose(got, expected, rtol=1e-1, atol=1e-2)   # True   (new)

Tests

pytest named_arrays/tests/test_pdf.py::test_pdf_argpercentile_gaussian passes (16 cases).

🤖 Generated with Claude Code

test_pdf_argpercentile_gaussian compares a percentile read off the
discretized grid `x` against the analytic Gaussian percentile. The
grid value has an absolute error of up to one grid spacing
(10 * std / (num - 1) ~ 1e-4). For the random NormalUncertainScalarArray
mean, when a distribution sample makes the expected percentile fall near
zero, that error exceeds rtol * |expected| and the relative-only
np.allclose intermittently fails.

Add atol=1e-2 (well above the ~1e-4 grid error, far below the order-1
scale) so near-zero samples are compared on an absolute basis.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@codecov

codecov Bot commented Jun 12, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.24%. Comparing base (82d7639) to head (6f5faf5).

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #187   +/-   ##
=======================================
  Coverage   97.24%   97.24%           
=======================================
  Files         105      105           
  Lines       14210    14210           
=======================================
  Hits        13819    13819           
  Misses        391      391           
Flag Coverage Δ
unittests 97.24% <100.00%> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@roytsmart roytsmart merged commit d60267c into main Jun 12, 2026
20 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant