Skip to content

Fix flaky ChainRules AD tests by keeping finite differences off domain boundaries#215

Open
devmotion wants to merge 1 commit into
masterfrom
dmw/finitedifferences
Open

Fix flaky ChainRules AD tests by keeping finite differences off domain boundaries#215
devmotion wants to merge 1 commit into
masterfrom
dmw/finitedifferences

Conversation

@devmotion

@devmotion devmotion commented Jun 16, 2026

Copy link
Copy Markdown
Member

Fixes #176

Problem

The AD tests in test/chainrules.jl intermittently fail in CI with DomainErrors from loggamma/digamma/log, e.g. this run. The failures have appeared on master and several PRs and are not caused by any recent change.

The analytic rules in ext/StatsFunsChainRulesCoreExt.jl are correct. The flakiness comes from the numerical reference: test_frule/test_rrule perturb the inputs with finite differences, and with unseeded exp(randn())/logistic(randn()) draws an argument occasionally lands close to a domain boundary, so the stencil crosses it and the primal throws (e.g. logbeta(α, β) with β < 0).

Fix

Keep every finite-difference stencil inside the domain by anchoring the test points well away from their boundaries and relying on ChainRulesTestUtils' defaults — no custom fdm or tangents needed.

test_frule/test_rrule evaluate the reference at input ± reach, where reach = max_range · |tangent|. With CRTU's defaults the reach is bounded:

  • max_range = 1e-2 (CRTU's _fdm = central_fdm(5, 1; max_range = 1e-2));
  • Float64 tangents are drawn from -9:0.01:9, so |tangent| ≤ 9 (they are not unbounded randn — that applies only to the generic/BigFloat methods, not to the Float64 arguments here).

Hence reach ≤ 1e-2 · 9 = 0.09. Drawing every differentiated argument ≥ 0.1 from its boundary (pos() = 0.1 + randexp(), unit01() = 0.1 + 0.8rand()) makes a boundary crossing impossible by construction (0.09 < 0.1), for any draw or seed. Each log-pdf domain is an independent box constraint (α>0, β>0, x∈(0,1), ν>0, λ>0, …), so per-coordinate anchoring suffices under simultaneous perturbation. The wider margin also keeps the points better-conditioned than the singular regions, so the finite-difference reference stays accurate at the default rtol. A fixed seed is added for reproducibility, and integer arguments resolve to NoTangent() automatically.

This margin depends on CRTU's defaults; a comment in the test records that if max_range or the tangent range grows, the 0.1 anchor must grow with it. FiniteDifferences is not added as a dependency.

Verification

  • 0 failures over 2500 seeds × 7 distributions (frule + rrule) at the default tolerance (240000/240000 checks pass).
  • Full Pkg.test() passes locally (chainrules | 98 98); the QA testset (Aqua/ExplicitImports/JET) stays green after dropping the FiniteDifferences test dependency.

@codecov-commenter

codecov-commenter commented Jun 16, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 72.39%. Comparing base (f8d7f5b) to head (709893b).
⚠️ Report is 2 commits behind head on master.

Additional details and impacted files
@@           Coverage Diff           @@
##           master     #215   +/-   ##
=======================================
  Coverage   72.39%   72.39%           
=======================================
  Files          22       22           
  Lines        1007     1007           
=======================================
  Hits          729      729           
  Misses        278      278           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@devmotion devmotion force-pushed the dmw/finitedifferences branch from 52f67f3 to 338d918 Compare June 16, 2026 22:58
…n boundaries

The AD tests in `test/chainrules.jl` intermittently failed in CI with
`DomainError`s from `loggamma`/`digamma`/`log`. The analytic rules in the
ChainRulesCore extension are correct; the failures came from the numerical
reference: `test_frule`/`test_rrule` perturb the inputs with finite differences,
and with unseeded `exp(randn())`/`logistic(randn())` draws an argument
occasionally landed close to a domain boundary, so the stencil crossed it.

The fix keeps every stencil in-domain by construction, relying on
ChainRulesTestUtils' defaults rather than a custom `fdm` or tangents.
`test_frule`/`test_rrule` evaluate the reference at `input ± reach` with
`reach = max_range * |tangent|`, and with CRTU's defaults both factors are
bounded:

* `max_range = 1e-2` (CRTU's `_fdm = central_fdm(5, 1; max_range = 1e-2)`);
* `Float64` tangents are drawn from `-9:0.01:9`, so `|tangent| <= 9`.

Hence `reach <= 0.09`. Drawing every differentiated argument >= 0.1 from its
boundary (`0.1 + randexp()`, `0.1 + 0.8rand()`) then makes a boundary crossing
impossible (`0.09 < 0.1`), for any draw or seed; each log-pdf domain is an
independent box constraint, so per-coordinate anchoring suffices. The wide
margin also keeps the points well-conditioned, so the reference stays accurate
at the default tolerance. A fixed seed is added for reproducibility, and integer
arguments resolve to `NoTangent()` automatically.

The margin relies on CRTU's defaults; a comment records that the `0.1` anchor
must grow if `max_range` or the tangent range grows. No new dependency is added.

Verified: 0 failures over 2500 seeds x 7 distributions (frule + rrule);
full Pkg.test() passes (chainrules | 98 98, QA | 19 19).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@devmotion devmotion force-pushed the dmw/finitedifferences branch from 338d918 to 709893b Compare June 16, 2026 23:33
@devmotion devmotion marked this pull request as ready for review June 17, 2026 08:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

random betalogpdf failure

2 participants