Skip to content

Simplify quantile root-finding to Roots defaults#2070

Merged
devmotion merged 11 commits into
masterfrom
dmw/roots
Jun 25, 2026
Merged

Simplify quantile root-finding to Roots defaults#2070
devmotion merged 11 commits into
masterfrom
dmw/roots

Conversation

@devmotion

@devmotion devmotion commented Jun 19, 2026

Copy link
Copy Markdown
Member

First, thanks to @Cedriq1astaken for the work in #2066 — diagnosing the quantile_newton infinite loop, introducing Roots.find_zero, and assembling the regression cases. This PR builds directly on that branch (its commits are preserved here) and pushes the idea a little further.

I started looking into this more closely after hitting the same class of bug from a different direction in #2069 (adding cdf/ccdf/quantile to SkewNormal). There, the hand-rolled Newton iteration hangs whenever the root is near zero: SkewNormal's support straddles 0, and the relative-only stopping rule abs(x - x0) > max(abs(x), abs(x0)) * tol shrinks below the cdf/pdf evaluation noise as the root approaches 0, so the iterates oscillate in their last bits and never converge. That's the same underlying issue as the extreme-tail oscillation in #2061, just triggered by the root's location rather than by p. Seeing two distinct symptoms of one root cause is what motivated converging on a single, options-free formulation.

1) What changes compared to master

This adds a dependency on Roots v3 (compat Roots = "3").

master solves quantiles with hand-rolled Newton/bisection loops and hand-tuned tolerances. This replaces them with Roots.find_zero, relying on Roots' default tolerances plus a bracketing fallback — no tol/max_iters/floor parameters:

  • quantile_bisectfind_zero(x -> cdf(d, x) - p, (lx, rx), ITP())
  • quantile_newton / cquantile_newton / invlogcdf_newton / invlogccdf_newtonfind_zero((F, f), x, Newton(), ITP()). Passing a bracketing method after Newton() makes Roots switch to ITP the moment the iteration brackets the root (a sign change of F).
  • Chernoff quantiles now route through quantile_newton (with the same Normal-based warm start) instead of three bespoke newton(...) branches. Those branches called a newton function that was never defined anywhere in the package, so quantile(Chernoff(), τ) threw UndefVarError for any non-tabulated τ (e.g. truncated(Chernoff(); lower=0.1)) — present since the original Chernoff PR (Chernoff distribution #840). This removes the dead code and fixes Error with Truncated Chernoff quantiles; missing function (newton) #1999.

The reason defaults suffice is that Roots' tolerances supply three distinct ingredients, each covering one of the failure modes:

Failure mode Issue(s) Ingredient that fixes it
quantile_newton stalls/oscillates for a root near zero #2069 (SkewNormal) Newton's absolute xatol = eps (replaces #2069's hand-added eps^(3/4) floor)
quantile_newton oscillates in the extreme tail (p ≈ 1) #1571, #1898, #2061 the ITP bracketing fallback (switch on sign change)
quantile_bisect can't terminate for a bracket far from zero #1611, #1807 ITP's relative xrtol = eps (the old absolute cbrt(eps)^2 fell below the float spacing)

Roots' full defaults for reference:

Newton() ITP() / bracketing
xatol eps(T) eps(T)^3
xrtol eps(T) eps(T)
atol / rtol 4·eps(S) 0 / 0
maxiters / maxevals 40 60
strict false true

I confirmed the mapping empirically: for SkewNormal's near-zero roots, Newton() converges without the ITP fallback (the absolute xatol does the work); the fallback is what rescues the extreme-tail cases. Because Roots' defaults already cover the near-zero regime, #2069 no longer needs its own quantile_newton change.

Bracketing algorithm: standardized on ITP — NonlinearSolve's documented recommendation, explicitly suited to smooth, well-behaved functions. CDF roots are simple (d(cdf)/dx = pdf > 0), so it's a natural fit. (For reference: Roots' robust default is Bisection, its performance picks are A42/AlefeldPotraShi; the newer ModAB was recently added to both packages but isn't yet a documented recommendation, and its strength — roots of higher multiplicity — doesn't apply here.)

Fixes: #1571, #1611, #1807, #1898, #1999, #2061 (and removes the need for #2069's separate quantile_newton fix).

2) What changes compared to #2066

#2066 also moves to Roots.find_zero, but keeps hand-tuned machinery on top. Relative to that branch, this PR:

🤖 Generated with Claude Code

Cedriq1astaken and others added 9 commits June 9, 2026 16:06
- Add magnitude-aware tolerance floor to prevent chasing floating-point granularity
- Reduce max_iter to 50 (proper stopping rule should rarely need it)
- Add clear warnings for non-convergence with fallback to bisection
- Detect 2-cycles explicitly as sign of ill-conditioning
Delegates core root-finding logic in quantile_newton functions to Roots.find_zero using Newton(). Upgrades quantile_bisect to use the ITP() bracketing solver as suggested. Retains precision-based xrtol tolerance scaling.
…equal

- the `chernoff` distribution now uses `quantile_newton` for none precomputed quantiles
- Added test cases for the following issues (#1571, #1611, #1807, #1869, #1999, #2061)
@codecov-commenter

codecov-commenter commented Jun 19, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 86.74%. Comparing base (03ddda7) to head (2561f2e).
⚠️ Report is 2 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #2070      +/-   ##
==========================================
+ Coverage   86.63%   86.74%   +0.11%     
==========================================
  Files         149      149              
  Lines        8921     8882      -39     
==========================================
- Hits         7729     7705      -24     
+ Misses       1192     1177      -15     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Building on #2066, simplify the quantile solvers to rely on Roots'
default tolerances instead of hand-tuned options:

- `quantile_bisect` delegates to `find_zero(g, (lx, rx), ITP())`. ITP's
  relative `xrtol = eps` converges even for brackets far from zero, where
  the previous absolute `cbrt(eps)^2` tolerance fell below the
  floating-point spacing and looped forever (#1611, #1807).
- The four `_newton` functions delegate to
  `find_zero((F, f), x, Newton(), ITP())`. Roots switches to the ITP
  bracketing method as soon as Newton brackets the root, which resolves
  the oscillation/stalling seen for extreme quantiles (#1571, #1898,
  #2061).

The `tol`/`max_iters` parameters and the explicit tolerance floors are
dropped: Roots' Newton defaults combine an absolute (`xatol = eps`) and a
relative (`xrtol = eps`) x-tolerance, so they converge both near `p ≈ 1`
and for roots near zero without bespoke floors.

`quantile_bisect` keeps only the exact `rx == lx` degenerate guard
(#1501); the approximate "collapsed bracket" handling is dropped, as a
non-bracketing interval is a separate concern from the solver choice.

Tests use `isapprox`'s type-appropriate defaults rather than fixed
tolerances, and are split one regression per testset. Adds coverage for
#1898 and a `Float32` round-trip.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
devmotion added a commit that referenced this pull request Jun 19, 2026
Implement the previously-missing (c)cdf for SkewNormal, now that StatsFuns
2.2 provides Owen's T function `owens_t`. The closed forms are

    cdf(x)  = Φ(z) - 2·T(z, α)
    ccdf(x) = Φ̄(z) + 2·T(z, α),   z = (x - ξ)/ω

with `logcdf`/`logccdf` using the generic `log(cdf)`/`log(ccdf)` fallbacks.
`pdf`/`logpdf` are refactored for consistency (destructure once, reuse `z`).

`quantile`/`cquantile` are added via Newton iteration from the mode, using
`quantile_newton`/`cquantile_newton`. This branch is stacked on #2070, whose
Roots-based solver handles the near-zero-root convergence that SkewNormal's
support (straddling 0) requires, so no change to `src/quantilealgs.jl` is
needed here. `invlogcdf`/`invlogccdf` are left to the generic
`quantile(d, exp(lp))` fallback: the log-space `invlogcdf_newton` overshoots
into the underflowed tail from the approximate mode `m_0` and returns NaN
there, whereas the generic fallback through `quantile_newton` stays robust.

Require StatsFuns 2.2 and bump the package version.

Testing: add a `sn`-based reference class (test/ref/continuous/skewnormal.R)
to the R reference framework, pin `sn` and its deps in renv.lock, and add the
generated entries to continuous_test.ref.json. The dedicated testset is
reduced to construction, support boundaries and the α=0 reduction to Normal.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@devmotion devmotion requested a review from andreasnoack June 22, 2026 08:26
@devmotion devmotion merged commit 6b5aa0d into master Jun 25, 2026
14 checks passed
@devmotion devmotion deleted the dmw/roots branch June 25, 2026 13:37
@devmotion

Copy link
Copy Markdown
Member Author

Thank you @Cedriq1astaken!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Error with Truncated Chernoff quantiles; missing function (newton)

4 participants