Skip to content

Add systematic distributional validation for distribution samplers #1282

Description

@synesenom

Problem

sample() is the library's primary interface — it is what ran is named for — and it has the weakest correctness verification of any part of the numeric API.

Today every distribution's sampler is checked by a ksTest (continuous) or chiTest (discrete) call in test/dist-runner.js, at whatever sample size a unit test can afford. That is a real check and it catches gross breakage, but it is weak in a specific, well-understood way: the Kolmogorov-Smirnov statistic is driven by the maximum deviation near the centre of the distribution and has notoriously poor power in the tails. A sampler that is correct in the bulk and wrong in the extremes passes comfortably. That is exactly the failure profile the library's sampling machinery is prone to:

  • Rejection samplers (src/algorithms/rejection.js) — an envelope that is slightly too tight in the tail silently truncates or under-weights extreme values.
  • Numerical inverse-transform samplers — any distribution without a closed-form _q(p) draws by inverting the CDF numerically, so it inherits the tail-inversion inaccuracy that Add quantile accuracy sweep to the differential-testing harness #1269 exists to measure. Bad conditioning near p -> 0 and p -> 1 maps directly to a misshapen tail.
  • Compound and mixture samplers — Normal-Inverse Gaussian via an inverse-Gaussian mixture, negative binomial via gamma-Poisson, and similar constructions. A mis-mapped parameter between the mixing and mixed components produces a distribution that is close but not correct.
  • PreComputed table distributions — where the table is truncated determines the largest value that can ever be drawn.

None of these show up in a centre-weighted goodness-of-fit test at unit-test sample sizes.

Scope

Add an out-of-band distributional validation harness for distribution samplers, alongside (not replacing) the existing in-suite ksTest/chiTest smoke checks.

  1. Large sample sizes. Orders of magnitude beyond what npm test can afford. This is the single biggest lever on test power and the reason it must run out-of-band.
  2. Tail-weighted statistics. Use Anderson-Darling rather than KS as the primary continuous statistic — the AD weighting function is precisely designed to give the tails the power KS lacks. andersonDarling already exists in src/test/, so this is reuse, not new math. Report both so the difference is visible.
  3. Explicit tail checks. Compare empirical quantiles against the distribution's own quantiles at p and 1-p for p down to the smallest the sample size supports. Report the extreme values drawn versus the theoretically expected extremes for that sample size.
  4. Moment matching. Compare sample moments against the analytical moments verified by the moment precision gate. This is an independent axis from goodness-of-fit and catches scale/location errors that a shape-focused test can miss.
  5. Discrete distributions. Chi-squared with deliberate attention to tail bins, which conventional binning rules tend to pool away — the pooling is exactly what hides a truncated tail.

Methodological requirement: multiple testing

Running a hypothesis test at alpha = 0.05 across ~165 distributions produces roughly 8 spurious failures per run by construction. A harness that reports 8 red distributions every run is one nobody reads, and the real failure hides among them.

Handle this explicitly — do not skip it and do not paper over it by loosening alpha until the output looks clean. Either control the false-discovery rate across the family of tests (Benjamini-Hochberg), or run with fixed seeds and a recorded expected outcome so any change in the pass/fail set is itself the signal. State the choice and the reasoning in the harness documentation.

What this does and does not prove

This harness validates the sampler against the library's own CDF. That is a consistency check, not an absolute one: if _generator and _cdf were wrong in the same way, it passes. It is still strong evidence, because the two are almost always independently implemented from different formulas.

The absolute guarantee only arrives in composition with #1265: once a distribution's CDF has been verified against mpmath, a sampler verified against that CDF is verified absolutely. Document this explicitly so nobody reads a green run as more than it is, and so the accuracy table (#1266) can distinguish "sampler consistent with CDF" from "sampler verified, CDF verified".

Out of scope

Calibration requirement

The harness must be demonstrated to detect a deliberately biased sampler — for example one whose rejection envelope is truncated at the 99.9th percentile, or an inverse-transform sampler with p clamped away from 0 and 1. A distributional test suite that has never been observed to reject has not been shown to have any power at all.

Acceptance criteria

  • Harness runs all distributions with a sample() method at large sample sizes, out-of-band from npm test.
  • Anderson-Darling reported alongside KS for continuous distributions; chi-squared with tail-bin attention for discrete.
  • Empirical-vs-theoretical tail quantile comparison and sample-vs-analytical moment comparison included.
  • Multiple-testing policy implemented and documented.
  • Negative control demonstrates detection of a deliberately biased sampler.
  • Reproducible under fixed seeds.
  • Documentation states plainly that this is consistency-with-CDF, not absolute correctness, absent Extend differential-testing harness to distribution pdf/pmf and cdf sweeps #1265 coverage for that distribution.
  • npm run standard and npm test pass.

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions