test(predictive): shape statistics in the posterior-predictive drift fence - #236
Open
thomaspinder wants to merge 4 commits into
Open
test(predictive): shape statistics in the posterior-predictive drift fence#236thomaspinder wants to merge 4 commits into
thomaspinder wants to merge 4 commits into
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## feat/56-predictive-apis #236 +/- ##
=======================================================
Coverage 93.8% 93.8%
=======================================================
Files 41 41
Lines 2534 2534
Branches 300 300
=======================================================
Hits 2378 2378
Misses 112 112
Partials 44 44 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
thomaspinder
force-pushed
the
feat/56-predictive-apis
branch
2 times, most recently
from
July 29, 2026 21:16
1eebac5 to
42f6757
Compare
…uction) (#218) * feat(identification): ZeroSignRestriction — combined zero-and-sign restrictions (#144) Implements the recursive orthogonalisation of Arias, Rubio-Ramirez & Waggoner (2018). Writing P = L Q, an impact zero restriction "variable i does not respond to shock j" is the linear condition e_i' L q_j = 0 on the j-th column of Q. Columns are built one at a time, each drawn uniformly from the unit sphere of the null space of R_k = [ Z_k L ; q_1' ; ... ; q_{k-1}' ] (m = z_k + (k-1) rows) so the zeros hold exactly (to SVD precision) and orthogonality to the earlier columns holds by construction. Only the sign restrictions need accept/reject. Details worth flagging for review: - Shocks are ordered internally by zero count descending (stable, so ties keep user order and the unidentified_* padding stays last); columns are permuted back to shock_names order before returning, and rows are never permuted. The Rubio-Ramirez, Waggoner & Zha (2010) rank condition z_j <= n - j is checked on that sorted padded sequence at identify() entry, before any sampling. - On an early impact-sign failure the WHOLE candidate is abandoned and the recursion restarts from column 1. Redrawing only the offending column would be a distribution bug — q_k's law is conditional on q_1..q_{k-1}. Documented in a code comment at the failure site. - Failed draws become NaN with one summary warning; there is no fallback to L, which would silently violate the zero guarantee. on_failure="raise" is opt-in. This diverges from SignRestriction deliberately. - Diagnostics land on _last_diagnostics under a zero_sign_ prefix, not on _last_acceptance_rate, which the pipeline surfaces under the misleading name sign_restriction_acceptance_rate. - Candidates are unweighted: no volume-element correction for the ARW uniform-conditional prior. Called out in the class docstring; the two test-anchored regimes (no zeros -> Haar; exact identification -> point up to signs) are weight-free. Anchor test: with full triangular zeros every null space is one-dimensional, so P must be lower triangular with P P' = Sigma, i.e. the Cholesky factor up to column signs. Acceptance is 1.0 and |P| reproduces |cholesky(Sigma)| to 0.0 (bit-exact — the SVD returns the canonical basis vectors for that structure). Adding "+" diagonal signs pins P == cholesky(Sigma) exactly. SignRestriction is untouched. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Egjd7ToFeb9TQqFnfRQZxV * docs(identification): zero-and-sign restrictions — reference, explanation, how-to (#144) - reference/identification.md: ZeroSignRestriction in the autosummary. - explanation/identification.md: new section covering the ARW recursive construction (labelled equations for the zero condition, the recursion, and the rank condition), the exact-vs-set spectrum with Cholesky at one end and pure sign restrictions at the other, and a warning admonition stating the three distributional caveats honestly: draws are unweighted (no volume-element correction for the ARW uniform-conditional prior), retries are per-theta rather than joint, and sampling is over O(n) rather than SO(n). - how-to/zero-sign-restrictions.md: climate worked example — an activity shock has zero contemporaneous impact on the temperature anomaly, which is a physical-lag exclusion rather than a sign — plus the admissible-zero counting rule, the diagnostics attrs, and a warning admonition spelling out that failed draws are NaN and never fall back to Cholesky. - references.bib: ariasRubioRamirezWaggoner2018, rubioRamirezWaggonerZha2010. - CONTEXT.md: "Zero-and-sign restrictions (ARW construction)" entry, with "penalty-function zeros" listed under _Avoid_ (that is a different, approximate construction); IdentificationScheme adapter list refreshed. Sphinx build with warnings-as-errors reports no warnings from any of these pages; equations, citations, and both new pages render. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Egjd7ToFeb9TQqFnfRQZxV * refactor(identification): drop unused n_vars parameter from _require_coefficients (#144) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Egjd7ToFeb9TQqFnfRQZxV * style: normalise blank lines at rebase keep-both junctions Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Egjd7ToFeb9TQqFnfRQZxV * fix(docs): restore closing brace lost resolving the references.bib rebase conflict The rebase conflict resolution merged the new bibliography entry into the blanchardQuah1989 entry, dropping its closing brace. sphinxcontrib-bibtex then failed to parse references.bib, breaking build-docs and docs-linkcheck. --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
…ttedVAR (#230) * refactor(spec): extract _build_pymc_model and fitted_values seams Behaviour-neutral extraction ahead of the predictive APIs (#56): * `VAR._build_pymc_model(data) -> (pm.Model, n_lags)` lifts the graph construction out of `VAR.fit` verbatim (the `sv/spec.py` precedent). `fit` now builds, samples, and wraps; `prior_predictive` will build and draw from the same graph. * `_residuals.fitted_values(posterior, data, n_lags)` lifts the conditional-mean computation out of `reduced_form_residuals`, which is now `y_obs - fitted_values(...)`. The same seam feeds `posterior_predictive`. No graph, no posterior, and no numerical output changes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Egjd7ToFeb9TQqFnfRQZxV * feat(predictive): VAR.prior_predictive (#56) `VAR.prior_predictive(data, *, draws=500, random_seed=None)` builds the same graph `fit` builds via `_build_pymc_model` and calls `pymc.sample_prior_predictive` on it, so the simulated prior is by construction the prior that gets sampled — no hand-rolled simulator to drift out of sync (the anti-pattern the issue calls out). Semantics: the simulated `obs` paths are one-step-ahead given the OBSERVED lags, `y_t = c + B x_t^obs (+ B_exog z_t) + L_t eps_t`, because the design matrices are baked into the graph. That is what `az.plot_ppc(..., group="prior")` expects and what puts the prior on the data's own time axis. Returns PyMC's InferenceData as-is: `prior`, `prior_predictive` (dims `(chain, draw, time, var)`, chain=1) and `observed_data`. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Egjd7ToFeb9TQqFnfRQZxV * feat(predictive): FittedVAR.posterior_predictive (#56) `FittedVAR.posterior_predictive(*, simulate_innovations=True, seed=None)` replicates the estimation sample from the posterior: y_rep[t] = intercept + B x_t^obs (+ B_exog z_t) + L_t eps_t Two deliberate readings, both asserted by tests: * One-step-ahead conditioned on the OBSERVED lags — the object `pm.sample_posterior_predictive` returns on the fitted graph and the one `az.plot_ppc` consumes. Not an iterated simulated path; that is `forecast()`. * `L_t` comes from the volatility seam (`cholesky_path`), so the innovations use the model's own Sigma — per-draw and per-t under SV. The issue's "residual covariance of each posterior draw" wording would flatten exactly the heteroscedasticity an SV fit exists to capture. Computed in NumPy, not on a PyMC graph: ConjugateVAR posteriors (which have no graph) get the method for free, `simulate_innovations=False` becomes a plain mean, and there is no backend divergence. The drift risk that buys is fenced by a slow moment-comparison against `pm.sample_posterior_predictive` on a fit with strongly correlated shocks (a near-diagonal Sigma would let a Cholesky-orientation bug through). `self.idata` is never mutated — a fresh InferenceData with `posterior_predictive` and `observed_data` comes back, and the docstring shows the `fitted.idata.extend(ppc)` recipe. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Egjd7ToFeb9TQqFnfRQZxV * docs(predictive): how-to page, ADR-0011 and CONTEXT entries (#56) * `docs/how-to/predictive-checks.md` (unexecuted MyST, zero CI cost): prior check via `az.plot_ppc(..., group="prior")`, posterior check plus a coverage snippet, mean mode for residual diagnostics, the `idata.extend` recipe, the memory note, and a table separating the two checks from `forecast` / `conditional_forecast`. * ADR-0011 records why the posterior predictive is NumPy rather than `pm.sample_posterior_predictive` (ConjugateVAR parity, mean mode, backend independence, the volatility seam) and names the slow test that fences the resulting drift risk. * CONTEXT.md gains a "Predictive check" term, a relationship line and a dialogue entry, all insisting these are estimation-window objects and not forecasts. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Egjd7ToFeb9TQqFnfRQZxV * fix(predictive): keep the new annotations beartype-resolvable (#56) `impulso.enable_runtime_checks()` beartype-wraps the public classes, and beartype resolves return annotations on call. It cannot resolve a DOTTED forward reference whose module is TYPE_CHECKING-only: `-> tuple["pm.Model", int]` on `_build_pymc_model` raised BeartypeCallHintForwardRefException on the first `prior_predictive` call under checks. Nothing in the suite exercised that combination. * `_build_pymc_model` now returns `tuple[Any, int]` — the same reason `FittedVAR.pymc_model` is `Any`, and PyMC stays lazily imported. * `spec.py` imports arviz at module level (it is already loaded by the time `import impulso` returns, so this costs nothing) and annotates `prior_predictive` with the real `az.InferenceData`. * New slow test drives both methods in a throwaway interpreter with checks enabled, and asserts a bad argument is still flagged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Egjd7ToFeb9TQqFnfRQZxV * test(predictive): pin the scale-adaptive exog prior to the shared graph seam (#56) Rebasing onto main brought in #237, which computes the `B_exog` prior standard deviation from the data (`_exog_prior_sigma`) inside the region this branch extracted into `VAR._build_pymc_model`. The extraction is what makes the merge safe: the scaled prior lives in the shared seam, so `prior_predictive` draws from the same graph `fit` samples. Had it stayed in `fit`, a prior-predictive check would have described a unit-scale model that was never estimated — silently. Nothing pinned that, so: * `TestExogPriorWiredIntoModel` gains two graph tests mirroring the existing `dist_params`-based fit-path ones — the `_build_pymc_model` graph's `B_exog` sigma equals `_exog_prior_sigma(...)`, honours `exog_prior_scale`, and is identical to the sigma the fit path builds. * `TestPriorPredictive` gains an end-to-end check that the simulated `B_exog` draws actually spread at that sigma rather than at 1. Also refreshes the two docstrings the merge made stale: `exog_prior_scale` no longer claims to apply to `fit` alone, and `_build_pymc_model` now states that every prior — the exog one included — lives in the seam. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Egjd7ToFeb9TQqFnfRQZxV * fix(spec): keep the sampling tail in fit and rethread error_dist after rebase The rebase onto main duplicated fit's sampling tail into _build_pymc_model and dropped error_dist from fit's model_construct call. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Egjd7ToFeb9TQqFnfRQZxV --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
…unts (#223) Closes #187, closes #188. #187: `counterfactual` and `structural_scenario` are linear solves in the structural shock matrix P. With NaN draws — reachable today via `LongRunRestriction(on_undefined="nan")`, and soon via other schemes — the in-sample engine inverted P and returned an all-NaN counterfactual silently, while the forecast engine died inside LAPACK as "SVD did not converge" from `matrix_rank`. Neither named the cause. A single guard, `_require_finite_shock_matrix`, now runs where P enters each engine (`structural_shock_context` and `_forecast_shock_matrices`), counts the NaN draws, names the `on_undefined="nan"` policy as the likely cause, and points at `on_undefined="raise"` to catch it at identification time. #188: `from_zero_restrictions` rejected genuinely recursive patterns whose `shock_names` were not pre-sorted, though the order is recoverable. The variable ordering already came from the restriction counts; the same count read down the other axis gives the shock order — in a triangular pattern the variable at position i is restricted by the shocks at positions i+1..n-1, so the shock at position k appears in exactly k restriction lists. Ties (non-recursive patterns) sort stably and are still caught by the unchanged exact per-position set check, so the "not recursive" error paths keep their messages. The caller's `shock_names` order is now irrelevant to validity; the names still label the columns. Claude-Session: https://claude.ai/code/session_01Egjd7ToFeb9TQqFnfRQZxV Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
The ADR-0011 fence compared means, SDs and the innovation covariance only, so a Student-t observation likelihood rescaled to matched variance (branch #168) would have passed it while the replicate law silently diverged — confirmed against the real pipeline: an injected t(7) pool clears the existing covariance assertion outright. Standardise the innovations by each draw's own L (an invertible per-draw map applied identically to both samples), pool over (chain, draw, time), and compare excess kurtosis plus a two-sample KS statistic. Both bounds are Monte Carlo-error calibrated: at N = 79,600 the pools are iid standard normal under the null, so the kurtosis difference has SE = sqrt(48 / N) and the bound is 6 sigma + 0.02 slack (0.167); the KS p-value is uniform, so 1e-6 is a 1e-6 flake budget. Measured over three seeds x two variables: null |dkurt| <= 0.065 and KS p >= 0.15; the injected t(7) alternative gives |dkurt| 1.86-2.06 and KS p ~ 1e-22..1e-32 — an 11x margin at the bound. ADR-0011's cost paragraph now records that the fence covers moments 1, 2 and 4 plus the full standardised marginal, and what it still cannot catch (anything matching every compared statistic — notably serial dependence across t, to which a pooled marginal is blind). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Egjd7ToFeb9TQqFnfRQZxV
thomaspinder
force-pushed
the
test/232-drift-fence-tail
branch
from
July 29, 2026 21:58
ffd092d to
db05473
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Follow-up on the predictive-APIs branch (#230), stacked on
feat/56-predictive-apis:The NumPy-vs-PyMC drift fence compared first and second moments only, so a future matched-variance Student-t likelihood (#168 will merge) would pass while the replicate law silently diverged. Premise confirmed empirically: injected matched-variance t(7) replicates pass the old fence's covariance/SD assertions in all three seed configurations (sd ratios 1.001–1.007).
The fence now also compares, on innovations standardised per draw by the model's own
L(a common invertible map — cannot manufacture or hide a difference, but strips the across-draw scale mixture that would swamp the signal) and pooled to N≈80k:6·√(48/N) + 0.02 = 0.167— note√(48/N), the SE of a difference of two pools, not the single-pool√(24/N): at the latter the 200-replication null simulation's worst case reached 62% of the bound, too close for a never-flake claim. As committed: worst null 39% of bound, weakest injected t(7) signal 11× over it.ADR-0011's cost paragraph updated: the fence now covers first, second, and fourth moments plus the KS statistic, and states what it still cannot catch.
Closes #232
Gates
Fast predictive tests 17 passed; slow fence suite 4 passed (43s); ruff/ty clean.
🤖 Generated with Claude Code
https://claude.ai/code/session_01Egjd7ToFeb9TQqFnfRQZxV