feat(pooling): static predictive-density pooling - #164
Open
thomaspinder wants to merge 6 commits into
Open
Conversation
This was referenced Jul 29, 2026
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #164 +/- ##
=======================================
+ Coverage 95.0% 95.5% +0.4%
=======================================
Files 45 47 +2
Lines 3099 3430 +331
Branches 380 436 +56
=======================================
+ Hits 2945 3276 +331
Misses 111 111
Partials 43 43 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
thomaspinder
force-pushed
the
feat/151-predictive-pooling
branch
from
July 29, 2026 13:45
c35fc2d to
ba8129b
Compare
ADR-0006 fixes the design: score each candidate's predictive density on an explicit held-out window, weight by stacking or log score, keep estimation (pool_forecasts) and application (PredictivePool.combine) separate. Records the exact Rao-Blackwellised mixture score, diagonal-only densities, joint-path scoring, rolling origins, and dynamic weights as considered and deferred, with the seams they slot into. CONTEXT.md gains the load-bearing terms — predictive pool, held-out window, forecast origin, log-score weights, stacking weights — each with the synonyms to avoid. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Egjd7ToFeb9TQqFnfRQZxV
pool_forecasts(fits, holdout) forecasts every candidate from the shared estimation end, scores each horizon's predictive density at the held-out realisations, and turns the resulting (H, M) log-score matrix into weights. Two rules: stacking (default — maximise the log score of the *pooled* predictive; convex on the simplex, so the SLSQP optimum is global) and log_score (softmax of per-model totals, i.e. pseudo-BMA). The returned PredictivePool is frozen and carries the weights, the full score matrix, a pooled predictive sample over the held-out window, and combine(), which applies the frozen weights to new forecasts from full-sample refits. Densities are Gaussians moment-matched to the forecast draws, joint across variables by default with a density="diagonal" escape hatch. Degenerate covariances are rejected up front — the smallest eigenvalue must clear the jitter — rather than scored as garbage. Exponentiation is done after a per-row maximum shift (stacking) or a global shift (log score). The row shift changes the objective by a w-independent constant, so the optimum is untouched, but it keeps the pool finite at log scores where an unshifted implementation underflows to zero or overflows to inf; both cases are regression-tested. The pool takes FittedVARs, not ForecastResults: only a FittedVAR carries the estimation metadata that makes "same origin, genuinely held out" checkable. It never refits, and it works across estimators — a conjugate (1, 200) posterior pools with a NUTS (2, 100) one. 98 tests, all fast, no MCMC. The solvers are pinned against closed-form optima (stacking's interior 7/9, log score's 0.8/0.2), the score matrix against scipy's multivariate_normal and norm, and the weights against az.compare for both methods — plus a direct scipy re-solve of the published score matrix, so ArviZ API drift cannot silently weaken the suite. 100% line and branch coverage on both new modules. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Egjd7ToFeb9TQqFnfRQZxV
A reference page mirroring scenario.md, and a how-to that walks the split, the candidate fits, the weights, and the refit-then-combine step. The stacking-versus-log-score contrast uses the numbers the test fixture actually produces: on two complementary models scoring -153.8 and -131.1 alone, stacking splits 0.50/0.50 and pools to -40.8, while log-score weights collapse to 0.00/1.00 and pool to -119.0 — no better than the model they picked. Also documents what the scores are not: a Gaussian approximation to a heavier-tailed mixture, summed per horizon from one fixed origin, with static weights and no way to detect a peeked-at holdout. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Egjd7ToFeb9TQqFnfRQZxV
…abel (#151) Three review findings. Mixed-frequency fits pooled silently, and whether they did depended on dict order. _check_alignment only ever saw the first fit's estimation index, so a quarterly and a monthly model sharing the end date 2014-10-01 passed when the quarterly one came first and failed when it came second — the monthly model being scored on dates three times further out than it forecasts. The check now collects every fit's inferred frequency, rejects the pool outright when they disagree (naming each model and its frequency), and warns when some model's index has no inferable frequency while the others do. Parametrised over both mapping orders so the order that used to slip through is pinned. A model labelled "pooled" was silently corrupted: to_dataframe() writes the combined predictive into a "pooled" column, overwriting that model's scores. The label is now reserved, rejected both early in pool_forecasts and in the PredictivePool validator so direct construction cannot smuggle it in. The how-to's stacking-versus-log-score section described a two-model mirrored setup its snippet never built — it reused the three-model macro fits from the section above — and then claimed log-score weights "pool no better than that model does" while its own table showed a 12-point gain. Replaced with a self-contained snippet that runs verbatim and reproduces its table exactly (weights 0.000/1.000 and 0.004/0.996, pooled -51.2, members -56.7/-51.2). The complementary-models case is now described by mechanism, without invented numbers, and the summary table above it no longer presents fabricated scores as measured output. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Egjd7ToFeb9TQqFnfRQZxV
#237 made VARData reject exog columns that are constant within the sample, so the all-ones "x" column the `_fitted`/`_holdout` helpers and the name-mismatch test used no longer constructs. Swap it for a deterministic ramp; B_exog is pinned to zero in these fixtures, so the exog values never enter the predictive mean — only their presence, count and names matter to the assertions. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Egjd7ToFeb9TQqFnfRQZxV
thomaspinder
force-pushed
the
feat/151-predictive-pooling
branch
from
July 29, 2026 21:03
06a0c67 to
65d9329
Compare
…base The rebase conflict resolution appended TestPoolingPublicAPI directly after the preceding class body with no separating blank lines, so ruff format (and therefore the quality job) failed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
New public leaf module
src/impulso/pooling.py:pool_forecasts(fits, holdout, *, method, density, n_draws, seed)scores each fitted model's predictive density on an explicit held-outVARDatawindow and returns a frozenPredictivePoolwith normalised weights, the full per-horizon log-score matrix, a pooled predictive sample (as a regularForecastResult), and optimiser diagnostics.PredictivePool.combine()applies the frozen weights to new forecasts from full-sample refits."stacking"(default — convex SLSQP solve of the mixture log score on the simplex, log-score-restart on failure, loudRuntimeErrorif both starts fail) and"log_score"(max-shifted softmax of summed scores, the pseudo-BMA analogue). Both are overflow-safe by construction (per-row max shift) — strictly safer than ArviZ's unshifted exponentiation, and tested as such at ±1e4–1e5 log-score magnitudes.density="diagonal"escape hatch), with a min-eigenvalue degeneracy guard that rejects covariances only the jitter makes decomposable. The exact mixture score is deferred behind the same keyword (ADR-0006).Generator.spawncontract across member forecasts and the mixture draw; bit-identical under a fixed seed.docs/reference/pooling.md, a how-to whose fenced examples were executed verbatim and whose tables are real output, CONTEXT.md vocabulary (pool / held-out window / stacking vs log-score / forecast origin),plot_pool_weights.Closes #151
Review
Planned by a dedicated planning agent (numerics empirically validated against installed arviz/scipy before coding); implementation adversarially reviewed by an independent reviewer that re-derived the stacking first-order conditions and stress-tested the degeneracy guard over 20k adversarial covariance trials. Three P2 findings (mixed-frequency validation hole, doc/number contradiction, reserved-label collision) fixed in a follow-up round; reviewer's enhancement suggestions filed as separate issues.
Conflicts note: trivially overlaps PR #160 in
__init__.py,docs/reference/index.md, andCONTEXT.md(additive entries in each).Tests
103 tests in
tests/test_pooling.pyplus public-API and plotting additions — all fast, no MCMC. Solvers pinned to closed-form optima (stacking interior optimum 7/9 exact from the KKT conditions; log-score 0.8/0.2), ArviZaz.compareparity for both methods via a constant-log-likelihood shim, a version-proof direct scipy re-solve, mixture membership statistics, RNG spawn-contract reconstruction, and one test per validation rule.pooling.pyandplotting/_pooling.pyat 100% line+branch coverage.ruff/ty/ fast suite: all green (636 passed, 29 deselected). Docs smoke build clean.🤖 Generated with Claude Code
https://claude.ai/code/session_01Egjd7ToFeb9TQqFnfRQZxV