Skip to content

feat(pooling): static predictive-density pooling - #164

Open
thomaspinder wants to merge 6 commits into
mainfrom
feat/151-predictive-pooling
Open

feat(pooling): static predictive-density pooling#164
thomaspinder wants to merge 6 commits into
mainfrom
feat/151-predictive-pooling

Conversation

@thomaspinder

Copy link
Copy Markdown
Owner

Summary

New public leaf module src/impulso/pooling.py: pool_forecasts(fits, holdout, *, method, density, n_draws, seed) scores each fitted model's predictive density on an explicit held-out VARData window and returns a frozen PredictivePool with normalised weights, the full per-horizon log-score matrix, a pooled predictive sample (as a regular ForecastResult), and optimiser diagnostics. PredictivePool.combine() applies the frozen weights to new forecasts from full-sample refits.

  • Two weight methods: "stacking" (default — convex SLSQP solve of the mixture log score on the simplex, log-score-restart on failure, loud RuntimeError if both starts fail) and "log_score" (max-shifted softmax of summed scores, the pseudo-BMA analogue). Both are overflow-safe by construction (per-row max shift) — strictly safer than ArviZ's unshifted exponentiation, and tested as such at ±1e4–1e5 log-score magnitudes.
  • Density evaluation: moment-matched joint Gaussian per horizon fitted to each model's forecast draws (density="diagonal" escape hatch), with a min-eigenvalue degeneracy guard that rejects covariances only the jitter makes decomposable. The exact mixture score is deferred behind the same keyword (ADR-0006).
  • Hard compatibility validation: ≥2 models, identical ordered variable names, one shared forecast origin, holdout postdating it, date-exact step alignment, and a shared estimation frequency across all fits (mixed monthly/quarterly candidates are rejected in any order). Every rule has an actionable error naming the offending label.
  • Determinism: single-seed Generator.spawn contract across member forecasts and the mixture draw; bit-identical under a fixed seed.
  • Docs: ADR-0006, docs/reference/pooling.md, a how-to whose fenced examples were executed verbatim and whose tables are real output, CONTEXT.md vocabulary (pool / held-out window / stacking vs log-score / forecast origin), plot_pool_weights.

Closes #151

Review

Planned by a dedicated planning agent (numerics empirically validated against installed arviz/scipy before coding); implementation adversarially reviewed by an independent reviewer that re-derived the stacking first-order conditions and stress-tested the degeneracy guard over 20k adversarial covariance trials. Three P2 findings (mixed-frequency validation hole, doc/number contradiction, reserved-label collision) fixed in a follow-up round; reviewer's enhancement suggestions filed as separate issues.

Conflicts note: trivially overlaps PR #160 in __init__.py, docs/reference/index.md, and CONTEXT.md (additive entries in each).

Tests

103 tests in tests/test_pooling.py plus public-API and plotting additions — all fast, no MCMC. Solvers pinned to closed-form optima (stacking interior optimum 7/9 exact from the KKT conditions; log-score 0.8/0.2), ArviZ az.compare parity for both methods via a constant-log-likelihood shim, a version-proof direct scipy re-solve, mixture membership statistics, RNG spawn-contract reconstruction, and one test per validation rule. pooling.py and plotting/_pooling.py at 100% line+branch coverage.

ruff / ty / fast suite: all green (636 passed, 29 deselected). Docs smoke build clean.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Egjd7ToFeb9TQqFnfRQZxV

@codecov-commenter

codecov-commenter commented Jul 29, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 95.5%. Comparing base (866ca8f) to head (1deb946).

Additional details and impacted files
@@           Coverage Diff           @@
##            main    #164     +/-   ##
=======================================
+ Coverage   95.0%   95.5%   +0.4%     
=======================================
  Files         45      47      +2     
  Lines       3099    3430    +331     
  Branches     380     436     +56     
=======================================
+ Hits        2945    3276    +331     
  Misses       111     111             
  Partials      43      43             

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

thomaspinder and others added 5 commits July 29, 2026 23:01
ADR-0006 fixes the design: score each candidate's predictive density on an
explicit held-out window, weight by stacking or log score, keep estimation
(pool_forecasts) and application (PredictivePool.combine) separate. Records
the exact Rao-Blackwellised mixture score, diagonal-only densities,
joint-path scoring, rolling origins, and dynamic weights as considered and
deferred, with the seams they slot into.

CONTEXT.md gains the load-bearing terms — predictive pool, held-out window,
forecast origin, log-score weights, stacking weights — each with the
synonyms to avoid.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Egjd7ToFeb9TQqFnfRQZxV
pool_forecasts(fits, holdout) forecasts every candidate from the shared
estimation end, scores each horizon's predictive density at the held-out
realisations, and turns the resulting (H, M) log-score matrix into weights.
Two rules: stacking (default — maximise the log score of the *pooled*
predictive; convex on the simplex, so the SLSQP optimum is global) and
log_score (softmax of per-model totals, i.e. pseudo-BMA). The returned
PredictivePool is frozen and carries the weights, the full score matrix, a
pooled predictive sample over the held-out window, and combine(), which
applies the frozen weights to new forecasts from full-sample refits.

Densities are Gaussians moment-matched to the forecast draws, joint across
variables by default with a density="diagonal" escape hatch. Degenerate
covariances are rejected up front — the smallest eigenvalue must clear the
jitter — rather than scored as garbage.

Exponentiation is done after a per-row maximum shift (stacking) or a global
shift (log score). The row shift changes the objective by a w-independent
constant, so the optimum is untouched, but it keeps the pool finite at log
scores where an unshifted implementation underflows to zero or overflows to
inf; both cases are regression-tested.

The pool takes FittedVARs, not ForecastResults: only a FittedVAR carries the
estimation metadata that makes "same origin, genuinely held out" checkable.
It never refits, and it works across estimators — a conjugate (1, 200)
posterior pools with a NUTS (2, 100) one.

98 tests, all fast, no MCMC. The solvers are pinned against closed-form
optima (stacking's interior 7/9, log score's 0.8/0.2), the score matrix
against scipy's multivariate_normal and norm, and the weights against
az.compare for both methods — plus a direct scipy re-solve of the published
score matrix, so ArviZ API drift cannot silently weaken the suite. 100%
line and branch coverage on both new modules.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Egjd7ToFeb9TQqFnfRQZxV
A reference page mirroring scenario.md, and a how-to that walks the split,
the candidate fits, the weights, and the refit-then-combine step. The
stacking-versus-log-score contrast uses the numbers the test fixture
actually produces: on two complementary models scoring -153.8 and -131.1
alone, stacking splits 0.50/0.50 and pools to -40.8, while log-score weights
collapse to 0.00/1.00 and pool to -119.0 — no better than the model they
picked.

Also documents what the scores are not: a Gaussian approximation to a
heavier-tailed mixture, summed per horizon from one fixed origin, with
static weights and no way to detect a peeked-at holdout.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Egjd7ToFeb9TQqFnfRQZxV
…abel (#151)

Three review findings.

Mixed-frequency fits pooled silently, and whether they did depended on dict
order. _check_alignment only ever saw the first fit's estimation index, so a
quarterly and a monthly model sharing the end date 2014-10-01 passed when the
quarterly one came first and failed when it came second — the monthly model
being scored on dates three times further out than it forecasts. The check now
collects every fit's inferred frequency, rejects the pool outright when they
disagree (naming each model and its frequency), and warns when some model's
index has no inferable frequency while the others do. Parametrised over both
mapping orders so the order that used to slip through is pinned.

A model labelled "pooled" was silently corrupted: to_dataframe() writes the
combined predictive into a "pooled" column, overwriting that model's scores.
The label is now reserved, rejected both early in pool_forecasts and in the
PredictivePool validator so direct construction cannot smuggle it in.

The how-to's stacking-versus-log-score section described a two-model mirrored
setup its snippet never built — it reused the three-model macro fits from the
section above — and then claimed log-score weights "pool no better than that
model does" while its own table showed a 12-point gain. Replaced with a
self-contained snippet that runs verbatim and reproduces its table exactly
(weights 0.000/1.000 and 0.004/0.996, pooled -51.2, members -56.7/-51.2). The
complementary-models case is now described by mechanism, without invented
numbers, and the summary table above it no longer presents fabricated scores
as measured output.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Egjd7ToFeb9TQqFnfRQZxV
#237 made VARData reject exog columns that are constant within the
sample, so the all-ones "x" column the `_fitted`/`_holdout` helpers and
the name-mismatch test used no longer constructs. Swap it for a
deterministic ramp; B_exog is pinned to zero in these fixtures, so the
exog values never enter the predictive mean — only their presence,
count and names matter to the assertions.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Egjd7ToFeb9TQqFnfRQZxV
@thomaspinder
thomaspinder force-pushed the feat/151-predictive-pooling branch from 06a0c67 to 65d9329 Compare July 29, 2026 21:03
…base

The rebase conflict resolution appended TestPoolingPublicAPI directly after
the preceding class body with no separating blank lines, so ruff format
(and therefore the quality job) failed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat: add static predictive-density pooling

2 participants