Skip to content

feat(deterministic): calendar-anchored deterministic climate regressors - #191

Open
thomaspinder wants to merge 6 commits into
mainfrom
feat/135-deterministic-regressors
Open

feat(deterministic): calendar-anchored deterministic climate regressors#191
thomaspinder wants to merge 6 commits into
mainfrom
feat/135-deterministic-regressors

Conversation

@thomaspinder

Copy link
Copy Markdown
Owner

Summary

New public module src/impulso/deterministic.py: composable deterministic regressors with deterministic column names, built for the climate workflow the issue describes — and built around the one property that makes them safe to forecast with:

design.build(index[:T+h]).iloc[T:] equals design.extend(index[:T], h) — trends continue their count, harmonics stay in phase, dummies stay calendar-correct. Tested as a single property, parametrised ~90 ways across term types, frequencies, and horizons.

  • Terms: Trend(degree≤3, scale), Fourier(period, order) (Nyquist-validated), SeasonalDummies(month|quarter|dayofweek, drop_first), BreakDummy(date, level|pulse). All calendar-anchored via period ordinals — one (origin, alias) resolved from the estimation index makes three of the four terms pure functions of the timestamp, so gaps in the index are handled correctly (a trend jumps across a gap rather than compressing it).
  • DeterministicDesign (frozen): build(index) for estimation (with a single rank guard against the always-present intercept — catching drop-first violations, degenerate Nyquist sines, dummies-vs-harmonics overlap, and dead break dummies with targeted messages), extend(index, steps) for the horizon, and exog_future(fitted, steps) which reorders columns by name against fitted.data.exog_names — closing the silent-wrong-forecast path (forecast consumes exog_future positionally).
  • Refuse-don't-guess frequency handling: explicit freqindex.freqpd.infer_freq verified by regenerate-and-compare (it demonstrably lies on short irregular indices) → error. Cycle periods are always explicit. Multiplied offsets (15D, 2h) count in sampling periods, matching the documented contract; off-anchor final timestamps extend without skipping a period.
  • In-scope hardening: FittedVAR.forecast now validates exog_future's shape (matching conditional_forecast's existing check) instead of failing with an opaque einsum error.
  • Docs: an onboarding how-to ("Deterministic regressors for climate VARs") covering the transform-vs-model routes, the no-NaN order of operations, the column-name contract, and the ConjugateVAR boundary (rejection message quoted verbatim, with both workarounds) — its end-to-end snippet is byte-identical to the slow integration test and says so. Reference page, protocols entry, CONTEXT.md term.

Closes #135

Review

Planned with live pandas 3.0 probes (the datetime64[us] unit trap, Day-not-Tick, infer_freq false positives — all designed around). Independently reviewed: verdict approve after two P2 fixes (multiplied-frequency elapsed counting; off-anchor extension skipping a period), both probe-verified by the reviewer and fixed with exact-arithmetic solutions rather than guards.

Tests

341 tests in tests/test_deterministic.py (incl. one slow MCMC round-trip asserting the design's column names survive into posterior["B_exog"].coords), plus forecast-validation and public-API tests. Coverage on deterministic.py: 99%.

ruff / ty / fast suite: all green (875 passed, 30 deselected). Docs build (warnings-as-errors) clean. (The only local slow-test failure is the known broken-marimo-husk nutpie import in the venv — environmental, affects every slow test in the repo.)

🤖 Generated with Claude Code

https://claude.ai/code/session_01Egjd7ToFeb9TQqFnfRQZxV

@codecov-commenter

codecov-commenter commented Jul 29, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 98.03150% with 5 lines in your changes missing coverage. Please review.
✅ Project coverage is 95.3%. Comparing base (866ca8f) to head (3e747c7).

Files with missing lines Patch % Lines
src/impulso/fitted.py 73.3% 2 Missing and 2 partials ⚠️
src/impulso/deterministic.py 99.5% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@           Coverage Diff           @@
##            main    #191     +/-   ##
=======================================
+ Coverage   95.0%   95.3%   +0.3%     
=======================================
  Files         45      46      +1     
  Lines       3099    3349    +250     
  Branches     380     427     +47     
=======================================
+ Hits        2945    3194    +249     
  Misses       111     111             
- Partials      43      44      +1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

thomaspinder and others added 6 commits July 29, 2026 23:03
Adds `impulso.deterministic`: `Trend`, `Fourier`, `SeasonalDummies` and
`BreakDummy`, composed by `DeterministicDesign` into an exogenous design
matrix for `VARData`. Climate series carry an annual cycle, a warming
trend and dated instrument breaks; modelled as exogenous regressors they
get posterior coefficients instead of being differenced away.

Terms anchor on integer *period ordinals* measured from the first
timestamp of the estimation index, never on row position. That gives the
continuation property —

    design.build(index[: T + h]).iloc[T:] == design.extend(index[:T], h)

— which is what makes `design.exog_future(fitted, h)` the same block the
coefficients were fitted against, reordered by name to match
`exog_names` so a permuted design cannot silently produce a wrong
forecast.

`build` rank-checks the design against the intercept every estimator
fits and names the likely cause (undropped dummy level, a Nyquist-degenerate
top harmonic, dummies and harmonics of the same cycle, a break outside the
sample). `extend` does not: short horizons are legitimately deficient.

Frequency resolution is explicit-first, then the index's own `freq`, then
inference *verified* by regenerating the index — pandas returns confident
false positives on short irregular indices. Business-day sampling is
refused with a named alternative rather than a deprecation warning.

Adds the `DeterministicTerm` protocol as the extension point, and exports
the five concrete names plus the protocol from `impulso`.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Egjd7ToFeb9TQqFnfRQZxV
…#135)

`conditional_forecast` already coerced and shape-checked `exog_future`;
`forecast` indexed it positionally and let a mismatch surface as an
opaque einsum error several loop iterations in. Now that the block is
routinely generated (`DeterministicDesign.exog_future`), the asymmetry is
a real hazard: extract `_resolve_exog_future` and apply the same contract
— posterior must carry `B_exog`, and the array must be exactly
`(steps, n_exog)`.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Egjd7ToFeb9TQqFnfRQZxV
…135)

Adds `docs/how-to/deterministic-regressors.md`: why the annual cycle and
warming trend belong somewhere other than the lag structure, the two
routes (subtract a climatology vs model it as exog, and what each costs),
the column-name contract, the no-NaN invariant with the order of
operations that preserves it, and the collinearity rules.

The end-to-end snippet is byte-identical to the body of
`tests/test_deterministic.py::test_deterministic_design_end_to_end`, so
the documented recipe is the one under test; the page says so.

Also adds the autosummary reference page, `DeterministicTerm` to the
protocols page, both toctrees, a pointer from the data-preparation guide,
and the `Deterministic design` entry in CONTEXT.md — including the
_Avoid_ that keeps "exogenous data" for real covariates (forcings, ENSO,
CO2) rather than generated calendar terms.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Egjd7ToFeb9TQqFnfRQZxV
…sion (#135)

Two probe-verified defects in the time helpers.

`_elapsed` counted period-ordinal ticks, not sampling periods. pandas
stores `15D` ordinals in days and `2h` ordinals in hours, so elapsed time
advanced by 15 (or 2) per observation on any multiplied offset. That made
`Fourier`'s documented contract — "cycle length in sampling periods" —
false: a user on 2-hourly data writing `Fourier(period=12)` for a daily
cycle silently fitted a 12-hour one. Divide by the multiplier, read off
`pd.Period(origin, alias).freq.n`. The origin is subtracted first, so an
on-grid index still yields exact integers, and unmultiplied frequencies
are bit-for-bit unchanged (`n == 1`).

`_extend_index` skipped a period when the sample ended off the offset's
anchor — the irregular-index-with-explicit-freq path the docstring
advertises. `pd.date_range` rolls an off-anchor start forward, so the
walk's first entry is *already* the first future period; dropping it
unconditionally lost one. An index ending 2000-03-15 under `MS` forecast
from May, silently losing April. Select the entries strictly after the
last observation instead, which is correct on and off anchor.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Egjd7ToFeb9TQqFnfRQZxV
…135)

Main gained a scale-adaptive prior on B_exog (#237): the prior standard
deviation is now `exog_prior_scale * sigma_i / s_j`, not a fixed
`Normal(0, 1)` in coefficient space. Both the `Trend` docstring and the
how-to justified `scale` by the old fixed prior, which is no longer true.

`Trend.scale` is still worth setting — it fixes the coefficient's units
("change per decade") and keeps the design column O(1) for the sampler —
so the guidance stands; only its reason changes.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Egjd7ToFeb9TQqFnfRQZxV
@thomaspinder
thomaspinder force-pushed the feat/135-deterministic-regressors branch from 54ee46a to 3e747c7 Compare July 29, 2026 21:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat: add deterministic climate regressors with an onboarding recipe

2 participants