NHL Total Goals Forecasting (Probabilistic)

A production-minded pipeline for probabilistic forecasting of NHL total goals. Instead of only predicting a point estimate, it can produce a full distribution over goal totals (e.g., P(over 6.5)), evaluate with proper scoring rules, and ship an API + dashboard for nightly inference.

Current benchmark release

• Release: benchmark-v1 using protocol nhl-total-goals-v1
• Untouched holdout: 20252026 regular season (1281 games after the common-feature filter)
• Champion: team_strength under blocked_holm_equivalence_prefer_simpler
• Holdout MAE: 1.8475; CRPS: 1.2855; Brier (>6.5): 0.2503
• Indistinguishable set: team_strength, double_poisson, xgb_tuned, xgb_current

The release manifest and technical report are tracked under reports/releases/benchmark-v1/. Decision/ROI diagnostics are opt-in and are not used for champion selection.

Features

22+ engineered features including:

• Rolling goals for/against (20-game window)
• Win percentage and streaks
• Rest days and back-to-back indicators
• Goalie save percentage and GAA
• Head-to-head historical matchup stats (new)
• Venue-specific scoring trends (new)

Probabilistic Forecasting

Given a point forecast mu = E[total_goals], the project can emit a discrete distribution over totals:

• Poisson
• Negative Binomial (NB2; overdispersed vs Poisson)
• Poisson mixture (extra dispersion / heavy tails)

From the distribution you get:

• P(total_goals > 6.5), P(total_goals > 7.5), etc.
• Predictive intervals (distribution quantiles)
• Conformal intervals (distribution-free uncertainty around the point forecast)

Quick Start

# Create a virtual environment (recommended)
python -m venv .venv
source .venv/bin/activate  # Windows: .venv\Scripts\activate

# Install dependencies
pip install -r requirements.txt

# Run the notebook
jupyter notebook notebooks/eda.ipynb

The notebook will:

1. Download 4 seasons of NHL data (2021-2025) with caching
2. Fetch starting goalie statistics for all games
3. Train an optimized XGBoost model
4. Show feature importance and predictions

Run tests:

pytest -q

Development

Tooling config is centralized in pyproject.toml and shared by CI and local runs. Install the dev dependencies and use the Makefile shortcuts:

make install     # runtime + dev deps (ruff, mypy, stubs)
make lint        # ruff lint (F, E, W, I)
make typecheck   # mypy over the typed surface
make test        # pytest
make cov         # pytest with coverage report
make check       # lint + typecheck + test (the CI gate)

Linting and formatting use Ruff and both are enforced in CI.

Type checking runs mypy over a deliberately small typed surface (listed under [tool.mypy]) that is fully clean today. The intent is to grow that list module by module rather than flip the whole pandas/xgboost codebase to strict at once — incremental typing that stays green.

Run the locked release benchmark:

python -m src.benchmark --tune-trials 40

This tunes on the locked historical seasons, scores the later holdout exactly once, promotes a release-grade artifact, and renders every public claim from one manifest. src.portfolio remains only as a compatibility alias and cannot change the locked cohort.

Prediction CLI

Make predictions for upcoming games:

# Predict games for the next 7 days with the promoted registry artifact
python -m src.predict --days 7

# Save predictions to CSV
python -m src.predict --output predictions.csv

# Specify historical data seasons
python -m src.predict --seasons 20242025 20252026

Probabilistic forecasts (over/under probabilities + intervals):

python -m src.predict --probabilistic --dist-model nb2 --thresholds 5.5 6.5 7.5

Honest Evaluation (Time-Series CV)

Run expanding-window CV with proper scoring rules and calibration diagnostics:

python -m src.evaluate --seasons 20222023 20232024 20242025 --point-model xgb --dist-model nb2

Outputs (saved under reports/):

• Fold metrics + aggregates (.json)
• Reliability curve for P(total_goals > 6.5) (.png)
• Randomized PIT histogram for distribution calibration (.png)
• Champion summary report (champion_model_report.md)

REST API

Run the prediction API server:

# Start the server
uvicorn src.api:app --reload

# Visit http://localhost:8000/docs for interactive documentation

Endpoints:

• GET /predict?days_ahead=7 - Get predictions for upcoming games
• GET /predict/probabilistic?days_ahead=7 - Distribution forecast + P(over/under)
• GET /predict/live?days_ahead=1 - Pregame + live-adjusted probabilities
• GET /dashboard - Tonight’s slate (simple HTML table)
• GET /dashboard/live - Auto-refreshing live slate (20s cadence)
• GET /model/info - Get model metadata and performance metrics
• GET /monitoring/summary - Deduplicated realized metrics and reference-based drift
• GET /metrics - In-process request metrics (counts, error rate, latency p50/p95/p99, counters)
• GET /health - Health check

By default, inference loads the promoted release-grade artifact from models/registry.json, loads the previous and active NHL seasons, and refreshes the active-season cache every six hours. NHL_MODEL_PATH is a migration-only override and is still required to pass release-grade validation. Use NHL_REGISTRY_PATH and NHL_HISTORICAL_SEASONS for normal configuration.

Observability: every request is timed and counted by an in-process metrics registry (src/metrics.py); responses carry X-Request-ID and X-Response-Time-ms headers, and unhandled errors return a safe JSON envelope ({"error", "request_id"}) instead of leaking a stack trace. Scrape GET /metrics for latency percentiles, per-route counts, error rate, and the predictions_served business counter.

Live Dashboard

The live dashboard uses NHL game state (score, period, clock) plus a residual-goals NB2 updater:

• remaining_frac = remaining_minutes / 60
• pace_mult = 1 + 0.04 * abs(goal_diff)
• mu_remaining = max(0.05, mu_pregame * remaining_frac * pace_mult)
• Remaining goals are modeled with NB2 and shifted by current goals.

Explainability + Stability

Feature stability across seasons (and optional SHAP summary plots if shap is installed):

python -m src.explain --seasons 20222023 20232024 20242025 --outdir reports/explain

Project Structure

├── src/
│   ├── config.py       # Centralized configuration
│   ├── data.py         # NHL API data fetching with caching
│   ├── features.py     # Feature engineering (rolling stats, H2H, venue)
│   ├── goalies.py      # Goalie data fetching from boxscores
│   ├── model.py        # XGBoost training, CV, Optuna optimization
│   ├── predict.py      # CLI for predictions
│   ├── api.py          # FastAPI REST API
│   ├── probabilistic.py # PMFs, log score, CRPS, calibration helpers
│   ├── evaluation.py   # Time-series CV with proper scoring rules
│   ├── evaluate.py     # Evaluation CLI
│   ├── explainability.py # SHAP (optional) + stability across seasons
│   ├── explain.py      # Explainability CLI
│   ├── conformal.py    # Split-conformal intervals
│   ├── team_strength.py # Shrinkage baseline on team IDs
│   ├── artifacts.py    # Model persistence with metadata
│   ├── registry.py     # Model versioning and management
│   ├── validation.py   # Input validation utilities
│   └── logging_config.py # Structured logging
├── notebooks/
│   └── eda.ipynb       # Full pipeline demonstration
├── data/               # Cached data (not in git)
└── models/             # Trained models (not in git)

Resume Bullets (Template)

• Built an NHL total-goals probabilistic forecaster producing calibrated over/under probabilities (e.g., P(total_goals > 6.5)), evaluated with expanding-window time-series CV and proper scoring rules (MAE 1.876, CRPS 1.298, Brier 0.247 at 6.5), and shipped a FastAPI service with a “Tonight’s slate” dashboard for nightly inference.

Hyperparameter Optimization

Use Optuna for Bayesian hyperparameter optimization:

from src import build_dataset, add_features, optimize_hyperparameters

df = build_dataset(['20232024', '20242025'])
df = add_features(df)

# Run 100 trials of optimization
best_params = optimize_hyperparameters(df, n_trials=100)
print(best_params)

Model Versioning

Track and manage model versions:

from src import get_registry, ModelArtifact

# Get the registry
registry = get_registry()

# Register a new model
artifact = ModelArtifact.from_training_result(result, seasons=['20232024'])
version = registry.register(artifact, name="xgboost", promote_to_production=True)

# Load production model
prod_model = registry.get_production_model()

# List all models
for model in registry.list_models():
    print(f"{model['version']}: MAE={model['mae']:.4f}")

Evaluation principles

1. Historical time-series CV is tuning-only; the later locked season is the release authority.
2. Championing is weighted and probabilistic - MAE, CRPS, dist-NLL, and Brier are normalized to the team-strength baseline.
3. Team-strength baseline is competitive - useful as a robust sanity check for overfitting.
4. Uncertainty changes the choice - ISO-week block bootstrap, Holm correction, and a simpler-model preference prevent tiny noisy gaps from being marketed as gains.

The exact winning configuration is stored in the tracked release manifest; README parameters are never maintained by hand.

Configuration

All parameters are centralized in src/config.py:

from src import config

# Feature settings
config.features.rolling_window  # 20 (games for rolling stats)
config.features.goalie_window   # 10 (games for goalie stats)
config.features.min_games       # 3 (minimum games before features valid)

# Model settings
config.model.test_size          # 0.2
config.model.cv_folds           # 5
config.model.xgb_params         # Optimized XGBoost parameters

Dependencies

Core:

• pandas, numpy, scikit-learn, xgboost
• requests, tqdm (data fetching)
• matplotlib (visualization)

Optional:

• optuna (hyperparameter optimization)
• fastapi, uvicorn, pydantic (REST API)

Data Sources

Game data is fetched from the NHL Web API (api-web.nhle.com). First run caches data locally for fast subsequent runs.

License

Educational purposes. Game data from NHL's public API.