English | 中文

Foresight

Multivariate Time Series Forecasting

LSTM · Transformer · XGBoost · Leakage Prevention

---

Headline

XGBoost wins: MAE = 0.256 and RMSE = 0.380 in log1p space — MASE 0.70, i.e. ~30% better than repeating last week. The useful negative result is that LSTM and Transformer do not beat the gradient-boosted baseline on this structured forecasting task after leakage fixes; all models do beat the seasonal-naive benchmark (MASE < 1).

The pipeline systematically benchmarks a seasonal-naive (t-7) baseline, XGBoost, Prophet, LSTM, and Transformer on tabular retail time-series data with heavy feature engineering (40+ lag / rolling / holiday / oil-price features). Once hand-crafted features cover the long-range dependencies, gradient boosting exploits those structured features more efficiently — deep learning does not necessarily beat a well-built baseline, which is itself a valuable engineering finding.

Model	MAE ↓	RMSE ↓	MAPE ↓	sMAPE	MASE ↓
Seasonal Naive (t-7)	0.361	0.569	17.61%	23.31%	0.991
XGBoost	0.256	0.380	11.98%	39.46%	0.702
LSTM	0.269	0.399	12.71%	40.66%	0.738
Transformer	0.282	0.410	12.76%	40.61%	0.774

The rerun fixes three leakage paths: oil lags are computed on the date-unique series, missing oil values use causal filling, and deep-learning validation targets are restricted to the holdout window. The contract is covered by tests/test_pipeline.py::TestLeakagePrevention; metrics come from reports/model_results.json.

Overview

End-to-end pipeline for multivariate time series forecasting. Benchmarks classical methods (XGBoost, Prophet) against modern neural architectures (LSTM, Transformer) on the Kaggle Store Sales dataset.

Key Highlights

• Baseline Models: Seasonal-naive (t-7), XGBoost Regressor + Facebook Prophet for benchmarking
• Deep Learning: LSTM with embedding layers + Transformer with multi-head self-attention and positional encoding
• Feature Engineering: Lag features (1/7/14/28/364d), rolling statistics, cyclical seasonal encodings, promo aggregates
• Evaluation: MAE, RMSE, MAPE, sMAPE + MASE/RMSSE scaled errors vs. the seasonal-naive benchmark
• Delivery: Streamlit dashboard comparing forecast vs. actual

Leakage Prevention

Leakage systematically inflates time-series metrics. This pipeline fixes three leakage paths:

• Oil lag computed on the daily series: an early version applied shift(1) directly on the long table, which crossed (store, family) group boundaries. The lag is now computed on a date-unique frame and merged back, so oil_lag_1 is always the previous day's oil price.
• Causal filling of missing oil prices: interpolate(method="linear") is bidirectional (it could pull future oil prices into the validation window); it has been replaced with ffill().bfill() only, so a given day's oil price never comes from later observations.
• DL validation-target filtering: TimeSeriesDataset(min_target_date=val_start) only emits samples whose prediction target is ≥ val_start. The 28 days of training tail ahead of the validation window serve as window input only — never as prediction targets mixed into validation loss/MAE.

Architecture

Raw CSVs (train, stores, oil, holidays, transactions)
    |
    v
Preprocess ──> Date features, log-transform, external merges
    |
    v
Feature Eng ──> Lags, rolling mean/std, seasonal encoding, promo features
    |
    +---> XGBoost / Prophet (baselines)
    +---> LSTM + Embeddings (deep learning)
    +---> Transformer + Positional Encoding (deep learning)
    |
    v
Evaluate ──> MAE, RMSE, MAPE, sMAPE, residual analysis
    |
    v
Dashboard ──> Forecast comparison, error distribution, residual analysis

Tech Stack

Layer	Tools	Notes
ETL	pandas, numpy	Time-based train/val split (no random shuffle)
Feature Eng	pandas rolling, sklearn preprocessing	Lag/rolling features with shift(1) to prevent leakage
Baselines	XGBoost, Prophet	Additive regression + tree-based benchmark
Deep Learning	PyTorch, PyTorch Lightning	LSTM + Transformer with categorical embeddings
Evaluation	sklearn metrics	MAE, RMSE, MAPE, sMAPE
Delivery	Streamlit	Side-by-side forecast comparison
Quality	pytest, ruff, GitHub Actions	CI validates pipeline end-to-end

Quick Start

git clone https://github.com/MeaFew/foresight.git
cd foresight

# Create and activate a Python 3.11 virtual environment
python -m venv .venv
# Linux / macOS: source .venv/bin/activate
# Windows PowerShell: .venv\Scripts\Activate.ps1

# Install dependencies, the package, and development tools
make setup
# Windows without GNU Make: python -m pip install -r requirements.txt
#                           python -m pip install -e ".[dev]"

# Download real dataset (GitHub Releases, ~21MB)
bash download_data.sh

# Run full pipeline
python run_all.py

# Or step by step
make preprocess
make features
make train-baseline     # XGBoost + Prophet
make train-lstm         # LSTM model
make train-transformer  # Transformer model
make evaluate

# Launch dashboard
make dashboard

# Quality gates
make verify

Project Structure

.
├── src/foresight/
│   ├── generate_mock_data.py     # Synthetic retail sales data
│   ├── preprocess.py              # Date parsing, log-transform, external merges
│   ├── config.py                  # Project-wide paths and modeling constants
│   ├── feature_engineering.py     # Lags, rolling stats, seasonal encoding
│   ├── train_baseline.py          # XGBoost + Prophet
│   ├── train_lstm.py              # LSTM with PyTorch Lightning
│   ├── train_transformer.py       # Transformer with positional encoding
│   ├── train_common.py            # Shared DL training/eval plumbing (Lightning)
│   ├── evaluate.py                # Model comparison & residual analysis
│   ├── backtest.py                # Rolling-origin (walk-forward) backtest
│   ├── predict.py                 # Model loading and inference
│   ├── metrics.py                 # MAE/RMSE/MAPE/sMAPE, TimeSeriesDataset
│   ├── metrics_utils.py           # torch-free mape/smape (lightweight testability)
│   └── audit_consistency.py       # Cross-reference README claims vs outputs
├── dashboard/
│   └── app.py                     # Streamlit forecast comparison
├── tests/
│   ├── test_core.py               # Core-module unit tests
│   ├── test_pipeline.py           # Unit + integration tests
│   ├── test_metrics.py            # mape/smape numerical contract + TimeSeriesDataset consistency
│   ├── test_backtest.py           # Rolling-origin split geometry, leakage contract, MASE hand-check
│   └── test_audit_consistency.py  # README/artifact consistency audit tests
├── Makefile                       # Workflow orchestration
└── requirements.txt

Model Comparison

Evaluation protocol

Kaggle scores original-scale sales with RMSLE, while this repository currently reports MAE, RMSE, MAPE, and sMAPE in log1p(sales) space, plus MASE/RMSSE against a seasonal-naive (t-7) benchmark. These protocols are not directly comparable, so this README shows only local results backed by reports/model_results.json on the shared chronological holdout; unsupported RMSLE and leaderboard estimates have been removed.

Seasonal-naive & MASE: the naive baseline predicts ŷ(t) = y(t−7) per (store, family) series — the same "condition on actual recent history" protocol as the lag-feature models. MASE = holdout MAE / in-sample seasonal-naive MAE (pooled over all training series, scale = 0.365 in log1p space); MASE < 1 means better than repeating last week. Details: reports/seasonal_naive_baseline.md.

Results

Model	MAE	RMSE	MAPE	sMAPE	MASE	Dataset
Seasonal Naive (t-7)	0.361	0.569	17.61%	23.31%	0.991	Full processed training set, final 16-day holdout
XGBoost	0.256	0.380	11.98%	39.46%	0.702	Full processed training set, final 16-day holdout
Prophet (aggregated)	—	—	—	—	—	(requires pystan compilation toolchain; verified in Docker/Linux CI)
LSTM	0.269	0.399	12.71%	40.66%	0.738*	Same chronological holdout
Transformer	0.282	0.410	12.76%	40.61%	0.774*	Same chronological holdout

* LSTM/Transformer MASE is derived from their aggregate MAE and the pooled seasonal-naive scale (per-row predictions from the GPU runs are not persisted).

LSTM/Transformer metrics are actual full-dataset results produced by PyTorch Lightning training on the same validation window as XGBoost. Run make train-lstm and make train-transformer to regenerate them; metrics are written to reports/model_results.json under "lstm_results" / "transformer_results" keys. The XGBoost MAE in this table is cross-checked against reports/model_results.json by python -m foresight.audit_consistency (make audit).

All MAE/RMSE/MAPE values are computed in log1p(sales) space. Every model — XGBoost included — is evaluated on the same chronological 16-day holdout at the tail of the training series (no shuffle, no cross-validation).

📊 Detailed experiment notes (training speed, improvement ideas)

Training speed (2.3M samples / RTX 4060)

An early single-threaded DataLoader (num_workers=0) left the GPU waiting between steps and was the main reason DL training was slow. With multi-process data loading enabled, per-epoch time dropped significantly:

Optimization	Notes
num_workers=4 + persistent_workers	4 processes run __getitem__ in parallel; workers are spawned once and reused across epochs
prefetch_factor=4	Prefetch queue keeps the GPU fed
pin_memory + non_blocking	Host→device copies overlap with compute
cudnn.benchmark=True	Auto-selects the fastest kernel for the fixed 28-step windows
bf16-mixed precision	Ada Tensor Core acceleration
batch_size=1024	Cuts kernel-launch overhead (bs=128: 72s/epoch → bs=1024: 38s/epoch)

Tune via CLI: python -m foresight.train_lstm --num_workers 8 --batch_size 2048. If a long run is interrupted, --resume continues from the latest checkpoint in reports/checkpoints/.

Why DL did not win, and improvement ideas

LSTM comes close to XGBoost (MAE gap 0.012); Transformer lags slightly. Improvement ideas: longer training, a larger d_model, or dedicated time-series architectures such as N-BEATS / TFT.

Rolling-Origin Backtest

A single chronological holdout is one draw from one window — the score can be lucky or unlucky depending on which 16 days happen to be at the tail. Random k-fold CV is not a valid alternative for time series: shuffling puts future rows into the training set and leaks autocorrelated neighbours across the fold boundary, biasing scores optimistically. The standard fix is rolling-origin (walk-forward) evaluation: an expanding training window re-scored on several non-overlapping future horizons. src/foresight/backtest.py retrains XGBoost from scratch on 5 origins (16-day horizon each, same as the single holdout) and re-scores the seasonal-naive baseline per fold, so the MASE denominator (in-sample naive MAE) is recomputed on each fold's own training window:

date ────────────────────────────────────────────────────────►
fold 1: [════════ train ════════][val]   val 2017-05-28 ~ 06-12
fold 2: [════════ train ════════════][val]   val 2017-06-13 ~ 06-28
fold 3: [════════ train ═════════════════][val]   val 2017-06-29 ~ 07-14
fold 4: [════════ train ═════════════════════][val]   val 2017-07-15 ~ 07-30
fold 5: [════════ train ═════════════════════════][val]   val 2017-07-31 ~ 08-15
                                     ▲ train never touches its val window or anything after

Per-fold metrics (log1p space; fold 5 is exactly the single-holdout window and reproduces its numbers — MAE 0.2562 / MASE 0.7023 vs. 0.256 / 0.702 — as a built-in consistency check):

Fold	Val window	XGB MAE	XGB RMSE	XGB MAPE	XGB MASE	Naive MASE	MASE scale
1	2017-05-28 ~ 06-12	0.255	0.379	12.32%	0.696	0.955	0.366
2	2017-06-13 ~ 06-28	0.254	0.378	12.18%	0.694	0.927	0.366
3	2017-06-29 ~ 07-14	0.246	0.369	11.90%	0.674	0.932	0.366
4	2017-07-15 ~ 07-30	0.250	0.374	11.92%	0.685	0.882	0.365
5	2017-07-31 ~ 08-15	0.256	0.381	11.98%	0.702	0.991	0.365

Mean ± std across the 5 folds vs. the single-holdout numbers from the table above (full per-fold values in reports/backtest_results.json; reproduce with make backtest):

Metric	XGB rolling mean ± std	XGB single holdout	Single within ±1 std?
MAE	0.252 ± 0.004	0.256	✓ (0.96σ)
RMSE	0.376 ± 0.005	0.380	✓ (0.91σ)
MAPE	12.06% ± 0.18%	11.98%	✓ (−0.43σ)
sMAPE	39.30% ± 0.55%	39.46%	✓ (0.29σ)
MASE	0.690 ± 0.011	0.702	✗ — marginally outside (1.08σ)
Naive MASE	0.937 ± 0.040	0.991	✗ — marginally outside (1.34σ)

Conclusion: the model is stable, the split was (very) slightly unlucky. XGBoost fold-to-fold spread is tiny (MASE std ≈ 1.6% of the mean), so the ranking over the naive baseline is robust; the single-holdout MASE of 0.702 sits ~1.1σ above the rolling mean of 0.690, i.e. the final 16-day window was marginally harder than a typical window — the honest "model quality" number is MASE ≈ 0.69 ± 0.01, not a single point estimate.

Data

The project uses the Kaggle Store Sales - Time Series Forecasting dataset:

• 54 stores across Ecuador
• 33 product families
• Daily sales from 2013 to 2017
• External variables: oil prices, holidays, promotions

For local testing without Kaggle credentials, run python -m foresight.generate_mock_data to create a statistically similar synthetic dataset.

Related Projects

Project	Repo	Description
E-commerce User Analytics	MeaFew/shoplytics	29M real user behavior records, 10 analytical modules
Marketing Attribution & MMM	MeaFew/attributor	MMM + multi-touch attribution + budget optimization
Credit Risk Scoring	MeaFew/riskscore	WOE/IV + XGBoost/LightGBM + SHAP interpretability
GNN Fraud Detection	MeaFew/graphguard	GraphSAGE + GNNExplainer on graph-structured fraud data

License

MIT