Time Series Analysis: A Multi-Domain Comparison of Statistical & Machine Learning Models

Jesse Arzate & Megan Lavender

---

Overview

This project is a comprehensive, cross-domain comparison of time series forecasting methodologies — spanning classical statistics, ensemble machine learning, and deep learning. We applied 15+ model variants across four real-world domains to answer a central question:

When does a complex model actually outperform a simpler one — and by how much?

Each domain was strategically selected to push our models to their limits across four distinct temporal challenges: rigid seasonality, high-frequency volatility, external intervention shocks, and irregular human performance. For the high-density datasets—Bike and Household Power—we enforced a rigorous validation framework using expanding-window cross-validation (EWCV) and paired statistical significance testing. For the Airport and Tennis domains, we utilized dedicated test holdouts to allow for a more qualitative analysis of specific time series behaviors.

---

Domains & Datasets

Domain	Granularity	Observations	Key Challenge	Source
🚲 Bike Sharing	Hourly	~17,000	Multi-seasonal demand with weather covariates	UCI ML Repository
⚡ Household Power	15-minute	~70,000	High-frequency volatility & heteroscedasticity	UCI ML Repository
✈️ Airport Passengers	Monthly	~50	COVID-19 structural break & exogenous regressors	Custom-compiled AI & search trend data
🎾 Tennis Performance	Weekly	~52/player	Irregular, sparse human performance dynamics	Jeff Sackmann ATP Dataset

---

Models Implemented

The full model library spans three paradigms, tailored to each domain's structure:

Statistical Models

• ARIMA / SARIMA / SARIMAX — Box-Jenkins family with seasonal differencing, exogenous regressors, and AIC-based grid search
• ARIMAX-GARCH — Hybrid combining mean (ARIMAX) and variance (GARCH) equations to explicitly model heteroscedasticity
• VAR — Vector Autoregression for multivariate interaction modeling (Tennis)

Machine Learning Models

• Random Forest — Bagged ensemble with lag-engineered features and cyclical time encodings
• XGBoost — Gradient-boosted trees with quantile regression for prediction intervals
• Prophet — Meta's additive regression model with automatic seasonality decomposition

Deep Learning Models

• LSTM — Long Short-Term Memory network with sequence-to-one architecture for non-linear temporal dependencies
• Transformer-GARCH — Multi-head self-attention encoder with positional encoding, paired with a GARCH(1,1) residual volatility layer for dynamic confidence intervals
• NNAR — Neural Network Autoregression (Tennis)

---

Key Results

🚲 Bike Sharing — Hourly Demand Forecasting

Model	Test MAE	Test R²	CV Avg MAE	CV Avg R²
ARIMAX	51.90	0.786	55.85	0.798
SARIMAX	38.67	0.881	39.76	0.894
Random Forest	21.02	0.958	32.30	0.907
XGBoost	20.89	0.960	30.23	0.922
Prophet	44.46	0.834	49.94	0.834

Finding: XGBoost dominated this domain with a test R² of 0.96, beating SARIMAX by a significant margin. A paired t-test confirmed XGBoost's superiority over Random Forest was statistically significant at α = 0.10 (p = 0.074). The gradient boosting approach effectively captured non-linear interactions between weather covariates, lag features, and cyclical time encodings that linear models fundamentally cannot represent.

---

⚡ Household Power — 15-Minute Interval Forecasting

Model	Test MAE (kW)	Test R²	CV Avg MAE	CV Avg R²
SARIMAX	0.324	—	0.306	0.780
ARIMAX-GARCH	0.316	0.760	0.306	0.780
LSTM	0.328	0.683	0.347	0.661
Transformer-GARCH	0.292	0.712	0.314	0.701

Finding: Despite deploying attention-based deep learning, the ARIMAX-GARCH hybrid remained competitive — and a paired t-test returned p = 0.68, meaning the Transformer-GARCH's apparent advantage was not statistically significant. The explicit seasonal harmonics in the statistical model provided a stronger linear foundation than the Transformer could learn implicitly. The ARIMAX-GARCH also trained ~5× faster, making it the clear production recommendation for this 15-minute interval use case.

---

✈️ Airport Passengers — Monthly Throughput with Structural Breaks

Models: ARIMA → SARIMA → SARIMAX → XGBoost

Finding: The Airport domain tested model robustness to an extreme structural break — the COVID-19 pandemic — which rendered pure autoregressive models unreliable. Incorporating exogenous regressors (Google search trends, COVID indicators) via SARIMAX significantly improved forecasts. XGBoost captured the non-linear recovery dynamics that the linear SARIMAX formulation could not.

---

🎾 Tennis — Weekly Player Performance Dynamics

Players: Carlos Alcaraz, Novak Djokovic, Jannik Sinner Models: ARIMA → VAR → NNAR

Finding: This domain exposed a fundamental challenge: human athletic performance is inherently low signal-to-noise. All models struggled with the volatile week-to-week swings in game win percentage. ARIMA captured general trend levels but overestimated consistency. VAR introduced cross-player dynamics (does Djokovic's form predict Sinner's?) but added complexity without proportional accuracy gains. The NNAR showed marginal improvement but could not fully model the erratic volatility.

---

Residual Diagnostics

Beyond point-forecast accuracy, we applied a formal residual diagnostic battery to every fitted model. A well-specified model should produce residuals that resemble white noise — no leftover serial correlation, no conditional heteroscedasticity, and ideally approximate normality. Three hypothesis tests were applied systematically:

Test	Null Hypothesis (H₀)	Reject Means…
Ljung-Box (Lag 20)	No serial correlation in residuals	Model has unexplained temporal structure
ARCH-LM	No ARCH effects (constant variance)	Volatility clusters remain — GARCH warranted
Shapiro-Wilk	Residuals are normally distributed	Heavy tails or skew present

🚲 Bike Sharing — Residual Tests

Model	Ljung-Box p	ARCH-LM p	Shapiro-Wilk p	Interpretation
ARIMAX	0.0000	0.0000	0.0000	All three nulls rejected — significant serial correlation, ARCH effects, and non-normality remain
SARIMAX	0.0000	0.0000	0.0000	Seasonal differencing reduces error but cannot eliminate non-linear structure
Random Forest	0.0000	0.0000	0.0000	Tree ensembles capture non-linear mean well, but residuals still exhibit volatility clustering
XGBoost	0.0000	0.0000	0.0000	Same pattern — excellent point forecasts, but the variance process is unmodeled
Prophet	0.0000	0.0000	0.0000	Additive decomposition leaves structured noise; most autocorrelation among all models

Takeaway: Every Bike model rejected all three nulls at any conventional significance level. This is not unusual for high-frequency hourly data with ~17,000 observations — the sheer sample size gives the tests enormous power to detect even minor departures. The key insight is that while all models leave some structure in the residuals, the magnitude of that structure varies dramatically: XGBoost's residual ACF decays far faster than ARIMAX's, indicating that its "leftover" autocorrelation is economically negligible even if statistically detectable.

---

⚡ Household Power — Residual Tests

Model	Ljung-Box p	ARCH-LM p	Interpretation
SARIMAX	0.0000	0.0000	Strong serial correlation and ARCH effects — motivates the GARCH extension
ARIMAX-GARCH	0.0000	0.0000	GARCH absorbs some volatility, but 70,000 15-min observations still detect residual structure
LSTM	0.0000	0.0000	Neural network captures non-linear trends effectively, but does not inherently "whiten" residuals
Transformer (pre-GARCH)	0.0000	0.0000	ARCH-LM p = 0.0000 on raw Transformer residuals explicitly justified the GARCH add-on
Transformer-GARCH	0.0000	0.0000	Even after GARCH layering, the ultra-high-frequency data retains detectable structure

Takeaway: The ARCH-LM test was the diagnostic workhorse in this domain. After fitting the SARIMAX mean model, the ARCH-LM p-value of 0.0000 provided the formal statistical mandate to layer a GARCH(1,1) variance model on top — producing the ARIMAX-GARCH hybrid. The same logic applied to the Transformer: raw Transformer residuals showed p = 0.0000 on the ARCH-LM test, justifying the two-stage Transformer-GARCH architecture. A BIC-based grid search over GARCH(p,q) orders consistently selected GARCH(1,1) as optimal across all cross-validation folds.

---

✈️ Airport Passengers — Residual Tests

Model	Ljung-Box p	Interpretation
ARIMA	1.39 × 10⁻⁶	Significant autocorrelation — unmodeled seasonal cycle leaking into residuals
SARIMA	< 2.2 × 10⁻¹⁶	Seasonal terms improve forecasts but residuals still reject white noise
SARIMAX	< 2.2 × 10⁻¹⁶	Exogenous regressors reduce error magnitude; serial structure persists
XGBoost	0.1562	Fail to reject H₀ — residuals consistent with white noise at α = 0.05

Takeaway: XGBoost was the only Airport model whose residuals passed the Ljung-Box test (p = 0.156), meaning it was the only model that adequately captured the full temporal structure — including the non-linear COVID-19 recovery dynamics that the Box-Jenkins family fundamentally could not represent. This is a strong result: not only did XGBoost produce the lowest error metrics (MAE = 0.062, RMSE = 0.076), it was also the only model to produce formally adequate residuals.

---

🎾 Tennis — Residual Tests

VAR Residuals (Ljung-Box per player):

Player	Ljung-Box p	Interpretation
Carlos Alcaraz	0.9869	Fail to reject — residuals consistent with white noise
Novak Djokovic	0.7322	Fail to reject — no remaining serial correlation
Jannik Sinner	0.9671	Fail to reject — residuals well-behaved

NNAR Residuals (Ljung-Box per player):

Player	Ljung-Box p	Interpretation
Carlos Alcaraz	1.09 × 10⁻⁶	Reject H₀ — significant autocorrelation remains
Novak Djokovic	1.31 × 10⁻⁸	Reject H₀ — neural net fails to capture temporal dependencies
Jannik Sinner	4.35 × 10⁻¹⁰	Reject H₀ — strongest autocorrelation signal

Takeaway: A surprising reversal — the VAR model produced the cleanest residuals of any model in any domain (all p > 0.73), meaning it fully captured the linear temporal dependencies between players. The NNAR, despite being the more "flexible" model, left significant autocorrelation in all three players' residuals. This suggests that for this sparse, weekly dataset, the VAR's explicit multivariate structure was more appropriate than the neural network's implicit feature learning. The VAR's high diagnostic adequacy, however, did not translate to superior forecast accuracy — a reminder that white-noise residuals are necessary but not sufficient for good out-of-sample performance.

---

Methodology

Evaluation Framework

All models were evaluated using a consistent, rigorous protocol:

1. Expanding-Window Cross-Validation (EWCV) — 5-fold rolling splits where the training window grows with each fold, preserving temporal ordering
2. Standardized Metrics — MAE (error magnitude), R² (predictive power), Fit Time (computational efficiency)
3. Statistical Significance Testing — Paired t-tests on CV fold MAEs between top candidate models to distinguish real performance gains from noise
4. Residual Diagnostics — Ljung-Box (serial correlation), ARCH-LM (heteroscedasticity), and Shapiro-Wilk (normality) tests applied to every model's residuals
5. Prediction Interval Calibration — 95% interval coverage evaluated for all probabilistic models

Feature Engineering

• Cyclical Encodings: Hour-of-day and day-of-week encoded as sin/cos pairs to preserve circular continuity
• Multi-Period Lags: Features at t-1 through t-24 (hourly) and t-1 through t-96 (15-min) to capture both short and long memory
• Seasonal Harmonics: Fourier terms for ARIMAX/SARIMAX to model complex seasonal patterns without excessive differencing
• Exogenous Variables: Weather (temperature, humidity, windspeed), Google Trends, COVID indicators — domain-specific

---

Project Structure

├── BikeMain.ipynb          # 🚲 Complete Bike Sharing pipeline (Python)
├── PowerMain.ipynb         # ⚡ Complete Household Power pipeline (Python)
├── AirportMain.ipynb       # ✈️ Complete Airport Passenger pipeline (R/Python)
├── TennisMain.ipynb        # 🎾 Complete Tennis Performance pipeline (R)
│
├── Jesse/
│   ├── Code/
│   │   ├── Bike/           # Individual model notebooks (ARIMAX → XGBoost → Prophet)
│   │   └── Power/          # Individual model notebooks (SARIMAX → Transformer-GARCH)
│   └── EDACode/            # Exploratory data analysis notebooks
│
└── Megan/
    └── Tennis/
        ├── EDA.Rmd         # Tennis exploratory analysis
        ├── ARIMA.Rmd       # ARIMA grid search & diagnostics
        ├── VAR.Rmd         # Vector Autoregression modeling
        └── NNAR.Rmd        # Neural Network Autoregression

Each *Main.ipynb notebook is fully self-contained — all models are trained from scratch within the notebook with no external file dependencies. These are the primary deliverables.

The Jesse/Code/ and Megan/Tennis/ directories contain the individual development notebooks where each model was built, tuned, and validated independently before consolidation.

---

Technical Stack

Component	Tools
Statistical Modeling	statsmodels, pmdarima, arch, R forecast, R vars
Machine Learning	scikit-learn, xgboost, prophet
Deep Learning	PyTorch (LSTM, Transformer)
Visualization	matplotlib, seaborn, R base graphics
Data Processing	pandas, numpy, R dplyr, tidyr, lubridate

---

Takeaways

1. Complexity ≠ accuracy. In two of four domains, well-specified statistical hybrids matched or beat deep learning architectures — at a fraction of the computational cost.

2. GARCH matters. Modeling the variance alongside the mean (via GARCH residual layers) consistently produced better-calibrated prediction intervals, regardless of whether the mean model was statistical or neural.

3. Ensemble ML excels with rich features. When abundant exogenous covariates and engineered lag features were available (Bike), tree-based ensembles dominated. When the feature space was thin (Tennis), they offered no advantage.

4. Statistical testing is non-negotiable. Without paired t-tests on CV folds, we would have incorrectly declared the Transformer-GARCH the "winner" in the Power domain — a conclusion that p = 0.68 firmly rejects.

5. Domain structure dictates model choice. There is no universal best model. The optimal approach depends on data frequency, seasonal complexity, feature availability, and whether the application demands point forecasts or calibrated uncertainty bands.

---

Authors

• Jesus Arzate — Bike Sharing & Household Power domains (Python: ARIMAX, SARIMAX, ARIMAX-GARCH, Random Forest, XGBoost, Prophet, LSTM, Transformer-GARCH)
• Megan Lavender — Airport Passengers & Tennis Performance domains (R/Python: ARIMA, SARIMA, SARIMAX, XGBoost, VAR, NNAR)

---

Georgia Institute of Technology — ISYE 6402: Time Series Analysis — Spring 2026