A win-probability audit of every NFL 4th-down decision since 2018. Coaches are ranked by how much win probability they leave on the field by punting when they should have gone for it, kicking field goals they shouldn't, and so on. Bootstrap confidence intervals are baked in because a single season of 4th downs is a small sample and the rankings are noisy.
Built on the public nflverse play-by-play data. The WP model is an XGBoost classifier that gets a 2024 log-loss of 0.465, within 0.3% of nflfastR's bundled WP model (0.463) on the same held-out season.
For every 4th-down play in 2018-2024 with a real snap (no penalty aborts, no kneeldowns), the pipeline computes three counterfactual expected win probabilities, all from the offense's pre-snap perspective:
E[WP | go] = P(convert) · WP(after gain) + (1 − P(convert)) · [1 − WP(opponent ball at our spot)]
E[WP | punt] = 1 − WP(opponent at expected post-punt yardline)
E[WP | FG] = P(make) · [1 − WP(opponent at kickoff, +3 against)] + (1 − P(make)) · [1 − WP(opponent at spot of kick)]
The model's recommendation is whichever option maximizes E[WP]. The cost of the coach's actual decision is the gap between the best option and the chosen option. Costs are non-negative by construction.
The Next.js dashboard lets you:
- Rank coaches by total WP lost on 4th down, with 90% bootstrap CIs on each total
- Filter by season (2018-2024) or look at all-time
- Click a coach to drill into every 4th down they faced, filtered by situation (red zone, two-minute drill, own territory, FG range) or actual decision
- Click any play to see the model's recommendation broken down by E[WP], the actual outcome, and an animated WP curve for the surrounding plays
- Python + XGBoost for the WP model, plus logistic regressions for 4th-down conversion probability and FG make probability, and an empirical lookup for expected net punt yards
- DuckDB as the analytical store. ~50MB file holds all of 2018-2024 pbp plus the scored 4th-down table and per-coach aggregates
- FastAPI for local development. The dev API serves the same JSON shapes that ship statically in production.
- Next.js 16 + TypeScript App Router frontend with
output: "export"for fully static deploy. Recharts for the WP curve. - Tailwind for styling.
In production every endpoint is precomputed: leaderboards live at /data/coaches/<season>.json, per-coach play lists at /data/plays/<slug>.json, and per-game pbp slices at /data/game/<game_id>.json. The browser fetches whichever it needs, all filtering happens client-side. No backend in the deploy.
| log-loss | Brier | AUC | |
|---|---|---|---|
| nflfastR (baseline) | 0.4634 | 0.1570 | n/a |
| our WP model | 0.4649 | 0.1579 | 0.848 |
| log-loss | Brier | base rate | |
|---|---|---|---|
| 4th-down conversion | 0.633 | 0.222 | 53.4% |
| Field-goal make | 0.380 | 0.119 | 84.5% |
Per-coach decision quality (2024, top 5 / bottom 5):
John Harbaugh BAL 103 plays loss=0.69 [0.46, 0.97] agree=60%
Shane Steichen IND 131 plays loss=0.80 [0.61, 1.02] agree=58%
Todd Bowles TB 98 plays loss=0.83 [0.58, 1.10] agree=62%
Dan Campbell DET 107 plays loss=0.87 [0.61, 1.14] agree=58%
Kevin O'Connell MIN 117 plays loss=0.95 [0.62, 1.30] agree=66%
...
Jim Harbaugh LAC 134 plays loss=1.86 [1.50, 2.28] agree=49%
Sean McVay LA 133 plays loss=1.89 [1.49, 2.36] agree=48%
Antonio Pierce LV 129 plays loss=1.91 [1.42, 2.46] agree=57%
Matt Eberflus CHI 142 plays loss=1.95 [1.47, 2.44] agree=48%
Coaches and the model agreed about 57% of the time in 2024. The single biggest disagreement category: 900 plays where the model said go for it and the coach punted instead.
Calling this a coach scorecard is honest only if I'm explicit about what it misses:
- Opponent strength. A 4th-and-2 against the Lions defense is a different proposition than against the Patriots. Our P(convert) model doesn't see the opponent. That's probably the biggest gap.
- Weather. Cold-weather punts come up short. Crosswind FGs miss. We don't model wind, temperature, or precipitation. The punt bucketing absorbs some of this on average, but not for a specific game.
- Personnel. Down to your backup kicker after a hamstring pull? The model says try the 52-yarder anyway, because the league-average kicker makes it.
- Decision quality vs outcome quality. We don't penalize a coach for the outcome of a play, only for the option they picked. A 4th-and-goal stuff counts the same as a 5-yard scamper if the model agreed with going. That's intentional, but it's worth saying out loud.
The bootstrap CIs make the noise legible: the gap between the 5th and 25th ranked coach in any given season is usually inside the noise band, even though the gap from 1st to 30th is real. Trust the bars more than the rank.
# 1. Backend (data pipeline + dev API)
cd backend
python3 -m venv .venv
.venv/bin/pip install -r requirements.txt
.venv/bin/pip install -e .
.venv/bin/python scripts/fetch.py # ~140MB of nflverse parquet
.venv/bin/python scripts/train.py # ~3 min on a laptop
.venv/bin/python scripts/score_season.py # ~30s, including bootstrap
.venv/bin/python scripts/export_static.py # writes JSON to frontend/public/data/
# 2. Frontend (static dev server)
cd ../frontend
npm install
npm run dev # http://localhost:3000
npm run build # produces frontend/out/ for static deploybackend/
src/fda/
data/ nflverse download + DuckDB ingest + feature engineering
model/ wp (xgboost), conversion (logit), fg (logit), punt (empirical)
decision/ counterfactual scorer + bootstrap CIs
api/ FastAPI app, one read-only DuckDB conn per request
scripts/ fetch / train / score_season entrypoints
data/ DuckDB file + raw nflverse parquet (gitignored)
models/ trained model artifacts + metrics JSON (gitignored)
frontend/
src/app/ App Router pages: leaderboard, /coach/[coach], /methodology
src/components/ Leaderboard, PlayList, PlayDetailDrawer, WpCurve, filters
src/lib/ API client + shared types + formatters
Vercel static deploy. vercel.json builds the frontend (output: "export") and uploads the precomputed JSON in frontend/public/data/. No runtime backend; the pipeline only runs locally to regenerate that data when a new season lands.
MIT. Data: nflverse releases, which inherit nflfastR's original license.