Limitations & what’s next — NFL Outcome Predictor

1. What we don’t have

The model uses only nflverse’s free public release. The following are real signals it’s flying without — each is something the market has and we don’t.

Injury reports

Mid-week practice statuses, game-time scratches, return-from-IR — none of it. A starting QB ruled out an hour before kickoff would not show up in our prediction.

Plausible lift if added: 1–3 pts of accuracy, mostly on a handful of games per week where status flipped late.

Weather

Temperature, wind, precipitation, dome vs outdoor. Wind in particular materially changes win probability for pass-heavy offenses.

Bigger effect on totals (over/under) than on who wins; modest lift on game-outcome accuracy.

Coaching changes & depth-chart turnover

A new head coach, OC, or DC shifts identity overnight. Elo & QBElo treat the team as the same entity that ended last season; the only nod to change is the off-season regression toward the mean.

Hard to quantify, but the first 2–4 games of any season carry above-average uncertainty for teams with regime change.

Situational EPA

Our rolling EPA is across all plays. It doesn’t split by down, distance, red zone, garbage time, or score state — so a team padding stats in blowouts looks the same as one converting on 3rd-and-long.

Garbage-time filtering alone would noticeably tighten the EPA features.

Camp news & leading indicators

Preseason performance, training-camp reports, suspensions, holdouts, contract drama. Markets price these in days before our weekly cron sees anything.

Travel, rest beyond days, dome/outdoor

We carry home_rest / away_rest in days. We do not carry travel distance, time-zone changes, short-week vs Thursday-night, or stadium type. HFA is a flat 48 for every venue.

2. Modeling choices I made

Decisions that shaped the model. Each was a real fork; I tried to take the option that protects the honesty of the benchmark.

No betting features in the ML model

The spread, totals, and moneylines are in the assembled feature frame but the ML model is trained on the 13 non-betting columns only. If I let the model see the closing line at train time it would learn to copy Vegas — and the “ML vs market” comparison would be meaningless. Keeping them out preserves an honest benchmark, at the cost of accuracy.

Walk-forward retraining instead of one frozen model

Each holdout season retrains the ML on all prior data and Platt-calibrates on the immediately-preceding season. This gives the ML the same online-adaptation budget that Elo and QBElo enjoy by design. Without it, ML lost to QBElo badly — with it, they’re roughly tied.

Off-the-shelf 538 weights for QB game value

The QB-value formula (+3.7/completion, +11.3/passing TD, −14.1/INT, …) is FiveThirtyEight’s, not re-derived for the current era. Rule changes (more passing, fewer big hits, kickoff tweaks) probably mean those weights are slightly stale.

Constant home-field advantage (HFA = 48)

League-average home edge has been drifting down since 2010 — we use a single constant that’s biased toward the historical average. No per-team HFA either: a Seattle home game and a Jacksonville home game get the same +48.

Coarse rest, no special teams, no coaching

Rest is just days. There’s no separate feature for “coming off a bye,” “short week,” or “Thursday night.” Kicking, returning, and coverage units don’t enter the model at all. Coaching changes are invisible to the pipeline.

Logistic over gradient boosting for the ML head

The gradient-boosted version actually had marginally higher straight-up accuracy in the walk-forward backtest. I lead with the calibrated logistic because its probabilities are sharper (better Brier), and probabilities are the project’s point.

3. Why we don’t beat Vegas (and don’t try)

The closing NFL line is one of the sharpest priced markets on Earth. It incorporates injury news, sharp money, public bias, and the books’ own balance sheets — in real time. A free-public-data model running a weekly cron is structurally a step behind on every one of those.

Across 2015–2025, Vegas beats our model in nearly every season on Brier and log loss. Single-season variance occasionally lets us nudge it on accuracy alone, but on the proper scoring rules — the only honest scoreboard — the market wins.

Even matching the closing line isn’t the same as beating it. The vig is roughly 4–5% on standard juice; a model would need a 3–5% systematic edge over true probability just to break even after the book’s margin. The calibration plot on the live tracker shows we’re calibrated and well above a coin flip; that is the win this project goes for, not an edge that doesn’t exist.

If a portfolio piece tells you it consistently beats Vegas, it’s either being graded on too few games or quietly looking at the line at training time. The honest claim is calibration.

4. What I’d build next

In priority order, with what each would realistically buy and rough effort. Each closes part of the gap to Vegas; none of them would let the project legitimately claim an edge.

‡ The plausible lift numbers below are estimates, not measured results — informed by the holdout backtest gap and the public literature on NFL modeling, but they’re ranges I’d expect, not values I’ve measured.

1

Injury features from nflverse’s injuries table

Join the practice-status / game-status data per team and turn it into “starter out,” “questionable count,” and a positional severity score. This is the single biggest gap between us and the market.

Plausible lift: 1–3 pts of accuracy, ~0.003–0.006 of Brier. Effort: ~half a day.
2

Per-team HFA + trending

Drop the constant 48 and fit an HFA per team-season (with shrinkage to the league mean). Decay older seasons. Probably reveals teams whose “home edge” is mostly altitude/weather/turf vs crowd noise.

Plausible lift: small but real (0.5–1 pt accuracy). Effort: a day, plus a backtest re-run.
3

Ensemble QBElo + ML at calibration time

Both models are individually well-calibrated and complementary — QBElo wins Brier, ML wins straight-up accuracy. A simple average (or a logistic blend tuned on the calibration year) usually closes a chunk of the gap to Vegas without overfitting.

Plausible lift: closes 30–50% of the QBElo→Vegas Brier gap. Effort: a few hours.
4

Situational EPA splits

Split the rolling EPA by down/distance, drop garbage-time plays (e.g., score margin ≥ 24 in Q4), and weight neutral-game-state plays higher. The current single-EPA number is noisy.

Plausible lift: tightens features more than improves predictions on its own — bigger gains when stacked with #1. Effort: a day.
5

Weather features for outdoor games

Join a free weather API on stadium location and kickoff time. Wind > 20mph is the variable that moves win probabilities (especially for pass-heavy offenses).

Plausible lift: small on game outcomes, larger on totals (a separate model we don’t ship). Effort: a day, plus testing the API in CI.
6

Bootstrap confidence intervals on the backtest metrics

Right now the holdout numbers (Brier 0.2205, accuracy 63.8%) look like point estimates. Bootstrapping the season-by-season results gives a real ± for each one — honest about how much of the gap to Vegas is noise.

Plausible lift: no lift to the model itself, but the reporting is much sharper. Effort: a few hours.
7

Re-derive the QB-value weights for the current era

Refit FiveThirtyEight’s box-score weights against modern post-2010 QB outcomes — the linear weights are old enough to have aged through major rule changes.

Plausible lift: small but cleanly attributable. Effort: a day.