Model Performance — The Honest Scoreboard
Every experiment reports the model's RMSE against actual outcomes on
held-out seasons next to the closing line's RMSE on the same games.
If the closing line wins, there is no edge — regardless of how good training loss looks.
Validation is walk-forward by season (never random k-fold). Confidence intervals come
from a 10,000-resample bootstrap of the RMSE difference.
Total 2020 excluded
Model RMSE 16.47
vs Closing Line RMSE 15.79
· diff -0.68 [-0.84, -0.51]
· n=3944
The closing line wins. No edge on this market overall.
| Holdout Season | n |
Model RMSE | Closing Line RMSE |
Diff (95% CI) |
|---|
| 2021 |
770 |
17.12 |
16.00 |
-1.12 [-1.57, -0.68] |
| 2022 |
776 |
16.33 |
15.34 |
-1.00 [-1.38, -0.61] |
| 2023 |
792 |
16.32 |
15.84 |
-0.47 [-0.83, -0.11] |
| 2024 |
798 |
16.54 |
16.21 |
-0.33 [-0.66, -0.01] |
| 2025 |
808 |
16.02 |
15.55 |
-0.48 [-0.80, -0.15] |
Segment breakdown (potential bet filters)
conf_tier
| Segment | n |
Model RMSE | Market RMSE | Model wins? |
|---|
| G5vG5 | 1641 |
16.91 |
16.22 |
no |
| P4vG5 | 594 |
16.50 |
15.87 |
no |
| P4vP4 | 1709 |
16.01 |
15.34 |
no |
week_band
| Segment | n |
Model RMSE | Market RMSE | Model wins? |
|---|
| wk1-4 | 1234 |
16.68 |
16.10 |
no |
| wk5-9 | 1338 |
16.38 |
15.46 |
no |
| wk10+ | 1372 |
16.36 |
15.83 |
no |
wind_band
| Segment | n |
Model RMSE | Market RMSE | Model wins? |
|---|
| calm | 3215 |
16.57 |
15.99 |
no |
| 10-15 | 599 |
16.15 |
14.94 |
no |
| 15-20 | 118 |
15.25 |
14.52 |
no |
total_band
| Segment | n |
Model RMSE | Market RMSE | Model wins? |
|---|
| <45 | 552 |
15.42 |
13.84 |
no |
| 45-52 | 1249 |
16.31 |
15.90 |
no |
| 52-59 | 1316 |
16.23 |
15.69 |
no |
| 59+ | 827 |
17.70 |
16.96 |
no |
fav_band
| Segment | n |
Model RMSE | Market RMSE | Model wins? |
|---|
| pickem-3 | 827 |
16.99 |
15.89 |
no |
| 3-7 | 1028 |
16.87 |
16.21 |
no |
| 7-14 | 1015 |
16.76 |
16.35 |
no |
| 14+ | 1074 |
15.35 |
14.74 |
no |
Total 2020 included
Model RMSE 16.31
vs Closing Line RMSE 15.79
· diff -0.52 [-0.67, -0.37]
· n=3944
The closing line wins. No edge on this market overall.
| Holdout Season | n |
Model RMSE | Closing Line RMSE |
Diff (95% CI) |
|---|
| 2021 |
770 |
16.71 |
16.00 |
-0.71 [-1.10, -0.32] |
| 2022 |
776 |
16.24 |
15.34 |
-0.90 [-1.25, -0.55] |
| 2023 |
792 |
16.30 |
15.84 |
-0.46 [-0.80, -0.12] |
| 2024 |
798 |
16.36 |
16.21 |
-0.15 [-0.45, +0.16] |
| 2025 |
808 |
15.94 |
15.55 |
-0.39 [-0.70, -0.08] |
Segment breakdown (potential bet filters)
conf_tier
| Segment | n |
Model RMSE | Market RMSE | Model wins? |
|---|
| G5vG5 | 1641 |
16.71 |
16.22 |
no |
| P4vG5 | 594 |
16.30 |
15.87 |
no |
| P4vP4 | 1709 |
15.92 |
15.34 |
no |
week_band
| Segment | n |
Model RMSE | Market RMSE | Model wins? |
|---|
| wk1-4 | 1234 |
16.51 |
16.10 |
no |
| wk5-9 | 1338 |
16.11 |
15.46 |
no |
| wk10+ | 1372 |
16.31 |
15.83 |
no |
wind_band
| Segment | n |
Model RMSE | Market RMSE | Model wins? |
|---|
| calm | 3215 |
16.39 |
15.99 |
no |
| 10-15 | 599 |
15.95 |
14.94 |
no |
| 15-20 | 118 |
15.78 |
14.52 |
no |
total_band
| Segment | n |
Model RMSE | Market RMSE | Model wins? |
|---|
| <45 | 552 |
15.04 |
13.84 |
no |
| 45-52 | 1249 |
16.23 |
15.90 |
no |
| 52-59 | 1316 |
16.11 |
15.69 |
no |
| 59+ | 827 |
17.49 |
16.96 |
no |
fav_band
| Segment | n |
Model RMSE | Market RMSE | Model wins? |
|---|
| pickem-3 | 827 |
16.67 |
15.89 |
no |
| 3-7 | 1028 |
16.94 |
16.21 |
no |
| 7-14 | 1015 |
16.49 |
16.35 |
no |
| 14+ | 1074 |
15.19 |
14.74 |
no |
Margin 2020 excluded
Model RMSE 16.52
vs Closing Line RMSE 15.29
· diff -1.23 [-1.43, -1.03]
· n=3944
The closing line wins. No edge on this market overall.
| Holdout Season | n |
Model RMSE | Closing Line RMSE |
Diff (95% CI) |
|---|
| 2021 |
770 |
17.53 |
15.59 |
-1.93 [-2.48, -1.38] |
| 2022 |
776 |
16.49 |
15.28 |
-1.21 [-1.67, -0.77] |
| 2023 |
792 |
15.97 |
15.20 |
-0.77 [-1.17, -0.38] |
| 2024 |
798 |
16.50 |
15.38 |
-1.11 [-1.50, -0.74] |
| 2025 |
808 |
16.11 |
15.01 |
-1.11 [-1.49, -0.72] |
Segment breakdown (potential bet filters)
conf_tier
| Segment | n |
Model RMSE | Market RMSE | Model wins? |
|---|
| G5vG5 | 1641 |
16.73 |
15.61 |
no |
| P4vG5 | 594 |
17.42 |
14.98 |
no |
| P4vP4 | 1709 |
16.00 |
15.09 |
no |
week_band
| Segment | n |
Model RMSE | Market RMSE | Model wins? |
|---|
| wk1-4 | 1234 |
17.18 |
15.15 |
no |
| wk5-9 | 1338 |
16.05 |
15.09 |
no |
| wk10+ | 1372 |
16.38 |
15.60 |
no |
wind_band
| Segment | n |
Model RMSE | Market RMSE | Model wins? |
|---|
| calm | 3215 |
16.48 |
15.25 |
no |
| 10-15 | 599 |
16.79 |
15.56 |
no |
| 15-20 | 118 |
16.76 |
15.11 |
no |
total_band
| Segment | n |
Model RMSE | Market RMSE | Model wins? |
|---|
| <45 | 552 |
15.93 |
14.84 |
no |
| 45-52 | 1249 |
16.49 |
15.45 |
no |
| 52-59 | 1316 |
16.46 |
15.23 |
no |
| 59+ | 827 |
17.05 |
15.45 |
no |
fav_band
| Segment | n |
Model RMSE | Market RMSE | Model wins? |
|---|
| pickem-3 | 827 |
16.16 |
15.16 |
no |
| 3-7 | 1028 |
16.38 |
15.07 |
no |
| 7-14 | 1015 |
16.49 |
15.50 |
no |
| 14+ | 1074 |
16.96 |
15.40 |
no |
Margin 2020 included
Model RMSE 16.48
vs Closing Line RMSE 15.29
· diff -1.19 [-1.39, -0.99]
· n=3944
The closing line wins. No edge on this market overall.
| Holdout Season | n |
Model RMSE | Closing Line RMSE |
Diff (95% CI) |
|---|
| 2021 |
770 |
17.33 |
15.59 |
-1.73 [-2.25, -1.22] |
| 2022 |
776 |
16.43 |
15.28 |
-1.15 [-1.61, -0.70] |
| 2023 |
792 |
16.06 |
15.20 |
-0.86 [-1.26, -0.47] |
| 2024 |
798 |
16.51 |
15.38 |
-1.13 [-1.52, -0.75] |
| 2025 |
808 |
16.06 |
15.01 |
-1.06 [-1.44, -0.68] |
Segment breakdown (potential bet filters)
conf_tier
| Segment | n |
Model RMSE | Market RMSE | Model wins? |
|---|
| G5vG5 | 1641 |
16.65 |
15.61 |
no |
| P4vG5 | 594 |
17.58 |
14.98 |
no |
| P4vP4 | 1709 |
15.91 |
15.09 |
no |
week_band
| Segment | n |
Model RMSE | Market RMSE | Model wins? |
|---|
| wk1-4 | 1234 |
17.12 |
15.15 |
no |
| wk5-9 | 1338 |
15.90 |
15.09 |
no |
| wk10+ | 1372 |
16.45 |
15.60 |
no |
wind_band
| Segment | n |
Model RMSE | Market RMSE | Model wins? |
|---|
| calm | 3215 |
16.45 |
15.25 |
no |
| 10-15 | 599 |
16.67 |
15.56 |
no |
| 15-20 | 118 |
16.70 |
15.11 |
no |
total_band
| Segment | n |
Model RMSE | Market RMSE | Model wins? |
|---|
| <45 | 552 |
16.00 |
14.84 |
no |
| 45-52 | 1249 |
16.41 |
15.45 |
no |
| 52-59 | 1316 |
16.35 |
15.23 |
no |
| 59+ | 827 |
17.10 |
15.45 |
no |
fav_band
| Segment | n |
Model RMSE | Market RMSE | Model wins? |
|---|
| pickem-3 | 827 |
16.16 |
15.16 |
no |
| 3-7 | 1028 |
16.19 |
15.07 |
no |
| 7-14 | 1015 |
16.37 |
15.50 |
no |
| 14+ | 1074 |
17.09 |
15.40 |
no |
Model Registry
| Version | Target | Trained |
Promoted | Spec hash |
|---|
| model_2026-09-04_v1 | total |
2026-09-04 22:21:58 |
yes |
4ac109ff2f4e |
| model_2026-09-04_v1 | margin |
2026-09-04 22:21:58 |
yes |
4ac109ff2f4e |