REVIEW 4 major objections 5 minor 16 references
Beyond Winning: Margin of Victory Relative to Expectation Unlocks Accurate Skill Ratings
T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read A margin-minus-expectation update rule for ELO improves NBA skill ratings and converges faster.
desk verdict Clean idea—use the residual between actual and expected margin in an ELO update—but the evaluation as written leaks test information and the convergence metric is tautological, so the empirical claims don't hold. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the MOV differential $\Delta_{MOV} = T_{MOV} - E_{MOV}$, computed from a scaled hyperbolic-tangent expectation model $E_{MOV}(\Delta R, I_{HA}) = \alpha \tanh(\beta \Delta R) + \gamma + \delta I_{HA}$. It isolates the unexpected component of a team's performance margin, and the algorithm injects it into the rating update as $\lambda \Delta_{MOV}$ alongside the standard ELO term $K(S_A - E_A)$. This lets the rating system absorb performance magnitude while keeping the update zero-sum and nearly as cheap as ELO.
What would settle it
Take the 2013-2023 NBA games in chronological order, fit the EMOV parameters using only the first 70% of games and the ratings ELO produces up to each match, then run MOVDA on the remaining 30% with no further fitting. If the whole-dataset fit leaked test information, the Brier score or convergence gain will shrink relative to the paper's Table I; if the fit was clean, the numbers should match.
Extended reading notes
Core claim
The paper's central claim is that margin-of-victory information should enter a rating system as a differential: the true margin minus a learned expected margin, rather than as a raw margin-scaled update. For a matchup with rating difference $\Delta R = R_A - R_B$ and home indicator $I_{HA}$, the expected margin is $E_{MOV}(\Delta R, I_{HA}) = \alpha \tanh(\beta \Delta R) + \gamma + \delta I_{HA}$, with the four parameters fit by least squares on historical games. The update becomes $R'_A = R_A + K(S_A - E_A) + \lambda(T_{MOV} - E_{MOV})$, and $R'_B$ moves by the negative of the same amount, preserving ELO's zero-sum structure. On the paper's holdout NBA data, this yields a Brier score of 0.2258 (1.54% below TrueSkill's 0.2294), outcome accuracy of 63.32% (0.58 percentage points above TrueSkill), and convergence in 166 games versus 193 for standard ELO.
Load-bearing premise
The four-parameter expectation curve is fit to ratings produced by standard ELO run on the whole dataset, so the load-bearing premise is that this fit used only training information and stays valid inside the MOVDA update loop; if the test games contributed to the fit, the reported improvements are inflated.
Editorial extensions
If this is right
- On the paper's holdout NBA data, MOVDA's Brier score of 0.2258 is the lowest among ELO, linear-MOV ELO, Glicko-2, and TrueSkill, meaning its probabilistic win predictions are better calibrated.
- MOVDA stabilizes a new team's rating in about 166 games, versus 193 for standard ELO and 189 for Glicko-2, so ratings adapt faster to genuine strength changes.
- Because the update remains zero-sum and requires only four pre-fit parameters plus one hyperparameter $\lambda$, MOVDA can run in real-time rating pipelines at roughly ELO's computational cost.
- The framework is retrainable: a new season or a new league requires only refitting $(\alpha, \beta, \gamma, \delta)$ and choosing a new $\lambda$, which the paper leaves to future work.
Reading between the lines
- The same differential idea could be applied to other scalar performance measures, such as goals, runs, or kill counts, by choosing a domain-appropriate expectation function; the paper demonstrates only point margins in the NBA.
- Because $\lambda$ controls how much weight the margin surprise carries, one could tune $\lambda$ per team or over time to handle volatility, roster turnover, or tanking behavior; the paper uses a single global $\lambda$.
- If the expectation curve is refit on a rolling window rather than once on historical data, MOVDA could track slow changes in league-wide scoring environments, such as pace or rule changes, without losing its computational simplicity.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper proposes Margin of Victory Differential Analysis (MOVDA), an ELO-style rating update that augments the standard binary outcome error with a term proportional to the difference between the observed margin and an expected margin modeled as E_MOV(ΔR, I_HA) = α·tanh(β·ΔR) + γ + δ·I_HA. The four parameters are fit on NBA data, the hyperparameter λ is tuned, and the method is compared with standard ELO, linear MOV-scaled ELO, Glicko-2, and TrueSkill on a chronological split of 13,619 NBA games. The paper reports a 1.54% lower Brier score than TrueSkill, 0.58 percentage points higher outcome accuracy, and 13.5% faster rating convergence, all at computational cost comparable to ELO. The update rule is simple and the zero-sum structure is preserved, but the empirical validation has several load-bearing problems: the convergence metric is self-referential, the fitting protocol for the expectation model is ambiguous and potentially leaky, the train/validation/test split and hyperparameter tuning are not described precisely, and no uncertainty quantification is provided for the small reported differences.
Significance. If the empirical claims were cleanly established, MOVDA would be a practically useful contribution: it offers a transparent, computationally cheap way to inject margin information into ELO via a learned residual, and the saturating tanh form for expected margin with a home-advantage term is a reasonable modeling choice. The paper also deserves credit for a simple ablation showing that the margin differential term changes the results. However, the most emphasized contribution—faster convergence—is not supported by the reported metric, and the predictive improvements are small and presented without confidence intervals. The manuscript therefore currently establishes a plausible idea rather than a verified result, and the experiments need to be rerun or substantially clarified before the claims can be accepted.
major comments (4)
- [Section IV-C] The convergence metric is self-referential: for each simulated new team, the target band is centered on the final stable rating computed from the final 200 games of the same test-set trajectory that is being evaluated. A system that never updates its ratings would satisfy the band condition immediately, because its constant rating equals its own final rating. Consequently, the comparison in Table I (MOVDA 166 games vs. Standard ELO 193 games) does not measure convergence to true skill; it measures when a trajectory settles near its own tail average. The 13.5% convergence improvement claimed in the abstract and conclusion is therefore unsupported. Please replace this metric with an external ground truth (for example, end-of-season strength measures or rolling predictive accuracy) or explicitly reframe the claim as self-consistency rather than convergence.
- [Section IV-D] The EMOV parameters are said to be estimated on the training data 'using ratings generated by standard ELO run on the whole dataset.' This contradicts Section III-B, which states that the ratings used for estimation are derived from processing the training data chronologically 'up to match i.' If 'the whole dataset' includes the testing and holdout partitions, then Eq. (6) uses future information to fit the expectation model, and all reported out-of-sample improvements are inflated. Please state exactly which games contribute to the ELO run used for the fit, and if the current wording is leaky, re-estimate the parameters using only information available up to each training match.
- [Sections IV-A and IV-D] The evaluation protocol is ambiguous. The data are described as split into 'training (first 70%) and testing (second 20%) and hold out test (remaining 10%),' but Table I is described as being on the 'hold out set' while Section IV-E refers to results 'on the test sets.' Hyperparameters are 'tuned on the same data set by optimizing for the Brier score' without specifying whether that set is a validation partition or the final test partition. If tuning used the final test set, the Brier and accuracy results are not honest out-of-sample estimates. Please report the exact partition boundaries, the tuning set, and the selected hyperparameter values for every model.
- [Table I] No confidence intervals or significance tests are reported for the Brier-score or accuracy differences. Since some differences are very small (Brier 0.2258 vs. 0.2264 for Glicko-2, accuracy 63.32% vs. 63.18% for Linear MOV ELO), the statement that MOVDA 'significantly outperforms' the baselines is not statistically supported as presented. Please provide error bars, paired tests, or at least a sensitivity analysis over the chronological split.
minor comments (5)
- [Abstract, Section IV-E, Section VI] Numerical claims are inconsistent: the abstract and conclusion report 1.54% Brier improvement and 13.5% faster convergence, while Section IV-E reports a 0.66% Brier improvement over standard ELO and 13.9% fewer games than standard ELO; the accuracy gain in Table I is 0.66 percentage points over TrueSkill, not 0.58. Please harmonize all reported percentages.
- [Section IV-E] The text says the results are shown 'across the four domains on the test sets,' but the paper reports only NBA data. Either add the missing domains or correct the sentence.
- [Figure 2] The caption describes the MOV distribution as approximately normal and centered near zero, while the text notes clear positive skew for home teams and negative skew for away teams. Please reconcile the caption with the distributional limitation acknowledged in Section V.
- [References] Reference [8] is a neuroimaging study on developmental coordination disorder and does not support the basketball MOV K-factor scaling claim; please replace it with the intended citation.
- [Throughout] There are minor typographical errors, including 'datset' in Section IV-A, 'Scale hyperbolic tangent' in Section V, and 'the computational efficiency comparable' in Section V.
Circularity Check
Convergence metric is self-referential: the ±20-point band is centered on the run's own final rating, so the headline 13.5% acceleration does not measure convergence to true skill; separately, the EMOV expectation is fitted using ELO ratings 'run on the whole dataset', risking leakage of evaluation-period outcomes into the fitted curve.
-
self definitional
[Section IV-C, Evaluation Metrics (Convergence Speed); interpreted in the Abstract and Section V]
"Convergence Speed: Measured by tracking hypothetical new teams introduced at the start of the test set with the default rating (1500). We calculated the average number of games required for such a team's rating to enter and remain continuously within a predefined band (e.g., ± 20 rating points) around its final stable rating over the final 200 games of the test set."
The metric's target is the final rating of the very trajectory being evaluated: a team 'converges' when it enters a band centered on its own final-200-game average, a quantity that is produced by the same run. A system that never updated its ratings would score zero games, because a constant rating equals its own final rating, so the metric measures self-agreement rather than agreement with any external or true skill. The paper nonetheless interprets the result as faster adaptation to team strength ('most importantly accelerates rating convergence by 13.5%' in the Abstract; 'ratings likely reflect current team strength more quickly' in Section V). Table I's comparison (MOVDA 166 games vs.
-
fitted input called prediction
[Section IV-D, Implementation Details; contrast with Section III-B]
"The EMOV parameters (α, β, γ, δ) for MOVDA were estimated using non-linear least squares implemented with SciPy's 'curve fit' on the training data, using ratings generated by standard ELO run on the whole dataset."
Section III-B specifies that the ratings entering the EMOV fit are 'derived from processing the training data chronologically using a baseline rating system ... up to match i', i.e., training information only. Section IV-D instead says the fit uses ratings from standard ELO 'run on the whole dataset'. If 'the whole dataset' includes the testing and holdout periods, then the ELO ratings — and hence the fitted EMOV of Eq. 4 — embed outcomes of the very games later used to claim out-of-sample improvements. Because EMOV enters every MOVDA update (Algorithm 1, lines 3–5) and every margin and probability evaluation, the reported 1.54% lower Brier score and 0.58% higher accuracy would be partially in-sample rather than predictions made from training information alone.
full rationale
The paper's empirical program splits into two families. The Brier score, outcome accuracy, and margin MAE comparisons are computed from pre-match ratings and do not reduce by construction: the EMOV parameters are fitted once by nonlinear least squares, the update rule (Eq. 8) is a genuine combination of ELO's outcome term and a margin residual, and the baselines (TrueSkill, Glicko-2) are external systems. Those comparisons are not circular. The convergence claim, however, is self-referential: the ±20-point band is centered on the same run's final-200-game average rating, so the metric is defined in terms of the trajectory it purports to validate, and any constant-rating system would trivially achieve zero games. Table I's 166 versus 193 games therefore does not establish faster adaptation to true team strength. Separately, Section IV-D says EMOV was fitted using ratings from standard ELO 'run on the whole dataset', which contradicts Section III-B's chronological-training description; if the whole dataset includes testing and holdout games, the fitted expectation embeds outcomes of the games used to claim out-of-sample gains. The ambiguity is load-bearing and unresolved, so the out-of-sample status of the Brier and accuracy numbers is unverified. Overall, partial circularity is concentrated in the headline convergence metric, with a genuine leakage risk in the expectation fit; the underlying Brier and accuracy machinery is not itself circular.
Assumptions & free parameters
free parameters (6)
- alpha
- beta
- gamma
- delta
- lambda
- K
assumptions (5)
- domain assumption Observed margins are approximately normally distributed around the expected margin (Eq. 5).
- ad hoc to paper The scaled tanh function is the correct functional form for expected margin (Eq. 4).
- domain assumption An expectation model fit to standard ELO ratings remains valid inside the MOVDA update loop.
- ad hoc to paper Home advantage enters only as an additive delta * I_HA term.
- domain assumption The Kaggle NBA dataset is accurate, complete, and correctly ordered chronologically.
Cite this review
Pith. "Pith review of Beyond Winning: Margin of Victory Relative to Expectation Unlocks Accurate Skill Ratings." pith.science (2026). https://pith.science/paper/OLQX76IW
@misc{pith2026250600348,
author = {Pith},
title = {Pith review of: Beyond Winning: Margin of Victory Relative to Expectation Unlocks Accurate Skill Ratings},
year = {2026},
howpublished = {\url{https://pith.science/paper/OLQX76IW}},
note = {Machine review of arXiv:2506.00348}
}
abstract
Knowledge of accurate relative skills in any competitive system is essential, but foundational approaches such as ELO discard extremely relevant performance data by concentrating exclusively on binary outcomes. While margin of victory (MOV) extensions exist, they often lack a definitive method for incorporating this information. We introduce Margin of Victory Differential Analysis (MOVDA), a framework that enhances traditional rating systems by using the deviation between the true MOV and a $\textit{modeled expectation}$. MOVDA learns a domain-specific, non-linear function (a scaled hyperbolic tangent that captures saturation effects and home advantage) to predict expected MOV based on rating differentials. Crucially, the $\textit{difference}$ between the true and expected MOV provides a subtle and weighted signal for rating updates, highlighting informative deviations in all levels of contests. Extensive experiments on professional NBA basketball data (from 2013 to 2023, with 13,619 games) show that MOVDA significantly outperforms standard ELO and Bayesian baselines. MOVDA reduces Brier score prediction error by $1.54\%$ compared to TrueSkill, increases outcome accuracy by $0.58\%$, and most importantly accelerates rating convergence by $13.5\%$, while maintaining the computational efficiency of the original ELO updates. MOVDA offers a theoretically motivated, empirically superior, and computationally lean approach to integrating performance magnitude into skill rating for competitive environments like the NBA.
Figures
Reference graph
Works this paper leans on
-
[8]
K. K. Gill, D. Lang, and J. G. Zwicker, “Cerebellar and Brainstem Differences in Children with Developmental Coordination Disorder: A V oxel-Based Morphometry Study,” Front. Hum. Neurosci. , vol. 16, p. 921505, 2022. [Online]. Available: https://doi.org/10.3389/fnhum.2022. 921505
-
[10]
D. N. Osherson, J. Stern, O. Wilkie, M. Stob, and E. E. Smith, “Default Probability,” Cogn. Sci. , vol. 15, no. 2, pp. 251–269, 1991. [Online]. Available: https://doi.org/10.1207/s15516709cog1502 3
-
[12]
Reinforcement Learning in Dynamic Environments,
Y . Sismanis, “Reinforcement Learning in Dynamic Environments,” in Proc. 2010 Int. Conf. Machine Learning , pp. 123–130, 2010. [Online]. Available: https://doi.org/10.1145/1234567.1234568
arXiv 2010
-
[1]
The Methodology of Officially Recognized International Sports Rating Systems,
R. Stefani, “The Methodology of Officially Recognized International Sports Rating Systems,” J. Quant. Anal. Sports , vol. 7, no. 4, 2011. [Online]. Available: https://doi.org/10.2202/1559-0410.1347
-
[2]
TrueSkill™: A Bayesian Skill Rating System,
R. Herbrich, T. Minka, and T. Graepel, “TrueSkill™: A Bayesian Skill Rating System,” in Advances in Neural Information Processing Systems , vol. 19, pp. 569–576, 2007. [Online]. Available: https://papers.nips.cc/ paper/3079-trueskilltm-a-bayesian-skill-rating-system
work page 2007
-
[3]
Whole-History Rating: A Bayesian Rating System for Players of Time-Varying Strength,
R. Coulom, “Whole-History Rating: A Bayesian Rating System for Players of Time-Varying Strength,” in Proc. 6th Int. Conf. Computers and Games (CG 2008) , Lecture Notes in Computer Science, vol. 5131, pp. 113–124. Springer, 2008. [Online]. Available: https://doi.org/10. 1007/978-3-540-87608-3 11
work page 2008
-
[4]
A. E. Elo, The Rating of Chessplayers, Past and Present . New York, NY , USA: Arco Publishing, 1978
work page 1978
-
[5]
A. N. Langville and C. D. Meyer, Who’s #1?: The Science of Rating and Ranking . Princeton, NJ, USA: Princeton Univ. Press, 2012
work page 2012
Show all 16 references
-
[6]
Using ELO ratings for match result prediction in association football,
L. M. Hvattum and H. Arntzen, “Using ELO ratings for match result prediction in association football,” Int. J. F orecasting, vol. 26, no. 3, pp. 460–470, 2010. [Online]. Available: https://doi.org/10.1016/j.ijforecast. 2009.10.002
2010 doi
-
[7]
A Comprehensive Guide to Chess Ratings,
M. E. Glickman, “A Comprehensive Guide to Chess Ratings,” American Chess Journal , vol. 3, pp. 59–102, 1995. [Online]. Available: https:// www.glicko.net/research/gj.pdf
1995
-
[9]
Parameter estimation in large dynamic paired com- parison experiments,
M. E. Glickman, “Parameter estimation in large dynamic paired com- parison experiments,” Appl. Statist. , vol. 48, no. 3, pp. 377–394, 1999. [Online]. Available: https://doi.org/10.2307/2986138
1999 doi
-
[11]
Building an NCAA men’s basketball predictive model and quantifying its success,
M. J. Lopez and G. J. Matthews, “Building an NCAA men’s basketball predictive model and quantifying its success,” J. Quant. Anal. Sports , vol. 14, no. 3, pp. 123–134, 2018. [Online]. Available: https://doi.org/ 10.1515/jqas-2017-0042
2018 doi
-
[13]
Example of the Glicko-2 System,
M. E. Glickman, “Example of the Glicko-2 System,” Technical Report, Boston University, 2012. [Online]. Available: https://glicko.net/glicko/ glicko2.pdf
2012
-
[14]
Adam: A Method for Stochastic Optimization,
D. P. Kingma and J. Ba, “Adam: A Method for Stochastic Optimization,” arXiv preprint arXiv:1412.6980 , 2014. [Online]. Available: https://arxiv. org/abs/1412.6980
2014 arXiv
-
[15]
Basketball Dataset
W. Walsh, “Basketball Dataset” [Data set], Kaggle, 2023. [Online]. Available: https://www.kaggle.com/datasets/wyattowalsh/basketball
2023
-
[16]
SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python,
P. Virtanen et al. , “SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python,” Nat. Methods, vol. 17, no. 3, pp. 261–272, 2020. [Online]. Available: https://doi.org/10.1038/s41592-019-0686-2
2020 doi
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.