Pith. sign in

REVIEW 4 major objections 5 minor 16 references

Beyond Winning: Margin of Victory Relative to Expectation Unlocks Accurate Skill Ratings

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Pith's one-line read A margin-minus-expectation update rule for ELO improves NBA skill ratings and converges faster.

desk verdict Clean idea—use the residual between actual and expected margin in an ELO update—but the evaluation as written leaks test information and the convergence metric is tautological, so the empirical claims don't hold. read the letter →

arxiv 2506.00348 v1 pith:OLQX76IW submitted 2025-05-31 stat.ML cs.AIcs.LG

classification stat.MLcs.AIcs.LG
keywords ELOmarginofvictorysportsanalyticsskillratingTrueGlickoNBABayesianinference
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that skill ratings become more accurate when they are updated not just from who won, but from how the actual margin of victory compares with the margin that the pre-match rating gap would predict. The proposed MOVDA method adds a second term to the standard ELO update, proportional to the difference between true margin and an expected margin modeled as a scaled hyperbolic tangent of the rating differential plus a home-advantage shift. Using 13,619 NBA games from 2013 to 2023, the paper reports that MOVDA lowers Brier score by 1.54% relative to TrueSkill, raises outcome accuracy by 0.58 percentage points, and speeds rating convergence by 13.5%, all while keeping ELO's computational cost. A sympathetic reader would care because the method turns margin of victory into a principled, calibrated signal using only four fitted parameters.

What carries the argument

The load-bearing object is the MOV differential $\Delta_{MOV} = T_{MOV} - E_{MOV}$, computed from a scaled hyperbolic-tangent expectation model $E_{MOV}(\Delta R, I_{HA}) = \alpha \tanh(\beta \Delta R) + \gamma + \delta I_{HA}$. It isolates the unexpected component of a team's performance margin, and the algorithm injects it into the rating update as $\lambda \Delta_{MOV}$ alongside the standard ELO term $K(S_A - E_A)$. This lets the rating system absorb performance magnitude while keeping the update zero-sum and nearly as cheap as ELO.

What would settle it

Take the 2013-2023 NBA games in chronological order, fit the EMOV parameters using only the first 70% of games and the ratings ELO produces up to each match, then run MOVDA on the remaining 30% with no further fitting. If the whole-dataset fit leaked test information, the Brier score or convergence gain will shrink relative to the paper's Table I; if the fit was clean, the numbers should match.

Watch

Extended reading notes

Core claim

The paper's central claim is that margin-of-victory information should enter a rating system as a differential: the true margin minus a learned expected margin, rather than as a raw margin-scaled update. For a matchup with rating difference $\Delta R = R_A - R_B$ and home indicator $I_{HA}$, the expected margin is $E_{MOV}(\Delta R, I_{HA}) = \alpha \tanh(\beta \Delta R) + \gamma + \delta I_{HA}$, with the four parameters fit by least squares on historical games. The update becomes $R'_A = R_A + K(S_A - E_A) + \lambda(T_{MOV} - E_{MOV})$, and $R'_B$ moves by the negative of the same amount, preserving ELO's zero-sum structure. On the paper's holdout NBA data, this yields a Brier score of 0.2258 (1.54% below TrueSkill's 0.2294), outcome accuracy of 63.32% (0.58 percentage points above TrueSkill), and convergence in 166 games versus 193 for standard ELO.

Load-bearing premise

The four-parameter expectation curve is fit to ratings produced by standard ELO run on the whole dataset, so the load-bearing premise is that this fit used only training information and stays valid inside the MOVDA update loop; if the test games contributed to the fit, the reported improvements are inflated.

Editorial extensions

If this is right

  • On the paper's holdout NBA data, MOVDA's Brier score of 0.2258 is the lowest among ELO, linear-MOV ELO, Glicko-2, and TrueSkill, meaning its probabilistic win predictions are better calibrated.
  • MOVDA stabilizes a new team's rating in about 166 games, versus 193 for standard ELO and 189 for Glicko-2, so ratings adapt faster to genuine strength changes.
  • Because the update remains zero-sum and requires only four pre-fit parameters plus one hyperparameter $\lambda$, MOVDA can run in real-time rating pipelines at roughly ELO's computational cost.
  • The framework is retrainable: a new season or a new league requires only refitting $(\alpha, \beta, \gamma, \delta)$ and choosing a new $\lambda$, which the paper leaves to future work.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same differential idea could be applied to other scalar performance measures, such as goals, runs, or kill counts, by choosing a domain-appropriate expectation function; the paper demonstrates only point margins in the NBA.
  • Because $\lambda$ controls how much weight the margin surprise carries, one could tune $\lambda$ per team or over time to handle volatility, roster turnover, or tanking behavior; the paper uses a single global $\lambda$.
  • If the expectation curve is refit on a rolling window rather than once on historical data, MOVDA could track slow changes in league-wide scoring environments, such as pace or rule changes, without losing its computational simplicity.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This paper proposes Margin of Victory Differential Analysis (MOVDA), an ELO-style rating update that augments the standard binary outcome error with a term proportional to the difference between the observed margin and an expected margin modeled as E_MOV(ΔR, I_HA) = α·tanh(β·ΔR) + γ + δ·I_HA. The four parameters are fit on NBA data, the hyperparameter λ is tuned, and the method is compared with standard ELO, linear MOV-scaled ELO, Glicko-2, and TrueSkill on a chronological split of 13,619 NBA games. The paper reports a 1.54% lower Brier score than TrueSkill, 0.58 percentage points higher outcome accuracy, and 13.5% faster rating convergence, all at computational cost comparable to ELO. The update rule is simple and the zero-sum structure is preserved, but the empirical validation has several load-bearing problems: the convergence metric is self-referential, the fitting protocol for the expectation model is ambiguous and potentially leaky, the train/validation/test split and hyperparameter tuning are not described precisely, and no uncertainty quantification is provided for the small reported differences.

Significance. If the empirical claims were cleanly established, MOVDA would be a practically useful contribution: it offers a transparent, computationally cheap way to inject margin information into ELO via a learned residual, and the saturating tanh form for expected margin with a home-advantage term is a reasonable modeling choice. The paper also deserves credit for a simple ablation showing that the margin differential term changes the results. However, the most emphasized contribution—faster convergence—is not supported by the reported metric, and the predictive improvements are small and presented without confidence intervals. The manuscript therefore currently establishes a plausible idea rather than a verified result, and the experiments need to be rerun or substantially clarified before the claims can be accepted.

major comments (4)
  1. [Section IV-C] The convergence metric is self-referential: for each simulated new team, the target band is centered on the final stable rating computed from the final 200 games of the same test-set trajectory that is being evaluated. A system that never updates its ratings would satisfy the band condition immediately, because its constant rating equals its own final rating. Consequently, the comparison in Table I (MOVDA 166 games vs. Standard ELO 193 games) does not measure convergence to true skill; it measures when a trajectory settles near its own tail average. The 13.5% convergence improvement claimed in the abstract and conclusion is therefore unsupported. Please replace this metric with an external ground truth (for example, end-of-season strength measures or rolling predictive accuracy) or explicitly reframe the claim as self-consistency rather than convergence.
  2. [Section IV-D] The EMOV parameters are said to be estimated on the training data 'using ratings generated by standard ELO run on the whole dataset.' This contradicts Section III-B, which states that the ratings used for estimation are derived from processing the training data chronologically 'up to match i.' If 'the whole dataset' includes the testing and holdout partitions, then Eq. (6) uses future information to fit the expectation model, and all reported out-of-sample improvements are inflated. Please state exactly which games contribute to the ELO run used for the fit, and if the current wording is leaky, re-estimate the parameters using only information available up to each training match.
  3. [Sections IV-A and IV-D] The evaluation protocol is ambiguous. The data are described as split into 'training (first 70%) and testing (second 20%) and hold out test (remaining 10%),' but Table I is described as being on the 'hold out set' while Section IV-E refers to results 'on the test sets.' Hyperparameters are 'tuned on the same data set by optimizing for the Brier score' without specifying whether that set is a validation partition or the final test partition. If tuning used the final test set, the Brier and accuracy results are not honest out-of-sample estimates. Please report the exact partition boundaries, the tuning set, and the selected hyperparameter values for every model.
  4. [Table I] No confidence intervals or significance tests are reported for the Brier-score or accuracy differences. Since some differences are very small (Brier 0.2258 vs. 0.2264 for Glicko-2, accuracy 63.32% vs. 63.18% for Linear MOV ELO), the statement that MOVDA 'significantly outperforms' the baselines is not statistically supported as presented. Please provide error bars, paired tests, or at least a sensitivity analysis over the chronological split.
minor comments (5)
  1. [Abstract, Section IV-E, Section VI] Numerical claims are inconsistent: the abstract and conclusion report 1.54% Brier improvement and 13.5% faster convergence, while Section IV-E reports a 0.66% Brier improvement over standard ELO and 13.9% fewer games than standard ELO; the accuracy gain in Table I is 0.66 percentage points over TrueSkill, not 0.58. Please harmonize all reported percentages.
  2. [Section IV-E] The text says the results are shown 'across the four domains on the test sets,' but the paper reports only NBA data. Either add the missing domains or correct the sentence.
  3. [Figure 2] The caption describes the MOV distribution as approximately normal and centered near zero, while the text notes clear positive skew for home teams and negative skew for away teams. Please reconcile the caption with the distributional limitation acknowledged in Section V.
  4. [References] Reference [8] is a neuroimaging study on developmental coordination disorder and does not support the basketball MOV K-factor scaling claim; please replace it with the intended citation.
  5. [Throughout] There are minor typographical errors, including 'datset' in Section IV-A, 'Scale hyperbolic tangent' in Section V, and 'the computational efficiency comparable' in Section V.

Circularity Check

2 steps flagged · score 6.0 of 10

Convergence metric is self-referential: the ±20-point band is centered on the run's own final rating, so the headline 13.5% acceleration does not measure convergence to true skill; separately, the EMOV expectation is fitted using ELO ratings 'run on the whole dataset', risking leakage of evaluation-period outcomes into the fitted curve.

  1. self definitional [Section IV-C, Evaluation Metrics (Convergence Speed); interpreted in the Abstract and Section V]
    "Convergence Speed: Measured by tracking hypothetical new teams introduced at the start of the test set with the default rating (1500). We calculated the average number of games required for such a team's rating to enter and remain continuously within a predefined band (e.g., ± 20 rating points) around its final stable rating over the final 200 games of the test set."

    The metric's target is the final rating of the very trajectory being evaluated: a team 'converges' when it enters a band centered on its own final-200-game average, a quantity that is produced by the same run. A system that never updated its ratings would score zero games, because a constant rating equals its own final rating, so the metric measures self-agreement rather than agreement with any external or true skill. The paper nonetheless interprets the result as faster adaptation to team strength ('most importantly accelerates rating convergence by 13.5%' in the Abstract; 'ratings likely reflect current team strength more quickly' in Section V). Table I's comparison (MOVDA 166 games vs.

  2. fitted input called prediction [Section IV-D, Implementation Details; contrast with Section III-B]
    "The EMOV parameters (α, β, γ, δ) for MOVDA were estimated using non-linear least squares implemented with SciPy's 'curve fit' on the training data, using ratings generated by standard ELO run on the whole dataset."

    Section III-B specifies that the ratings entering the EMOV fit are 'derived from processing the training data chronologically using a baseline rating system ... up to match i', i.e., training information only. Section IV-D instead says the fit uses ratings from standard ELO 'run on the whole dataset'. If 'the whole dataset' includes the testing and holdout periods, then the ELO ratings — and hence the fitted EMOV of Eq. 4 — embed outcomes of the very games later used to claim out-of-sample improvements. Because EMOV enters every MOVDA update (Algorithm 1, lines 3–5) and every margin and probability evaluation, the reported 1.54% lower Brier score and 0.58% higher accuracy would be partially in-sample rather than predictions made from training information alone.

full rationale

The paper's empirical program splits into two families. The Brier score, outcome accuracy, and margin MAE comparisons are computed from pre-match ratings and do not reduce by construction: the EMOV parameters are fitted once by nonlinear least squares, the update rule (Eq. 8) is a genuine combination of ELO's outcome term and a margin residual, and the baselines (TrueSkill, Glicko-2) are external systems. Those comparisons are not circular. The convergence claim, however, is self-referential: the ±20-point band is centered on the same run's final-200-game average rating, so the metric is defined in terms of the trajectory it purports to validate, and any constant-rating system would trivially achieve zero games. Table I's 166 versus 193 games therefore does not establish faster adaptation to true team strength. Separately, Section IV-D says EMOV was fitted using ratings from standard ELO 'run on the whole dataset', which contradicts Section III-B's chronological-training description; if the whole dataset includes testing and holdout games, the fitted expectation embeds outcomes of the games used to claim out-of-sample gains. The ambiguity is load-bearing and unresolved, so the out-of-sample status of the Brier and accuracy numbers is unverified. Overall, partial circularity is concentrated in the headline convergence metric, with a genuine leakage risk in the expectation fit; the underlying Brier and accuracy machinery is not itself circular.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The central method rests on four fitted curve parameters plus a tuned lambda and an unreported K, two strong distributional and functional assumptions, and the assumption that ELO-derived expectations transfer into the MOVDA loop.

free parameters (6)
  • alpha
    Scale of the tanh expected-margin curve (Eq. 4), estimated by nonlinear least squares on NBA training data (Eq. 6).
  • beta
    Steepness of the tanh expected-margin curve (Eq. 4), estimated by nonlinear least squares on NBA training data (Eq. 6).
  • gamma
    Baseline offset in expected margin (Eq. 4), estimated by nonlinear least squares on NBA training data (Eq. 6).
  • delta
    Home advantage adjustment in expected margin (Eq. 4), estimated by nonlinear least squares on NBA training data (Eq. 6).
  • lambda
    Weight of the MOV differential term in Eq. 8, tuned by grid search over [0.1, 3.0] optimizing Brier score; the selected value is not reported.
  • K
    ELO K-factor for the outcome-based update term; the value used is not reported in the paper.
assumptions (5)
  • domain assumption Observed margins are approximately normally distributed around the expected margin (Eq. 5).
    Used to justify least-squares estimation; the paper acknowledges skewness in actual NBA margins (Section V).
  • ad hoc to paper The scaled tanh function is the correct functional form for expected margin (Eq. 4).
    Chosen for monotonicity, symmetry, and saturation; no derivation from a generative model of basketball scoring.
  • domain assumption An expectation model fit to standard ELO ratings remains valid inside the MOVDA update loop.
    EMOV is estimated using ratings from standard ELO, then used to compute residuals for MOVDA updates; the paper does not justify this transfer.
  • ad hoc to paper Home advantage enters only as an additive delta * I_HA term.
    Other contextual factors such as injuries, rest, and travel are excluded (Section V), yet treated as negligible in the expectation model.
  • domain assumption The Kaggle NBA dataset is accurate, complete, and correctly ordered chronologically.
    All results depend on this external data source; no validation of scores, team identities, or home status is reported.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Beyond Winning: Margin of Victory Relative to Expectation Unlocks Accurate Skill Ratings." pith.science (2026). https://pith.science/paper/OLQX76IW

@misc{pith2026250600348,
  author       = {Pith},
  title        = {Pith review of: Beyond Winning: Margin of Victory Relative to Expectation Unlocks Accurate Skill Ratings},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OLQX76IW}},
  note         = {Machine review of arXiv:2506.00348}
}
abstract

Knowledge of accurate relative skills in any competitive system is essential, but foundational approaches such as ELO discard extremely relevant performance data by concentrating exclusively on binary outcomes. While margin of victory (MOV) extensions exist, they often lack a definitive method for incorporating this information. We introduce Margin of Victory Differential Analysis (MOVDA), a framework that enhances traditional rating systems by using the deviation between the true MOV and a $\textit{modeled expectation}$. MOVDA learns a domain-specific, non-linear function (a scaled hyperbolic tangent that captures saturation effects and home advantage) to predict expected MOV based on rating differentials. Crucially, the $\textit{difference}$ between the true and expected MOV provides a subtle and weighted signal for rating updates, highlighting informative deviations in all levels of contests. Extensive experiments on professional NBA basketball data (from 2013 to 2023, with 13,619 games) show that MOVDA significantly outperforms standard ELO and Bayesian baselines. MOVDA reduces Brier score prediction error by $1.54\%$ compared to TrueSkill, increases outcome accuracy by $0.58\%$, and most importantly accelerates rating convergence by $13.5\%$, while maintaining the computational efficiency of the original ELO updates. MOVDA offers a theoretically motivated, empirically superior, and computationally lean approach to integrating performance magnitude into skill rating for competitive environments like the NBA.

Figures

Figures reproduced from arXiv: 2506.00348 by the authors.

Figure 1
Figure 1. Comparison of the actual and fitted margin of victory [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Distribution of margin of victory (MOV) for (a) home [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Comparison of (a) rating convergence speed and (b) [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

16 extracted references · 11 canonical work pages

  1. [8]

    Cerebellar and Brainstem Differences in Children with Developmental Coordination Disorder: A V oxel-Based Morphometry Study,

    K. K. Gill, D. Lang, and J. G. Zwicker, “Cerebellar and Brainstem Differences in Children with Developmental Coordination Disorder: A V oxel-Based Morphometry Study,” Front. Hum. Neurosci. , vol. 16, p. 921505, 2022. [Online]. Available: https://doi.org/10.3389/fnhum.2022. 921505

  2. [10]

    Default Probability,

    D. N. Osherson, J. Stern, O. Wilkie, M. Stob, and E. E. Smith, “Default Probability,” Cogn. Sci. , vol. 15, no. 2, pp. 251–269, 1991. [Online]. Available: https://doi.org/10.1207/s15516709cog1502 3

  3. [12]

    Reinforcement Learning in Dynamic Environments,

    Y . Sismanis, “Reinforcement Learning in Dynamic Environments,” in Proc. 2010 Int. Conf. Machine Learning , pp. 123–130, 2010. [Online]. Available: https://doi.org/10.1145/1234567.1234568

  4. [1]

    The Methodology of Officially Recognized International Sports Rating Systems,

    R. Stefani, “The Methodology of Officially Recognized International Sports Rating Systems,” J. Quant. Anal. Sports , vol. 7, no. 4, 2011. [Online]. Available: https://doi.org/10.2202/1559-0410.1347

  5. [2]

    TrueSkill™: A Bayesian Skill Rating System,

    R. Herbrich, T. Minka, and T. Graepel, “TrueSkill™: A Bayesian Skill Rating System,” in Advances in Neural Information Processing Systems , vol. 19, pp. 569–576, 2007. [Online]. Available: https://papers.nips.cc/ paper/3079-trueskilltm-a-bayesian-skill-rating-system

  6. [3]

    Whole-History Rating: A Bayesian Rating System for Players of Time-Varying Strength,

    R. Coulom, “Whole-History Rating: A Bayesian Rating System for Players of Time-Varying Strength,” in Proc. 6th Int. Conf. Computers and Games (CG 2008) , Lecture Notes in Computer Science, vol. 5131, pp. 113–124. Springer, 2008. [Online]. Available: https://doi.org/10. 1007/978-3-540-87608-3 11

  7. [4]

    A. E. Elo, The Rating of Chessplayers, Past and Present . New York, NY , USA: Arco Publishing, 1978

  8. [5]

    A. N. Langville and C. D. Meyer, Who’s #1?: The Science of Rating and Ranking . Princeton, NJ, USA: Princeton Univ. Press, 2012

Show all 16 references
  1. [6]

    Using ELO ratings for match result prediction in association football,

    L. M. Hvattum and H. Arntzen, “Using ELO ratings for match result prediction in association football,” Int. J. F orecasting, vol. 26, no. 3, pp. 460–470, 2010. [Online]. Available: https://doi.org/10.1016/j.ijforecast. 2009.10.002

  2. [7]

    A Comprehensive Guide to Chess Ratings,

    M. E. Glickman, “A Comprehensive Guide to Chess Ratings,” American Chess Journal , vol. 3, pp. 59–102, 1995. [Online]. Available: https:// www.glicko.net/research/gj.pdf

  3. [9]

    Parameter estimation in large dynamic paired com- parison experiments,

    M. E. Glickman, “Parameter estimation in large dynamic paired com- parison experiments,” Appl. Statist. , vol. 48, no. 3, pp. 377–394, 1999. [Online]. Available: https://doi.org/10.2307/2986138

  4. [11]

    Building an NCAA men’s basketball predictive model and quantifying its success,

    M. J. Lopez and G. J. Matthews, “Building an NCAA men’s basketball predictive model and quantifying its success,” J. Quant. Anal. Sports , vol. 14, no. 3, pp. 123–134, 2018. [Online]. Available: https://doi.org/ 10.1515/jqas-2017-0042

  5. [13]

    Example of the Glicko-2 System,

    M. E. Glickman, “Example of the Glicko-2 System,” Technical Report, Boston University, 2012. [Online]. Available: https://glicko.net/glicko/ glicko2.pdf

  6. [14]

    Adam: A Method for Stochastic Optimization,

    D. P. Kingma and J. Ba, “Adam: A Method for Stochastic Optimization,” arXiv preprint arXiv:1412.6980 , 2014. [Online]. Available: https://arxiv. org/abs/1412.6980

  7. [15]

    Basketball Dataset

    W. Walsh, “Basketball Dataset” [Data set], Kaggle, 2023. [Online]. Available: https://www.kaggle.com/datasets/wyattowalsh/basketball

  8. [16]

    SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python,

    P. Virtanen et al. , “SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python,” Nat. Methods, vol. 17, no. 3, pp. 261–272, 2020. [Online]. Available: https://doi.org/10.1038/s41592-019-0686-2

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.