Rating competitors in games with strength-dependent tie probabilities

Mark E. Glickman

REVIEW 3 major objections 4 minor 26 references

Rating competitors in games with strength-dependent tie probabilities

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that draw probability can be modeled as increasing with average player strength, and that a one-step Newton-Raphson filter makes the resulting Bayesian rating updates practical.

desk verdict A transparent, useful account of a deployed rating system, but the implemented algorithm quietly departs from the stated model; worth refereeing for the rating community. read the letter →

arxiv 2506.11354 v2 pith:54DR6FEI submitted 2025-06-12 stat.ME

Mark E. Glickman This is my paper

classification stat.ME

keywords BayesiandynamicgeneralizedlinearmodelBradley-Terrychesstournamentspairedcomparisonmodelsrankingtieoutcomesstrength-dependentdrawprobabilityNewton-Raphsonapproximation

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper develops a rating system for games that can end in a win, loss, or draw, in which the chance of a draw is allowed to grow with the average strength of the two players. It extends an existing paired-comparison model into a Bayesian dynamic framework where each player's latent strength follows a normal random walk across rating periods. To keep updates fast enough for large federations, the posterior is approximated by a single Newton-Raphson step evaluated at the prior mean, with opponent strengths integrated out by Gauss-Hermite quadrature. The system is tested on hundreds of thousands of correspondence-chess games and, with adjusted hyperparameters, was adopted for official ratings in 2023. If the approximation is accurate, the method offers a scalable way to make drawn results informative, especially for elite players whose games are mostly draws.

What carries the argument

The load-bearing object is the strength-dependent multinomial outcome model of Eq. (3), together with the one-step Newton-Raphson filter of Eqs. (22)-(23). The draw exponent $\beta_0 + (1+\beta_1)(\theta_i+\theta_j)/2$ is what carries the new information: it lets a draw between two strong players be less surprising, and therefore less rating-neutral, than a draw between weak players. The filter approximates the posterior of each player by a normal density whose mean is one Newton-Raphson step from the prior mean and whose variance is the reciprocal of the log-posterior curvature at that point; opponents' strengths are entered through their prior means and variances, with the resulting integrals handled by two-point Gauss-Hermite quadrature. An implemented detail resets the draw score to $1/2$ when $\beta_1\neq0$ so that equal-strength players who draw do not both gain points.

What would settle it

Simulate a rating period in which some players play ten or more games, then compare the one-step Newton-Raphson posterior means and variances with those from full Bayesian MCMC or high-order quadrature; if the discrepancies grow with the number of games per period and materially alter ratings, the approximation that carries the system is false.

Watch

Extended reading notes

Core claim

The central claim is that a rating system can explicitly model a tie as a strength-dependent outcome and still be updated in real time. The paper proposes a multinomial outcome model in which the draw probability is proportional to $\exp\{\beta_0 + (1+\beta_1)(\theta_i+\theta_j)/2\}$, so that a positive $\beta_1$ makes draws increasingly likely as the average latent strength rises; the win and loss probabilities are shaped similarly, with a possible white-piece advantage. Player strengths evolve as a normal random walk between periods. After replacing each opponent's posterior by her prior and approximating the required integrals with two-point Gauss-Hermite quadrature, the paper derives closed-form one-step Newton-Raphson updates for each player's posterior mean and variance. It validates these updates against higher-order quadrature on every game in a final validation period and reports close agreement, and it describes the hyperparameter values that were ultimately implemented in a federation rating system.

Load-bearing premise

The whole system leans on the assumption that one quick numerical correction, started from the pre-period estimate, stays accurate even for a player who plays many games in one rating period; the paper only checks this correction game by game.

Editorial extensions

If this is right

If the central claim is correct, a federation can update thousands of players' ratings in parallel, because each player's posterior update depends only on her own prior and her opponents' priors.
Explicitly modeling strength-dependent draws should reduce rating stagnation at the top: a draw between two 2500-level players is treated as a highly expected result rather than as half a win.
The hyperparameters (draw baseline, draw-strength interaction, and innovation volatility) can be chosen by maximizing one-step-ahead predictive likelihood, giving a principled calibration procedure.
The implemented system marks the first operational rating algorithm of a major correspondence-chess body that treats draws as a distinct strength-dependent outcome.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The same single-step filtering logic should transfer to other settings with skill-dependent ties, such as certain online games or penalty shootouts, whenever a multinomial outcome model can be written down.
The paper validates the normal approximation only on one game at a time; testing it on players with many games within a single rating period, against MCMC or high-order quadrature, is a natural next check.
The 'quasi-optimized' parameter choice suggests that predictive accuracy alone may not define a usable rating scale; the paper's account of why the optimized parameters were rejected invites further study of how rating scales are made acceptable to users.
One could test the model's core assumption directly on other large game datasets by fitting the same smooth of average strength against draw log-odds and checking whether the positive slope replicates.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Mark, this paper is worth your time if you care about rating systems. Glickman develops a dynamic Bayesian rating system for games with draws that depend on player strength, and it has been running at the ICCF since 2023. The genuinely new piece is the filtering approximation: a one-step Newton-Raphson update in a multinomial logit model, with a two-point Gauss-Hermite quadrature to integrate out opponents. That is not in the earlier literature. The math in Sections 4.1-4.3 and Appendix A is coherent; Appendix B checks the normal approximation against Gauss-Hermite quadrature on 17,414 real games and the agreement is good, at least for single-game updates. The paper is also unusually candid about the compromises made for deployment.

The soft spot is real: the deployed algorithm does not actually implement the model in Section 3. In Appendix A, the draw score a_d is overridden from (1+beta_1)/2 to 1/2. That means the Newton-Raphson update no longer targets the posterior of the stated model; it optimizes a different objective. The paper explains why (draws between equal players would otherwise inflate ratings) but does not re-derive the update under the override or show that it corresponds to any coherent statistical model. So the central claim in the abstract that the ICCF-deployed system is a valid instantiation of this approach is too strong. This is not fatal—the system is explicit about what it does—but it needs to be acknowledged in the conclusions.

A second issue is that hyperparameters are 'quasi-optimized' for face validity after seeing the validation data, and Figure 5 uses those same parameters. The paper admits this, but it means the predictive validity evidence is partly circular. There is also no comparison to Elo or Glicko on predictive performance, which would be the natural benchmark.

Proportionately: the filtering approximation is likely to be accurate under the stated model, and the implementation details are all on paper. The paper is honest about its departures. For the rating-systems subfield, this is a significant contribution—a deployed system with strength-dependent draws—and the derivations are solid enough to take seriously. I would send it to a referee, provided the referee is asked to weigh the a_d override and the absence of a baseline comparison. It deserves a revision rather than a rejection.

Referee Report

3 major / 4 minor

Summary. The paper proposes a Bayesian dynamic rating system for head-to-head games in which the outcome model of Eq. (3) allows the probability of a draw and the advantage of playing white to depend on the average strength of the two players. The algorithm approximates each period's posterior by replacing opponent posteriors with opponent priors, applying two-point Gauss-Hermite quadrature over each opponent's strength, and performing a one-step Newton-Raphson update from the prior mean, as given in Eqs. (22)-(23). Hyperparameters are selected either by maximizing predictive likelihood or by quasi-optimized choices made for practical acceptability. The method is demonstrated on 392,658 ICCF games, and the paper states that the ICCF adopted the system in 2023. The central claim is that the fast one-step approximation is sufficiently accurate for large-scale rating updates and that the deployed system is a valid instantiation of the Glickman (2025) model.

Significance. If the approximation is valid, the paper addresses a real gap in rating methodology: strength-dependent tie probabilities matter in correspondence chess, and the proposed model is a natural extension of existing Bradley-Terry-type frameworks. The analytic derivative computations in Appendix A are coherent, and the comparison with Gauss-Hermite quadrature in Appendix B is a useful sanity check. The paper also demonstrates a serious large-scale application. However, the deployed algorithm deviates from the stated model in a way that is disclosed but not analyzed, and the numerical validation does not cover the cumulative multi-game regime actually used. These gaps affect the central claims and need to be resolved before the paper can be accepted.

major comments (3)

[Appendix A, paragraph after Eq. (37)] The implemented system overrides the draw score from a_d = (1+beta_1)/2 to exactly 1/2. Because a_d enters the gradient and Hessian of the log-posterior through the expressions in Eqs. (29)-(30), the updates in Eqs. (36)-(37) are no longer the Newton-Raphson approximation to the posterior derived from Eq. (19). The paper neither re-derives the algorithm under this override nor shows that the modified updates correspond to a coherent statistical model. This is not a purely cosmetic issue: the abstract and Section 5 claim that the deployed system is based on the Glickman (2025) model, but the actual deployed update optimizes a different objective. The authors should either present the implemented algorithm as a separate heuristic variant, derive its properties, or remove the override from the reported implementation.
[Appendix B, Table 3] The validation of the one-step Newton-Raphson approximation covers only single-game updates, one game at a time, for the 17,414 validation games. The actual filtering procedure in Section 4.3 processes all games in a rating period simultaneously through the sums in Eqs. (21)-(23). When a player has many games in one period, the posterior mode can be far from the prior mean, and a single Newton step evaluated at mu_it may be inaccurate. The paper does not test the cumulative multi-game-per-period setting or quantify the resulting bias in posterior means and variances. Since the claimed adequacy of the fast approximation is load-bearing for the whole method, this gap should be addressed, for example by simulating periods with dense schedules or by comparing sequential multi-game updates against a more accurate benchmark.
[Section 5.2 and Figure 5] The quasi-optimized hyperparameters in Table 2 were chosen partly by inspecting the ratings they produce on the ICCF data, specifically to keep top ratings below 3000 and to adjust draw-related rating changes. Figure 5 then reports predictive summaries on the five validation periods using those same parameters. This is not an out-of-sample validation: the parameters were selected with knowledge of the validation-period outcomes and the resulting ratings. The paper should either use a genuinely held-out period for the validation reported in Figure 5 or explicitly describe Figure 5 as a calibration check rather than as independent evidence of predictive validity.

minor comments (4)

[Section 4.2] The sentence 'it is important to retain both computational tractability' appears to be missing the second item in the pair; it should say something like 'retain computational tractability and analytical convenience.'
[Appendix A, Eqs. (34)-(35)] The displayed formulas contain an extra closing parenthesis, for example 'P+wj(aw-s+1j))/Pj' should be 'P+wj(aw-s+1j)/Pj.'
[Table 3] The header 'Log Std Dev comparions' contains a typo, and the row label 'Games, mu_1 <= 3.43' should read 'All games, mu_1 <= 3.43' for consistency with the other strength-tercile rows.
[Section 4.3, Eq. (17)] A brief explanation that the two-point Gauss-Hermite rule with nodes at mu +/- sigma follows from the standard nodes +/-1/sqrt(2) would help readers not familiar with the change of variables.

Circularity Check

1 steps flagged · score 4.0 of 10

One moderate circularity: the Figure 5 validation uses the same five-period window that shaped the quasi-optimized parameters; the core model derivation is otherwise self-contained.

fitted input called prediction [Section 5.1-5.2 (hyperparameter tuning and Figure 5 validation)]
"The quasi-optimized hyperparameter values in Table 2 yield ratings with strong predictive validity, as demonstrated in Figure 5. The analysis is based on the five validation periods of ICCF game results."

These same five periods were used in Section 5.1 to optimize the model hyperparameters ('the remaining 5 periods were used to evaluate the predictive log-likelihood'), and Section 5.2 explains that Table 2's quasi-optimized values were adopted by balancing predictive accuracy against the practical acceptance of the resulting ratings. Figure 5 is then presented as validation of predictions made with those very parameters. Because the parameter values were chosen with this validation window in view, the draw-probability separation and calibration shown in Figure 5 are partly a restatement of the tuning choices rather than an independent out-of-sample prediction.

full rationale

The central derivation chain—Eq. (3) model, Eq. (16) marginalization, Eqs. (21)-(23) one-step Newton-Raphson updates, and Appendix A's computational details—is self-contained. The model is explicitly defined in Eq. (3), so citing Glickman (2025) for 'further details' is not load-bearing; no uniqueness theorem or hidden ansatz is imported from the companion paper. Appendix B honestly compares the normal approximation to Gauss-Hermite quadrature on 17,414 single-game updates, which is an independent numerical check of the approximation rather than a circular prediction. The one real circularity concern is the validation of quasi-optimized parameters on the same five-period window used for tuning; that makes Figure 5's claims of predictive validity less clean. Separately, Appendix A's disclosure that the implemented system overrides a_d=(1+beta_1)/2 to exactly 1/2 means the deployed updates do not exactly implement the stated model, but this is an internal-consistency or correctness issue, not a circularity: the override is disclosed and not disguised as a prediction. Overall, the paper's core derivation does not reduce to its inputs; the evidence is weakened only by the in-sample-tuned validation, giving a moderate circularity score.

Assumptions & free parameters 6 free parameters · 6 assumptions · 0 invented entities

The central claim rests on a companion model by the same author, a normal random walk, and several numerical approximations. The free parameters in Tables 1 and 2 are fitted and then adjusted by hand, and the a_d override means the deployed algorithm is not exactly the stated Bayesian model. No invented entities are needed.

free parameters (6)

beta_0 (baseline draw intercept) = 0.35338 optimized; 1.09861 quasi-optimized
Controls overall draw frequency; fitted by maximizing predictive log-likelihood and then adjusted by hand in the quasi-optimized version.
beta_1 (strength dependence of draw probability) = 0.57041 optimized; 0.17037 quasi-optimized
Controls how draw probability increases with average strength; fitted to ICCF data and then reduced for the deployed system.
tau (innovation standard deviation) = 0.46040 optimized; 0.14391 quasi-optimized
Random walk step size for strength evolution; fitted, then reduced to 25 Elo points for stability.
alpha_0 and alpha_1 (white advantage parameters) = set to 0 by ICCF decision
Part of the general model (3) but fixed at zero in the implemented version, removing order effects.
Uncertainty cap for prior standard deviation = 0.691 on latent scale (120 Elo points)
Ad hoc rule: posterior SD above threshold is carried forward unchanged instead of adding tau^2; chosen for stability.
Initial prior standard deviations = 0.576 for rated players, 1.439 for unrated players
Chosen to reflect 100 and 250 Elo uncertainty; author says reasonable variations had minimal impact.

assumptions (6)

domain assumption Outcome model (3) from Glickman (2025) is the true data-generating process for game outcomes.
The whole rating system is built on this multinomial logit parameterization; Section 3 states 'Our starting point is the model proposed by Glickman (2025)'.
domain assumption Player strengths follow a normal random walk with constant innovation variance tau^2 (Eq. 5).
Assumed in Section 3 to model ability change over time.
ad hoc to paper Opponent posterior can be replaced by opponent prior (Eq. 13).
Section 4.1: 'we begin by replacing the posterior distributions of player i's opponents with their corresponding prior distributions'; justified only by well-informed prior estimates and relatively few games.
standard math 2-point Gauss-Hermite quadrature is accurate for the opponent integrals in (16).
Section 4.2 and Appendix B compare against 9-point quadrature; the approximation is a numerical choice, not a proven bound.
ad hoc to paper The implemented update should treat a draw as exactly half a point (a_d = 1/2), overriding the model's (1 + beta_1)/2.
Appendix A: 'the implemented system overrides the default value of a_d, setting it to exactly 1/2' so that equal players drawing do not both increase.
ad hoc to paper One Newton-Raphson step from the prior mean gives an adequate normal posterior approximation.
Section 4.3, Eqs. (22)-(23); validated only for single-game updates in Appendix B.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Rating competitors in games with strength-dependent tie probabilities." pith.science (2026). https://pith.science/paper/54DR6FEI

@misc{pith2026250611354,
  author       = {Pith},
  title        = {Pith review of: Rating competitors in games with strength-dependent tie probabilities},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/54DR6FEI}},
  note         = {Machine review of arXiv:2506.11354}
}

read the original abstract

Competitor rating systems for head-to-head games are typically used to measure playing strength from game outcomes. Ratings computed from these systems are often used to select top competitors for elite events, for pairing players of similar strength in online gaming, and for players to track their own strength over time. Most implemented rating systems assume only win/loss outcomes, and treat occurrences of ties as the equivalent to half a win and half a loss. However, in games such as chess, the probability of a tie (draw) is demonstrably higher for stronger players than for weaker players, so that rating systems ignoring this aspect of game results may produce strength estimates that are unreliable. We develop a new rating system for head-to-head games based on a model by Glickman (2025) that explicitly acknowledges that a tie may depend on the strengths of the competitors. The approach uses a Bayesian dynamic modeling framework. Within each time period, posterior updates are computed in closed form using a single Newton-Raphson iteration evaluated at the prior mean. The approach is demonstrated on a large dataset of chess games played in International Correspondence Chess Federation tournaments.

Figures

Figures reproduced from arXiv: 2506.11354 by the authors.

**Figure 2.** Each panel shows the rating change resulting from a win (blue), draw (green), or [PITH_FULL_IMAGE:figures/full_fig_p040_2.png] view at source ↗

**Figure 3.** Comparison of new ratings in March 2022 based on the optimized hyperparameters [PITH_FULL_IMAGE:figures/full_fig_p041_3.png] view at source ↗

Figures from the paper (2 more)

**Figure 4.** Figure 4: Comparison of new ratings in March 2022 using hyperparameters from Table 2 [PITH_FULL_IMAGE:figures/full_fig_p042_4.png]

**Figure 5.** Figure 5: Top panel: boxplots of predicted draw probabilities for the 17,414 games in the [PITH_FULL_IMAGE:figures/full_fig_p043_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

26 extracted references · 24 canonical work pages

[1]

Bhat, C. R. (1995). A heteroscedastic extreme value model of intercity travel mode choice. Transportation Research Part B: Methodological , 29(6):471--483

work page 1995
[2]

and Dunsmore, I

Boys, R. and Dunsmore, I. (1987). Diagnostic and sampling models in screening. Biometrika , 74(2):365--374

work page 1987
[3]

Bradley, R. A. and Terry, M. E. (1952). Rank analysis of incomplete block designs: I . The method of paired comparisons. Biometrika , pages 324--345

work page 1952
[4]

Crouch, E. A. and Spiegelman, D. (1990). The evaluation of integrals of the form _ - ^ f(t) (-t^2) : Application to logistic-normal models. Journal of the American Statistical Association , 85(410):464--469

work page 1990
[5]

David, H. (1988). The method of paired comparisons . Charles Griffin & Company, London

work page 1988
[6]

Davidson, R. R. (1970). On Extending the Bradley - Terry Model to Accommodate Ties in Paired Comparison Experiments . Journal of the American Statistical Association , 65(329):317--328

work page 1970
[7]

and Koopman, S

Durbin, J. and Koopman, S. J. (2012). Time series analysis by state space methods , volume 38. OUP Oxford

work page 2012
[8]

Elo, A. E. (1978). The rating of chess players, past and present . Arco Publishing, New York

work page 1978

Show all 26 references

[9]

and Tutz, G

Fahrmeir, L. and Tutz, G. (1994). Dynamic stochastic models for time-dependent ordered paired comparison systems. Journal of the American Statistical Association , 89(428):1438--1449

1994
[10]

Glickman, M. E. (1993). Paired comparison models with time-varying parameters . Harvard University

1993
[11]

Glickman, M. E. (1999). Parameter estimation in large dynamic paired comparison experiments. Journal of the Royal Statistical Society Series C: Applied Statistics , 48(3):377--394

1999
[12]

Glickman, M. E. (2001). Dynamic paired comparison models with stochastic variances. Journal of Applied Statistics , 28(6):673--689

2001
[13]

Glickman, M. E. (2025). Paired comparison models with strength-dependent ties and order effects. arXiv:2505.24783

2025 arXiv
[14]

Glickman, M. E. and Jones, A. C. (2024). Models and rating systems for head-to-head competition. Annual Review of Statistics and Its Application , 12:259--282

2024
[15]

J., and Lit, R

Gorgi, P., Koopman, S. J., and Lit, R. (2019). The analysis and forecasting of tennis matches by using a high dimensional dynamic model. Journal of the Royal Statistical Society Series A: Statistics in Society , 182(4):1393--1409

2019
[16]

Harding, M. C. and Hausman, J. (2007). Using a L aplace approximation to estimate the random coefficients logit model by nonlinear least squares. International Economic Review , 48(4):1311--1328

2007
[17]

and Tibshirani, R

Hastie, T. and Tibshirani, R. (1986). Generalized additive models. Statistical science , 1(3):297--310

1986
[18]

Ingram, M. (2021). How to extend E lo: a B ayesian perspective. Journal of Quantitative Analysis in Sports , 17(3):203--219

2021
[19]

Knorr-Held, L. (2000). Dynamic rating of sports teams. Journal of the Royal Statistical Society: Series D (The Statistician) , 49(2):261--276

2000
[20]

Nelder, J. A. and Mead, R. (1965). A simplex method for function minimization. The Computer Journal , 7(4):308--313

1965
[21]

Pirjol, D. (2013). The logistic-normal integral and its generalizations. Journal of Computational and Applied Mathematics , 237(1):460--469

2013
[22]

and Zigova, K

Pryanishnikov, I. and Zigova, K. (2016). Multinomial logit models for the austrian labor market. Austrian Journal of Statistics , 32(4):267--282

2016
[23]

Steen, N., Byrne, G., and Gelbard, E. (1969). Gaussian quadratures for the integrals _0^ e^ -x^2 f(x) dx and _0^b e^ -x^2 f(x) dx . Mathematics of Computation , 23(107):661--671

1969
[24]

and Djebbi, A

Szczecinski, L. and Djebbi, A. (2020). Understanding draws in E lo rating algorithm. Journal of Quantitative Analysis in Sports , 16(3):211--220

2020
[25]

J., and Migon, H

West, M., Harrison, P. J., and Migon, H. S. (1985). Dynamic generalized linear models and B ayesian forecasting. Journal of the American Statistical Association , 80(389):73--83

1985
[26]

Wood, S. N. (2011). Fast stable restricted maximum likelihood and marginal likelihood estimation of semiparametric generalized linear models. Journal of the Royal Statistical Society (B) , 73(1):3--36

2011

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

Bhat, C. R. (1995). A heteroscedastic extreme value model of intercity travel mode choice. Transportation Research Part B: Methodological , 29(6):471--483

work page 1995

[2] [2]

and Dunsmore, I

Boys, R. and Dunsmore, I. (1987). Diagnostic and sampling models in screening. Biometrika , 74(2):365--374

work page 1987

[3] [3]

Bradley, R. A. and Terry, M. E. (1952). Rank analysis of incomplete block designs: I . The method of paired comparisons. Biometrika , pages 324--345

work page 1952

[4] [4]

Crouch, E. A. and Spiegelman, D. (1990). The evaluation of integrals of the form _ - ^ f(t) (-t^2) : Application to logistic-normal models. Journal of the American Statistical Association , 85(410):464--469

work page 1990

[5] [5]

David, H. (1988). The method of paired comparisons . Charles Griffin & Company, London

work page 1988

[6] [6]

Davidson, R. R. (1970). On Extending the Bradley - Terry Model to Accommodate Ties in Paired Comparison Experiments . Journal of the American Statistical Association , 65(329):317--328

work page 1970

[7] [7]

and Koopman, S

Durbin, J. and Koopman, S. J. (2012). Time series analysis by state space methods , volume 38. OUP Oxford

work page 2012

[8] [8]

Elo, A. E. (1978). The rating of chess players, past and present . Arco Publishing, New York

work page 1978