EPP: interpretable score of model predictive power

Alicja Gosiewska; Katarzyna Woznica; Maciej Zwolinski; Mateusz Bakala; Przemyslaw Biecek

REVIEW 4 major objections 7 minor 9 references

EPP: interpretable score of model predictive power

T0 review · 4 major / 7 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper introduces EPP, a model-rating score whose differences are logits of win probabilities, and argues it makes performance comparisons interpretable and comparable across data sets.

desk verdict A reasonable Bradley-Terry proposal for model scoring whose main selling point—cross-dataset comparability—is undone by the estimation protocol. read the letter →

arxiv 1908.09213 v1 pith:TXBJY7NK submitted 2019-08-24 cs.LG stat.ML

Alicja Gosiewska , Mateusz Bakala , Katarzyna Woznica , Maciej Zwolinski , Przemyslaw Biecek This is my paper · ORCID

classification cs.LGstat.ML

keywords Eloratingmodelevaluationpredictiveperformanceprobabilisticinterpretationlogisticregressionhyperparametertuningdatasetembeddingscross-validationstability

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces EPP (Elo-based Predictive Power), a rating score for predictive models built by treating pairs of models on the same data set as playing matches on train/test splits, with higher test AUC counting as a win. The central claim is that EPP differences have a direct probabilistic reading: for models $M_i$ and $M_j$ with scores $\beta_{M_i}$ and $\beta_{M_j}$, the difference $\beta_{M_i} - \beta_{M_j}$ is the logit of the probability that $M_i$ beats $M_j$. The authors argue this fixes the weaknesses of raw metrics such as AUC: differences become interpretable, significance can be assessed, fold-to-fold stability is reflected, and scores can be compared between data sets. They then demonstrate uses for hyperparameter tuning and for constructing embeddings of data sets from their models' EPP scores.

What carries the argument

The central machinery is an Elo-style rating estimated by logistic regression on pairwise match results. A match is a comparison of two models on the same data set and a train/test split, and the winner is the model with higher AUC. The design uses indicators $x_{M_a}$ equal to $1$ for the first model, $-1$ for the second, and $0$ otherwise, so the logistic model $\operatorname{logit}(p_{i,j}) = \beta_{M_1} x_{M_1} + \cdots + \beta_{M_n} x_{M_n}$ reduces to $\operatorname{logit}(p_{i,j}) = \beta_{M_i} - \beta_{M_j}$. The fitted coefficients are the EPP scores, and the identity between rating differences and log odds is what turns every reported comparison into a probability.

What would settle it

Fit EPP scores separately on two data sets that share a pair of algorithms and add a constant to every score on one data set; all within-data-set win probabilities are unchanged while the cross-data-set score difference shifts, so the claimed cross-data-set comparability collapses unless a calibration constraint is imposed. Running the procedure with a shared reference model on both data sets and checking whether the same probability is recovered would settle the claim directly.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is a rating system for predictive models with an interpretable scale. Instead of summarizing a model by an average metric, EPP estimates one score per model from pairwise comparisons across many train/test splits, using the identification $\operatorname{logit}(p_{i,j}) = \beta_{M_i} - \beta_{M_j}$, where $p_{i,j}$ is the probability that model $M_i$ outperforms model $M_j$. The EPP difference is therefore the log odds of better performance, and the paper claims this gives every difference the same probabilistic meaning regardless of data set, while also reflecting how often a model wins across folds rather than just its average. The authors apply the score to eleven data sets, four algorithms, and eleven hyperparameter settings each, and use the resulting rankings to discuss tunability and to embed data sets in a low-dimensional space.

Load-bearing premise

The load-bearing assumption is that a fixed difference between two EPP scores means the same thing on every data set; the fitting procedure itself only plays matches within a data set, so an arbitrary constant can be added to all scores on one data set without changing any fitted probability, and nothing in the method fixes these constants across data sets.

Editorial extensions

If this is right

A practitioner can read an EPP difference as a win probability: if model A is two logits above model B, then A is expected to beat B with probability $\operatorname{invlogit}(2)$, giving a concrete significance statement.
EPP can support hyperparameter tuning by showing which settings consistently beat others and which data sets reverse the usual ordering, signalling that tuning effort should be spent differently.
Fold stability is incorporated: a model that wins more folds receives a higher EPP score than a model with a higher average but inconsistent per-fold performance.
The claimed cross-data-set comparability lets EPP scores be pooled across benchmarks; the same score gap on different data sets is asserted to carry the same probabilistic meaning, enabling data set embeddings and meta-analysis of model performance.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Because all matches are played within a data set, the EPP scale on each data set is identified only up to an additive constant; a strict reading of the method leaves cross-data-set comparisons underdetermined unless an anchor or calibration constraint is added.
The logit identity places EPP in the family of paired-comparison rating models, so established techniques for handling ties, varying match strengths, and uncertainty in ratings could be imported to give confidence intervals for EPP differences.
A natural testable extension is to restrict comparisons to identical train/test splits (the paper notes this alternative); this would remove split-induced noise and make the win probabilities sharper.
Data set embeddings built from EPP vectors could serve as transfer-learning features: for a new data set, locate its nearest neighbours in embedding space and use their best hyperparameter settings as a starting point.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The paper applies Elo/Bradley-Terry to model performance evaluation: fit a logistic regression on pairwise model wins to get interpretable score differences. That is the actual novelty, and it is a legitimate idea worth discussing. The authors correctly point out real weaknesses in common measures like AUC and F1 (no interval-scale differences, no significance, no fold-stability), and the within-dataset probabilistic interpretation of EPP differences is clean and useful. Using multiple train/test splits as repeated matches is a natural way to incorporate stability.

The soft spot is the cross-dataset comparability claim (Ad 2.3). Section 5.1 generates comparisons only within datasets—'on the same data set, yet not necessary on the same train/test split.' No match ever pairs models from different datasets. That means the logistic regression has one connected component per dataset, and within each component the EPP scores are identifiable only up to an additive constant. You can shift all scores for dataset 3 by +5 and all scores for dataset 151 by -3 and the fitted probabilities never change. So the EPP values across datasets have no absolute meaning, and comparisons like 'random forest has higher EPP on dataset 3 than GBM on dataset 151' are empty. This undermines the headline advantage and the dataset embeddings in Figure 4. It is a load-bearing flaw, not a minor omission.

Two smaller issues. The probabilistic interpretation is true by construction, since Equation 2 defines the model that way; it is a restatement, not an empirical finding. And the claim that 'Elo was not previously used to assess predictive models' is asserted without any literature survey; it may be true, but it needs support. No code or data is provided, and no confidence intervals are given for the EPP estimates, which weakens the significance claims.

That said, the core idea is sound for within-dataset model comparison and hyperparameter tuning. Figures 2 and 3 give a useful sense of tunability, and the interpretation of score differences as log-odds is a genuine improvement over raw AUC differences. The paper is clearly an extended abstract, but the flaws are fixable: restrict the claims to within-dataset comparisons, add anchors or cross-dataset matches if you want comparability, or explicitly say that the additive offset is a limitation.

I would send this to peer review, not desk-reject it. The idea deserves scrutiny, and the authors have a real chance to turn it into a solid contribution if they address the identifiability problem. But as it stands, the cross-dataset comparability claim should not be accepted.

Referee Report

4 major / 7 minor

Summary. The paper proposes EPP (Elo-Based Predictive Power), a rating score for predictive models inspired by the Elo chess rating system. The authors define the logit of the probability that model i outperforms model j as the difference of their EPP scores and estimate these scores by logistic regression on pairwise comparisons derived from AUC values. They claim that differences in EPP scores have a direct probabilistic interpretation, that EPP scores are directly comparable across datasets, that EPP accounts for cross-validation fold stability, and that EPP can support hyperparameter tuning and dataset embeddings. The experimental section applies the method to 11 OpenML datasets, 4 algorithms, and 11 hyperparameter settings, presenting plots of EPP scores and a PCA biplot of dataset embeddings.

Significance. The paper identifies genuine limitations of standard metrics, notably the lack of an interval-scale interpretation and the neglect of fold-level stability. The idea of casting model comparison as a Bradley-Terry / Elo-style rating problem is reasonable and could be useful for ranking and tuning. If the cross-dataset comparability claim were supported, the proposal would be a meaningful contribution. However, the central probabilistic interpretation is established by definition rather than by evidence, and the cross-dataset comparability claim is not supported by the estimation procedure because the experimental design creates no comparisons across datasets. The paper also provides no validation of predicted win probabilities, no uncertainty quantification for the significance claim, and no comparison with existing model-comparison methods. The contribution is therefore an interesting idea with a clear presentation, but the advertised advantages are not demonstrated.

major comments (4)

[Section 4, Eq. (2) and Section 4.1, Ad 2.1] The claimed probabilistic interpretation is a tautology, not an empirical finding. Equation (2) defines the model as logit(p_ij) = beta_Mi - beta_Mj, so the statement in Ad 2.1 that "the EPP difference ... is the logit of the probability" is true by construction. To make this a substantive advantage, the paper would need to show that the logistic model actually fits the observed pairwise outcomes, for example by evaluating the calibration of predicted win probabilities against empirical frequencies, or by comparing the model's estimates with a nonparametric estimate of P(Mi beats Mj). No such validation is provided.
[Section 5.1 and Section 4.1, Ad 2.3] The cross-dataset comparability claim is not identifiable from the described estimation procedure. In Section 5.1, matches are constructed only "on the same data set, yet not necessary on the same train/test split." Because no match involves models from different datasets, the likelihood in Equations (2)-(3) is invariant to adding an arbitrary constant c_d to every EPP score within each dataset d. The paper describes no anchor, normalization, cross-dataset match, or calibration step that would fix these per-dataset constants. Consequently, statements such as "random forest has higher EPP on dataset 3 than GBM on dataset 151" have no empirical content, and Ad 2.3's claim that EPP differences have the same meaning across datasets is unsupported.
[Section 5.1 and Section 4.1, Ad 2.2 and Ad 2.4] The experimental protocol is underspecified and does not support the fold-stability and significance claims. The paper does not state how many pairwise observations are created, how the design matrix is constructed when train/test splits differ, or whether the comparisons are paired. Ad 2.4 says EPP "takes into consideration how many times one model beat another," but comparing model performances on different splits does not constitute a paired comparison; one model may appear better simply because it was evaluated on an easier split. Additionally, no standard errors or confidence intervals are reported for the estimated beta coefficients, so the claim in Ad 2.2 that EPP "allows to assess the significance via probability of better performance" is not substantiated.
[Section 5.2, Figure 2] The statement that "Models with EPP equals 0 have average performances" is not justified. The logistic regression likelihood is invariant to adding a constant to all EPP scores within a dataset, so the zero point is arbitrary and depends on the identifiability constraint chosen by the software (e.g., sum-to-zero or a reference model). Without specifying this constraint, the interpretation of zero as "average performance" is not meaningful, and this affects the qualitative conclusions drawn from Figures 2 and 3.

minor comments (7)

[Abstract] The abstract contains grammatical and spelling errors: "First is, that they are not on an interval scale" should be "First, they are not on an interval scale," and "Forth" should be "Fourth."
[Section 2.3] "There at least three points of view" is missing the verb "are": it should read "There are at least three points of view."
[Section 3] "Elo scores can be interpret in terms of probability" should be "can be interpreted."
[Section 4] There are typographical errors: "wining" should be "winning," and the extension in Equation (3) appears to contain a typo, as the final term reads "... + beta_Mk x_Mn" instead of using the correct index for the model.
[Section 5.1] "As a single round, we consider comparison of performances" should read "we consider the comparison of performances." More importantly, the description of how rounds and matches are defined is too vague to reproduce the experiments; the paper should specify the exact pairing scheme and the number of observations used in the logistic regression.
[Section 6] There are several typos: "extentions" should be "extensions," "The idea od EPP" should be "The idea of EPP," and "may by extended" should be "may be extended."
[Figures 2 and 3] The figures would be clearer with labeled axes and a description of the identifiability constraint used to set the zero of the EPP scale; as noted in the major comments, the current "0 equals average" interpretation is not established.

Circularity Check

1 steps flagged · score 6.0 of 10

EPP's probabilistic interpretation is its defining equation; Section 4.1 restates Eq. 2 as an advantage rather than deriving it.

self definitional [Section 4, Eq. 1-2; Section 4.1, Ad 2.1]
"The desired property is that log(odds(i,j)) = βMi − βMj. (1)... The following procedure satisfies Property 1. ... logit(pi,j) = βMi − βMj. (2)... The EPP difference for models Mi and Mj is the logit of the probability that Mi achieves better performance than Mj (see Formula 2 and Formula 5)."

The advertised interpretation of EPP differences is not derived from independent evidence; it is imposed as the defining equation. Equation 2 is proposed specifically to satisfy the 'desired property' in Equation 1, so saying that the EPP difference is the logit of a win probability is a restatement of the model assumption. The subsequent advantages in Section 4.1 (significance testing, cross-dataset comparability, fold-stability) all inherit this same definitional property, making the 'demonstration' of advantages a validation by construction rather than by empirical or mathematical derivation.

full rationale

The paper is transparent that Equation 1 is a 'desired property' and Equation 2 is a proposed procedure satisfying it, so this is not a hidden data-driven circularity of the kind where a fitted parameter is relabeled as a prediction. However, the claimed central advantage, that EPP differences are logits of win probabilities, is exactly the defining logistic-regression equation; presenting this as an 'advantage' in Ad 2.1 is self-definitional. The later advantages (Ad 2.2, Ad 2.3, Ad 2.4) are all restatements of this same definition, so the paper's validation of the score's properties reduces to its own postulate. Separately, the experimental protocol in Section 5.1 only compares models 'on the same data set', so the cross-dataset comparability claim (Ad 2.3) is not identified by the described estimation procedure; that is an identifiability/correctness concern rather than a circularity, so it is not scored as a separate circular step. Overall, the central interpretive claim is true by construction, but the paper does not overclaim an empirical derivation; thus the circularity is partial rather than total.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The central claim rests on the Bradley-Terry model as a construction, plus the experimental protocol for defining wins. No novel physical or formal entities are introduced.

assumptions (3)

ad hoc to paper The probability that model Mi beats model Mj follows a Bradley-Terry model: logit(p_ij) = beta_Mi - beta_Mj.
This is the defining assumption of EPP (Equation 2). It forces the desired log-odds interpretation but is not derived from any external principle.
domain assumption A 'match' between two models can be defined by comparing their AUC on the same dataset but on possibly different train/test splits.
Section 5.1 states that comparisons are made on the same dataset but not necessarily the same split. This assumes split mismatches do not bias the win indicator.
domain assumption AUC is a valid underlying performance measure for defining wins and losses.
All experiments use AUC as the raw performance score; the EPP properties are conditional on this choice, which is not justified beyond standard practice.

how reviews work

0 comments

Cite this review

Pith. "Pith review of EPP: interpretable score of model predictive power." pith.science (2026). https://pith.science/paper/TXBJY7NK

@misc{pith2026190809213,
  author       = {Pith},
  title        = {Pith review of: EPP: interpretable score of model predictive power},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TXBJY7NK}},
  note         = {Machine review of arXiv:1908.09213}
}

read the original abstract

The most important part of model selection and hyperparameter tuning is the evaluation of model performance. The most popular measures, such as AUC, F1, ACC for binary classification, or RMSE, MAD for regression, or cross-entropy for multilabel classification share two common weaknesses. First is, that they are not on an interval scale. It means that the difference in performance for the two models has no direct interpretation. It makes no sense to compare such differences between datasets. Second is, that for k-fold cross-validation, the model performance is in most cases calculated as an average performance from particular folds, which neglects the information how stable is the performance for different folds. In this talk, we introduce a new EPP rating system for predictive models. We also demonstrate numerous advantages for this system, First, differences in EPP scores have probabilistic interpretation. Based on it we can assess the probability that one model will achieve better performance than another. Second, EPP scores can be directly compared between datasets. Third, they can be used for navigated hyperparameter tuning and model selection. Forth, we can create embeddings for datasets based on EPP scores.

Figures

Figures reproduced from arXiv: 1908.09213 by the authors.

**Figure 2.** EPP scores of different hyperparameter settings for kknn across data sets. Each panel corresponds to [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗

**Figure 3.** Boxplots of EPP scores for different models across data sets. [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: PCA biplot of EPP scores for different hyperparameter settings for k-nearest neighbours model. Groups [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

9 extracted references · 8 canonical work pages

[1]

P. Biecek. DALEX: Explainers for Complex Predictive Models in R . 2018. URL http://jmlr.org/papers/v19/18-416.html

work page 2018
[2]

Bischl, G

B. Bischl, G. Casalicchio, M. Feurer, F. Hutter, M. Lang, R. G. Mantovani, J. N. van Rijn, and J. Vanschoren. OpenML benchmarking suites and the OpenML100 . 2017

work page 2017
[3]

Elo and S

A. Elo and S. Sloan. The Rating of Chess Players, Past and Present . 2008. URL https://books.google.pl/books?id=syjcPQAACAAJ

work page 2008
[4]

Goutte and E

C. Goutte and E. Gaussier. A probabilistic interpretation of precision, recall and F-score, with implication for evaluation . In European Conference on Information Retrieval, 2005

work page 2005
[5]

Herbrich, T

R. Herbrich, T. Minka, and T. Graepel. TrueSkill(TM): A Bayesian Skill Rating System . In Advances in Neural Information Processing Systems 20, 2007. URL https://www.microsoft.com/en-us/research/publication/trueskilltm-a-bayesian-skill-rating-system/

work page 2007
[6]

Tunability: Importance of hyperparameters of machine learning algorithms

Philipp Probst and Anne-Laure Boulesteix and Bernd Bischl . Tunability: Importance of hyperparameters of machine learning algorithms. 2019. URL http://jmlr.org/papers/v20/18-444.html

work page 2019
[7]

D. Powers. Evaluation: From Precision, Recall and F-Factor to ROC, Informedness, Markedness & Correlation . 2008

work page 2008
[8]

Sokolova and G

M. Sokolova and G. Lapalme. A systematic analysis of performance measures for classification tasks. Information Processing & Management, 2009. URL http://www.sciencedirect.com/science/article/pii/S0306457309000259

work page 2009

Show all 9 references

[9]

Vanschoren, J

J. Vanschoren, J. N. van Rijn, B. Bischl, and L. Torgo. OpenML: Networked Science in Machine Learning . SIGKDD Explorations, 2013. URL http://doi.acm.org/10.1145/2641190.2641198

2013

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [1]

P. Biecek. DALEX: Explainers for Complex Predictive Models in R . 2018. URL http://jmlr.org/papers/v19/18-416.html

work page 2018

[2] [2]

Bischl, G

B. Bischl, G. Casalicchio, M. Feurer, F. Hutter, M. Lang, R. G. Mantovani, J. N. van Rijn, and J. Vanschoren. OpenML benchmarking suites and the OpenML100 . 2017

work page 2017

[3] [3]

Elo and S

A. Elo and S. Sloan. The Rating of Chess Players, Past and Present . 2008. URL https://books.google.pl/books?id=syjcPQAACAAJ

work page 2008

[4] [4]

Goutte and E

C. Goutte and E. Gaussier. A probabilistic interpretation of precision, recall and F-score, with implication for evaluation . In European Conference on Information Retrieval, 2005

work page 2005

[5] [5]

Herbrich, T

R. Herbrich, T. Minka, and T. Graepel. TrueSkill(TM): A Bayesian Skill Rating System . In Advances in Neural Information Processing Systems 20, 2007. URL https://www.microsoft.com/en-us/research/publication/trueskilltm-a-bayesian-skill-rating-system/

work page 2007

[6] [6]

Tunability: Importance of hyperparameters of machine learning algorithms

Philipp Probst and Anne-Laure Boulesteix and Bernd Bischl . Tunability: Importance of hyperparameters of machine learning algorithms. 2019. URL http://jmlr.org/papers/v20/18-444.html

work page 2019

[7] [7]

D. Powers. Evaluation: From Precision, Recall and F-Factor to ROC, Informedness, Markedness & Correlation . 2008

work page 2008

[8] [8]

Sokolova and G

M. Sokolova and G. Lapalme. A systematic analysis of performance measures for classification tasks. Information Processing & Management, 2009. URL http://www.sciencedirect.com/science/article/pii/S0306457309000259

work page 2009