{"id":"f88f24c4-9a0b-40e8-b699-5744007ec849","arxiv_id":"1908.09213","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"EPP applies a Bradley-Terry logistic model to pairwise model wins, making differences in scores log-odds of win probabilities, but the claimed cross-dataset comparability is not established.","lead":"This paper proposes a new score, EPP, that ranks machine learning models using an Elo-style logistic regression on pairwise comparisons of model performance. The score is intended to make differences between models interpretable as probabilities and comparable across datasets.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Cross-dataset EPP comparability is unidentifiable: §5.1 creates no matches across datasets, so arbitrary additive offsets per dataset leave the likelihood unchanged and Ad 2.3 is unsupported.","rationale":"The reader's rejection is supported. The probabilistic interpretation in Equation 5 is true by construction once Equation 2 is assumed, and within a single dataset the EPP score is a Bradley-Terry ranking. The load-bearing, non-tautological claim is cross-dataset comparability. Since the match-generation rule in Section 5.1 only pairs models on the same dataset, the likelihood is invariant to adding any constant to all EPP coefficients of a dataset. No anchor or normalization is provided. Therefore the abstract's claim that 'EPP scores can be directly compared between data sets' and Section 4.1's Ad 2.3 are unsupported. This also propagates to the dataset embeddings in Section 5.3, which treat EPP scores from different datasets as commensurable. This is a correctness and identifiability problem in the central claim, not a stylistic issue. I agree with the reader's weakest-assumption analysis, and the REJECT verdict should stand unchanged.","tokens_in":6324,"tokens_out":5186,"duration_ms":56656,"concrete_test":"Refit the Section 5.1 logistic regression with one free intercept per OpenML dataset, and then refit again after adding a different constant to all EPP coefficients within, say, datasets 3 and 50. The two fitted log-likelihoods will be identical because the within-dataset probabilities are unchanged by these shifts; now compare the cross-dataset EPP differences, e.g., randomForest on dataset 3 versus GBM on dataset 151. If those differences change while the likelihood stays constant, the cross-dataset ordering is arbitrary and the Ad 2.3 claim is not identified.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central advertised advantage beyond a per-tournament ranking is that EPP scores can be directly compared between data sets (abstract and Section 4.1, Ad 2.3). But the experimental protocol in Section 5.1 generates comparisons only 'on the same data set, yet not necessary on the same train/test split.' Because no match ever pairs a model from one dataset with a model from another, the logistic regression in Equations 2–3 partitions the data into disconnected components. Within each dataset, the vector of EPP coefficients is identifiable only up to an additive constant: adding any constant c_d to every EPP score for dataset d leaves every fitted logit and every within-dataset probability unchanged. The paper describes no anchor, no normalization, no cross-dataset match, and no calibration step that would fix these per-dataset constants. Consequently, the numerical EPP values for different datasets can be shifted arbitrarily without changing the fitted model, so statements such as 'random forest has higher EPP on dataset 3 than GBM on dataset 151' have no empirical content. The Ad 2.3 claim that differences between EPP scores have the same meaning across data sets is not supported by the estimation procedure as described. This is an identifiability failure in the central claim, not merely a missing baseline or a disagreement with common practice.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes EPP (Elo-Based Predictive Power), a rating score for predictive models inspired by the Elo chess rating system. The authors define the logit of the probability that model i outperforms model j as the difference of their EPP scores and estimate these scores by logistic regression on pairwise comparisons derived from AUC values. They claim that differences in EPP scores have a direct probabilistic interpretation, that EPP scores are directly comparable across datasets, that EPP accounts for cross-validation fold stability, and that EPP can support hyperparameter tuning and dataset embeddings. The experimental section applies the method to 11 OpenML datasets, 4 algorithms, and 11 hyperparameter settings, presenting plots of EPP scores and a PCA biplot of dataset embeddings.","tokens_in":6531,"tokens_out":4051,"duration_ms":42249,"significance":"The paper identifies genuine limitations of standard metrics, notably the lack of an interval-scale interpretation and the neglect of fold-level stability. The idea of casting model comparison as a Bradley-Terry / Elo-style rating problem is reasonable and could be useful for ranking and tuning. If the cross-dataset comparability claim were supported, the proposal would be a meaningful contribution. However, the central probabilistic interpretation is established by definition rather than by evidence, and the cross-dataset comparability claim is not supported by the estimation procedure because the experimental design creates no comparisons across datasets. The paper also provides no validation of predicted win probabilities, no uncertainty quantification for the significance claim, and no comparison with existing model-comparison methods. The contribution is therefore an interesting idea with a clear presentation, but the advertised advantages are not demonstrated.","major_comments":[{"comment":"The claimed probabilistic interpretation is a tautology, not an empirical finding. Equation (2) defines the model as logit(p_ij) = beta_Mi - beta_Mj, so the statement in Ad 2.1 that \"the EPP difference ... is the logit of the probability\" is true by construction. To make this a substantive advantage, the paper would need to show that the logistic model actually fits the observed pairwise outcomes, for example by evaluating the calibration of predicted win probabilities against empirical frequencies, or by comparing the model's estimates with a nonparametric estimate of P(Mi beats Mj). No such validation is provided.","section":"Section 4, Eq. (2) and Section 4.1, Ad 2.1"},{"comment":"The cross-dataset comparability claim is not identifiable from the described estimation procedure. In Section 5.1, matches are constructed only \"on the same data set, yet not necessary on the same train/test split.\" Because no match involves models from different datasets, the likelihood in Equations (2)-(3) is invariant to adding an arbitrary constant c_d to every EPP score within each dataset d. The paper describes no anchor, normalization, cross-dataset match, or calibration step that would fix these per-dataset constants. Consequently, statements such as \"random forest has higher EPP on dataset 3 than GBM on dataset 151\" have no empirical content, and Ad 2.3's claim that EPP differences have the same meaning across datasets is unsupported.","section":"Section 5.1 and Section 4.1, Ad 2.3"},{"comment":"The experimental protocol is underspecified and does not support the fold-stability and significance claims. The paper does not state how many pairwise observations are created, how the design matrix is constructed when train/test splits differ, or whether the comparisons are paired. Ad 2.4 says EPP \"takes into consideration how many times one model beat another,\" but comparing model performances on different splits does not constitute a paired comparison; one model may appear better simply because it was evaluated on an easier split. Additionally, no standard errors or confidence intervals are reported for the estimated beta coefficients, so the claim in Ad 2.2 that EPP \"allows to assess the significance via probability of better performance\" is not substantiated.","section":"Section 5.1 and Section 4.1, Ad 2.2 and Ad 2.4"},{"comment":"The statement that \"Models with EPP equals 0 have average performances\" is not justified. The logistic regression likelihood is invariant to adding a constant to all EPP scores within a dataset, so the zero point is arbitrary and depends on the identifiability constraint chosen by the software (e.g., sum-to-zero or a reference model). Without specifying this constraint, the interpretation of zero as \"average performance\" is not meaningful, and this affects the qualitative conclusions drawn from Figures 2 and 3.","section":"Section 5.2, Figure 2"}],"minor_comments":[{"comment":"The abstract contains grammatical and spelling errors: \"First is, that they are not on an interval scale\" should be \"First, they are not on an interval scale,\" and \"Forth\" should be \"Fourth.\"","section":"Abstract"},{"comment":"\"There at least three points of view\" is missing the verb \"are\": it should read \"There are at least three points of view.\"","section":"Section 2.3"},{"comment":"\"Elo scores can be interpret in terms of probability\" should be \"can be interpreted.\"","section":"Section 3"},{"comment":"There are typographical errors: \"wining\" should be \"winning,\" and the extension in Equation (3) appears to contain a typo, as the final term reads \"... + beta_Mk x_Mn\" instead of using the correct index for the model.","section":"Section 4"},{"comment":"\"As a single round, we consider comparison of performances\" should read \"we consider the comparison of performances.\" More importantly, the description of how rounds and matches are defined is too vague to reproduce the experiments; the paper should specify the exact pairing scheme and the number of observations used in the logistic regression.","section":"Section 5.1"},{"comment":"There are several typos: \"extentions\" should be \"extensions,\" \"The idea od EPP\" should be \"The idea of EPP,\" and \"may by extended\" should be \"may be extended.\"","section":"Section 6"},{"comment":"The figures would be clearer with labeled axes and a description of the identifiability constraint used to set the zero of the EPP scale; as noted in the major comments, the current \"0 equals average\" interpretation is not established.","section":"Figures 2 and 3"}],"recommendation":"reject","confidential_remarks":"The manuscript is essentially an idea proposal with illustrative experiments. The central advertised advantage, cross-dataset comparability, fails because the estimation procedure contains no cross-dataset link, making the per-dataset additive constants unidentified. The probabilistic interpretation is a definitional property of the Bradley-Terry model rather than a demonstrated empirical property. Given that these issues are load-bearing for the paper's main claims and cannot be fixed by minor revisions, rejection is appropriate. The authors may wish to consider reframing the contribution as a within-dataset ranking tool and adding explicit validation of the model's fit and uncertainty estimates."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper applies Elo/Bradley-Terry to model performance evaluation: fit a logistic regression on pairwise model wins to get interpretable score differences. That is the actual novelty, and it is a legitimate idea worth discussing. The authors correctly point out real weaknesses in common measures like AUC and F1 (no interval-scale differences, no significance, no fold-stability), and the within-dataset probabilistic interpretation of EPP differences is clean and useful. Using multiple train/test splits as repeated matches is a natural way to incorporate stability.\n\nThe soft spot is the cross-dataset comparability claim (Ad 2.3). Section 5.1 generates comparisons only within datasets—'on the same data set, yet not necessary on the same train/test split.' No match ever pairs models from different datasets. That means the logistic regression has one connected component per dataset, and within each component the EPP scores are identifiable only up to an additive constant. You can shift all scores for dataset 3 by +5 and all scores for dataset 151 by -3 and the fitted probabilities never change. So the EPP values across datasets have no absolute meaning, and comparisons like 'random forest has higher EPP on dataset 3 than GBM on dataset 151' are empty. This undermines the headline advantage and the dataset embeddings in Figure 4. It is a load-bearing flaw, not a minor omission.\n\nTwo smaller issues. The probabilistic interpretation is true by construction, since Equation 2 defines the model that way; it is a restatement, not an empirical finding. And the claim that 'Elo was not previously used to assess predictive models' is asserted without any literature survey; it may be true, but it needs support. No code or data is provided, and no confidence intervals are given for the EPP estimates, which weakens the significance claims.\n\nThat said, the core idea is sound for within-dataset model comparison and hyperparameter tuning. Figures 2 and 3 give a useful sense of tunability, and the interpretation of score differences as log-odds is a genuine improvement over raw AUC differences. The paper is clearly an extended abstract, but the flaws are fixable: restrict the claims to within-dataset comparisons, add anchors or cross-dataset matches if you want comparability, or explicitly say that the additive offset is a limitation.\n\nI would send this to peer review, not desk-reject it. The idea deserves scrutiny, and the authors have a real chance to turn it into a solid contribution if they address the identifiability problem. But as it stands, the cross-dataset comparability claim should not be accepted.","headline":"A reasonable Bradley-Terry proposal for model scoring whose main selling point—cross-dataset comparability—is undone by the estimation protocol.","tokens_in":7094,"tokens_out":2758,"would_cite":false,"duration_ms":27725,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper introduces EPP, a model-rating score whose differences are logits of win probabilities, and argues it makes performance comparisons interpretable and comparable across data sets.","keywords":["Elo rating","model evaluation","predictive performance","probabilistic interpretation","logistic regression","hyperparameter tuning","dataset embeddings","cross-validation stability"],"falsifier":"Fit EPP scores separately on two data sets that share a pair of algorithms and add a constant to every score on one data set; all within-data-set win probabilities are unchanged while the cross-data-set score difference shifts, so the claimed cross-data-set comparability collapses unless a calibration constraint is imposed. Running the procedure with a shared reference model on both data sets and checking whether the same probability is recovered would settle the claim directly.","tokens_in":6085,"feed_emoji":"🏆","tokens_out":8515,"duration_ms":81718,"temperature":0.7,"pith_summary":"The paper introduces EPP (Elo-based Predictive Power), a rating score for predictive models built by treating pairs of models on the same data set as playing matches on train/test splits, with higher test AUC counting as a win. The central claim is that EPP differences have a direct probabilistic reading: for models $M_i$ and $M_j$ with scores $\\beta_{M_i}$ and $\\beta_{M_j}$, the difference $\\beta_{M_i} - \\beta_{M_j}$ is the logit of the probability that $M_i$ beats $M_j$. The authors argue this fixes the weaknesses of raw metrics such as AUC: differences become interpretable, significance can be assessed, fold-to-fold stability is reflected, and scores can be compared between data sets. They then demonstrate uses for hyperparameter tuning and for constructing embeddings of data sets from their models' EPP scores.","feed_headline":"Model ratings turn performance gaps into win probabilities","feed_subtitle":"Pairwise model matches produce scores whose differences read directly as odds of one model beating another.","key_machinery":"The central machinery is an Elo-style rating estimated by logistic regression on pairwise match results. A match is a comparison of two models on the same data set and a train/test split, and the winner is the model with higher AUC. The design uses indicators $x_{M_a}$ equal to $1$ for the first model, $-1$ for the second, and $0$ otherwise, so the logistic model $\\operatorname{logit}(p_{i,j}) = \\beta_{M_1} x_{M_1} + \\cdots + \\beta_{M_n} x_{M_n}$ reduces to $\\operatorname{logit}(p_{i,j}) = \\beta_{M_i} - \\beta_{M_j}$. The fitted coefficients are the EPP scores, and the identity between rating differences and log odds is what turns every reported comparison into a probability.","core_discovery":"On the paper's own terms, the discovery is a rating system for predictive models with an interpretable scale. Instead of summarizing a model by an average metric, EPP estimates one score per model from pairwise comparisons across many train/test splits, using the identification $\\operatorname{logit}(p_{i,j}) = \\beta_{M_i} - \\beta_{M_j}$, where $p_{i,j}$ is the probability that model $M_i$ outperforms model $M_j$. The EPP difference is therefore the log odds of better performance, and the paper claims this gives every difference the same probabilistic meaning regardless of data set, while also reflecting how often a model wins across folds rather than just its average. The authors apply the score to eleven data sets, four algorithms, and eleven hyperparameter settings each, and use the resulting rankings to discuss tunability and to embed data sets in a low-dimensional space.","pith_inferences":["Because all matches are played within a data set, the EPP scale on each data set is identified only up to an additive constant; a strict reading of the method leaves cross-data-set comparisons underdetermined unless an anchor or calibration constraint is added.","The logit identity places EPP in the family of paired-comparison rating models, so established techniques for handling ties, varying match strengths, and uncertainty in ratings could be imported to give confidence intervals for EPP differences.","A natural testable extension is to restrict comparisons to identical train/test splits (the paper notes this alternative); this would remove split-induced noise and make the win probabilities sharper.","Data set embeddings built from EPP vectors could serve as transfer-learning features: for a new data set, locate its nearest neighbours in embedding space and use their best hyperparameter settings as a starting point."],"forward_implications":["A practitioner can read an EPP difference as a win probability: if model A is two logits above model B, then A is expected to beat B with probability $\\operatorname{invlogit}(2)$, giving a concrete significance statement.","EPP can support hyperparameter tuning by showing which settings consistently beat others and which data sets reverse the usual ordering, signalling that tuning effort should be spent differently.","Fold stability is incorporated: a model that wins more folds receives a higher EPP score than a model with a higher average but inconsistent per-fold performance.","The claimed cross-data-set comparability lets EPP scores be pooled across benchmarks; the same score gap on different data sets is asserted to carry the same probabilistic meaning, enabling data set embeddings and meta-analysis of model performance."],"supporting_citations":[{"why":"supplies the Elo rating formula and the interpretation of rating differences as expected scores that EPP adapts to models.","marker":"Elo and Sloan (2008)"},{"why":"provides the OpenML100 benchmark data sets used in the EPP experiments.","marker":"Bischl et al. (2017)"},{"why":"provides the OpenML database from which the data sets and their identifiers are taken.","marker":"Vanschoren et al. (2013)"},{"why":"gives the notion of hyperparameter tunability that the paper compares against when using EPP to judge how much an algorithm benefits from tuning.","marker":"Philipp Probst and Anne-Laure Boulesteix and Bernd Bischl (2019)"},{"why":"introduces TrueSkill, the multi-player rating extension the paper proposes for future ensemble evaluation.","marker":"Herbrich et al. (2007)"},{"why":"motivates the weaknesses of common performance measures that EPP is designed to overcome.","marker":"Powers (2008)"}],"fun_headline_variants":["EPP rating turns model gaps into win odds","Interpretable EPP scores map to win probabilities","Model ranking becomes probability of winning","Win odds from pairwise model ratings"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that a fixed difference between two EPP scores means the same thing on every data set; the fitting procedure itself only plays matches within a data set, so an arbitrary constant can be added to all scores on one data set without changing any fitted probability, and nothing in the method fixes these constants across data sets.","fun_headline_variants_meta":{"raw":{"variants":["EPP rating turns model gaps into win odds","Interpretable EPP scores map to win probabilities","Model ranking becomes probability of winning","Win odds from pairwise model ratings"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000196,"raw_usage":{"total_tokens":1347,"prompt_tokens":918,"completion_tokens":429,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":534,"completion_tokens_details":{"reasoning_tokens":376}},"tokens_in":534,"tokens_out":429,"duration_ms":5474,"temperature":1.0,"reasoning_tokens":376,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T11:17:23.987622+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Fit EPP scores separately on two data sets that share a pair of algorithms and add a constant to every score on one data set; all within-data-set win probabilities are unchanged while the cross-data-set score difference shifts, so the claimed cross-data-set comparability collapses unless a calibration constraint is imposed. Running the procedure with a shared reference model on both data sets and checking whether the same probability is recovered would settle the claim directly.","supporting_citations":[{"cited_title":"Tunability: Importance of hyperparameters of machine learning algorithms","cited_arxiv_id":null,"evidence_quote":"gives the notion of hyperparameter tunability that the paper compares against when using EPP to judge how much an algorithm benefits from tuning."}],"review_version":1}