{"id":"a9ca55c3-82aa-46ba-b604-143b624481d9","arxiv_id":"2505.12844","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"AGI-Elo applies Elo/Glicko-style ratings to model-versus-test-case matches, producing joint difficulty and competency scores and competency-gap estimates across six AI benchmarks.","lead":"This paper introduces AGI-Elo, a rating system that scores individual test questions and AI models on the same scale, so harder questions and stronger models receive higher ratings. It then estimates how many rating points separate today's best models from an oracle that perfectly masters each of six benchmark tasks.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Validation of AGI-Elo is in-sample (resubstitution): ratings are fit to match outcomes and then evaluated on those same outcomes, so the high Table 2 correlations do not establish predictive validity or the transitivity assumption.","rationale":"The most load-bearing condition for the central claim is that the jointly estimated ratings generalize beyond the matches used to fit them. The paper's Table 2 and Figure 4 are self-consistency checks: each test-case rating is derived from the same average performance with which it is then correlated, and the bin-averaged MAE/MSE compare the model's predictions against the data that produced them. Under the logistic/Glicko update, near-perfect in-sample correlation is unsurprising, so the empirical support for the transitivity assumption cited in Section 3.1 is circular. A held-out split with a mean-accuracy baseline is the minimal experiment that would resolve this. If AGI-Elo does not outperform the baseline on held-out test cases, the system's claimed predictive utility and its competency-gap numbers rest on an unvalidated assumption. I agree with the reader's CONDITIONAL verdict; the request for held-out validation and external baselines is exactly what the paper needs. Separately, Equation 3 has an apparent sign inconsistency relative to Equations 7 and 10: with alpha=1/400 and beta=10, beta^{-alpha(R_t-R_a)} = 10^{(R_a-R_t)/400}, which gives high-rated agents a low probability on easy test cases. This should be corrected, but it is secondary to the validation concern.","tokens_in":21614,"tokens_out":12083,"duration_ms":126118,"concrete_test":"Hold out a random 20% of test cases before fitting. Refit AGI-Elo on the remaining 80% of matches (all agents vs. retained test cases), then use the fitted ratings to predict each agent's score on the held-out test cases. Compare held-out MAE and Spearman correlation against a trivial baseline that sets test-case difficulty to the empirical logit of mean model accuracy and predicts with a single fitted agent offset. If AGI-Elo is not materially better on held-out cases, the joint rating adds no predictive value beyond average performance, and the in-sample validation cannot support the central claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The reliability evidence in Section 4.3 is resubstitution, not prediction. Spearman rho_t correlates each estimated test-case rating R_t with the average agent performance on the same test cases that produced the match scores used to estimate R_t; rho_a does the same for agents. The MAE/MSE in Table 2 bin test cases by their fitted ratings and compare empirical binned performance with expectations derived from those same fitted ratings. High correlation and low error are expected when a model is evaluated on its own training data. These checks therefore cannot support the paper's claim in Section 3.1 that the transitivity assumption is 'later supported by our experimental results in subsection 4.3.' In particular, the aggregate correlations would not reveal systematic violations of transitivity, such as test cases where a lower-rated model succeeds while a higher-rated model fails and vice versa. Because the competency gaps in Table 1 and the long-tail analysis depend on a single scalar rating scale, the central claim is only as strong as the unidimensionality assumption, which the reported experiments never directly test.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes AGI-Elo, a rating system based on Glicko that jointly estimates the difficulty of individual test cases and the competency of AI agents (models or humans) by converting benchmark scores into match outcomes between test cases and agents. The fitted ratings are used to predict agent performance on test cases, identify long-tail difficult cases, and compute competency gaps to hypothetical oracles at confidence thresholds. The method is applied to six datasets spanning vision, language, and action domains, and the authors report high Spearman correlations and low prediction errors as evidence of reliability, together with rating distributions and competency-gap analyses.","tokens_in":21791,"tokens_out":4385,"duration_ms":43927,"significance":"If the validity of the ratings were established, this framework would offer a useful difficulty-aware complement to aggregate performance metrics. The paper has notable strengths: it releases code and computed ratings, covers a diverse set of tasks with 7-20 agents per task, includes a human expert in the motion-planning evaluation, and builds on standard, correctly stated Glicko update equations. However, the current validation is in-sample, so the paper's central empirical claims—predictive accuracy and empirical support for the transitivity assumption—are not yet demonstrated. The significance of the work therefore hinges on whether the validation can be repaired with held-out evaluations and direct tests of the model's assumptions.","major_comments":[{"comment":"The reliability evidence is resubstitution. Ratings are estimated from the full set of match scores, and the same match scores are then used to compute the Spearman correlations (rho_t, rho_a) and the binned MAE/MSE. Because each test-case rating R_t is essentially a monotone transform of the average agent performance on that test case under the logistic model, high rho_t values are expected by construction; rho_a and the MAE/MSE comparisons suffer the same circularity. These metrics therefore cannot distinguish a genuinely predictive rating system from a lookup table of training outcomes. To support predictive validity, the authors should evaluate on held-out test cases or held-out agents (for example, fit on a subset of matches and evaluate on the remainder), or report cross-validated prediction of individual match outcomes.","section":"§4.3, Table 2, Fig. 4"},{"comment":"The paper states in Section 3.1 that the transitivity assumption 'is later supported by our experimental results in subsection 4.3,' but the experiments in 4.3 are the same in-sample consistency checks described above. Aggregate correlations and binned prediction error would not reveal systematic transitivity violations, such as a lower-rated agent solving a test case that a higher-rated agent fails and vice versa across many pairs. A direct test is needed—for example, comparing expected and observed pairwise outcome agreement across all agent-test pairs, or checking whether rating differences consistently predict solving probability in disjoint subgroups of test cases. Without such a test, the single-scale interpretation underlying the rankings and competency gaps in Table 1 is not empirically supported.","section":"§3.1, §4.3"},{"comment":"The oracle competency-gap computation in Eq. (12) extrapolates the logistic model far beyond the observed rating range: for an oracle at 99% confidence, the required rating is roughly Rt,max + 800, while all measured agents lie below Rt,max. The paper assumes that the logistic model remains valid at these extreme rating differences and that the hardest observed test case anchors the real-world distribution, but neither assumption is validated. At minimum, the authors should report the range of rating differences actually observed in the matches, test the logistic fit near the extremes (for example, on held-out matches with large rating differences), and discuss the sensitivity of the Table 1 gaps to the choice of S_theta.","section":"§3.4, Eq. (12), Table 1"}],"minor_comments":[{"comment":"The scoring functions for COCO, MMLU, and LiveCodeBench are each described as 'used on Waymo dataset'; these appear to be copy-paste errors and should be corrected to name the respective datasets.","section":"Appendix B.2.3, B.3.3, B.4.3"},{"comment":"The rho_t and rho_a panels have y-axis labels that appear only as 't' and 'a'; expanding these labels would improve readability.","section":"Figure 4"},{"comment":"The sentence stating that for each agent 'the portion of the test cases and the percentile curve that lies to the right represents the fraction of the dataset that remains difficult' is ambiguous; clarifying that test cases with rating greater than the agent's rating are solved with below 50% confidence would help.","section":"§4.2"},{"comment":"The initialization is written as R∼N(1500, 3502), which is likely intended as N(1500, 350^2); please clarify the notation and, for completeness, state these values in Section 3.3 where initialization is first described.","section":"§4.1"}],"recommendation":"major_revision","confidential_remarks":"The circularity of the validation is the main obstacle to acceptance. I believe the underlying method is plausible and worth pursuing, but Section 4.3 needs to be rebuilt around out-of-sample evaluation and a direct test of transitivity. The copy-paste errors in the appendix tables should also be fixed. I do not see evidence of misconduct; the issue is a methodological design flaw in the validation section."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing you should know: this is a workmanlike, clearly written application of Glicko/IRT to model-vs-test-case matches, run across six benchmarks with code and ratings released. The competency-gap numbers and long-tail curves are genuinely new and interpretable, and Eq. 12 makes the oracle-mastery definition explicit. That is a real contribution to evaluation practice.\n\nThe main soft spot is the validation. Ratings are fit to match outcomes and then checked against those same outcomes. Table 2's near-perfect Spearman correlations are therefore weaker evidence than the prose suggests: test-case ratings are nearly a monotone transform of average agent performance, so high rho is unsurprising. The MAE/MSE checks bin by fitted ratings, which is also resubstitution. Section 3.1 says the transitivity assumption is \"later supported by our experimental results,\" but the experiments are exactly the ones that generated the ratings. There are no held-out agents, held-out test cases, or external baselines like a simple IRT fit. The paper's own limitations paragraph mentions compute constraints but not this gap.\n\nOne minor technical point: the scoring function is supposed to map to [0,1], but for Waymo mAP and NA VSIM PDMS that isn't guaranteed, and the paper does not discuss it. That is fixable but should be addressed.\n\nWho is this for? Benchmark builders and anyone who wants per-instance difficulty labels. It is not a new theory, but it is a practical tool with a useful data release. With held-out validation it would be a solid contribution.\n\nI would send it to peer review. Ask for out-of-sample prediction, external IRT/Elo baselines, and uncertainty estimates on the competency gaps. If the authors can show predictive validity on held-out cases, the paper is genuinely useful.","headline":"Clean Glicko/IRT application to benchmark items and models with useful released ratings, but the validation is in-sample and doesn't test the transitivity claim.","tokens_in":22371,"tokens_out":2570,"would_cite":false,"duration_ms":28718,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"AGI-Elo claims that treating benchmark evaluation as matches between test cases and agents produces a single rating scale for difficulty and competency, with predictive power and measurable gaps to task mastery.","keywords":["AGI evaluation","rating system","test case difficulty","model competency","Elo","item response theory","competency gap","long-tail distribution"],"falsifier":"One concrete test: hold out a subset of the agents and test cases, fit ratings on the remaining matches, then check whether predicted performance on the held-out pairs matches the observations; if the predictions degrade sharply, the ratings are memorising the match data rather than encoding difficulty and competency. A sharper adversarial check is to construct two agents A and B and test cases such that A solves a hard set, B solves an easy set, and B beats A head-to-head, and ask whether the single-scale ratings still predict the head-to-head outcome.","tokens_in":21371,"feed_emoji":"📊","tokens_out":7795,"duration_ms":76304,"temperature":0.7,"pith_summary":"AGI-Elo treats every evaluation as a match between a test case and an agent, so that a single rating scale carries both the difficulty of individual images, questions, code problems, or driving scenarios and the competency of AI models and humans. The paper argues this joint rating lets a benchmark say not just who wins on average, but which test cases are hard, which agents are strong, and how many rating points separate current systems from an oracle that masters the task at a chosen confidence level. The authors validate the approach on six established datasets across vision, language, and action, reporting high rank correlations with standard metrics and low prediction errors. The headline result is that the best models sit roughly 233–387 rating points below oracle-level mastery at 50% confidence and over 1,000 points below at 99% confidence, while a human expert on the motion-planning task is within about 21 points of the oracle.","feed_headline":"AGI-Elo puts every test case and AI model on one rating scale","feed_subtitle":"The chess-style system predicts model performance and measures the gap to full task mastery across six benchmarks.","key_machinery":"The machinery is a two-sided rating system in which test cases and agents are the two player types and only cross-type matches are played. Each player's rating is a normal distribution $N(\\mu,\\sigma^2)$, updated by a Glicko-style rule: the expected score follows the logistic curve $E_{ij}=1/(1+10^{-g(\\sigma_j)(\\mu_i-\\mu_j)/400})$, and the update moves $\\mu$ and $\\sigma$ by the difference between actual and expected match score, weighted by the opponent's uncertainty. Because every agent is matched against every test case, the ratings anchor model competency to the empirical difficulty distribution of test cases, and the inverse scoring function $f^{-1}$ projects predicted match scores back into the original metric space. The oracle gap is computed from the hardest test case's rating as $R_{\\text{oracle@}S_\\theta}\\ge R_{t,\\max}-400\\log_{10}((1-S_\\theta)/S_\\theta)$, defining how far an agent is from solving every test case at confidence $S_\\theta$.","core_discovery":"The central discovery claimed is that a Glicko-style rating system can be pointed at heterogeneous players—test cases on one side, agents on the other—and still produce a single, transitive rating scale. Because every agent plays every test case, each win or loss updates both the perceived competency of the agent and the perceived difficulty of the test case, and transitivity lets ratings be compared across players that never meet directly. The paper reports that on ImageNet, COCO, MMLU, LiveCodeBench, Waymo, and NAVSIM, the resulting ratings rank test cases and agents consistently with the original metrics (Spearman correlations near ±0.97–1.00) and predict binned agent performance with low mean absolute error (0.017–0.066). It frames these results as evidence that the system can answer three questions at once: how hard each test case is, how competent each agent is, and how far current models are from full task mastery.","pith_inferences":["A natural extension the paper leaves implicit: if the single-scale transitivity holds across datasets, ratings from different benchmarks could be chained through shared test cases to compare models and humans across tasks.","The oracle gap inherits the assumption that the hardest test case in a benchmark approximates the worst case in the real world; on datasets with truncated difficulty ranges, the true gap to full mastery would be understated.","The same rating signal could drive adaptive evaluation, pairing each agent with test cases near its current rating to reach a stable rating with fewer matches than the full cross-product used here.","Per-test-case ratings could serve as curriculum or active-learning signals, steering future data collection toward the long tail the paper identifies."],"forward_implications":["Benchmarks can report a difficulty distribution per test case instead of one average score, exposing long tails and easy clusters.","Competency gaps to oracle-level mastery become a single number per agent and confidence threshold, allowing progress to be tracked over time: the paper estimates current best models are about 233–387 rating points from the oracle at 50% confidence and 1,031–1,185 points at 99% confidence.","The rating system predicts an agent's expected performance on any individual test case, and can list the test cases on which an agent falls below any chosen performance threshold.","Rating gains across model generations quantify progress, e.g., ImageNet classification moved from AlexNet at 1586 to ConvNeXt-Large at 2035, mastering an additional 18% of the dataset over roughly a decade."],"supporting_citations":[{"why":"Supplies the Gaussian rating and rating-deviation update equations that AGI-Elo adapts for cross-type matches.","marker":"[19]"},{"why":"Supplies the logistic expected-score model and the 400-point rating convention used throughout.","marker":"[15]"},{"why":"Provides the item-response-theory formulation of ability-versus-difficulty that motivates heterogeneous test-case–agent matches.","marker":"[54]"},{"why":"Extends IRT to instance-level analysis of machine learning classifiers, the prior approach AGI-Elo generalises.","marker":"[58]"},{"why":"ImageNet is the vision classification benchmark whose 50,000 test images and 20 models validate the ratings.","marker":"[13]"},{"why":"COCO is the object-detection benchmark providing test cases and model performance for rating validation.","marker":"[48]"},{"why":"MMLU is the question-answering benchmark used to estimate model and question difficulty ratings.","marker":"[29]"},{"why":"LiveCodeBench is the code-generation benchmark whose 880 problems and 20 models test the language domain.","marker":"[37]"},{"why":"Waymo Open Motion Dataset provides the motion-prediction test cases and trajectory metrics used in the action domain.","marker":"[17]"},{"why":"NAVSIM provides the motion-planning benchmark and the human expert whose near-oracle gap supports the mastery analysis.","marker":"[12]"}],"fun_headline_variants":["AGI-Elo ranks every test and model on one scale","One rating system measures AI task difficulty and skill","AGI-Elo quantifies the gap to full task mastery","Chess-style Elo rates AI models and test cases together","AGI-Elo: a single scale for tasks and agents"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that test cases and agents live on one transitive difficulty–competency axis, so wins and losses in cross-type matches can be converted into a single rating scale; if transitivity fails, the predicted performances and the competency gaps cease to be meaningful.","fun_headline_variants_meta":{"raw":{"variants":["AGI-Elo ranks every test and model on one scale","One rating system measures AI task difficulty and skill","AGI-Elo quantifies the gap to full task mastery","Chess-style Elo rates AI models and test cases together","AGI-Elo: a single scale for tasks and agents"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000689,"raw_usage":{"total_tokens":3090,"prompt_tokens":883,"completion_tokens":2207,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":499,"completion_tokens_details":{"reasoning_tokens":2124}},"tokens_in":499,"tokens_out":2207,"duration_ms":17071,"temperature":1.0,"reasoning_tokens":2124,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:25:36.723530+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"One concrete test: hold out a subset of the agents and test cases, fit ratings on the remaining matches, then check whether predicted performance on the held-out pairs matches the observations; if the predictions degrade sharply, the ratings are memorising the match data rather than encoding difficulty and competency. A sharper adversarial check is to construct two agents A and B and test cases such that A solves a hard set, B solves an easy set, and B beats A head-to-head, and ask whether the single-scale ratings still predict the head-to-head outcome.","supporting_citations":[{"cited_title":"Glickman","cited_arxiv_id":null,"evidence_quote":"Supplies the Gaussian rating and rating-deviation update equations that AGI-Elo adapts for cross-type matches."},{"cited_title":"Item response theory in ai: Analysing machine learning classifiers at the instance level","cited_arxiv_id":null,"evidence_quote":"Extends IRT to instance-level analysis of machine learning classifiers, the prior approach AGI-Elo generalises."}],"review_version":1}