Pith. sign in

REVIEW 3 major objections 4 minor 92 references

AGI-Elo: How Far Are We From Mastering A Task?

T0 review · 3 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read AGI-Elo claims that treating benchmark evaluation as matches between test cases and agents produces a single rating scale for difficulty and competency, with predictive power and measurable gaps to task mastery.

desk verdict Clean Glicko/IRT application to benchmark items and models with useful released ratings, but the validation is in-sample and doesn't test the transitivity claim. read the letter →

arxiv 2505.12844 v2 pith:MPAPB77Z submitted 2025-05-19 cs.AI cs.RO

Shuo Sun , Yimin Zhao , Christina Dao Wen Lee , Jiawei Sun , Chengran Yuan , Zefan Huang , Dongen Li , Justin KW Yeoh
show 3 more authors
Alok Prakash Thomas W. Malone Marcelo H. Ang Jr
This is my paper · ORCID
classification cs.AIcs.RO
keywords AGIevaluationratingsystemtestcasedifficultymodelcompetencyEloitemresponsetheorygaplong-taildistribution
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

AGI-Elo treats every evaluation as a match between a test case and an agent, so that a single rating scale carries both the difficulty of individual images, questions, code problems, or driving scenarios and the competency of AI models and humans. The paper argues this joint rating lets a benchmark say not just who wins on average, but which test cases are hard, which agents are strong, and how many rating points separate current systems from an oracle that masters the task at a chosen confidence level. The authors validate the approach on six established datasets across vision, language, and action, reporting high rank correlations with standard metrics and low prediction errors. The headline result is that the best models sit roughly 233–387 rating points below oracle-level mastery at 50% confidence and over 1,000 points below at 99% confidence, while a human expert on the motion-planning task is within about 21 points of the oracle.

What carries the argument

The machinery is a two-sided rating system in which test cases and agents are the two player types and only cross-type matches are played. Each player's rating is a normal distribution $N(\mu,\sigma^2)$, updated by a Glicko-style rule: the expected score follows the logistic curve $E_{ij}=1/(1+10^{-g(\sigma_j)(\mu_i-\mu_j)/400})$, and the update moves $\mu$ and $\sigma$ by the difference between actual and expected match score, weighted by the opponent's uncertainty. Because every agent is matched against every test case, the ratings anchor model competency to the empirical difficulty distribution of test cases, and the inverse scoring function $f^{-1}$ projects predicted match scores back into the original metric space. The oracle gap is computed from the hardest test case's rating as $R_{\text{oracle@}S_\theta}\ge R_{t,\max}-400\log_{10}((1-S_\theta)/S_\theta)$, defining how far an agent is from solving every test case at confidence $S_\theta$.

What would settle it

One concrete test: hold out a subset of the agents and test cases, fit ratings on the remaining matches, then check whether predicted performance on the held-out pairs matches the observations; if the predictions degrade sharply, the ratings are memorising the match data rather than encoding difficulty and competency. A sharper adversarial check is to construct two agents A and B and test cases such that A solves a hard set, B solves an easy set, and B beats A head-to-head, and ask whether the single-scale ratings still predict the head-to-head outcome.

Watch

Extended reading notes

Core claim

The central discovery claimed is that a Glicko-style rating system can be pointed at heterogeneous players—test cases on one side, agents on the other—and still produce a single, transitive rating scale. Because every agent plays every test case, each win or loss updates both the perceived competency of the agent and the perceived difficulty of the test case, and transitivity lets ratings be compared across players that never meet directly. The paper reports that on ImageNet, COCO, MMLU, LiveCodeBench, Waymo, and NAVSIM, the resulting ratings rank test cases and agents consistently with the original metrics (Spearman correlations near ±0.97–1.00) and predict binned agent performance with low mean absolute error (0.017–0.066). It frames these results as evidence that the system can answer three questions at once: how hard each test case is, how competent each agent is, and how far current models are from full task mastery.

Load-bearing premise

The load-bearing premise is that test cases and agents live on one transitive difficulty–competency axis, so wins and losses in cross-type matches can be converted into a single rating scale; if transitivity fails, the predicted performances and the competency gaps cease to be meaningful.

Editorial extensions

If this is right

  • Benchmarks can report a difficulty distribution per test case instead of one average score, exposing long tails and easy clusters.
  • Competency gaps to oracle-level mastery become a single number per agent and confidence threshold, allowing progress to be tracked over time: the paper estimates current best models are about 233–387 rating points from the oracle at 50% confidence and 1,031–1,185 points at 99% confidence.
  • The rating system predicts an agent's expected performance on any individual test case, and can list the test cases on which an agent falls below any chosen performance threshold.
  • Rating gains across model generations quantify progress, e.g., ImageNet classification moved from AlexNet at 1586 to ConvNeXt-Large at 2035, mastering an additional 18% of the dataset over roughly a decade.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension the paper leaves implicit: if the single-scale transitivity holds across datasets, ratings from different benchmarks could be chained through shared test cases to compare models and humans across tasks.
  • The oracle gap inherits the assumption that the hardest test case in a benchmark approximates the worst case in the real world; on datasets with truncated difficulty ranges, the true gap to full mastery would be understated.
  • The same rating signal could drive adaptive evaluation, pairing each agent with test cases near its current rating to reach a stable rating with fewer matches than the full cross-product used here.
  • Per-test-case ratings could serve as curriculum or active-learning signals, steering future data collection toward the long tail the paper identifies.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes AGI-Elo, a rating system based on Glicko that jointly estimates the difficulty of individual test cases and the competency of AI agents (models or humans) by converting benchmark scores into match outcomes between test cases and agents. The fitted ratings are used to predict agent performance on test cases, identify long-tail difficult cases, and compute competency gaps to hypothetical oracles at confidence thresholds. The method is applied to six datasets spanning vision, language, and action domains, and the authors report high Spearman correlations and low prediction errors as evidence of reliability, together with rating distributions and competency-gap analyses.

Significance. If the validity of the ratings were established, this framework would offer a useful difficulty-aware complement to aggregate performance metrics. The paper has notable strengths: it releases code and computed ratings, covers a diverse set of tasks with 7-20 agents per task, includes a human expert in the motion-planning evaluation, and builds on standard, correctly stated Glicko update equations. However, the current validation is in-sample, so the paper's central empirical claims—predictive accuracy and empirical support for the transitivity assumption—are not yet demonstrated. The significance of the work therefore hinges on whether the validation can be repaired with held-out evaluations and direct tests of the model's assumptions.

major comments (3)
  1. [§4.3, Table 2, Fig. 4] The reliability evidence is resubstitution. Ratings are estimated from the full set of match scores, and the same match scores are then used to compute the Spearman correlations (rho_t, rho_a) and the binned MAE/MSE. Because each test-case rating R_t is essentially a monotone transform of the average agent performance on that test case under the logistic model, high rho_t values are expected by construction; rho_a and the MAE/MSE comparisons suffer the same circularity. These metrics therefore cannot distinguish a genuinely predictive rating system from a lookup table of training outcomes. To support predictive validity, the authors should evaluate on held-out test cases or held-out agents (for example, fit on a subset of matches and evaluate on the remainder), or report cross-validated prediction of individual match outcomes.
  2. [§3.1, §4.3] The paper states in Section 3.1 that the transitivity assumption 'is later supported by our experimental results in subsection 4.3,' but the experiments in 4.3 are the same in-sample consistency checks described above. Aggregate correlations and binned prediction error would not reveal systematic transitivity violations, such as a lower-rated agent solving a test case that a higher-rated agent fails and vice versa across many pairs. A direct test is needed—for example, comparing expected and observed pairwise outcome agreement across all agent-test pairs, or checking whether rating differences consistently predict solving probability in disjoint subgroups of test cases. Without such a test, the single-scale interpretation underlying the rankings and competency gaps in Table 1 is not empirically supported.
  3. [§3.4, Eq. (12), Table 1] The oracle competency-gap computation in Eq. (12) extrapolates the logistic model far beyond the observed rating range: for an oracle at 99% confidence, the required rating is roughly Rt,max + 800, while all measured agents lie below Rt,max. The paper assumes that the logistic model remains valid at these extreme rating differences and that the hardest observed test case anchors the real-world distribution, but neither assumption is validated. At minimum, the authors should report the range of rating differences actually observed in the matches, test the logistic fit near the extremes (for example, on held-out matches with large rating differences), and discuss the sensitivity of the Table 1 gaps to the choice of S_theta.
minor comments (4)
  1. [Appendix B.2.3, B.3.3, B.4.3] The scoring functions for COCO, MMLU, and LiveCodeBench are each described as 'used on Waymo dataset'; these appear to be copy-paste errors and should be corrected to name the respective datasets.
  2. [Figure 4] The rho_t and rho_a panels have y-axis labels that appear only as 't' and 'a'; expanding these labels would improve readability.
  3. [§4.2] The sentence stating that for each agent 'the portion of the test cases and the percentile curve that lies to the right represents the fraction of the dataset that remains difficult' is ambiguous; clarifying that test cases with rating greater than the agent's rating are solved with below 50% confidence would help.
  4. [§4.1] The initialization is written as R∼N(1500, 3502), which is likely intended as N(1500, 350^2); please clarify the notation and, for completeness, state these values in Section 3.3 where initialization is first described.

Circularity Check

2 steps flagged · score 6.0 of 10

AGI-Elo's headline reliability evidence is in-sample resubstitution: ratings are fit to match scores and then checked against those same match scores, so the high correlations and low errors do not establish predictive validity or the transitivity assumption.

  1. fitted input called prediction [Section 4.3 (Table 2), using Eq. (10) prediction]
    "For each agent, its average performance ¯Ma,B = 1 |B| P t∈B Ma,t on all test cases within the same rating bin B is computed and compared against the theoretical expectations E[Ma,B] derived from the rating system. The mean absolute error (MAE) and mean squared error (MSE) are used to quantify the deviation between the empirical performance ¯Ma,B and the theoretical expectation E[Ma,B]."

    The ratings R_a and R_t in Eq. (10) are estimated from the same match scores S_{a,t}=f(M_{a,t}) that define the empirical bin averages \bar M_{a,B}. Comparing E[M_{a,B}] with \bar M_{a,B} is therefore a resubstitution (training-set) calibration check, not an out-of-sample prediction. The same holds for rho_t and rho_a: the 'average agent performance on that test case' used for rho_t is exactly the data that produced R_t. High correlation and low error are expected when a fitted model is evaluated on its own training data, so Table 2 cannot establish predictive validity or the transitivity assumption.

  2. other [Section 3.1 (transitivity assumption), citing Section 4.3]
    "AGI-Elo leverages the transitivity property of rating systems, under the assumption that the transitivity property remains valid in our heterogeneous agent setting (an assumption later supported by our experimental results in subsection 4.3)."

    The only support offered for the transitivity assumption is Section 4.3's consistency and MAE/MSE numbers, which are computed on the same match outcomes used to fit the ratings. Those aggregated correlations cannot detect violations such as rating cycles or non-transitive triples, and they presuppose the single-scale logistic model they are meant to justify. Thus the load-bearing assumption is supported by evidence that itself assumes the assumption.

full rationale

The core fitting procedure of AGI-Elo is self-contained: ratings are estimated from match scores via a Glicko-like update, and the resulting rating distributions, long-tail descriptions, and competency-gap numbers are legitimate summaries of that fitted model. However, the paper's validation claims go beyond this. Section 4.3 calls Table 2 'predictive accuracy,' but the 'theoretical expectations' in that table are computed with Eq. (10) from ratings that were fit to the very same match scores being binned and averaged. Likewise, the Spearman correlations rho_t and rho_a correlate fitted ratings with average performances derived from the same match outcomes that generated those ratings. These are training-set calibration checks, so the near-perfect correlations and low errors are unsurprising and do not demonstrate out-of-sample prediction. The transitivity assumption, which is load-bearing for placing test cases and agents on one scale, is said to be supported by these same in-sample results, and the reported aggregate correlations would not reveal systematic transitivity violations. I found no load-bearing self-citation chain or uniqueness theorem smuggled in by citation; the circularity is concentrated in the validation step. Because the central 'prediction' claim reduces partly to a resubstitution check, while the rating-estimation method itself has independent content, a score of 6 is appropriate rather than a higher score.

Assumptions & free parameters 2 free parameters · 5 assumptions · 1 invented entities

The central computation depends on two hand-chosen rating parameters, the transitivity and logistic-model assumptions, the oracle extrapolation, and the assumption that benchmark datasets represent real-world difficulty. The oracle entity is a useful analytic device but has no independent empirical support.

free parameters (2)
  • Initial rating mean and deviation = 1500, 350
    All agents and test cases start at N(1500, 350^2). This choice affects absolute rating values but not relative ordinal rankings.
  • Logistic scale 400 = 400
    The scale factor alpha = 1/400 and beta = 10 are inherited from chess Elo conventions. They set the rating spread and the meaning of rating differences.
assumptions (5)
  • domain assumption Transitivity holds for heterogeneous matches between models and test cases.
    Section 3.1 states that the system relies on transitivity to place agents and test cases on a single rating scale from inter-category matches.
  • domain assumption The logistic model in Eq. 7 correctly describes the probability of solving a test case as a function of rating difference.
    Equation 3/7 assumes a one-parameter IRT-style logistic link for all tasks and all agents.
  • ad hoc to paper An oracle can be represented by the same logistic model and its rating can be extrapolated beyond the observed rating range.
    Equation 12 computes oracle ratings from R_t,max and S_theta without empirical data at those extreme rating differences.
  • domain assumption The dataset is a faithful miniature reflection of the real-world distribution of test cases.
    Section 3.4 makes this assumption explicitly when interpreting competency gaps as real-world mastery gaps.
  • domain assumption The task-specific scoring functions f are invertible and map metrics into a comparable [0,1] probability-like space.
    Section 3.2 defines f and f^-1; for several tasks f is the identity, but invertibility and cross-task comparability are assumed without dedicated validation.
invented entities (1)
  • Oracle agent at mastery threshold S_theta
    purpose: Defines the competency-gap target in Eq. 12-13.
    The oracle is a hypothetical player with no falsifiable handle outside the fitted rating model; the paper presents no independent evidence for its properties.

how reviews work

0 comments
Cite this review

Pith. "Pith review of AGI-Elo: How Far Are We From Mastering A Task?." pith.science (2026). https://pith.science/paper/MPAPB77Z

@misc{pith2026250512844,
  author       = {Pith},
  title        = {Pith review of: AGI-Elo: How Far Are We From Mastering A Task?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MPAPB77Z}},
  note         = {Machine review of arXiv:2505.12844}
}
read the original abstract

As the field progresses toward Artificial General Intelligence (AGI), there is a pressing need for more comprehensive and insightful evaluation frameworks that go beyond aggregate performance metrics. This paper introduces a unified rating system that jointly models the difficulty of individual test cases and the competency of AI models (or humans) across vision, language, and action domains. Unlike existing metrics that focus solely on models, our approach allows for fine-grained, difficulty-aware evaluations through competitive interactions between models and tasks, capturing both the long-tail distribution of real-world challenges and the competency gap between current models and full task mastery. We validate the generalizability and robustness of our system through extensive experiments on multiple established datasets and models across distinct AGI domains. The resulting rating distributions offer novel perspectives and interpretable insights into task difficulty, model progression, and the outstanding challenges that remain on the path to achieving full AGI task mastery.

Figures

Figures reproduced from arXiv: 2505.12844 by the authors.

Figure 1
Figure 1. In this paper, we address long-standing questions regarding the current capabilities of AGI [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Illustration of the proposed AGI-Elo rating system. However, our proposed rating system diverges significantly as it is designed for matches between heterogeneous agents, in a similar fashion to Item Response Theory (IRT) [54], which models the probability that an agent (human or model) with a certain ability level correctly solves a test case as: P(correct | α, β, Rt, Ra) = 1 1 + β−α(Rt−Ra) (3) where Rt and Ra pres… view at source ↗
Figure 3
Figure 3. Visualization of the estimated test case rating distribution and agent ratings on six distinct datasets. The percentile curve represents the cumulative percentage of test cases up to each rating level. For each agent, the portion of the test cases and the percentile curve that lies to the right represents the fraction of the dataset that remains difficult (below 50% confidence). more symmetrical distributions from t… view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Evaluation of the reliability as a function of the percentage of completed matches. [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Visualization of the predicted (theoretical) agent performances based on the differences [PITH_FULL_IMAGE:figures/full_fig_p016_5.png]
Figure 6
Figure 6. Figure 6: Model ratings over the percentage of matches on respective datasets. [PITH_FULL_IMAGE:figures/full_fig_p017_6.png]
Figure 7
Figure 7. Figure 7: System prediction errors and Spearman’s correlations over the percentage of matches on [PITH_FULL_IMAGE:figures/full_fig_p018_7.png]
Figure 8
Figure 8. Figure 8: System prediction errors and Spearman’s correlations over the percentage of matches on [PITH_FULL_IMAGE:figures/full_fig_p018_8.png]
Figure 9
Figure 9. Figure 9: System prediction errors and Spearman’s correlations over the percentage of matches on [PITH_FULL_IMAGE:figures/full_fig_p019_9.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

92 extracted references · 30 canonical work pages

  1. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  2. [2]

    Claude 3.7 sonnet and claude code

    Anthropic. Claude 3.7 sonnet and claude code. https://www.anthropic.com/news/ claude-3-7-sonnet , February 2025. Accessed: 2025-05-16

    2025

  3. [3]

    Unsu- pervised label noise modeling and loss correction

    Eric Arazo, Daniel Ortego, Paul Albert, Noel E O’Connor, and Kevin McGuinness. Unsu- pervised label noise modeling and loss correction. In International Conference on Machine Learning, pages 312–321, 2019

    2019

  4. [4]

    Curriculum learning

    Yoshua Bengio, Jerome Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th annual international conference on machine learning, pages 41–48, 2009

    2009

  5. [5]

    Elo uncovered: Robustness and best practices in language model evaluation

    Meriem Boubdir, Edward Kim, Beyza Ermis, Sara Hooker, and Marzieh Fadaee. Elo uncovered: Robustness and best practices in language model evaluation. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024

    2024

  6. [6]

    Lang, Sourabh V ora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom

    Holger Caesar, Varun Bankiti, Alex H. Lang, Sourabh V ora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In CVPR, 2020

    2020

  7. [7]

    Active bias: Training more accurate neural networks by emphasizing high variance samples

    Hwanjun Songkuk Chang and et al. Active bias: Training more accurate neural networks by emphasizing high variance samples. In Advances in Neural Information Processing Systems, pages 1002–1012, 2017

    2017

  8. [8]

    Mmdetection: Open mmlab detection toolbox and benchmark

    Kai Chen, Jiaqi Wang, Jiangmiao Pang, Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, et al. Mmdetection: Open mmlab detection toolbox and benchmark. arXiv preprint arXiv:1906.07155, 2019

    arXiv 1906

Show all 92 references
  1. [9]

    Chatbot arena: An open platform for evaluating llms by human preference

    Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Banghua Zhu, Hao Zhang, Michael Jordan, Joseph E Gonzalez, et al. Chatbot arena: An open platform for evaluating llms by human preference. In Forty-first International Conference...

    2024

  2. [10]

    Transfuser: Imitation with transformer-based sensor fusion for autonomous driving

    Kashyap Chitta, Aditya Prakash, Bernhard Jaeger, Zehao Yu, Katrin Renz, and Andreas Geiger. Transfuser: Imitation with transformer-based sensor fusion for autonomous driving. Pattern Analysis and Machine Intelligence (PAMI), 2023

    2023

  3. [11]

    Openscene: The largest up-to-date 3d occupancy prediction bench- mark in autonomous driving, 2023

    OpenScene Contributors. Openscene: The largest up-to-date 3d occupancy prediction bench- mark in autonomous driving, 2023

    2023

  4. [12]

    Navsim: Data-driven non-reactive autonomous vehicle simulation and benchmarking

    Daniel Dauner, Marcel Hallgarten, Tianyu Li, Xinshuo Weng, Zhiyu Huang, Zetong Yang, Hongyang Li, Igor Gilitschenski, Boris Ivanovic, Marco Pavone, Andreas Geiger, and Kashyap Chitta. Navsim: Data-driven non-reactive autonomous vehicle simulation and benchmarking. In Advances ...

    2024

  5. [13]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 248–255, 2009. 10

    2009

  6. [14]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv...

    2010 arXiv

  7. [15]

    The proposed uscf rating system, its development, theory, and applications

    Arpad E Elo. The proposed uscf rating system, its development, theory, and applications. Chess life, 22(8):242–247, 1967

    1967

  8. [16]

    Understanding dataset difficulty with v-usable information

    Kawin Ethayarajh, Yejin Choi, and Swabha Swayamdipta. Understanding dataset difficulty with v-usable information. In International Conference on Machine Learning, pages 5988–6008. PMLR, 2022

    2022

  9. [17]

    Large scale interactive motion forecasting for autonomous driving: The waymo open motion dataset

    Scott Ettinger, Shuyang Cheng, Benjamin Caine, Chenxi Liu, Hang Zhao, Sabeek Pradhan, Yuning Chai, Ben Sapp, Charles R Qi, Yin Zhou, et al. Large scale interactive motion forecasting for autonomous driving: The waymo open motion dataset. In Proceedings of the IEEE/CVF Internat...

    2021

  10. [18]

    Are we ready for autonomous driving? the kitti vision benchmark suite

    Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In Conference on Computer Vision and Pattern Recognition (CVPR), 2012

    2012

  11. [19]

    Glickman

    Mark E. Glickman. Parameter estimation in large dynamic paired comparison experiments. Journal of the Royal Statistical Society: Series C (Applied Statistics), 48(3):377–394, 1999

    1999

  12. [20]

    Gemini 2.5: Our most intelligent ai model

    Google DeepMind. Gemini 2.5: Our most intelligent ai model. https://blog.google/ technology/google-deepmind/gemini-model-thinking-updates-march-2025/ , March 2025. Accessed: 2025-05-16

    2025

  13. [21]

    The llama 3 herd of models

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  14. [22]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  15. [23]

    Deepseek-coder: When the large language model meets programming–the rise of code intelligence

    Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Yu Wu, YK Li, et al. Deepseek-coder: When the large language model meets programming–the rise of code intelligence. arXiv preprint arXiv:2401.14196, 2024

    2024 arXiv

  16. [24]

    Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A. Smith. Annotation artifacts in natural language inference data. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Huma...

    2018

  17. [25]

    Tan et al

    K. Tan et al. H. Caesar, J. Kabzan. Nuplan: A closed-loop ml-based planning benchmark for autonomous vehicles. In CVPR ADP3 workshop, 2021

    2021

  18. [26]

    Co-teaching: Robust training of deep neural networks with extremely noisy labels

    Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor W Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. In Advances in Neural Information Processing Systems, pages 8527–8537, 2018

    2018

  19. [27]

    Mask r-cnn

    Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. arXiv preprint arXiv:1703.06870, 2017

    2017 arXiv

  20. [28]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778, 2016

    2016

  21. [29]

    Measuring massive multitask language understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020. 11

    2009 arXiv

  22. [30]

    Trueskill ™: a bayesian skill rating system

    Ralf Herbrich, Tom Minka, and Thore Graepel. Trueskill ™: a bayesian skill rating system. Advances in neural information processing systems, 19, 2006

    2006

  23. [31]

    Learning whodunnit: Classification of event participants in news articles

    Dirk Hovy, Barbara Plank, and Anders Sogaard. Learning whodunnit: Classification of event participants in news articles. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 540–545, 2013

    2013

  24. [32]

    Densely connected convolutional networks

    Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4700–4708, 2017

    2017

  25. [33]

    Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jia- jun Zhang, Bowen Yu, Keming Lu, et al. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409.12186, 2024

    2024 arXiv

  26. [34]

    Gpt-4o system card

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024

    2024 arXiv

  27. [35]

    Squeezenet: Alexnet-level accuracy with 50x fewer parameters and< 0.5 mb model size

    Forrest N Iandola, Song Han, Matthew W Moskewicz, Khalid Ashraf, William J Dally, and Kurt Keutzer. Squeezenet: Alexnet-level accuracy with 50x fewer parameters and< 0.5 mb model size. arXiv preprint arXiv:1602.07360, 2016

    2016 arXiv

  28. [36]

    Openai o1 system card

    Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai o1 system card. arXiv preprint arXiv:2412.16720, 2024

    2024 arXiv

  29. [37]

    Livecodebench: Holistic and contamination free evaluation of large language models for code

    Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Ar- mando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.07974, 2024

    2024 arXiv

  30. [38]

    Ultralytics yolov8, 2023

    Glenn Jocher, Ayush Chaurasia, and Jing Qiu. Ultralytics yolov8, 2023

    2023

  31. [39]

    Ultralytics yolo11, 2024

    Glenn Jocher and Jing Qiu. Ultralytics yolo11, 2024

    2024

  32. [40]

    ultralytics/yolov5: v3

    Glenn Jocher, Alex Stoken, Jirka Borovec, Liu Changyu, Adam Hogan, Laurentiu Diaconu, Jake Poznanski, Lijun Yu, Prashant Rai, Russ Ferriday, et al. ultralytics/yolov5: v3. 0. Zenodo, 2020

    2020

  33. [41]

    Learning multiple layers of features from tiny images

    Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009

    2009

  34. [42]

    Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep con- volutional neural networks. In Advances in Neural Information Processing Systems (NeurIPS), volume 25, pages 1097–1105, 2012

    2012

  35. [43]

    Datasets: A community library for natural language processing

    Quentin Lhoest, Albert Villanova Del Moral, Yacine Jernite, Abhishek Thakur, Patrick V on Platen, Suraj Patil, Julien Chaumond, Mariama Drame, Julien Plu, Lewis Tunstall, et al. Datasets: A community library for natural language processing. arXiv preprint arXiv:2109.02846, 2021

    2021 arXiv

  36. [44]

    Quanti- fying ai psychology: A psychometrics benchmark for large language models

    Yuan Li, Yue Huang, Hongyi Wang, Xiangliang Zhang, James Zou, and Lichao Sun. Quanti- fying ai psychology: A psychometrics benchmark for large language models. arXiv preprint arXiv:2406.17675, 2024

    2024

  37. [45]

    Diffusiondrive: Truncated diffusion model for end-to-end autonomous driving

    Bencheng Liao, Shaoyu Chen, Haoran Yin, Bo Jiang, Cheng Wang, Sixu Yan, Xinbang Zhang, Xiangyu Li, Ying Zhang, Qian Zhang, and Xinggang Wang. Diffusiondrive: Truncated diffusion model for end-to-end autonomous driving. arXiv preprint arXiv:2411.15139, 2024

    2024 arXiv

  38. [46]

    Eda: Evolving and distinct anchors for multimodal motion prediction

    Longzhong Lin, Xuewu Lin, Tianwei Lin, Lichao Huang, Rong Xiong, and Yue Wang. Eda: Evolving and distinct anchors for multimodal motion prediction. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 3432–3440, 2024

    2024

  39. [47]

    Focal loss for dense object detection

    Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. arXiv preprint arXiv:1708.02002, 2017. 12

    2017 arXiv

  40. [48]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer vision–ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings...

    2014

  41. [49]

    Deepseek-v3 technical report

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024

    2024 arXiv

  42. [50]

    Reasoning multi-agent behavioral topology for interactive autonomous driving

    Haochen Liu, Li Chen, Yu Qiao, Chen Lv, and Hongyang Li. Reasoning multi-agent behavioral topology for interactive autonomous driving. In NeurIPS, 2024

    2024

  43. [51]

    Ssd: Single shot multibox detector

    Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yang Fu, and Alexander C Berg. Ssd: Single shot multibox detector. In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part I 14, p...

    2016

  44. [52]

    Swin transformer: Hierarchical vision transformer using shifted windows

    Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 10012–10022, 2021

    2021

  45. [53]

    A convnet for the 2020s

    Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11976–11986, 2022

    2022

  46. [54]

    Statistical theories of mental test scores

    Frederic M Lord and Melvin R Novick. Statistical theories of mental test scores. IAP, 2008

    2008

  47. [55]

    Starcoder 2 and the stack v2: The next generation

    Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Noua- mane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, et al. Starcoder 2 and the stack v2: The next generation. arXiv preprint arXiv:2402.19173, 2024

    2024 arXiv

  48. [56]

    Detrs beat yolos on real-time object detection (2023)

    Wenyu Lv, Shangliang Xu, Yian Zhao, Guanzhong Wang, Jinman Wei, Cheng Cui, Yuning Du, Qingqing Dang, and Yi Liu. Detrs beat yolos on real-time object detection (2023). arXiv preprint arXiv:2304.08069, 2023

    2023 arXiv

  49. [57]

    Shufflenet v2: Practical guidelines for efficient cnn architecture design

    Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun. Shufflenet v2: Practical guidelines for efficient cnn architecture design. In Proceedings of the European conference on computer vision (ECCV), pages 116–131, 2018

    2018

  50. [58]

    Item response theory in ai: Analysing machine learning classifiers at the instance level

    Fernando Martínez-Plumed, Ricardo BC Prudêncio, Adolfo Martínez-Usó, and José Hernández- Orallo. Item response theory in ai: Analysing machine learning classifiers at the instance level. Artificial intelligence, 271:18–42, 2019

    2019

  51. [59]

    The llama 4 herd: The beginning of a new era of natively multimodal ai innova- tion

    Meta AI. The llama 4 herd: The beginning of a new era of natively multimodal ai innova- tion. https://ai.meta.com/blog/llama-4-multimodal-intelligence/ , April 2025. Accessed: 2025-05-16

    2025

  52. [60]

    Hardness of samples need to be quantified for a reliable evaluation system: Exploring potential opportunities with a new task

    Swaroop Mishra, Anjana Arunkumar, Chris Bryan, and Chitta Baral. Hardness of samples need to be quantified for a reliable evaluation system: Exploring potential opportunities with a new task. arXiv preprint arXiv:2210.07631, 2022

    arXiv

  53. [61]

    Crosslin- gual generalization through multitask finetuning

    Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng-Xin Yong, Hailey Schoelkopf, et al. Crosslin- gual generalization through multitask finetuning. arXiv preprint arXiv:2211.01786, 2022

    2022 arXiv

  54. [62]

    Nanogpt api

    NanoGPT. Nanogpt api. https://nano-gpt.com/api, 2025. Accessed: 2025-05-15

    2025

  55. [63]

    Openai api

    OpenAI. Openai api. https://platform.openai.com, 2025. Accessed: 2025-05-15

    2025

  56. [64]

    Pytorch: An imperative style, high-performance deep learning library

    A Paszke. Pytorch: An imperative style, high-performance deep learning library. arXiv preprint arXiv:1912.01703, 2019

    1912 arXiv

  57. [65]

    Qwq: Reflect deeply on the boundaries of the unknown

    Qwen Team. Qwq: Reflect deeply on the boundaries of the unknown. https://qwenlm. github.io/blog/qwq-32b-preview/ , November 2024. Accessed: 2025-05-16. 13

    2024

  58. [66]

    Design- ing network design spaces

    Ilija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, and Piotr Dollár. Design- ing network design spaces. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10428–10436, 2020

    2020

  59. [67]

    Yolov3: An incremental improvement

    Joseph Redmon and Ali Farhadi. Yolov3: An incremental improvement. arXiv preprint arXiv:1804.02767, 2018

    2018 arXiv

  60. [68]

    Faster r-cnn: Towards real-time object detection with region proposal networks

    Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. arXiv preprint arXiv:1506.01497, 2015

    2015 arXiv

  61. [69]

    Mobilenetv2: Inverted residuals and linear bottlenecks

    Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4510–4520, 2018

    2018

  62. [70]

    Learning with bad training data via iterative trimmed loss minimization

    Yao Shen and Sujay Sanghavi. Learning with bad training data via iterative trimmed loss minimization. In International Conference on Machine Learning, pages 5739–5748, 2019

    2019

  63. [71]

    Motion transformer with global inten- tion localization and local movement refinement

    Shaoshuai Shi, Li Jiang, Dengxin Dai, and Bernt Schiele. Motion transformer with global inten- tion localization and local movement refinement. Advances in Neural Information Processing Systems, 2022

    2022

  64. [72]

    Very deep convolutional networks for large-scale image recognition

    Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014

    2014 arXiv

  65. [73]

    less is more

    Valentin I Spitkovsky, Hiyan Alshawi, and Dan Jurafsky. Baby steps: How "less is more" in unsupervised dependency parsing. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 751–759, 2010

    2010

  66. [74]

    Rmp-yolo: A robust motion predictor for partially observable scenarios even if you only look once

    Jiawei Sun, Jiahui Li, Tingchen Liu, Chengran Yuan, Shuo Sun, Zefan Huang, Anthony Wong, Keng Peng Tee, and Marcelo H Ang Jr. Rmp-yolo: A robust motion predictor for partially observable scenarios even if you only look once. arXiv preprint arXiv:2409.11696, 2024

    2024 arXiv

  67. [75]

    Jiawei Sun, Chengran Yuan, Shuo Sun, Shanze Wang, Yuhang Han, Shuailei Ma, Zefan Huang, Anthony Wong, Keng Peng Tee, and Marcelo H. Ang. Controlmtr: Control-guided motion transformer with scene-compliant intention points for feasible motion prediction. In 2024 IEEE 27th Intern...

    2024

  68. [76]

    Impact: Behavioral intention-aware multimodal trajectory prediction with adaptive context trimming

    Jiawei Sun, Xibin Yue, Jiahui Li, Tianle Shen, Chengran Yuan, Shuo Sun, Sheng Guo, Quanyun Zhou, and Marcelo H Ang Jr. Impact: Behavioral intention-aware multimodal trajectory prediction with adaptive context trimming. arXiv preprint arXiv:2504.09103, 2025

    2025

  69. [77]

    Dataset cartography: Mapping and diagnosing datasets with training dynamics

    Swabha Swayamdipta, Roy Schwartz, Nicholas Lourie, Yizhong Wang, Hannaneh Hajishirzi, Noah A Smith, and Yejin Choi. Dataset cartography: Mapping and diagnosing datasets with training dynamics. arXiv preprint arXiv:2009.10795, 2020

    2009 arXiv

  70. [78]

    Re- thinking the inception architecture for computer vision

    Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Re- thinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2818–2826, 2016

    2016

  71. [79]

    Mingxing Tan and Quoc V . Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6105–6114, 2019

    2019

  72. [80]

    Gemma 3 technical report

    Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, et al. Gemma 3 technical report. arXiv preprint arXiv:2503.19786, 2025

    2025 arXiv

  73. [81]

    Fcos: Fully convolutional one-stage object detection

    Zhi Tian, Chunhua Shen, Hao Chen, and Tong He. Fcos: Fully convolutional one-stage object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 9627–9636, 2019. 14

    2019

  74. [82]

    An empirical study of example forgetting during deep neural network learning

    Mariya Toneva, Alessandro Sordoni, Yulia Tsvetkov, Tommi Jaakkola, and Ellie Pavlick. An empirical study of example forgetting during deep neural network learning. In International Conference on Learning Representations, 2019

    2019

  75. [83]

    Ildae: Instance-level difficulty analysis of evaluation data

    Neeraj Varshney, Swaroop Mishra, and Chitta Baral. Ildae: Instance-level difficulty analysis of evaluation data. arXiv preprint arXiv:2203.03073, 2022

    2022 arXiv

  76. [84]

    Grandmaster level in starcraft ii using multi-agent reinforcement learning

    Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki, et al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 575(7782):350–354, 2019

    2019

  77. [85]

    Are all training examples created equal? an empirical study

    Kumar V odrahalli, Ganesh Ramakrishnan, and Balaraman Ravindran. Are all training examples created equal? an empirical study. In arXiv preprint arXiv:1803.07156, 2018

    2018 arXiv

  78. [86]

    Towards heterogeneous long-tailed learning: Benchmarking, metrics, and toolbox

    Haohui Wang, Weijie Guan, Jianpeng Chen, Zi Wang, and Dawei Zhou. Towards heterogeneous long-tailed learning: Benchmarking, metrics, and toolbox. In Advances in Neural Information Processing Systems 37 (NeurIPS 2024), Datasets and Benchmarks Track, 2024

    2024

  79. [87]

    Aggregated residual transformations for deep neural networks

    Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1492–1500, 2017

    2017

  80. [88]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115, 2024

    2024 arXiv

  81. [89]

    Drama: An efficient end-to-end motion planner for autonomous driving with mamba

    Chengran Yuan, Zhanqi Zhang, Jiawei Sun, Shuo Sun, Zefan Huang, Christina Dao Wen Lee, Dongen Li, Yuhang Han, Anthony Wong, Keng Peng Tee, et al. Drama: An efficient end-to-end motion planner for autonomous driving with mamba. arXiv preprint arXiv:2408.03601, 2024

    2024 arXiv

  82. [90]

    Manmatha, Mu Li, and Alexander Smola

    Hang Zhang, Chongruo Wu, Zhongyue Zhang, Yi Zhu, Zhi Zhang, Haibin Lin, Yue Sun, Tong He, Jonas Muller, R. Manmatha, Mu Li, and Alexander Smola. Resnest: Split-attention networks. arXiv preprint arXiv:2004.08955, 2020

    2004 arXiv

  83. [91]

    Dino: Detr with improved denoising anchor boxes for end-to-end object detection

    Hao Zhang, Feng Li, Shilong Liu, Lei Zhang, Hang Su, Jun Zhu, Lionel M Ni, and Heung- Yeung Shum. Dino: Detr with improved denoising anchor boxes for end-to-end object detection. arXiv preprint arXiv:2203.03605, 2022

    2022 arXiv

  84. [92]

    From static benchmarks to adaptive testing: Psychometrics in ai evaluation

    Yan Zhuang, Qi Liu, Yuting Ning, Weizhe Huang, Zachary A Pardos, Patrick C Kyllonen, Jiyun Zu, Qingyang Mao, Rui Lv, Zhenya Huang, et al. From static benchmarks to adaptive testing: Psychometrics in ai evaluation. arXiv preprint arXiv:2306.10512, 2023. 15 A Supplementary resul...

    2023 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.