Pith. sign in

REVIEW 3 major objections 4 minor 1 cited by

How Benchmark Prediction from Fewer Data Misses the Mark

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Benchmark prediction fails exactly where it is needed most: estimating new, stronger models.

desk verdict A careful empirical study whose central 'frontier' result is real under its accuracy-ranked proxy, but the abstract claims more than the design supports. read the letter →

arxiv 2506.07673 v2 pith:4JKIHM35 submitted 2025-06-09 cs.LG

Guanhua Zhang , Florian E. Dorner , Moritz Hardt This is my paper
classification cs.LG
keywords benchmarkpredictionefficientLLMevaluationextrapolationmodelsimilarityaugmentedinversepropensityweightingcore-setselectionfrontierrandomsampling
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tests whether benchmark prediction, estimating a model's full benchmark score from a small evaluated subset, can replace full evaluation. Across 19 benchmarks and 11 methods, it finds that a simple baseline, a random subset plus ridge regression, matches or beats most published methods in the interpolation setting where target models resemble known ones. In the extrapolation setting, where the strongest 30% of models are targets and the weakest 50% serve as sources, nearly every method's estimation gap balloons and none consistently beats simply averaging a random sample. The exception introduced here, an augmented inverse propensity weighting estimator, remains consistent and beats random sampling in 18 of 19 benchmarks, though its gains stay modest. The conclusion is that efficient evaluation works among familiar models but misestimates performance at the evaluation frontier.

What carries the argument

The machine doing the work is the augmented inverse propensity weighting (AIPW) estimator, defined as $$h_{\text{AIPW}}(f) = \bar{s}(f, C) + \frac{1}{1 + \frac{n}{N-n}}\left(\frac{1}{N-n}\sum_{z \in D \setminus C} \hat{s}(f, z) - \frac{1}{n}\sum_{z \in C} \hat{s}(f, z)\right),$$ where $C$ is a random core set of $n$ points and $\hat{s}(f, z)$ is a ridge-regression prediction of the target model's pointwise performance from the source models' pointwise performances. Because AIPW is a consistent estimator, it does not inherit the systematic bias of pure regression when the target model is unlike the sources, and its variance reduction is governed by the correlation between predicted and actual pointwise scores. The contrasting quantity is model similarity, which correlates strongly negatively with estimation gap for all learning-based methods but neutrally for random sampling.

What would settle it

Take a benchmark where new target models are known to be stronger but also architecturally distinct from all source models, and compare AIPW against random-sample averaging at $n = 50$; if AIPW's estimation gap is not below random sampling, the paper's claim that AIPW consistently outperforms under extrapolation fails.

Watch

Extended reading notes

Core claim

The paper's central discovery is that benchmark prediction's success is driven by model similarity, not by core-set selection. When source and target models are drawn from the same pool, fitting a ridge regression to a random subset reduces the estimation gap by 37.2% on average compared with the sample mean, and careful core-set selection adds little. When targets are chosen as the top 30% of models and sources as the bottom 50%, the same methods fail on average: Random-Sampling-Learn's gap rises by about 203%, and the previous best method, Random-Search-Learn, rises by 185.1%. Only the introduced AIPW estimator, a consistent estimator that corrects a regression proxy with the sample mean, beats random sampling on average in this extrapolation regime. The authors read this as evidence that these methods fail where benchmarking matters most, judging new models that outperform everything seen before.

Load-bearing premise

The extrapolation regime is modeled by ranking models on the full benchmark and using the bottom 50% as sources and the top 30% as targets, and the authors note that for HELM-Lite and GLUE the model pool is below 100; if real frontier models differ in kind, or the pool sizes change rankings, the measured failure pattern could differ.

Editorial extensions

If this is right

  • Users of benchmark prediction should include Random-Sampling-Learn as a baseline; in the interpolation regime it outperforms most published methods while costing almost nothing.
  • At the evaluation frontier, the reported gains of published methods do not transfer: none of the ten non-AIPW methods consistently beats a random-sample average when targets are stronger than sources.
  • AIPW offers a safe default among the tested methods, beating random sampling in 18 of 19 benchmarks under extrapolation, though its edge shrinks as the core set grows.
  • Even in interpolation, giving random sampling twice as much data makes it competitive with or better than every tested method, so the case for careful subset selection weakens.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If frontier models differ in kind rather than only in average score, the extrapolation failure observed here likely understates the risk, because the authors' split by full-benchmark rank cannot distinguish the two.
  • AIPW's remaining similarity dependence is tied to the quality of the pointwise regressor, so investing evaluation budget in better cross-model predictors may extend the frontier at which it holds.
  • The near-irrelevance of core-set selection suggests benchmark-prediction research should shift from subset optimization to estimator design and to reporting uncertainty along model-similarity axes.
  • A practical consequence the authors do not spell out is that leaderboards using efficient evaluation could quote both an estimate and the similarity of the new model to the source pool, signaling when the estimate is unreliable.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper studies benchmark prediction (efficient LLM evaluation) with 11 methods across 19 benchmarks, each including at least 84 models. In the interpolation regime (random source/target split), a simple Random-Sampling-Learn baseline—Ridge regression on a randomly sampled core set—outperforms most existing core-set-selection methods, suggesting that careful subset selection is not the main driver of success. In the extrapolation regime (lowest-performing 50% of models as sources, top 30% as targets), most methods fail to beat the simple random-sample average, while the proposed AIPW estimator consistently does, although with modest gains. The paper concludes that benchmark prediction is unreliable precisely at the evaluation frontier, where new and stronger models must be assessed.

Significance. If the empirical pattern holds, the paper is significant: it challenges the core-set-selection orthodoxy in efficient LLM evaluation, offers a strong and simple baseline, and provides a cautionary result about learned evaluators for frontier models. The study's strengths include a large and diverse benchmark suite (19 benchmarks, 84–448 models per benchmark), 100 trials with reported standard errors, ablations over core-set sizes n=50/100/200, and released code. The central limitation is that the extrapolation regime is a synthetic proxy constructed by ranking models on the same full benchmark, so the headline 'frontier' conclusion needs external validation before it can be taken as a general law of efficient evaluation.

major comments (3)
  1. [Section 4.3, 'Extrapolation model split'] The paper's headline conclusion that benchmark prediction 'fails just when it is most needed: at the evaluation frontier' rests entirely on an extrapolation split in which models are ranked by full-benchmark accuracy and the bottom 50% are sources while the top 30% are targets. This operationalizes 'new and better models' as 'higher-scoring members of the same cohort.' Because the split conditions on the target label (full accuracy), the observed failure pattern may be partly mechanical: high-accuracy targets have lower Bernoulli variance, which reduces the Random-Sampling baseline error, while source-trained regressors must extrapolate across a mean shift. Real frontier models may differ in kind (skill profile, error pattern, data distribution) rather than only in average accuracy. Appendix D lists only the small number of models as a limitation and does not flag this proxy threat. The authors should validate the central claim with a temporally or architecturally disjoint split (e.g., models ordered by release date or held-out model families) or otherwise rule out that the finding is an artifact of sorting on the target label.
  2. [Section 3.2, Eq. (3)] The AIPW estimator trains the outcome regression g on the same core-set C that is used in the debiasing term, so the term \bar{\hat{s}}(f,C) is an in-sample prediction. Standard consistency results for augmented inverse propensity weighting and prediction-powered inference require the prediction function to be trained on independent data or cross-fitted. Under overfitting, E[\bar{\hat{s}}(D\setminus C) - \bar{\hat{s}}(C)] is negative, which biases the correction term; the claim in §3.2 that 'AIPW is a consistent estimator' (citing [19]) is therefore not justified as implemented. The authors should either cross-fit g, or report a bias analysis and show that the empirical superiority of AIPW is robust to this issue.
  3. [Section 4.5 and Conclusion] The statement that 'Random-Sampling outperforms all other methods when given twice as much data, even in the interpolation model split' is contradicted by Table 1. In interpolation, Random-Sampling with n=100 has an average estimation gap of 3.3, while Random-Sampling-Learn (2.9) and Random-Search-Learn (2.7) achieve lower gaps at n=50; with n=200, Random-Sampling (2.1) ties Random-Sampling-Learn, P-IRT, and GP-IRT at n=100 and is worse than Random-Search-Learn (2.0). The claim appears to hold only in the extrapolation setting, and even there it needs qualification. This matters because the conclusion repeats the claim as practical advice to increase the sampling budget.
minor comments (4)
  1. [Throughout] There are several typos: 'accross' (Section 4.2), 'Broarder Impacts' (Appendix D title), 'Random-Selection-Learn' in Appendix A.2 for Random-Sampling-Learn, and 'Fon' in the definition of s(f,z) in Section 3.1 ('for any f in F on any data point').
  2. [Figure 4] Entries such as '-0' should be formatted as 0.00 to avoid implying a nonzero negative correlation for Random-Sampling, which the text describes as neutral.
  3. [Section 4.4] The use of Chebyshev's inequality in Eq. (5) explains the role of σ_f, but the subsequent correlation analysis treats the normalized gap as if it removes all variance effects; a sentence clarifying that the normalization is a heuristic would help.
  4. [Appendix A.2, Eq. (17)] The Lasso objective uses s(f,D)^T θ' for a target model f, but θ is fit on source models; the notation should make explicit that θ is trained on sources and applied to targets.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the paper's central extrapolation finding is an empirical result obtained from held-out target models, not a consequence of estimator definitions or self-citations.

full rationale

The paper's central claims are empirically measured rather than derived from their inputs. In the interpolation and extrapolation regimes, regression baselines and AIPW are trained on source-model data and evaluated on held-out target models via the estimation gap in Eq. 1, so the reported failure pattern is not forced by construction. The extrapolation split in Section 4.3 ranks models by full-benchmark accuracy to define sources and targets, which is a proxy assumption about the evaluation frontier; this is an external-validity concern, not circularity, because the full-benchmark labels are not fed into the estimators being tested. AIPW's consistency is cited to an external statistics reference [19], and its variance-reduction bound is attributed to [14], a paper sharing two authors; however, this self-citation only contextualizes AIPW's behavior, while the empirical comparisons in Figures 2-4 and Table 1 stand independently. The model-similarity analysis also guards against a mechanical confound by normalizing the estimation gap by sigma_f in Eq. 6, and the correlation result is explicitly checked against Random-Sampling, which shows neutral correlation. Limitation D about the small number of models is a robustness caveat, not a circular step. No equation in the paper reduces to its own inputs, no fitted parameter is renamed as a prediction, and no load-bearing uniqueness theorem is imported from the authors' prior work.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central empirical conclusions rest on the model-similarity framework, the accuracy-metric restriction, standard regression and AIPW theory, and the constructed extrapolation split. No new physical or conceptual entities are introduced.

free parameters (4)
  • Ridge regularization parameter for Random-Sampling-Learn and AIPW = not reported in text
    Standard hyperparameter for the ridge regressions used in the new baselines; likely default or cross-validated but not specified, so a modest source of ambiguity in replication.
  • Number of PCA components k = selected from {2, 5, 10, 20}
    Chosen by cross-validation for the PCA matrix-completion method; affects PCA imputation accuracy.
  • Lasso sparsity parameter lambda = set so that exactly n coefficients are nonzero
    The lambda value is implicitly chosen to satisfy the sparsity constraint; standard Lasso tuning but not fully specified.
  • GP-IRT mixing weight lambda = chosen heuristically per original paper
    Controls the combination of Anchor-Points-Weighted and P-IRT; inherited from prior work, not fitted in this paper.
assumptions (5)
  • standard math AIPW is a consistent estimator of the full-benchmark mean.
    Consistency is cited from Glynn and Quinn [19] and used to explain why AIPW outperforms Random-Sampling under extrapolation; the proof is not reproduced in this paper.
  • standard math Variance reduction of AIPW depends on the Pearson correlation between predicted and true per-point scores.
    Equation 3 references a variance reduction factor from Dorner et al. [14]; this background result supports the claim that AIPW helps less when predictions are uninformative.
  • domain assumption Model similarity as defined in Equation 4 is the right measure for interpolation difficulty.
    The paper uses Mania et al. [36] similarity to explain when benchmark prediction works; this is a modeling choice, not a derived fact.
  • ad hoc to paper The extrapolation split, bottom 50% of models by full accuracy as sources and top 30% as targets, faithfully represents the evaluation frontier.
    This proxy is central to the headline finding but is constructed from the same benchmark scores; it is not validated on chronologically new or qualitatively different frontier models.
  • domain assumption Per-point performance is Bernoulli, allowing normalization by sigma_f.
    The normalized estimation gap in Equation 6 divides by sqrt(p(1-p)), and the method set is restricted to accuracy benchmarks; non-accuracy metrics are out of scope.

how reviews work

0 comments
Cite this review

Pith. "Pith review of How Benchmark Prediction from Fewer Data Misses the Mark." pith.science (2026). https://pith.science/paper/4JKIHM35

@misc{pith2026250607673,
  author       = {Pith},
  title        = {Pith review of: How Benchmark Prediction from Fewer Data Misses the Mark},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4JKIHM35}},
  note         = {Machine review of arXiv:2506.07673}
}
read the original abstract

Large language model (LLM) evaluation is increasingly costly, prompting interest in methods that speed up evaluation by shrinking benchmark datasets. Benchmark prediction (also called efficient LLM evaluation) aims to select a small subset of evaluation points and predict overall benchmark performance from that subset. In this paper, we systematically assess the strengths and limitations of 11 benchmark prediction methods across 19 diverse benchmarks. First, we identify a highly competitive baseline: Take a random sample and fit a regression model on the sample to predict missing entries. Outperforming most existing methods, this baseline challenges the assumption that careful subset selection is necessary for benchmark prediction. Second, we discover that all existing methods crucially depend on model similarity. They work best when interpolating scores among similar models. The effectiveness of benchmark prediction sharply declines when new models have higher accuracy than previously seen models. In this setting of extrapolation, none of the previous methods consistently beat a simple average over random samples. To improve over the sample average, we introduce a new method inspired by augmented inverse propensity weighting. This method consistently outperforms the random sample average even for extrapolation. However, its performance still relies on model similarity and the gains are modest in general. This shows that benchmark prediction fails just when it is most needed: at the evaluation frontier, where the goal is to evaluate new models of unknown capabilities.

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Certified in Theory, Broken in Practice: Assumption Gaps in Cryptographic Model Certification

    cs.CR 2026-07 accept novelty 6.0 of 10

    Certifying a model only on a fixed audit dataset is vulnerable to data-forging; committing before sampling fresh audit data restores distributional security.

Reference graph

Works this paper leans on

66 extracted references · 30 canonical work pages · cited by 1 Pith paper

  1. [19]

    An introduction to the augmented inverse propensity weighted estimator

    Adam N Glynn and Kevin M Quinn. An introduction to the augmented inverse propensity weighted estimator. Political analysis, 18(1):36–56, 2010

    2010

  2. [1]

    Jordan, and Tijana Zrnic

    Anastasios Nikolas Angelopoulos, Stephen Bates, Clara Fannjiang, Michael I. Jordan, and Tijana Zrnic. Prediction-powered inference. Science, 382:669 – 674, 2023

    work page 2023

  3. [2]

    Duchi, and Tijana Zrnic

    Anastasios Nikolas Angelopoulos, John C. Duchi, and Tijana Zrnic. Ppi++: Efficient prediction-powered inference. ArXiv, abs/2311.01453, 2023

    arXiv 2023

  4. [3]

    Courville

    Yoshua Bengio, Nicholas Léonard, and Aaron C. Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. ArXiv, abs/1308.3432, 2013

    arXiv 2013

  5. [4]

    The fifth PASCAL recognizing textual entailment challenge

    Luisa Bentivogli, Ido Dagan, Hoa Trang Dang, Danilo Giampiccolo, and Bernardo Magnini. The fifth PASCAL recognizing textual entailment challenge. 2009

    2009

  6. [5]

    Autoeval done right: Using synthetic data for model evaluation

    Pierre Boyeau, Anastasios N Angelopoulos, Nir Yosef, Jitendra Malik, and Michael I Jor- dan. Autoeval done right: Using synthetic data for model evaluation. arXiv preprint arXiv:2403.07008, 2024

    arXiv 2024

  7. [6]

    A singular value thresholding algorithm for matrix completion

    Jian-Feng Cai, Emmanuel J Candès, and Zuowei Shen. A singular value thresholding algorithm for matrix completion. SIAM Journal on optimization, 20(4):1956–1982, 2010

    1956

  8. [7]

    Humans or llms as the judge? a study on judgement biases

    Guiming Hardy Chen, Shunian Chen, Ziche Liu, Feng Jiang, and Benyou Wang. Humans or llms as the judge? a study on judgement biases. arXiv preprint arXiv:2402.10669, 2024

    arXiv 2024

Show all 66 references
  1. [8]

    Think you have solved question answering? try arc, the ai 2 reasoning challenge

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai 2 reasoning challenge. arXiv:1803.05457v1, 2018

    2018 arXiv

  2. [9]

    Training verifiers to solve math word problems

    Karl Cobbe, Vineet Kosaraju, Mo Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schul- man. Training verifiers to solve math word problems. ArXiv, abs/2110.14168, 2021

    arXiv

  3. [10]

    Computing the testing error without a testing set

    Ciprian A Corneanu, Sergio Escalera, and Aleix M Martinez. Computing the testing error without a testing set. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2677–2685, 2020

    2020

  4. [11]

    The PASCAL recognising textual entailment challenge

    Ido Dagan, Oren Glickman, and Bernardo Magnini. The PASCAL recognising textual entailment challenge. In Machine learning challenges. evaluating predictive uncertainty, visual object classification, and recognising tectual entailment, pages 177–190. Springer, 2006

    2006

  5. [12]

    Are labels always necessary for classifier accuracy evaluation? In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 15069– 15078, 2021

    Weijian Deng and Liang Zheng. Are labels always necessary for classifier accuracy evaluation? In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 15069– 15078, 2021

    2021

  6. [13]

    Automatically constructing a corpus of sentential paraphrases

    William B Dolan and Chris Brockett. Automatically constructing a corpus of sentential paraphrases. In Proceedings of the International Workshop on Paraphrasing , 2005

    2005

  7. [14]

    Limits to scalable evaluation at the frontier: Llm as judge won’t beat twice the data

    Florian E Dorner, Vivian Yvonne Nastl, and Moritz Hardt. Limits to scalable evaluation at the frontier: Llm as judge won’t beat twice the data. In The Thirteenth International Conference on Learning Representations, 2025. 15

    2025

  8. [15]

    Facility location: concepts, models, algo- rithms and case studies

    Reza Zanjirani Farahani and Masoud Hekmatfar. Facility location: concepts, models, algo- rithms and case studies. 2009

    2009

  9. [16]

    Open llm leaderboard v2

    Clémentine Fourrier, Nathan Habib, Alina Lozovskaya, Konrad Szafer, and Thomas Wolf. Open llm leaderboard v2. https://huggingface.co/spaces/open-llm-leaderboard/open _ llm_leaderboard, 2024

    2024

  10. [17]

    Challenges in evaluating AI systems, 2023

    Deep Ganguli, Nicholas Schiefer, Marina Favaro, and Jack Clark. Challenges in evaluating AI systems, 2023

    2023

  11. [18]

    The third PASCAL recognizing textual entailment challenge

    Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and Bill Dolan. The third PASCAL recognizing textual entailment challenge. In Proceedings of the ACL-P ASCAL workshop on textual entailment and paraphrasing , pages 1–9. Association for Computational Linguistics, 2007

    2007

  12. [20]

    Chandra, Ponnurangam Kumaraguru, Douwe Kiela, Ameya Prabhu, Matthias Bethge, and Jonas Geiping

    Shashwat Goel, Joschka Struber, Ilze Amanda Auzina, Karuna K. Chandra, Ponnurangam Kumaraguru, Douwe Kiela, Ameya Prabhu, Matthias Bethge, and Jonas Geiping. Great models think alike and this undermines ai oversight. ArXiv, abs/2502.04313, 2025

    2025 arXiv

  13. [21]

    A survey on llm-as-a-judge

    Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, Yuanzhuo Wang, and Jian Guo. A survey on llm-as-a-judge. ArXiv, abs/2411.15594, 2024

    2024 arXiv

  14. [22]

    Ho, Christopher Ré, Adam Chilton, Aditya Narayana, Alex Chohlas-Wood, Austin Peters, Brandon Waldon, Daniel N

    Neel Guha, Julian Nyarko, Daniel E. Ho, Christopher Ré, Adam Chilton, Aditya Narayana, Alex Chohlas-Wood, Austin Peters, Brandon Waldon, Daniel N. Rockmore, Diego Zam- brano, Dmitry Talisman, Enam Hoque, Faiz Surani, Frank Fagan, Galit Sarfaty, Gregory M. Dickinson, Haggai Por...

    2023

  15. [23]

    Is gpt- 4 a reliable rater? evaluating consistency in gpt- 4 text ratings

    Veronika Hackl, Alexandra Elena Müller, Michael Granitzer, and Maximilian Sailer. Is gpt- 4 a reliable rater? evaluating consistency in gpt- 4 text ratings. ArXiv, abs/2308.02575, 2023

    2023 arXiv

  16. [24]

    Test-time training on nearest neighbors for large language models

    Moritz Hardt and Yu Sun. Test-time training on nearest neighbors for large language models. arXiv preprint arXiv:2305.18466, 2023

    2023 arXiv

  17. [25]

    Measuring massive multitask language understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Xiaodong Song, and Jacob Steinhardt. Measuring massive multitask language understanding. ArXiv, abs/2009.03300, 2020

    2009 arXiv

  18. [26]

    Measuring mathematical problem solving with the math dataset

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Xiaodong Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. ArXiv, abs/2103.03874, 2021. 16

    2021 arXiv

  19. [27]

    Categorical reparameterization with gumbel- softmax

    Eric Jang, Shixiang Shane Gu, and Ben Poole. Categorical reparameterization with gumbel- softmax. ArXiv, abs/1611.01144, 2016

    2016 arXiv

  20. [28]

    What disease does this patient have? a large-scale open domain question answering dataset from medical exams

    Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. What disease does this patient have? a large-scale open domain question answering dataset from medical exams. ArXiv, abs/2009.13081, 2020

    2009 arXiv

  21. [29]

    Scaling laws for neural language models

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020

    2001 arXiv

  22. [30]

    Active testing: Sample- efficient model evaluation

    Jannik Kossen, Sebastian Farquhar, Yarin Gal, and Tom Rainforth. Active testing: Sample- efficient model evaluation. In International Conference on Machine Learning, 2021

    2021

  23. [31]

    Active surrogate estimators: An active learning approach to label-efficient model evaluation

    Jannik Kossen, Sebastian Farquhar, Yarin Gal, and Tom Rainforth. Active surrogate estimators: An active learning approach to label-efficient model evaluation. ArXiv, abs/2202.06881, 2022

    2022 arXiv

  24. [32]

    Retrieval- augmented generation for knowledge-intensive nlp tasks

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Na- man Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval- augmented generation for knowledge-intensive nlp tasks. Advances in neural information processing syst...

    2020

  25. [33]

    Active evaluation acquisition for efficient llm benchmarking

    Yang Li, Jie Ma, Miguel Ballesteros, Yassine Benajiba, and Graham Horwood. Active evaluation acquisition for efficient llm benchmarking. ArXiv, abs/2410.05952, 2024

    2024 arXiv

  26. [34]

    Manning, Christopher R’e, Diana Acosta-Navas, Drew A

    Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Cosgrove, Christopher D. Manning, Christopher R’e, Diana Acosta-Nava...

    2023

  27. [35]

    Singh, Rylan Schaeffer, Andrew Poulton, Oluwasanmi Koyejo, Pontus Stenetorp, Sharan Narang, and Dieuwke Hupkes

    Lovish Madaan, Aaditya K. Singh, Rylan Schaeffer, Andrew Poulton, Oluwasanmi Koyejo, Pontus Stenetorp, Sharan Narang, and Dieuwke Hupkes. Quantifying variance in evaluation benchmarks. ArXiv, abs/2406.10229, 2024

    2024 arXiv

  28. [36]

    Model similarity mitigates test set overuse

    Horia Mania, John Miller, Ludwig Schmidt, Moritz Hardt, and Benjamin Recht. Model similarity mitigates test set overuse. ArXiv, abs/1905.12580, 2019

    1905 arXiv

  29. [37]

    Can a suit of armor conduct electricity? a new dataset for open book question answering

    Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun’ichi Tsujii, editors, Proceedings of the 2018 Conference on Empiri...

    2018

  30. [38]

    How predictable is language model benchmark performance? ArXiv, abs/2401.04757, 2024

    David Owen. How predictable is language model benchmark performance? ArXiv, abs/2401.04757, 2024

    2024 arXiv

  31. [39]

    Predictaboard: Benchmarking llm score predictability

    Lorenzo Pacchiardi, Konstantinos Voudouris, Ben Slater, Fernando Mart’inez-Plumed, Jos’e Hern’andez-Orallo, Lexin Zhou, and Wout Schellaert. Predictaboard: Benchmarking llm score predictability. ArXiv, abs/2502.14445, 2025

    2025 arXiv

  32. [40]

    Bowman, and Shi Feng

    Arjun Panickssery, Samuel R. Bowman, and Shi Feng. Llm evaluators recognize and favor their own generations. ArXiv, abs/2404.13076, 2024

    2024 arXiv

  33. [41]

    tinybenchmarks: evaluating llms with fewer examples

    Felipe Maia Polo, Lucas Weber, Leshem Choshen, Yuekai Sun, Gongjun Xu, and Mikhail Yurochkin. tinybenchmarks: evaluating llms with fewer examples. ArXiv, abs/2402.14992, 2024

    2024 arXiv

  34. [42]

    Efficient multi-prompt evaluation of llms

    Felipe Maia Polo, Ronald Xu, Lucas Weber, Mírian Silva, Onkar Bhardwaj, Leshem Choshen, Allysson Flavio Melo de Oliveira, Yuekai Sun, and Mikhail Yurochkin. Efficient multi-prompt evaluation of llms. arXiv preprint arXiv:2405.17202, 2024

    2024 arXiv

  35. [43]

    SQuAD: 100,000+ questions for machine comprehension of text

    Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of EMNLP, pages 2383–2392. Association for Computational Linguistics, 2016

    2016

  36. [44]

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. Gpqa: A graduate-level google-proof q&a benchmark. ArXiv, abs/2311.12022, 2023

    2023 arXiv

  37. [45]

    Semiparametric efficiency in multivariate regression models with missing data

    James M Robins and Andrea Rotnitzky. Semiparametric efficiency in multivariate regression models with missing data. Journal of the American Statistical Association , 90(429):122–129, 1995

    1995

  38. [46]

    Lalor, Robin Jia, and Jordan L

    Pedro Rodriguez, Joe Barrow, Alexander Miserlis Hoyle, John P . Lalor, Robin Jia, and Jordan L. Boyd-Graber. Evaluation examples are not equally informative: How should that change nlp leaderboards? In Annual Meeting of the Association for Computational Linguistics , 2021

    2021

  39. [47]

    Maddison, and Tatsunori B

    Yangjun Ruan, Chris J. Maddison, and Tatsunori B. Hashimoto. Observational scaling laws and the predictability of language model performance. ArXiv, abs/2405.10938, 2024

    2024 arXiv

  40. [48]

    Bernstein, Alexander C

    Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael S. Bernstein, Alexander C. Berg, and Li Fei-Fei. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, ...

    2014

  41. [49]

    Data distillation: A survey

    Noveen Sachdeva and Julian McAuley. Data distillation: A survey. Trans. Mach. Learn. Res., 2023, 2023

    2023

  42. [50]

    Scaling llm test-time compute optimally can be more effective than scaling model parameters.arXiv preprint arXiv:2408.03314, 2024

    Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling llm test-time compute optimally can be more effective than scaling model parameters.arXiv preprint arXiv:2408.03314, 2024

    2024 arXiv

  43. [51]

    Recursive deep models for semantic compositionality over a sentiment treebank

    Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of EMNLP, pages 1631–1642, 2013. 18

    2013

  44. [52]

    Musr: Testing the limits of chain-of-thought with multistep soft reasoning

    Zayne Sprague, Xi Ye, Kaj Bostrom, Swarat Chaudhuri, and Greg Durrett. Musr: Testing the limits of chain-of-thought with multistep soft reasoning. ArXiv, abs/2310.16049, 2023

    2023 arXiv

  45. [53]

    Test-time training with self-supervision for generalization under distribution shifts

    Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei Efros, and Moritz Hardt. Test-time training with self-supervision for generalization under distribution shifts. In International conference on machine learning, pages 9229–9248. PMLR, 2020

    2020

  46. [54]

    Le, Ed H

    Mirac Suzgun, Nathan Scales, Nathanael Scharli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V . Le, Ed H. Chi, Denny Zhou, and Jason Wei. Challenging big-bench tasks and whether chain-of-thought can solve them. In Annual Meeting of the Association fo...

    2022

  47. [55]

    Four lectures on probabilistic methods for data science

    Roman Vershynin. Four lectures on probabilistic methods for data science. ArXiv, abs/1612.06661, 2016

    2016 arXiv

  48. [56]

    Anchor points: Benchmarking models with much fewer examples

    Rajan Vivek, Kawin Ethayarajh, Diyi Yang, and Douwe Kiela. Anchor points: Benchmarking models with much fewer examples. ArXiv, abs/2309.08638, 2023

    2023 arXiv

  49. [57]

    Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. In BlackboxNLP@EMNLP, 2018

    2018

  50. [58]

    Ku, Kai Wang, Alex Zhuang, Rongqi "Richard" Fan, Xiang Yue, and Wenhu Chen

    Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max W.F. Ku, Kai Wang, Alex Zhuang, Rongqi "Richard" Fan, Xiang Yue, and Wenhu Chen. Mmlu-pro: A more robust and challenging multi-task ...

    2024 arXiv

  51. [59]

    Self-preference bias in llm-as-a-judge

    Koki Wataoka, Tsubasa Takahashi, and Ryokan Ri. Self-preference bias in llm-as-a-judge. ArXiv, abs/2410.21819, 2024

    2024 arXiv

  52. [60]

    Adina Williams, Nikita Nangia, and Samuel R. Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of NAACL-HLT, 2018

    2018

  53. [61]

    Skill-mix: a flexible and expandable family of evaluations for ai models

    Dingli Yu, Simran Kaur, Arushi Gupta, Jonah Brown-Cohen, Anirudh Goyal, and Sanjeev Arora. Skill-mix: a flexible and expandable family of evaluations for ai models. ArXiv, abs/2310.17567, 2023

    2023 arXiv

  54. [62]

    Automatic evaluation of attribution by large language models

    Xiang Yue, Boshi Wang, Kai Zhang, Ziru Chen, Yu Su, and Huan Sun. Automatic evaluation of attribution by large language models. In Conference on Empirical Methods in Natural Language Processing, 2023

    2023

  55. [63]

    Instruction-following evaluation for large language models

    Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-following evaluation for large language models. ArXiv, abs/2311.07911, 2023

    2023 arXiv

  56. [64]

    Belardi, Ruihan Wu, Travis Zhang, Carla P

    Jin Peng Zhou, Christian K. Belardi, Ruihan Wu, Travis Zhang, Carla P . Gomes, Wen Sun, and Kilian Q. Weinberger. On speeding up language model evaluation. ArXiv, abs/2407.06172, 2024

    2024 arXiv

  57. [65]

    Xiao Zhou, Renjie Pi, Weizhong Zhang, Yong Lin, Zonghao Chen, and T. Zhang. Probabilistic bilevel coreset selection. ArXiv, abs/2301.09880, 2023. 19

    2023 arXiv

  58. [66]

    How to select datapoints for efficient human evaluation of nlg models?, 2025

    Vilém Zouhar, Peng Cui, and Mrinmaya Sachan. How to select datapoints for efficient human evaluation of nlg models?, 2025. 20 A Details of Benchmark Prediction Methods A.1 Problem Formulation We repeat the notation and the problem formulation here for the reader’s convenience....

    2025

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.