Pith. sign in

REVIEW 3 major objections 5 minor 7 cited by

Cer-Eval: Certifiable and Cost-Efficient Evaluation Framework for LLMs

T0 review · 3 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Cer-Eval is an online evaluation algorithm that certifies LLM performance with confidence intervals while using 20-40% fewer test points than full-benchmark evaluation.

desk verdict Cer-Eval has a real idea—formal test sample complexity with an adaptive partition algorithm—but the main theorem's guarantee doesn't transfer to the implemented partition subroutine, so the central certification claim is currently unproven. read the letter →

arxiv 2505.03814 v1 pith:MA54ZLBF submitted 2025-05-02 stat.ML cs.AIcs.CLcs.LG

Ganghua Wang , Zhaorun Chen , Bo Li , Haifeng Xu This is my paper · ORCID
classification stat.MLcs.AIcs.CLcs.LG MSC 62L1062G1568T50
keywords LLMevaluationtestsamplecomplexityadaptiveonlineconfidenceintervalssequentialanalysisvariancereductionbenchmarkefficiencycertified
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper aims to replace the standard practice of evaluating a large language model by averaging over an entire benchmark with a sequential, adaptive process that stops as soon as the estimate is within a user-chosen error epsilon with confidence 1-delta. It introduces test sample complexity, the minimum number of test points any certified evaluator needs, and gives matching upper and lower bounds under a bounded-loss assumption. It then shows that partitioning the test data into low-variance groups can shrink the required sample size, and builds the Cer-Eval algorithm that learns such partitions adaptively via 1-nearest-neighbor clustering in embedding space. On MMLU, AlpacaEval, and MATH with models including GPT-4o, Cer-Eval reports comparable estimation error while using roughly 60-80% of the test data (20-40% savings) at the 95% confidence level.

What carries the argument

The load-bearing object is a confidence-interval radius per partition group, built from an adaptive Bernstein-type inequality. For each group k, Cer-Eval computes eta_k = $\sqrt$((2 ln(log(n_k)+1)+ln(16K/delta))/n_k) and epsilon_k = 2 $eta_k^{2}$/3 + 2 $\sqrt$((v_k + eta_k + $eta_k^{2}$) $eta_k^{2}$); termination occurs when the weighted average epsilon_hat = sum_j N_j epsilon_j / N falls below the user's epsilon. The 1-nearest-neighbor partition subroutine chooses, among candidate labelings by discretized loss values, the one that minimizes epsilon_hat. This machinery turns variance reduction into sample savings: a benign partition with small within-group variances v_k makes each epsilon_k shrink faster, so the algorithm certifies accuracy with fewer evaluated points.

What would settle it

Compute, on a real benchmark, the empirical per-group sample fractions n_k/n and variances v_k at Cer-Eval's termination; if for some group n_k/n < ln(K+1) max{v_k, $epsilon^{{2/3}}$} while the algorithm still claims certification, then the condition behind the sample-savings theorem is violated and the guarantee does not follow from the paper's theory. A direct experiment would run Cer-Eval on a dataset whose loss is determined by a feature absent from the embedding vectors and check whether the reported confidence interval covers the full-dataset average at the claimed 1-delta rate.

Watch

Extended reading notes

Core claim

The central claim is that certifiable LLM evaluation need not evaluate all test points: an (n, epsilon, delta)-certified online algorithm can stop early and still guarantee that its reported interval contains the true error with probability at least 1-delta. The paper proves that with only a bounded-loss assumption the test sample complexity is Theta($epsilon^{{-2}}$(ln(1/delta)+ln ln(1/epsilon))), and that with a benign partition that concentrates probability mass on low-variance regions the required sample count drops by a factor rho = O(ln(K+1) sum_k max{v_k, $epsilon^{{2/3}}$}). Cer-Eval implements this idea by repeatedly partitioning the embedding space with 1-nearest-neighbor, estimating per-group mean and variance, and sampling from the group that reduces the confidence-interval radius most. In experiments the algorithm matches static full-dataset evaluation error with an empirical failure probability of zero while saving 20-40% of test points on MMLU, AlpacaEval, and MATH for GPT-4o and similar models at 95% confidence.

Load-bearing premise

The 20-40% savings guarantee rests on the assumption that the algorithm's learned partition is benign—each group must contain enough points relative to its true loss variance—and the paper does not prove that its 1-nearest-neighbor partition subroutine produces such a partition.

Editorial extensions

If this is right

  • Users no longer need to evaluate an entire benchmark: with Cer-Eval they can stop once the confidence interval radius reaches a user-set epsilon, and the output carries a 1-delta coverage guarantee.
  • If the partition-based savings hold, benchmark curators can use test sample complexity to decide whether a dataset is large enough for a target error, or to tell users more data are needed (the paper reports AlpacaEval needs more points at small epsilon).
  • The apparent scaling law between model size and required test points is explained by accuracy near 0.5: as models improve beyond 50% accuracy, needed test points for fixed error and confidence need not grow with model size.
  • The framework replaces static averages with a sequential process that adapts when epsilon is implicit, such as deciding whether accuracy exceeds 0.5, or changes during evaluation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not show that its 1-nearest-neighbor partition always yields benign partitions; if it does not, the stated savings on a particular benchmark could fall below 20% even though the confidence guarantee may still hold empirically.
  • If better embeddings or task-specific attributes make learned partitions closer to benign, the observed savings could exceed 40%; the paper's own ablation points to partition quality, not the embedding model, as the limiting factor.
  • Applying the same certified stopping idea to model ranking could stop as soon as the confidence interval separates two models, using an error level tied to their performance gap.
  • Instrumenting Cer-Eval to report per-group n_k, v_k, and the benign-partition condition at termination would turn the theoretical assumption into a per-run diagnostic.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes Cer-Eval, an online, adaptive framework for LLM evaluation. It formalizes an (n, ε, δ)-certified evaluation algorithm, defines test sample complexity, establishes upper and lower bounds under a bounded-loss assumption (Theorem 4.2), and shows that with a 'benign partition' the required sample size can be reduced by a factor controlled by the within-partition variances (Theorem 5.2). The authors then present Algorithm 1 (Cer-Eval) with a 1-nearest-neighbor partition subroutine (Algorithm 2), and report simulations and real-benchmark experiments on MMLU, AlpacaEval, and MATH claiming 20–40% test-point savings at 95% confidence.

Significance. If the guarantees transferred to the implemented algorithm, this would be a useful contribution: the framework addresses a real gap in current LLM evaluation practice, which typically reports full-dataset averages without finite-sample reliability guarantees. The adaptive Hoeffding/Bernstein machinery is appropriate, and the test-sample-complexity view is a meaningful formalization. The main theoretical ideas are sensible. However, the certification of the implemented algorithm is not established: Theorem 5.2 applies only when Algorithm 1 is run with a known benign partition, while Algorithm 2 is not shown to produce one. In addition, the reported empirical failure probability validates coverage of the finite-dataset average rather than the distributional target in Definition 3.4. These gaps are load-bearing for the paper's headline claim, but they appear addressable rather than fatal.

major comments (3)
  1. [Theorem 4.2 and Appendix A] The lower-bound statement in Theorem 4.2 has a sign error. The theorem states that no algorithm can be (n', ε, δ)-certified whenever lim n' ε^2 / (ln(1/δ) + ln ln(1/ε)) > 0, but a lower bound should rule out sample sizes that grow slower than the benchmark rate, i.e., the limit should be 0. The proof in Appendix A indeed assumes equation (5) with the limit equal to 0, and the printed theorem is therefore inconsistent with its own proof. As stated, the theorem is false for any n' that scales faster than the benchmark rate. This must be corrected, and the statement and proof should be reconciled.
  2. [Definition 5.1, Theorem 5.2, Algorithm 2] The main certification result does not transfer to the implemented algorithm. Theorem 5.2 guarantees that Algorithm 1 is (n', ε, δ)-certified only when it operates with a benign partition satisfying n_k/n ≥ ln(K+1) max{v_k, ε^{2/3}} for the true conditional variances v_k, and the proof explicitly says 'the known benign partition will be used as the partition subroutine.' Algorithm 2, however, produces partitions by training a 1-NN classifier on quantized loss labels and choosing K to minimize the estimated CI radius; the paper never shows that these learned partitions satisfy Definition 5.1. Since the partition is a random function of the evaluated samples and is selected to minimize an estimated width, the per-group sample sizes n_k are not controlled against the true v_k. Consequently, the certified guarantee and the claimed 20–40% savings are not established for Cer-Eval as implemented. The paper should either prove a benign-partition property for Algorithm 2 under explicit conditions, or clearly separate the certified guarantees (oracle partition) from the empirical evaluation of the heuristic.
  3. [Section 6.2 and Definition 3.4] The empirical failure probability in the real-world experiments is computed as the proportion of trials where the confidence interval does not contain the model's average performance across the entire dataset, not the true expected loss R = E_{(X,Y)∼P_XY} ℓ(f(X),Y) targeted by Definition 3.4. As a result, the reported zero empirical failure probability does not validate the (n, ε, δ)-certification claim. The authors should either evaluate coverage on held-out test data drawn from P_XY, or explicitly state that the experiments validate coverage of the finite-dataset average and adjust the strength of the claim accordingly.
minor comments (5)
  1. [Algorithm 1, line 14] The word 'raduis' should be 'radius'.
  2. [Example 3.1] The word 'basline' should be 'baseline'.
  3. [Appendix A, Proof of Theorem 5.3] The proof refers to 'Theorem 5.1' and 'Theorem 4.1', but the relevant statements appear to be Theorem 5.2 and Theorem 4.2; the cross-references should be corrected.
  4. [Algorithm 2, line 3] The phrase 'a random subset of S' is underspecified; the size of the subset and the sampling procedure should be stated for reproducibility.
  5. [Appendix D] The ablation discussion states that Algorithm 2 'does not extract enough information from the embedding vectors'; this is a relevant caveat and should be acknowledged in the main text alongside the savings claims, since it further weakens the link between the theoretical savings and the implemented method.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the sample-complexity bounds are derived from external concentration inequalities, and the 20-40% savings are empirical stopping-rule outcomes, not fitted inputs or self-citation chains.

full rationale

The paper's central derivation chain is self-contained against external statistical results. The upper bound in Theorem 4.2 follows from an adaptive Hoeffding inequality (Lemma B.3, citing Zhao et al. [31]) and the lower bound from Farrell [11] and Mannor & Tsitsiklis [24]; neither is the authors' own prior work. Theorem 5.2 is a genuine conditional result: it proves that Algorithm 1, when supplied with a partition satisfying Definition 5.1 with the true conditional variances v_k, is (n', eps, delta)-certified with n'/n = O(ln(K+1) sum max{v_k, eps^{2/3}}). The quantities bR_k and bv_k in Algorithm 1 are online empirical means and variances used inside a Bernstein-type confidence interval; they are not parameters fitted to a subset and then reported as a prediction of a closely related quantity. The claimed 20-40% savings are read directly from the number of evaluated points at termination in simulations and real benchmarks, compared with the static baseline, so they are not manufactured by construction. The authors' own earlier works cited in the paper [5,6,7,27] are benchmark or safety-evaluation papers used for context, not as load-bearing support for the sample-complexity theorems, so there is no self-citation chain forcing the result. The main weakness is a correctness gap rather than circularity: the implemented partition subroutine (Algorithm 2) is not proved to produce the benign partition required by Theorem 5.2, and the empirical failure check in Section 6.2 targets the average over the finite dataset rather than the distributional risk R. But a validity gap between theorem assumptions and implementation is not an instance of a derivation reducing to its own inputs. Accordingly, no circular step is exhibited and the score is 0.

Assumptions & free parameters 2 free parameters · 7 assumptions · 0 invented entities

The central bounds depend on the standard bounded-IID assumption plus a benign-partition condition that the algorithm is not proven to satisfy. There are no fitted constants in the derivation, but the empirical method has unspecified hyperparameters (warm start m, adaptive K, embedding model).

free parameters (2)
  • Warm start size m in Algorithm 1 = not specified
    Controls how many points are evaluated before the first partition; the main text never states the value used in experiments, and it affects early stopping behavior.
  • Partition size K = chosen adaptively by Algorithm 2
    Algorithm 2 selects K by minimizing an estimated confidence interval radius over k = 1 to ceil(ln|S|)+1; this is a data-dependent model choice, not a fixed fitted constant, but it influences the realized savings.
assumptions (7)
  • domain assumption Loss function is bounded, 0 <= l(f(X),Y) <= 1.
    Assumption 4.1, the foundation for all Hoeffding and Bernstein bounds.
  • domain assumption Test data D_n are IID draws from the task distribution P_XY.
    Section 3 states this, and it underpins the concentration inequalities and the (n, epsilon, delta) certificate.
  • domain assumption The benchmark dataset D_n represents the underlying task distribution P_XY.
    Section 3 uses this to justify treating the full benchmark average as the object of inference in experiments.
  • ad hoc to paper A benign partition exists: n_k/n >= ln(K+1) * max{v_k, epsilon^{2/3}} for the true conditional variances v_k.
    Definition 5.1 and Theorem 5.2; the sample-efficiency ratio is only certified if this condition holds.
  • domain assumption Group probability masses mu_k are known, or can be estimated without cost, in the proof of Theorem 5.2.
    Appendix A says 'without loss of generality, we assume that mu_i is known', but this costs extra unlabeled draws and is not addressed in the experiments.
  • ad hoc to paper The 1-nearest-neighbor partition subroutine in Algorithm 2 produces approximately benign partitions.
    Section 5 and Algorithm 2; no theorem proves this, yet the real-world savings claims depend on it.
  • standard math Standard Hoeffding, Bernstein, and adaptive Hoeffding/Bernstein concentration inequalities are valid.
    Lemmas B.1-B.5 cite standard results used throughout the proofs.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Cer-Eval: Certifiable and Cost-Efficient Evaluation Framework for LLMs." pith.science (2026). https://pith.science/paper/MA54ZLBF

@misc{pith2026250503814,
  author       = {Pith},
  title        = {Pith review of: Cer-Eval: Certifiable and Cost-Efficient Evaluation Framework for LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MA54ZLBF}},
  note         = {Machine review of arXiv:2505.03814}
}
read the original abstract

As foundation models continue to scale, the size of trained models grows exponentially, presenting significant challenges for their evaluation. Current evaluation practices involve curating increasingly large datasets to assess the performance of large language models (LLMs). However, there is a lack of systematic analysis and guidance on determining the sufficiency of test data or selecting informative samples for evaluation. This paper introduces a certifiable and cost-efficient evaluation framework for LLMs. Our framework adapts to different evaluation objectives and outputs confidence intervals that contain true values with high probability. We use ``test sample complexity'' to quantify the number of test points needed for a certifiable evaluation and derive tight bounds on test sample complexity. Based on the developed theory, we develop a partition-based algorithm, named Cer-Eval, that adaptively selects test points to minimize the cost of LLM evaluation. Real-world experiments demonstrate that Cer-Eval can save 20% to 40% test points across various benchmarks, while maintaining an estimation error level comparable to the current evaluation process and providing a 95% confidence guarantee.

Figures

Figures reproduced from arXiv: 2505.03814 by the authors.

Figure 1
Figure 1. Overview of Cer-Eval, a partition-based adaptive evaluation algorithm. It iterates through [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Percentage of test points saved by Cer-Eval compared to the baselines on [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Percentage of test points saved by Cer-Eval compared to baselines in [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Number of needed test points v.s. model size when evaluating models from multiple fam￾ilies using Cer-Eval, with estimation error level ϵ = 0.07 and failure probability δ = 0.05. 0.0 0.2 0.4 0.6 0.8 Model accuracy 200 300 400 500 600 Test sample size Qwen1.5 Meta gemma…
Figure 6
Figure 6. Figure 6: Percentage of test points saved by the proposed algorithms compared to Base when [PITH_FULL_IMAGE:figures/full_fig_p017_6.png]
Figure 7
Figure 7. Figure 7: Percentage of test points saved by the proposed algorithms compared to Base when [PITH_FULL_IMAGE:figures/full_fig_p018_7.png]
Figure 8
Figure 8. Figure 8: Percentage of test points saved by the proposed algorithms compared to Base when [PITH_FULL_IMAGE:figures/full_fig_p019_8.png]
Figure 9
Figure 9. Figure 9: Number of needed test points v.s. model accuracy when evaluating models from multiple [PITH_FULL_IMAGE:figures/full_fig_p019_9.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 7 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. How Many Iterations to Jailbreak? Dynamic Budget Allocation for Multi-Turn LLM Evaluation

    cs.LG 2026-05 unverdicted novelty 7.0 of 10

    DAPRO provides the first dynamic, theoretically guaranteed way to allocate interaction budgets across test cases for bounding time-to-event in multi-turn LLM evaluations, achieving tighter coverage than static conform...

  2. RouteGuard: Certifying Routing Gain in LLM Multi-Agent Systems When Complementarity Is Not Enough

    stat.ML 2026-08 conditional novelty 6.0 of 10

    Routing gain among LLM advisors can be certified with a finite-sample bracket and a matching minimax lower bound, and certification fails on uninformative gates and statistically redundant advisor pools.

  3. BayesAME: Bayesian Active Model Evaluation

    cs.LG 2026-07 conditional novelty 6.0 of 10

    A sequential Bayesian method automatically grows a coreset until performance estimate and uncertainty stabilize, outperforming adapted baselines and showing active selection beats random when reference signals are rich.

  4. SocialMaze: A Benchmark for Evaluating Social Reasoning in Large Language Models

    cs.CL 2025-05 conditional novelty 6.0 of 10

    SocialMaze is a six-task benchmark that claims to evaluate LLM social reasoning along deep reasoning, dynamic interaction, and information uncertainty dimensions.

  5. Dynamically Allocating Evaluation Effort for Model Ranking

    cs.CL 2026-08 conditional novelty 5.0 of 10

    Rank-weighted adaptive sampling of which model to score next recovers top-model rankings from WMT human evaluation data with less budget than uniform evaluation.

  6. ParEvalLayer: When Partial LLM-Agent Evaluations Support a Decision

    cs.AI 2026-08 conditional novelty 5.0 of 10

    Partial LLM-agent evaluations can support a promote/reject decision after 15–25% of tasks on some benchmarks, but not others; reports should state the decision rule and unresolved count.

  7. ReasonBENCH: Benchmarking the (In)Stability of LLM Reasoning

    cs.AI 2025-12 reject novelty 5.0 of 10

    LLM reasoning benchmark scores vary substantially across repeated runs under the same model, strategy, and task, so single-run evaluation can misrank systems.

Reference graph

Works this paper leans on

32 extracted references · 13 canonical work pages · cited by 7 Pith papers

  1. [1]

    Explaining neural scaling laws

    Yasaman Bahri, Ethan Dyer, Jared Kaplan, Jaehoon Lee, and Utkarsh Sharma. Explaining neural scaling laws. Proceedings of the National Academy of Sciences, 121(27):e2311878121, 2024

    2024

  2. [2]

    Concentration Inequalities: A Nonasymptotic Theory of Independence

    Stéphane Boucheron, Gábor Lugosi, and Pascal Massart. Concentration Inequalities: A Nonasymptotic Theory of Independence . Oxford University Press, 02 2013. ISBN 9780199535255. doi: 10.1093/acprof:oso/9780199535255.001.0001. URL https://doi. org/10.1093/acprof:oso/9780199535255.001.0001

    arXiv 2013

  3. [3]

    Au- toeval done right: Using synthetic data for model evaluation

    Pierre Boyeau, Anastasios N Angelopoulos, Nir Yosef, Jitendra Malik, and Michael I Jordan. Au- toeval done right: Using synthetic data for model evaluation. arXiv preprint arXiv:2403.07008, 2024

    arXiv 2024

  4. [4]

    A survey on evaluation of large language models

    Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. A survey on evaluation of large language models. ACM Transactions on Intelligent Systems and Technology, 15(3):1–45, 2024

    2024

  5. [5]

    Mj-bench: Is your multimodal reward model really a good judge for text-to-image generation? arXiv preprint arXiv:2407.04842, 2024

    Zhaorun Chen, Yichao Du, Zichen Wen, Yiyang Zhou, Chenhang Cui, Zhenzhen Weng, Haoqin Tu, Chaoqi Wang, Zhengwei Tong, Qinglan Huang, et al. Mj-bench: Is your multimodal reward model really a good judge for text-to-image generation? arXiv preprint arXiv:2407.04842, 2024

    arXiv 2024

  6. [6]

    Safewatch: An efficient safety-policy following video guardrail model with transparent explanations

    Zhaorun Chen, Francesco Pinto, Minzhou Pan, and Bo Li. Safewatch: An efficient safety-policy following video guardrail model with transparent explanations. arXiv preprint arXiv:2412.06878, 2024

    arXiv 2024

  7. [7]

    Shieldagent: Shielding agents via verifiable safety policy reasoning

    Zhaorun Chen, Mintong Kang, and Bo Li. Shieldagent: Shielding agents via verifiable safety policy reasoning. arXiv preprint arXiv:2503.22738, 2025

    arXiv 2025

  8. [8]

    Angelopoulos, Tianle Li, Dacheng Li, Banghua Zhu, Hao Zhang, Michael I

    Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios N. Angelopoulos, Tianle Li, Dacheng Li, Banghua Zhu, Hao Zhang, Michael I. Jordan, Joseph E. Gonzalez, and Ion Stoica. Chatbot arena: an open platform for evaluating llms by human preference. In Proceedings of the 41st International Conference on Machine Learning, ICML’24. JMLR.org, 2024

    work page 2024

Show all 32 references
  1. [9]

    Bert: Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In North American Chapter of the Association for Computational Linguistics, 2019. URL https://api.semanticscholar. org/CorpusID:52967399

    2019

  2. [10]

    Length-controlled alpacaeval: A simple way to debias automatic evaluators

    Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B Hashimoto. Length-controlled alpacaeval: A simple way to debias automatic evaluators. arXiv preprint arXiv:2404.04475, 2024. 10

    2024 arXiv

  3. [11]

    Asymptotic behavior of expected sample size in certain one sided tests

    Roger H Farrell. Asymptotic behavior of expected sample size in certain one sided tests. The Annals of Mathematical Statistics, pages 36–72, 1964

    1964

  4. [12]

    Stratified prediction-powered inference for hybrid language model evaluation

    Adam Fisch, Joshua Maynez, R Alex Hofer, Bhuwan Dhingra, Amir Globerson, and William W Cohen. Stratified prediction-powered inference for hybrid language model evaluation. In Proc. NeurIPS, 2024

    2024

  5. [13]

    Llm-based nlg evaluation: Current status and challenges

    Mingqi Gao, Xinyu Hu, Jie Ruan, Xiao Pu, and Xiaojun Wan. Llm-based nlg evaluation: Current status and challenges. arXiv preprint arXiv:2402.01383, 2024

    2024 arXiv

  6. [14]

    Measuring massive multitask language understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum? id=d7KBjmI3GmQ

    2021

  7. [15]

    Measuring mathematical problem solving with the math dataset

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. NeurIPS, 2021

    2021

  8. [16]

    An empirical analysis of compute-optimal large language model training

    Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katherine Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon ...

    2022

  9. [17]

    Trustgpt: A benchmark for trustworthy and responsible large language models

    Yue Huang, Qihui Zhang, Lichao Sun, et al. Trustgpt: A benchmark for trustworthy and responsible large language models. arXiv preprint arXiv:2306.11507, 2023

    2023 arXiv

  10. [18]

    Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeff Wu, and Dario Amodei

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeff Wu, and Dario Amodei. Scaling laws for neural language mod- els. ArXiv, abs/2001.08361, 2020. URL https://api.semanticscholar.org/CorpusID: 210861095

    2001 arXiv

  11. [19]

    Noisy binary search and its applications

    Richard M Karp and Robert Kleinberg. Noisy binary search and its applications. In Proceed- ings of the eighteenth annual ACM-SIAM symposium on Discrete algorithms, pages 881–890. Citeseer, 2007

    2007

  12. [20]

    Péter Kevei and David M. Mason. A note on a maximal Bernstein inequality. Bernoulli, 17(3): 1054 – 1062, 2011. doi: 10.3150/10-BEJ304. URL https://doi.org/10.3150/10-BEJ304

    2011 doi

  13. [21]

    Metabench– a sparse benchmark to measure general ability in large language models

    Alex Kipnis, Konstantinos V oudouris, Luca M Schulze Buschoff, and Eric Schulz. Metabench– a sparse benchmark to measure general ability in large language models. arXiv preprint arXiv:2407.12844, 2024

    2024 arXiv

  14. [22]

    TruthfulQA: Measuring how models mimic human falsehoods

    Stephanie Lin, Jacob Hilton, and Owain Evans. TruthfulQA: Measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computa- tional Linguistics (Volume 1: Long Papers), pages 3214–3252. Association for Computational Linguisti...

    2022

  15. [23]

    Trustworthy llms: A survey and guideline for evaluating large language models’ alignment

    Yang Liu, Yuanshun Yao, Jean-Francois Ton, Xiaoying Zhang, Ruocheng Guo Hao Cheng, Yegor Klochkov, Muhammad Faaiz Taufiq, and Hang Li. Trustworthy llms: A survey and guideline for evaluating large language models’ alignment. arXiv preprint arXiv:2308.05374, 2023

    2023 arXiv

  16. [24]

    The sample complexity of exploration in the multi-armed bandit problem

    Shie Mannor and John N Tsitsiklis. The sample complexity of exploration in the multi-armed bandit problem. Journal of Machine Learning Research, 5(Jun):623–648, 2004

    2004

  17. [25]

    Adding error bars to evals: A statistical approach to language model evaluations

    Evan Miller. Adding error bars to evals: A statistical approach to language model evaluations. arXiv preprint arXiv:2411.00640, 2024

    2024 arXiv

  18. [26]

    tinybenchmarks: evaluating llms with fewer examples

    Felipe Maia Polo, Lucas Weber, Leshem Choshen, Yuekai Sun, Gongjun Xu, and Mikhail Yurochkin. tinybenchmarks: evaluating llms with fewer examples. In Proceedings of the 41st International Conference on Machine Learning, ICML’24. JMLR.org, 2025. 11

    2025

  19. [27]

    Mmdt: Decoding the trustworthiness and safety of multimodal foundation models

    Chejian Xu, Jiawei Zhang, Zhaorun Chen, Chulin Xie, Mintong Kang, Yujin Potter, Zhun Wang, Zhuowen Yuan, Alexander Xiong, Zidi Xiong, et al. Mmdt: Decoding the trustworthiness and safety of multimodal foundation models. arXiv preprint arXiv:2503.14827, 2025

    2025 arXiv

  20. [28]

    Data efficient evaluation of large language models and text-to-image models via adaptive sampling

    Cong Xu, Gayathri Saranathan, Mahammad Parwez Alam, Arpit Shah, James Lim, Soon Yee Wong, Foltin Martin, and Suparna Bhattacharya. Data efficient evaluation of large language models and text-to-image models via adaptive sampling. arXiv preprint arXiv:2406.15527, 2024

    2024 arXiv

  21. [29]

    Collaborative performance prediction for large language models

    Qiyuan Zhang, Fuyuan Lyu, Xue Liu, and Chen Ma. Collaborative performance prediction for large language models. arXiv preprint arXiv:2407.01300, 2024

    2024 arXiv

  22. [30]

    SafetyBench: Evaluating the safety of large language models

    Zhexin Zhang, Leqi Lei, Lindong Wu, Rui Sun, Yongkang Huang, Chong Long, Xiao Liu, Xuanyu Lei, Jie Tang, and Minlie Huang. SafetyBench: Evaluating the safety of large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (V...

    2024

  23. [31]

    Adaptive concentration inequalities for sequential decision problems

    Shengjia Zhao, Enze Zhou, Ashish Sabharwal, and Stefano Ermon. Adaptive concentration inequalities for sequential decision problems. In Advances in Neural Information Processing Systems, volume 29, 2016

    2016

  24. [32]

    Promptbench: Towards evaluating the robustness of large language models on adversarial prompts

    Kaijie Zhu, Jindong Wang, Jiaheng Zhou, Zichen Wang, Hao Chen, Yidong Wang, Linyi Yang, Wei Ye, Yue Zhang, Neil Zhenqiang Gong, et al. Promptbench: Towards evaluating the robustness of large language models on adversarial prompts. arXiv e-prints, pages arXiv–2306, 2023. 12 A M...

    2023

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.