REVIEW 5 cited by
Position: Don't Use the CLT in LLM Evals With Fewer Than a Few Hundred Datapoints
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Rigorous statistical evaluations of large language models (LLMs), including valid error bars and significance testing, are essential for meaningful and reliable performance assessment. Currently, when such statistical measures are reported, they typically rely on the Central Limit Theorem (CLT). In this position paper, we argue that while CLT-based methods for uncertainty quantification are appropriate when benchmarks consist of thousands of examples, they fail to provide adequate uncertainty estimates for LLM evaluations that rely on smaller, highly specialized benchmarks. In these small-data settings, we demonstrate that CLT-based methods perform very poorly, usually dramatically underestimating uncertainty (i.e. producing error bars that are too small). We give recommendations for alternative frequentist and Bayesian methods that are both easy to implement and more appropriate in these increasingly common scenarios. We provide a simple Python library for these Bayesian methods at https://github.com/sambowyer/bayes_evals .
Forward citations
Cited by 5 Pith papers
-
The multiply iterated law of the iterated logarithm: game-theoretic foundations of sequential detection boundaries
The multiply iterated LIL is derived as the minimax boundary of a sequential-detection game whose equalizer prior is the Jeffreys prior selected by the Erdős-Kolmogorov integral test, yielding a closed-form 3/2 coeffi...
-
The multiply iterated law of the iterated logarithm: game-theoretic foundations of sequential detection boundaries
A game-theoretic reformulation of sequential detection shows the LIL as the minimax boundary, with the optimal mixing prior being the Jeffreys prior on the scale-of-scales selected by the Erdős-Kolmogorov test, yieldi...
-
Don't Pass@k: A Bayesian Framework for Large Language Model Evaluation
A Dirichlet-prior Bayesian estimator for model success probability replaces Pass@k, delivering faster-converging and more stable rankings with credible intervals on math benchmarks.
-
Generative Responsible AI Data Evaluation Schema (GRAIDES) for AI Assurance in Local Government
GRAIDES is proposed as a standardized data model for generative AI observability, benchmarking and assurance in government settings with a focus on human-model alignment.
-
Correcting Prompt Dependence in LLM Benchmarks: A Bayesian Hierarchical Model with Embedding-Space Clustering
A Bayesian model that groups similar LLM test prompts into clusters gives better predictive scores than a no-clustering baseline but does not prove that it truly corrects prompt dependence.
Discussion (0). Sign in to comment.