REVIEW 13 cited by
Quantifying Variance in Evaluation Benchmarks
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
abstract
Evaluation benchmarks are the cornerstone of measuring capabilities of large language models (LLMs), as well as driving progress in said capabilities. Originally designed to make claims about capabilities (or lack thereof) in fully pretrained models, evaluation benchmarks are now also extensively used to decide between various training choices. Despite this widespread usage, we rarely quantify the variance in our evaluation benchmarks, which dictates whether differences in performance are meaningful. Here, we define and measure a range of metrics geared towards measuring variance in evaluation benchmarks, including seed variance across initialisations, and monotonicity during training. By studying a large number of models -- both openly available and pretrained from scratch -- we provide empirical estimates for a variety of variance metrics, with considerations and recommendations for practitioners. We also evaluate the utility and tradeoffs of continuous versus discrete performance measures and explore options for better understanding and reducing this variance. We find that simple changes, such as framing choice tasks (like MMLU) as completion tasks, can often reduce variance for smaller scale ($\sim$7B) models, while more involved methods inspired from human testing literature (such as item analysis and item response theory) struggle to meaningfully reduce variance. Overall, our work provides insights into variance in evaluation benchmarks, suggests LM-specific techniques to reduce variance, and more generally encourages practitioners to carefully factor in variance when comparing models.
Forward citations
Cited by 13 Pith papers
-
Fluid Language Model Benchmarking
Fluid Benchmarking, combining IRT-based ability estimation with Fisher-information-based adaptive item selection, improves LM evaluation across efficiency, validity, variance, and saturation in pretraining settings.
-
How Much of the Routing Gap Is Real? Decomposing the Router-to-Oracle Gap into Reproducible Specialist Advantage and Single-Draw Label Noise
Under stochastic decoding, 12–36% of the LLM router-to-oracle gap is single-draw noise unreachable by any single-commit router, while the rest is recoverable specialist advantage; best-of-K sampling closes the floor.
-
Structure-Aware Fill-in-the-Middle Pretraining for Code
AST-FIM masks complete syntax-tree subtrees during fill-in-the-middle pretraining, improving infilling performance on real-world code edits.
-
Messier: A High-Resolution Corpus for Cross-Benchmark Agent Evaluation
A unified corpus of 957k trial outcomes shows frontier progress is uneven and strict all-pass aggregation obscures capability and can reorder agents.
-
DeepSWE: Measuring Frontier Coding Agents on Original, Long-Horizon Engineering Tasks
Original, never-upstreamed multi-file engineering tasks with functional verifiers grade coding agents more faithfully and separate frontier models more widely than inherited-test SWE benchmarks.
-
FineWeb2: One Pipeline to Scale Them All -- Adapting Pre-Training Data Processing to Every Language
An adaptive, per-language data filtering and deduplication pipeline produces multilingual LLM pre-training corpora that beat prior public datasets on 11 of 14 evaluated languages, and a 20TB, 1,868 language-script dat...
-
How Benchmark Prediction from Fewer Data Misses the Mark
Benchmark prediction methods mostly work by interpolation among similar models and fail on better, unfamiliar models, where random sampling with an AIPW-style correction is the only consistent improvement.
-
MedHELM: Holistic Evaluation of Large Language Models for Medical Tasks
A clinician-validated taxonomy and 35-benchmark suite show that large language models vary widely across medical tasks, with reasoning models leading overall.
-
Laplace-PSN-IRT: Uncertainty Quantification for Neural Item Response Theory Models of LLM Benchmarks
Scoped last-layer Laplace posteriors for PSN-IRT yield calibrated ability/difficulty uncertainty, showing 64 of 66 model pairs are statistically indistinguishable and that posterior-averaged Fisher information survive...
-
AdaRoPE: Not All Attention Heads Should Rotate and Scale Equally
Head-wise learnable rotary frequencies and length-dependent attention scaling (AdaRoPE) beat uniform RoPE and YaRN schedules in pretraining and 8k-to-64k context extension up to 8B scale.
-
ReasonBENCH: Benchmarking the (In)Stability of LLM Reasoning
LLM reasoning benchmark scores vary substantially across repeated runs under the same model, strategy, and task, so single-run evaluation can misrank systems.
-
Correcting Prompt Dependence in LLM Benchmarks: A Bayesian Hierarchical Model with Embedding-Space Clustering
A Bayesian model that groups similar LLM test prompts into clusters gives better predictive scores than a no-clustering baseline but does not prove that it truly corrects prompt dependence.
-
Beyond Text Compression: Evaluating Tokenizers Across Scales
Tokenizer choice matters mostly for multilingual tasks, and 350M-parameter models can predict 2.7B model ranking on translation but not on English benchmarks.
Discussion (0). Continue with ORCID to comment.