Prefix caching makes LLM serving non-reproducible: repeated identical requests with caching enabled produce different outputs for a large fraction of episodes, a failure that coarser weight quantization amplifies substantially.
Quantifying Variance in Evaluation Benchmarks
25 Pith papers cite this work, alongside 2 external citations. Polarity classification is still indexing.
25
Pith papers citing it
2
external citations · Pith
abstract
Evaluation benchmarks are the cornerstone of measuring capabilities of large language models (LLMs), as well as driving progress in said capabilities. Originally designed to make claims about capabilities (or lack thereof) in fully pretrained models, evaluation benchmarks are now also extensively used to decide between various training choices. Despite this widespread usage, we rarely quantify the variance in our evaluation benchmarks, which dictates whether differences in performance are meaningful. Here, we define and measure a range of metrics geared towards measuring variance in evaluation benchmarks, including seed variance across initialisations, and monotonicity during training. By studying a large number of models -- both openly available and pretrained from scratch -- we provide empirical estimates for a variety of variance metrics, with considerations and recommendations for practitioners. We also evaluate the utility and tradeoffs of continuous versus discrete performance measures and explore options for better understanding and reducing this variance. We find that simple changes, such as framing choice tasks (like MMLU) as completion tasks, can often reduce variance for smaller scale ($\sim$7B) models, while more involved methods inspired from human testing literature (such as item analysis and item response theory) struggle to meaningfully reduce variance. Overall, our work provides insights into variance in evaluation benchmarks, suggests LM-specific techniques to reduce variance, and more generally encourages practitioners to carefully factor in variance when comparing models.
representative citing papers
AST-FIM masks complete syntax-tree subtrees during fill-in-the-middle pretraining, improving infilling performance on real-world code edits.
HTEB introduces dynamic, multi-axis evaluation of text embedding robustness using LLM transformations, finding decoupled profiles across models and that scaling does not close all robustness gaps.
A 400k+ GPU-hour study shows RL scaling in LLMs follows predictable sigmoidal trajectories, with most design choices affecting efficiency rather than the performance asymptote, enabling accurate large-scale predictions via the ScaleRL recipe.
CompanionSim provides a synthetic data framework showing that AI companionship behaviors decrease third-party perceptions of likability, humanlikeness, and trust, with significant demographic heterogeneity.
Expanding a retrieval corpus from one to seven shards causes 6.4 to 10.3 percentage points of answer churn beyond ordinary repeat variation in a fixed QA system, while exact-match accuracy moves less than two points.
Across three enterprise agent benchmarks, agent main effects are under 3% of total score variance, so leaderboard order reflects task specialization rather than a general capability advantage.
Original, never-upstreamed multi-file engineering tasks with functional verifiers grade coding agents more faithfully and separate frontier models more widely than inherited-test SWE benchmarks.
Benchmark signal-to-noise ratio, measured as score spread across models divided by checkpoint-to-checkpoint variability, predicts small-to-large model decision accuracy and can be improved by subtask filtering, checkpoint averaging, or switching to bits-per-byte.
Benchmark prediction methods mostly work by interpolation among similar models and fail on better, unfamiliar models, where random sampling with an AIPW-style correction is the only consistent improvement.
A clinician-validated taxonomy and 35-benchmark suite show that large language models vary widely across medical tasks, with reasoning models leading overall.
HARP, a new benchmark of 5,409 US math competition problems with human solutions and choices, keeps frontier LLMs far from saturation: the best model scores 75.9% overall and only 41.1% on the hardest 197 problems.
NLI benchmarks still discriminate between LLMs of different sizes and are not saturated, while the Jensen-Shannon distance between model and human label distributions shrinks with scale.
Losses of models trained on different datasets are related by shifted power laws, enabling translation of scaling laws and prediction of downstream performance from a few runs.
Establishes concentration bounds for infinitely exchangeable sequences with cancellation for zero-sum contrasts and applies the result to distribution-free uncertainty quantification in composite AI benchmarks.
Maps common low-compute research strategies for foundation models onto statistical, internal, external, and construct validity threats via a causal-inference lens.
Paired LLM leaderboard comparisons frequently lack resolution at conventional (alpha=0.05, power=0.8) levels, with a new per-pair ratio q=N/N* showing that common unpaired shortcuts underestimate required samples by roughly a factor of two.
Dynamic Boundary Evaluation locates each LLM's performance boundary at ~50% pass probability via a calibrated item bank and Skill-Guided Boundary Search algorithm to enable unified, adaptive evaluations across safety, capability, and truthfulness.
Empirical study of eight LLMs finds overuse of popular libraries like NumPy in up to 45% of unnecessary cases and strong default preference for Python even when suboptimal.
Pretraining data determines loss-to-loss scaling laws in LLMs, while model size, optimization, tokenizer, and architecture have limited impact.
Tokenizer choice matters mostly for multilingual tasks, and 350M-parameter models can predict 2.7B model ranking on translation but not on English benchmarks.
A 1.6B Arabic language model trained with synthetic multiple-choice instruction data outperforms 7B-13B models on several Arabic multiple-choice benchmarks.
Single-seed CRPS estimates in limited-data BDL show high variance and peaks for heteroscedastic methods, with local variance correlating above 0.96 to single-seed error.
Using generalized stochastic dominance, the authors find that human-written text completions are not significantly outperformed by five LLM decoding strategies across mixed cardinal and ordinal quality metrics.
citing papers explorer
-
Same Request, Different Answer: Quantization Amplifies Cache-Induced Divergence in LLM Serving
Prefix caching makes LLM serving non-reproducible: repeated identical requests with caching enabled produce different outputs for a large fraction of episodes, a failure that coarser weight quantization amplifies substantially.
-
Structure-Aware Fill-in-the-Middle Pretraining for Code
AST-FIM masks complete syntax-tree subtrees during fill-in-the-middle pretraining, improving infilling performance on real-world code edits.
-
The Harder Text Embedding Benchmark (HTEB): Beyond One-dimensional Static Robustness
HTEB introduces dynamic, multi-axis evaluation of text embedding robustness using LLM transformations, finding decoupled profiles across models and that scaling does not close all robustness gaps.
-
The Art of Scaling Reinforcement Learning Compute for LLMs
A 400k+ GPU-hour study shows RL scaling in LLMs follows predictable sigmoidal trajectories, with most design choices affecting efficiency rather than the performance asymptote, enabling accurate large-scale predictions via the ScaleRL recipe.
-
CompanionSim: Synthetic Data for Evaluating Anthropomorphism in Human-AI Relationships
CompanionSim provides a synthetic data framework showing that AI companionship behaviors decrease third-party perceptions of likability, humanlikeness, and trust, with significant demographic heterogeneity.
-
Same Agent, Different Answers: A Repeat-Aware Audit of Corpus-Induced Answer Churn in Retrieval-Augmented QA
Expanding a retrieval corpus from one to seven shards causes 6.4 to 10.3 percentage points of answer churn beyond ordinary repeat variation in a fixed QA system, while exact-match accuracy moves less than two points.
-
Deployment Decision Reliability: A Generalizability-Theory Framework for Sizing Long-Horizon Agent Evaluations
Across three enterprise agent benchmarks, agent main effects are under 3% of total score variance, so leaderboard order reflects task specialization rather than a general capability advantage.
-
DeepSWE: Measuring Frontier Coding Agents on Original, Long-Horizon Engineering Tasks
Original, never-upstreamed multi-file engineering tasks with functional verifiers grade coding agents more faithfully and separate frontier models more widely than inherited-test SWE benchmarks.
-
Signal and Noise: A Framework for Reducing Uncertainty in Language Model Evaluation
Benchmark signal-to-noise ratio, measured as score spread across models divided by checkpoint-to-checkpoint variability, predicts small-to-large model decision accuracy and can be improved by subtask filtering, checkpoint averaging, or switching to bits-per-byte.
-
How Benchmark Prediction from Fewer Data Misses the Mark
Benchmark prediction methods mostly work by interpolation among similar models and fail on better, unfamiliar models, where random sampling with an AIPW-style correction is the only consistent improvement.
-
MedHELM: Holistic Evaluation of Large Language Models for Medical Tasks
A clinician-validated taxonomy and 35-benchmark suite show that large language models vary widely across medical tasks, with reasoning models leading overall.
-
HARP: A challenging human-annotated math reasoning benchmark
HARP, a new benchmark of 5,409 US math competition problems with human solutions and choices, keeps frontier LLMs far from saturation: the best model scores 75.9% overall and only 41.1% on the hardest 197 problems.
-
Lost in Inference: Rediscovering the Role of Natural Language Inference for Large Language Models
NLI benchmarks still discriminate between LLMs of different sizes and are not saturated, while the Jensen-Shannon distance between model and human label distributions shrinks with scale.
-
Loss-to-Loss Prediction: Scaling Laws for All Datasets
Losses of models trained on different datasets are related by shifted power laws, enabling translation of scaling laws and prediction of downstream performance from a few runs.
-
Bounded Difference Concentration for Infinitely Exchangeable Sequences with Applications to AI Benchmark Uncertainty
Establishes concentration bounds for infinitely exchangeable sequences with cancellation for zero-sum contrasts and applies the result to distribution-free uncertainty quantification in composite AI benchmarks.
-
Validity Threats for Foundation Model Research
Maps common low-compute research strategies for foundation models onto statistical, internal, external, and construct validity threats via a causal-inference lens.
-
Resolution Diagnostics for Paired LLM Evaluation
Paired LLM leaderboard comparisons frequently lack resolution at conventional (alpha=0.05, power=0.8) levels, with a new per-pair ratio q=N/N* showing that common unpaired shortcuts underestimate required samples by roughly a factor of two.
-
Beyond Fixed Benchmarks and Worst-Case Attacks: Dynamic Boundary Evaluation for Language Models
Dynamic Boundary Evaluation locates each LLM's performance boundary at ~50% pass probability via a calibrated item bank and Skill-Guided Boundary Search algorithm to enable unified, adaptive evaluations across safety, capability, and truthfulness.
-
A Study of LLMs' Preferences for Libraries and Programming Languages
Empirical study of eight LLMs finds overuse of popular libraries like NumPy in up to 45% of unnecessary cases and strong default preference for Python even when suboptimal.
-
LLMs on the Line: Data Determines Loss-to-Loss Scaling Laws
Pretraining data determines loss-to-loss scaling laws in LLMs, while model size, optimization, tokenizer, and architecture have limited impact.
-
Beyond Text Compression: Evaluating Tokenizers Across Scales
Tokenizer choice matters mostly for multilingual tasks, and 350M-parameter models can predict 2.7B model ranking on translation but not on English benchmarks.
-
Arabic Stable LM: Adapting Stable LM 2 1.6B to Arabic
A 1.6B Arabic language model trained with synthetic multiple-choice instruction data outperforms 7B-13B models on several Arabic multiple-choice benchmarks.
-
A Tale of Two Variances: When Single-Seed Benchmarks Fail in Bayesian Deep Learning
Single-seed CRPS estimates in limited-data BDL show high variance and peaks for heteroscedastic methods, with local variance correlating above 0.96 to single-seed error.
-
Statistical Multicriteria Evaluation of LLM-Generated Text
Using generalized stochastic dominance, the authors find that human-written text completions are not significantly outperformed by five LLM decoding strategies across mixed cardinal and ordinal quality metrics.
- DataComp-VLM: Improved Open Datasets for Vision-Language Models