REVIEW 7 cited by
Accounting for Variance in Machine Learning Benchmarks
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Strong empirical evidence that one machine-learning algorithm A outperforms another one B ideally calls for multiple trials optimizing the learning pipeline over sources of variation such as data sampling, data augmentation, parameter initialization, and hyperparameters choices. This is prohibitively expensive, and corners are cut to reach conclusions. We model the whole benchmarking process, revealing that variance due to data sampling, parameter initialization and hyperparameter choice impact markedly the results. We analyze the predominant comparison methods used today in the light of this variance. We show a counter-intuitive result that adding more sources of variation to an imperfect estimator approaches better the ideal estimator at a 51 times reduction in compute cost. Building on these results, we study the error rate of detecting improvements, on five different deep-learning tasks/architectures. This study leads us to propose recommendations for performance comparisons.
Forward citations
Cited by 7 Pith papers
-
Where You Measure Decides What You Measure: Position Selection in Ablation-Based SAE Evaluation
In ablation-based SAE evaluation, the measurement token is selected by the dictionary under test, and holding that token fixed collapses most of the variance that is usually attributed to differences between dictionaries.
-
Governing Agentic AI in FinTech
Financial institutions can lose the ability to explain or reproduce agentic AI decisions even when the system is capable and seemingly stable; the paper names this a Verifiability Gap and dissects its mechanisms.
-
Data Quality Profiling at Scale with Progressive Sampling: A Benchmark for Data-Centric AI Pipelines
Random uniform sampling dominates proxy-guided MCMC for data-quality profiling at scale, with up to 49x lower error on real datasets.
-
Grokking Is Conditional and Fragile: A Fully-Tractable, Multi-Seed Study at 12K Parameters
In a fully tractable 12K Llama-style model, grokking is a conditional fragile phase transition gated by coverage (tracking modulus more than structure), weight decay, and floating-point reduction order, so evidence mu...
-
Beginning with You: Perceptual-Initialization Improves Vision-Language Representation and Alignment
Perceptually initializing a CLIP vision encoder with NIGHTS triplet judgments before YFCC15M contrastive training improves zero-shot accuracy and retrieval over an identical random-start baseline.
-
Evaluation of Seismic Artificial Intelligence with Uncertainty
A seismic AI evaluation framework jointly estimates training and data uncertainty and learning efficiency, and shows no single PhaseNet training approach wins at all data budgets.
-
SketchMamba: A Lightweight State-Space Model for Joint Progressive Sketch Classification and Stroke Auto-Completion
Dense per-step class loss on a 1.55M-parameter causal Mamba unifies progressive sketch recognition (94.93% final Top-1, 0.706 progressive AUC) and stroke auto-completion on 58-class Quick, Draw!.
Discussion (0). Continue with ORCID to comment.