The paper shows how bootstrapping and Bayesian hierarchical modeling can quantify uncertainty in task-aggregated benchmark scores, using simulated VTAB data, and that a low-ranked model can dominate under certain task weightings.
Time for a change: A tutorial for comparing multiple classifiers through Bayesian analysis
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
citation-role summary
background 1
citation-polarity summary
fields
stat.ML 1years
2025 1verdicts
CONDITIONAL 1roles
background 1polarities
background 1representative citing papers
citing papers explorer
-
Statistical Uncertainty Quantification for Aggregate Performance Metrics in Machine Learning Benchmarks
The paper shows how bootstrapping and Bayesian hierarchical modeling can quantify uncertainty in task-aggregated benchmark scores, using simulated VTAB data, and that a low-ranked model can dominate under certain task weightings.