Pith. sign in

REVIEW 4 major objections 4 minor 2 cited by

Statistical Uncertainty Quantification for Aggregate Performance Metrics in Machine Learning Benchmarks

T0 review · 4 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Benchmark rankings that ignore uncertainty can mislead: with task weighting and standard errors, a sixth-place model becomes the best on structured-image tasks.

desk verdict A well-written methodology demonstration whose VTAB 'insights' are not empirical findings, because the simulation ignores the cross-model correlation the paper itself flags. read the letter →

arxiv 2501.04234 v1 pith:M544ZWX5 submitted 2025-01-08 stat.ML cs.LGstat.AP

Rachel Longjohn , Giri Gopalan , Emily Casleton This is my paper · ORCID
classification stat.MLcs.LGstat.AP MSC 62F4062F1562F07
keywords uncertaintyquantificationmachinelearningbenchmarksbootstrapBayesianhierarchicalmodeltaskaggregationVTABleaderboardrankingsweighting
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Machine learning leaderboards typically report a single number that averages a model's accuracy across many benchmark tasks, and that number ignores the sampling uncertainty in each task's test set. This paper argues that such aggregate scores need confidence and credibility intervals, and demonstrates two standard statistical tools for building them: bootstrapping the evaluation data and a beta-binomial Bayesian hierarchical model. Applied to the VTAB benchmark's 16 models, the intervals show that the top two models are statistically indistinguishable in overall average accuracy, and that a model ranked sixth overall can be the best when structured-image tasks are weighted heavily. The paper also provides simplex visualizations that show which model wins for every combination of task-category weights, with uncertainty shading that indicates when the top two models are too close to call. The payoff is a way for benchmark users to attach honest uncertainty to leaderboard rankings and to see how task weighting changes the conclusion.

What carries the argument

The machinery is the weighted aggregate score $S_i = \sum_j w_j \hat{p}_{ij}$ with its variance decomposition $\mathrm{Var}(S_i) = \sum_j w_j^2 \mathrm{Var}(\hat{p}_{ij}) + 2\sum_{j<j'} w_j w_{j'} \mathrm{Cov}(\hat{p}_{ij}, \hat{p}_{ij'})$, which the paper uses to build standard errors for model differences on ternary simplex plots; the variance formula also shows that positive between-task correlation inflates the true variance relative to the independence assumption. Supporting this is the $\beta$-binomial hierarchical model in which the number of correct responses $Y_{ij}$ follows $\mathrm{Binomial}(\theta_{ij}, N_j)$ with $\theta_{ij} \sim \mathrm{Beta}(\alpha_i, \beta_i)$ and exponential hyperpriors, fitted by a Gibbs sampler with slice-sampling steps, which provides shrinkage for tasks with small test sets. The bootstrap serves as the nonparametric counterpart, resampling each task's test instances and recomputing the benchmark's own aggregation rule. Together these objects convert leaderboard point estimates into distributions over aggregate scores, differences, and ranks.

What would settle it

Obtain the actual per-task test responses for the 16 VTAB models and re-run the bootstrap, the hierarchical model, and the simplex weighting with the real data; if Rotation no longer occupies the structured-weighted winning region, or if the Sup-Rotation versus Sup-Exemplar difference becomes significant, the paper's empirical claims would be overturned.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that accounting for uncertainty in aggregated benchmark metrics changes the conclusions that leaderboards invite: the two leading supervised models, Sup-Rotation-100% and Sup-Exemplar-100%, have overlapping 83.4% intervals for unweighted average accuracy, so their apparent ordering is not reliable, while under a normalized-accuracy scheme their ranking reverses with non-overlapping intervals. More strikingly, rank-based aggregation and task-weighting visualizations both suggest that the Rotation model, sixth by average accuracy, is the best model when structured-image tasks receive heavy weight. These findings are demonstrated through bootstrapped test data, a beta-binomial hierarchical model, and standard-error-aware simplex plots rather than through new statistical theory; the contribution is the demonstration that established statistical machinery can be transplanted onto model benchmarking.

Load-bearing premise

The VTAB demonstration relies on test responses simulated from the published leaderboard accuracies with independent Bernoulli draws per model and task; if the real test data contain cross-model correlations or item-level structure that this simulation omits, the reported intervals and the Rotation result may not transfer to the actual benchmark.

Editorial extensions

If this is right

  • Practitioners can attach bootstrap or hierarchical-model intervals to any leaderboard that aggregates per-task accuracies, and should treat point-estimate orderings within those intervals as unresolved.
  • Task weighting can reverse model dominance: the same model can be worst overall yet best for a stakeholder who cares about one task category, as the Rotation model shows for structured images.
  • Normalizing task scores before aggregation changes rankings and should be reported alongside raw averages, since it rewards strength on harder tasks.
  • Rank aggregation methods that reward many small wins (average-rank schemes) can disagree sharply with mean-based rankings, so leaderboards should report more than one aggregation rule.
  • If tasks are positively correlated, the true uncertainty is larger than the independence-based intervals suggest, so the gray 'too close to call' regions in the plots would grow.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same toolkit transfers directly to LLM benchmarks that publish per-example responses: item-level resampling and a hierarchical model over tasks would let anyone reproduce these intervals without access to model training runs.
  • A natural next step the paper does not pursue is estimating the between-task and between-model covariance matrix from response-level data, which would replace the independence assumption and the $z = 2\sqrt{1/2}$ heuristic with an empirical correlation adjustment.
  • The simplex plots could be inverted into a decision aid: a stakeholder who can only articulate qualitative priorities ('mostly structured images') gets a region of weights, not a single weighting, for which a model is the best within uncertainty.
  • If benchmark owners published test responses with item-level metadata, the bootstrap step would no longer need simulation, and the whole framework could be run exactly by any practitioner.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The manuscript proposes statistical uncertainty quantification for aggregate performance metrics on machine learning benchmarks, illustrating bootstrap confidence intervals, Bayesian hierarchical modeling, and uncertainty-aware simplex visualizations on the Visual Task Adaptation Benchmark (VTAB). Because per-task test responses are unavailable, the authors generate synthetic Bernoulli data whose means match the published VTAB leaderboard accuracies and whose sample sizes match the benchmark's test sets, then compute interval estimates for average accuracy, pairwise differences, rank aggregation, and category-weighting plots. They also include a two-model simulation study intended to show that a beta-binomial BHM can outperform the bootstrap when one task has a small sample size.

Significance. The statistical identities in Section 2.3 are correct, and the simplex visualizations are a genuinely useful way to communicate weighting uncertainty. The paper is honest about its main limitation: all VTAB conclusions are derived from simulated data consistent with leaderboard accuracies, not from actual responses. However, because the BHM demonstration in Section 2.2 uses priors centered on the true parameter values, and because the VTAB application ignores cross-model correlation, the current manuscript does not establish the empirical claims about Rotation or about the relative merits of bootstrap versus BHM. If the correlation sensitivity analysis and the simulation study are fixed, the paper would be a useful pedagogical demonstration of existing statistical tools for benchmark aggregation.

major comments (4)
  1. [§2.2, Table 1] The numerical illustration is internally inconsistent. The text specifies θ_B1 = .525 and N2 = 10000, but Table 1 reports 115 correct out of 200 for Model B on Task 1 (0.575, not 0.525) and reports N2 = 1000 with 5,000 correct responses for both models, which is impossible. The bootstrap interval (-.059, .097) and the BHM interval (-.021, -.003) cannot be reproduced from the stated inputs, so this simulation does not support the claimed comparison.
  2. [§2.2, simulation study priors] The BHM advantage is manufactured by the prior. The simulation uses normal priors with means (2000, 2100, 2000, 1900) for (α1, α2, β1, β2), which correspond to prior success probabilities 0.50 for Model A and 0.525 for Model B — exactly the true generative values. This is not the Exp(1/10000) hyperprior specified earlier in the section, and it guarantees in advance that the posterior for θ_A - θ_B is centered below zero. To establish that shrinkage helps in small-sample settings, the authors should use the same prior for both methods or run a simulation study under a range of priors and report frequentist coverage and interval length, not a single cherry-picked case.
  3. [§3, simulated VTAB data] The empirical application conditions on an independence assumption the paper itself identifies as potentially misleading. Section 3 states that the authors 'used simulated data consistent with the per-task accuracies', and the simulation draws independent Bernoulli responses for each model-task pair; there is no cross-model correlation even though all models are evaluated on the same test items. Equation (3) shows that positive between-model correlation can reduce the standard deviation of a difference by roughly 30% at ρ = .5, yet the bootstrap and BHM intervals in Tables 2–5, 8, and 11 are computed under ρ = 0; only the simplex plots apply a crude z-scaling. As a result, the pairwise 'no meaningful difference' conclusions and the rank intervals are not robust checks under realistic dependence. Please add a correlated-data sensitivity analysis (e.g., a latent-variable model with common item effects) or, if actual responses cannot be released, state explicitly which conclusions survive a range of correlation values.
  4. [§3.3, headline VTAB insight] The headline Rotation finding is a deterministic consequence of the point estimates used to generate the data. In §3.3, the structured-category accuracies are 57.3 for Rotation versus 55.5 for SR-100%, so any weighting that puts large mass on structured images will favor Rotation in expectation; the intervals only reflect Bernoulli sampling noise. The paper should be explicit that the 'insights' are properties of the leaderboard point estimates under the simulation assumptions, not discoveries about unobserved response patterns.
minor comments (4)
  1. [§2.2] The notation uses Binom(θ_ij, N_j) while the usual convention is Binom(N_j, θ_ij); more importantly, the simulation study introduces a normal prior on (α_i, β_i) without explaining its relation to the Exp hyperpriors presented above. Please reconcile the two model specifications.
  2. [§2.4] Estimating the high and low normalization anchors as bootstrap maxima and minima makes the normalized metric depend strongly on the number of bootstrap replicates and on sampling noise; please report sensitivity to the choice of anchors or use fixed baselines.
  3. [Figures 3 and 4] The gray region is defined as 'within z standard errors of 0', but the choice of z = 2 versus z = 1.4 is only heuristically justified; please explain the relationship to Eq. (3) and to the actual standard errors used in the plots.
  4. [§4] There is a typo in the first sentence: 'adpated' should be 'adapted'.

Circularity Check

2 steps flagged · score 6.0 of 10

BHM toy example plants the answer in its priors and the VTAB 'Rotation insight' is a deterministic re-plot of the leaderboard accuracies used to generate the simulated data.

  1. fitted input called prediction [Section 2.2, 'Simulation study' (Bayesian Hierarchical Modeling of Evaluation Data)]
    "α2 ∼ N (2100, ́10); β2 ∼ N (1900, 10) ... Here, we find that the 95% credibility interval for ¯θA. − ¯θB. is (-0.021, -0.003). Since the interval is strictly negative, we are led to the correct conclusion that model B is better than model A."

    The priors are centered at the true parameter values: for model B, Beta(α2, β2) has mean 2100/(2100+1900)=0.525, exactly the true θB1=0.525, while model A's prior mean is 0.5. The 4000 prior counts dominate the small Task 1 sample and pull the posterior toward these planted values, so the posterior interval excludes 0 because the prior already encodes that B is better, not because the data demonstrate it. The bootstrap, which has no such planted prior, fails; the comparison therefore demonstrates the content of the prior rather than the BHM's ability to learn from data.

  2. renaming known result [Section 3 (Application to VTAB) and Section 3.3 (Visualizing Task Weightings with Uncertainty)]
    "Since we do not have the per-task responses across the 16 fine-tuned models compared for VTAB, we used simulated data consistent with the per-task accuracies for each model on the VTAB leaderboard ... A robust result across all of these plots is that when structured images are weighted heavily in the aggregation, Rotation is suggested as the best performing model."

    The simulated data are independent Bernoulli draws with success probabilities equal to the leaderboard per-task accuracies, so the bootstrap means, BHM posteriors, and weighted aggregates shown in the simplex plots are deterministic functions of those same input accuracies. The Rotation 'insight' is already present in the input: the leaderboard gives Rotation a structured-category accuracy of 57.3 versus 55.5 for SR-100% and 54.7 for SE-100%. The plots re-express this ordering in new coordinates and add uncertainty from the simulated draws, but the qualitative 'insight' is equivalent to the input by construction; no independent test responses are involved.

full rationale

The paper's core methodological machinery (bootstrap, BHM, simplex visualization) is standard and self-contained, and the authors are transparent about having simulated VTAB responses from leaderboard accuracies. However, two load-bearing demonstrations reduce to their inputs. First, the Section 2.2 simulation chooses beta-binomial priors centered at the true model difference (α2=2100, β2=1900 vs α1=2000, β1=2000), so the 'correct conclusion' that B beats A is encoded in the prior rather than learned; the bootstrap comparison is therefore not a fair evaluation of BHM. Second, the VTAB application generates independent Bernoulli data from the leaderboard accuracies and then reports as a 'robust result' that Rotation wins under heavy structured weighting, which is exactly the ordering already present in those leaderboard accuracies. This is a re-description of the input in new coordinates, not an independent prediction. The methodological proposal itself is not circular, and there is no load-bearing self-citation, so the paper is only partially circular.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The paper introduces no new entities. The main free parameters are the simulation study priors and the weakly informative BHM priors, both chosen by hand. The simulation's independence assumption is a domain assumption that may not hold in real benchmark data.

free parameters (3)
  • lambda_alpha, lambda_beta = 1/10000
    Chosen weakly informative for the BHM, with no sensitivity analysis or justification for the specific value.
  • Simulation study prior means (alpha1, alpha2, beta1, beta2) = 2000, 2100, 2000, 1900
    These priors are chosen to shrink task 1's variance and force the BHM to detect the true difference, so they are fitted to the desired conclusion.
  • Normalization high/low = Estimated from bootstrap samples
    The high and low scores for normalized accuracy are taken as the max and min over bootstrap samples, making the normalized metric data-dependent and arbitrary.
assumptions (5)
  • domain assumption Each task's responses are i.i.d. Bernoulli with probability equal to the reported leaderboard accuracy.
    This underlies the simulation of VTAB data in Section 3.
  • domain assumption Independence across models and tasks in the simulated data.
    The simulation generates responses independently per model and task, and the paper does not model cross-model correlations (Sections 2.3 and 3).
  • standard math The binomial model for correct response counts.
    Used in the BHM data model in Section 2.2.
  • domain assumption The VTAB leaderboard aggregation is an unweighted average of per-task accuracies.
    Stated in Appendix A.1.
  • domain assumption The test set sizes from the VTAB GitHub repository are correct.
    Used for both simulation and standard error computations (Section 3 and Appendix A.2).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Statistical Uncertainty Quantification for Aggregate Performance Metrics in Machine Learning Benchmarks." pith.science (2026). https://pith.science/paper/M544ZWX5

@misc{pith2026250104234,
  author       = {Pith},
  title        = {Pith review of: Statistical Uncertainty Quantification for Aggregate Performance Metrics in Machine Learning Benchmarks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/M544ZWX5}},
  note         = {Machine review of arXiv:2501.04234}
}
read the original abstract

Modern artificial intelligence is supported by machine learning models (e.g., foundation models) that are pretrained on a massive data corpus and then adapted to solve a variety of downstream tasks. To summarize performance across multiple tasks, evaluation metrics are often aggregated into a summary metric, e.g., average accuracy across 10 question-answering tasks. When aggregating evaluation metrics, it is useful to incorporate uncertainty in the aggregate metric in order to gain a more realistic understanding of model performance. Our objective in this work is to demonstrate how statistical methodology can be used for quantifying uncertainty in metrics that have been aggregated across multiple tasks. The methods we emphasize are bootstrapping, Bayesian hierarchical (i.e., multilevel) modeling, and the visualization of task weightings that consider standard errors. These techniques reveal insights such as the dominance of a specific model for certain types of tasks despite an overall poor performance. We use a popular ML benchmark, the Visual Task Adaptation Benchmark (VTAB), to demonstrate the usefulness of our approaches.

Figures

Figures reproduced from arXiv: 2501.04234 by the authors.

Figure 1
Figure 1. Illustration of challenges when evaluating foundation models. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Illustration of bootstrap procedure for aggregate metrics. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 4
Figure 4. Visualization of differences in model performance under different category weightings using the [PITH_FULL_IMAGE:figures/full_fig_p011_4.png] view at source ↗
Figures from the paper (2 more)
Figure 5
Figure 5. Figure 5: Posterior probabilities over model ranks for average accuracy weighted to favor structured image [PITH_FULL_IMAGE:figures/full_fig_p019_5.png]
Figure 6
Figure 6. Figure 6: Posterior probabilities over model ranks for unweighted average accuracy using the Bayesian [PITH_FULL_IMAGE:figures/full_fig_p019_6.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Rethinking LLM Parametric Knowledge as Post-retrieval Confidence for Dynamic Retrieval and Reranking

    cs.IR 2025-09 conditional novelty 6.0 of 10

    Shifts in an LLM's hidden-state confidence, before and after a retrieved context, are used as a preference signal to fine-tune a reranker and to trigger retrieval only when initial confidence is low.

  2. Quantifying Ranking Uncertainty in LLM Benchmarks

    cs.LG 2026-06 conditional novelty 5.0 of 10

    MMLU ranking uncertainty is dominated by subject-level variability; rank confidence intervals widen substantially when subjects are treated as the sampling unit.

Reference graph

Works this paper leans on

30 extracted references · 22 canonical work pages · cited by 2 Pith papers

  1. [1]

    GPT -4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. GPT -4 technical report. arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  2. [2]

    Bayesian inferences on uncertain ranks and orderings: Application to ranking players and lineups

    Andr \'e s F Barrientos, Deborshee Sen, Garritt L Page, and David B Dunson. Bayesian inferences on uncertain ranks and orderings: Application to ranking players and lineups. Bayesian Analysis, 18 0 (3): 0 777--806, 2023

    work page 2023

  3. [3]

    Time for a change: A tutorial for comparing multiple classifiers through Bayesian analysis

    Alessio Benavoli, Giorgio Corani, Janez Dem s ar, and Marco Zaffalon. Time for a change: A tutorial for comparing multiple classifiers through Bayesian analysis. Journal of Machine Learning Research, 18 0 (77): 0 1--36, 2017

    work page 2017

  4. [4]

    Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S

    Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, S. Buch, Dallas Card, Rodrigo Castellon, Niladri S. Chatterji, Annie S. Chen, Kathleen A. Creel, Jared Davis, Dora Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Stefano...

    work page 2021

  5. [5]

    Accounting for variance in machine learning benchmarks

    Xavier Bouthillier, Pierre Delaunay, Mirko Bronzi, Assya Trofimov, Brennan Nichyporuk, Justin Szeto, Nazanin Mohammadi Sepahvand, Edward Raff, Kanika Madan, Vikram Voleti, et al. Accounting for variance in machine learning benchmarks. Proceedings of Machine Learning and Systems , 3: 0 747--769, 2021

    work page 2021

  6. [6]

    What are the best systems? N ew perspectives on NLP benchmarking

    Pierre Colombo, Nathan Noiry, Ekhine Irurozki, and St \'e phan Cl \'e men c on. What are the best systems? N ew perspectives on NLP benchmarking. Advances in Neural Information Processing Systems, 35: 0 26915--26932, 2022

    work page 2022

  7. [7]

    The benchmark lottery

    Mostafa Dehghani, Yi Tay, Alexey A Gritsenko, Zhe Zhao, Neil Houlsby, Fernando Diaz, Donald Metzler, and Oriol Vinyals. The benchmark lottery. arXiv preprint arXiv:2107.07002, 2021

    arXiv 2021

  8. [8]

    Statistical comparisons of classifiers over multiple data sets

    Janez Dem s ar. Statistical comparisons of classifiers over multiple data sets. The Journal of Machine Learning Research , 7: 0 1--30, 2006

    work page 2006

Show all 30 references
  1. [9]

    Bayesian aggregation of order-based rank data

    Ke Deng, Simeng Han, Kate J Li, and Jun S Liu. Bayesian aggregation of order-based rank data. Journal of the American Statistical Association , 109 0 (507): 0 1023--1039, 2014

    2014

  2. [10]

    Approximate statistical tests for comparing supervised classification learning algorithms

    Thomas G Dietterich. Approximate statistical tests for comparing supervised classification learning algorithms. Neural computation, 10 0 (7): 0 1895--1923, 1998

    1923

  3. [11]

    Statistical significance testing for natural language processing

    Rotem Dror, Lotem Peled-Cohen, Segev Shlomov, and Roi Reichart. Statistical significance testing for natural language processing. Springer, 2020

    2020

  4. [12]

    An Introduction to the Bootstrap

    Bradley Efron and Robert J Tibshirani. An Introduction to the Bootstrap. Chapman and Hall/CRC, 1994

    1994

  5. [13]

    The graphical presentation of a collection of means

    Harvey Goldstein and Michael JR Healy. The graphical presentation of a collection of means. Journal of the Royal Statistical Society: Series A (Statistics in Society) , 158 0 (1): 0 175--177, 1995

    1995

  6. [14]

    League tables and their limitations: S tatistical issues in comparisons of institutional performance

    Harvey Goldstein and David J Spiegelhalter. League tables and their limitations: S tatistical issues in comparisons of institutional performance. Journal of the Royal Statistical Society Series A : S tatistics in Society , 159 0 (3): 0 385--409, 1996

    1996

  7. [15]

    Randomized significance tests in machine translation

    Yvette Graham, Nitika Mathur, and Timothy Baldwin. Randomized significance tests in machine translation. In Proceedings of the Ninth Workshop on Statistical Machine Translation , pages 266--274, 2014

    2014

  8. [16]

    Modeling the variability of rankings

    Peter Hall and Hugh Miller. Modeling the variability of rankings. The Annals of Statistics , pages 2652--2677, 2010

    2010

  9. [17]

    Statistical comparisons of classifiers by generalized stochastic dominance

    Christoph Jansen, Malte Nalenz, Georg Schollmeyer, and Thomas Augustin. Statistical comparisons of classifiers by generalized stochastic dominance. Journal of Machine Learning Research, 24 0 (231): 0 1--37, 2023

    2023

  10. [18]

    Active Bayesian assessment of black-box classifiers

    Disi Ji, Robert L Logan, Padhraic Smyth, and Mark Steyvers. Active Bayesian assessment of black-box classifiers. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 7935--7944, 2021

    2021

  11. [19]

    Theory of Point Estimation

    Erich L Lehmann and George Casella. Theory of Point Estimation . Springer Science & Business Media, 1998

    1998

  12. [20]

    Bayesian analysis of rank data with covariates and heterogeneous rankers

    Xinran Li, Dingdong Yi, and Jun S Liu. Bayesian analysis of rank data with covariates and heterogeneous rankers. Statistical Science , 37 0 (1): 0 1--23, 2022

    2022

  13. [21]

    Slice sampling

    Radford M Neal. Slice sampling. The Annals of Statistics, 31 0 (3): 0 705--767, 2003

    2003

  14. [22]

    Uncertainty in ranking

    Justin Rising. Uncertainty in ranking. arXiv preprint arXiv:2107.03459, 2021

    2021 arXiv

  15. [23]

    On comparing classifiers: Pitfalls to avoid and a recommended approach

    Steven L Salzberg. On comparing classifiers: Pitfalls to avoid and a recommended approach. Data mining and knowledge discovery, 1: 0 317--328, 1997

    1997

  16. [24]

    Martin R. Smith. Ternary: An R Package for Creating Ternary Plots . Comprehensive R Archive Network , 2017. doi:10.5281/zenodo.1068996

    2017 doi

  17. [25]

    Beyond the imitation game: Quantifying and extrapolating the capabilities of language models

    Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adri \`a Garriga-Alonso, et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprin...

    2022 arXiv

  18. [26]

    Classifier uncertainty: E vidence, potential impact, and probabilistic treatment

    Niklas T \"o tsch and Daniel Hoffmann. Classifier uncertainty: E vidence, potential impact, and probabilistic treatment . PeerJ Computer Science , 7: 0 e398, 2021

    2021

  19. [27]

    LLaMA : Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth \'e e Lacroix, Baptiste Rozi \`e re, Naman Goyal, Eric Hambro, Faisal Azhar, et al. LLaMA : Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023

    2023 arXiv

  20. [28]

    Follow the leader (board) with confidence: E stimating p-values from a single test set with item and response variance

    Shira Wein, Christopher Homan, Lora Aroyo, and Chris Welty. Follow the leader (board) with confidence: E stimating p-values from a single test set with item and response variance. In Findings of the Association for Computational Linguistics: ACL 2023 , pages 3138--3161, 2023

    2023

  21. [29]

    Confidence intervals for population ranks in the presence of ties and near ties

    Minge Xie, Kesar Singh, and Cun-Hui Zhang. Confidence intervals for population ranks in the presence of ties and near ties. Journal of the American Statistical Association, 104 0 (486): 0 775--788, 2009

    2009

  22. [30]

    A large-scale study of representation learning with the visual task adaptation benchmark

    Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, et al. A large-scale study of representation learning with the visual task adaptation benchmark. arXiv prep...

    1910 arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.