{"id":"413e5463-c263-4f70-94e8-2ef06267e647","arxiv_id":"2501.04234","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"The paper shows how bootstrapping and Bayesian hierarchical modeling can quantify uncertainty in task-aggregated benchmark scores, using simulated VTAB data, and that a low-ranked model can dominate under certain task weightings.","lead":"This paper applies standard statistical tools, bootstrapping and Bayesian hierarchical models, to attach uncertainty to average performance scores in machine learning benchmarks. It demonstrates the methods on the VTAB image benchmark and shows that ranking conclusions can change when task weights and uncertainty are considered.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"All VTAB conclusions rest on independent Bernoulli simulations, yet the paper's own Eq. (3) shows cross-model correlation materially shrinks difference variances; without a paired or correlated sensitivity check, the 'no meaningful difference' and Rotation-weighting findings are not established.","rationale":"I read the paper as a demonstration of standard statistical machinery rather than a new estimator. The bootstrap and beta-binomial hierarchical model are standard, and the variance identities in Section 2.3 are correct. The central claim, however, is that these methods reveal insights about VTAB models, and that claim rests entirely on simulated data. The reader's weakest assumption identifies this correctly: independent Bernoulli draws from leaderboard accuracies are treated as a faithful proxy for real per-task responses. My stress-test sharpens this into a concrete mechanism. Paired evaluations induce positive correlation between models' accuracies on the same test items, and the paper's own Eq. (3) shows that this correlation reduces the variance of model differences. The paper uses that fact only in the weighting plots, not in the bootstrap or BHM intervals that support the main tables. The consequence is that the reported uncertainty may be substantially too wide, which directly affects conclusions such as 'there are no meaningful differences between SR-100% and SE-100%.' The proposed sensitivity analysis would settle whether the qualitative conclusions survive realistic correlation. I agree with the reader's conditional verdict: the paper should be revised to present the VTAB findings as illustrative, correct the internal numerical inconsistencies, and either obtain or explicitly model paired response structure. I do not recommend rejection because the methodological core is sound and the limitations are clearly disclosed in the text, but the empirical claims should not be stated as established facts about VTAB.","tokens_in":19200,"tokens_out":6552,"duration_ms":65493,"concrete_test":"Run a correlation sensitivity analysis at VTAB scale: simulate paired test responses for the 16 models and 19 tasks using a latent Gaussian copula with margins equal to the VTAB leaderboard accuracies, equicorrelation rho in {0, 0.25, 0.5, 0.75} across models within each task, and the real per-task test sizes. For each rho, perform the same bootstrap resampling on test items jointly for all models, then recompute (a) the SR-100% minus SE-100% difference confidence interval from Table 3 and (b) Rotation's average-rank confidence interval from Table 4. If at rho=0.5 the SR/SE interval excludes zero or shifts by more than one standard error relative to rho=0, the paper's 'no meaningful difference' conclusion is an artifact of the independence simulation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's empirical demonstration depends on Section 3's statement that the authors 'used simulated data consistent with the per-task accuracies for each model on the VTAB leaderboard,' because per-task responses are unavailable. This simulation draws independent Bernoulli responses for each model and task, imposing zero cross-model and zero cross-task correlation. That assumption is load-bearing for three concrete outputs: (i) the pairwise confidence intervals in Table 3 for SR-100% versus SE-100%; (ii) the rank intervals in Tables 4, 5, 8, and 10; and (iii) the headline insight from Figures 3 and 4 that when structured images are weighted heavily, Rotation is the best model. In a real benchmark, all models are evaluated on the same test items, so their per-task accuracies are positively correlated. The paper's own Section 2.3, Eq. (3), acknowledges that positive correlation reduces the variance of a difference: for equal variances and rho=0.5, the standard deviation of the difference is about 70% of the independent-case value. The authors apply this correction only as a crude z-scaling in the simplex plots; the bootstrap and BHM intervals ignore it. The BHM in Section 2.2 likewise assumes independent Beta distributions across models and tasks, with no correlation term. The interval widths and rank distributions are therefore computed under an assumption the paper itself identifies as potentially misleading. A compounding internal issue is Table 1: Task 2 is listed as N2=1000 yet Model A has 5000 correct responses, and the text says N2=10000, so the one worked simulation advocating the BHM is not reproducible as printed. The methodological message may survive, but the empirical claims about actual VTAB models do not.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes statistical uncertainty quantification for aggregate performance metrics on machine learning benchmarks, illustrating bootstrap confidence intervals, Bayesian hierarchical modeling, and uncertainty-aware simplex visualizations on the Visual Task Adaptation Benchmark (VTAB). Because per-task test responses are unavailable, the authors generate synthetic Bernoulli data whose means match the published VTAB leaderboard accuracies and whose sample sizes match the benchmark's test sets, then compute interval estimates for average accuracy, pairwise differences, rank aggregation, and category-weighting plots. They also include a two-model simulation study intended to show that a beta-binomial BHM can outperform the bootstrap when one task has a small sample size.","tokens_in":19531,"tokens_out":6320,"duration_ms":65122,"significance":"The statistical identities in Section 2.3 are correct, and the simplex visualizations are a genuinely useful way to communicate weighting uncertainty. The paper is honest about its main limitation: all VTAB conclusions are derived from simulated data consistent with leaderboard accuracies, not from actual responses. However, because the BHM demonstration in Section 2.2 uses priors centered on the true parameter values, and because the VTAB application ignores cross-model correlation, the current manuscript does not establish the empirical claims about Rotation or about the relative merits of bootstrap versus BHM. If the correlation sensitivity analysis and the simulation study are fixed, the paper would be a useful pedagogical demonstration of existing statistical tools for benchmark aggregation.","major_comments":[{"comment":"The numerical illustration is internally inconsistent. The text specifies θ_B1 = .525 and N2 = 10000, but Table 1 reports 115 correct out of 200 for Model B on Task 1 (0.575, not 0.525) and reports N2 = 1000 with 5,000 correct responses for both models, which is impossible. The bootstrap interval (-.059, .097) and the BHM interval (-.021, -.003) cannot be reproduced from the stated inputs, so this simulation does not support the claimed comparison.","section":"§2.2, Table 1"},{"comment":"The BHM advantage is manufactured by the prior. The simulation uses normal priors with means (2000, 2100, 2000, 1900) for (α1, α2, β1, β2), which correspond to prior success probabilities 0.50 for Model A and 0.525 for Model B — exactly the true generative values. This is not the Exp(1/10000) hyperprior specified earlier in the section, and it guarantees in advance that the posterior for θ_A - θ_B is centered below zero. To establish that shrinkage helps in small-sample settings, the authors should use the same prior for both methods or run a simulation study under a range of priors and report frequentist coverage and interval length, not a single cherry-picked case.","section":"§2.2, simulation study priors"},{"comment":"The empirical application conditions on an independence assumption the paper itself identifies as potentially misleading. Section 3 states that the authors 'used simulated data consistent with the per-task accuracies', and the simulation draws independent Bernoulli responses for each model-task pair; there is no cross-model correlation even though all models are evaluated on the same test items. Equation (3) shows that positive between-model correlation can reduce the standard deviation of a difference by roughly 30% at ρ = .5, yet the bootstrap and BHM intervals in Tables 2–5, 8, and 11 are computed under ρ = 0; only the simplex plots apply a crude z-scaling. As a result, the pairwise 'no meaningful difference' conclusions and the rank intervals are not robust checks under realistic dependence. Please add a correlated-data sensitivity analysis (e.g., a latent-variable model with common item effects) or, if actual responses cannot be released, state explicitly which conclusions survive a range of correlation values.","section":"§3, simulated VTAB data"},{"comment":"The headline Rotation finding is a deterministic consequence of the point estimates used to generate the data. In §3.3, the structured-category accuracies are 57.3 for Rotation versus 55.5 for SR-100%, so any weighting that puts large mass on structured images will favor Rotation in expectation; the intervals only reflect Bernoulli sampling noise. The paper should be explicit that the 'insights' are properties of the leaderboard point estimates under the simulation assumptions, not discoveries about unobserved response patterns.","section":"§3.3, headline VTAB insight"}],"minor_comments":[{"comment":"The notation uses Binom(θ_ij, N_j) while the usual convention is Binom(N_j, θ_ij); more importantly, the simulation study introduces a normal prior on (α_i, β_i) without explaining its relation to the Exp hyperpriors presented above. Please reconcile the two model specifications.","section":"§2.2"},{"comment":"Estimating the high and low normalization anchors as bootstrap maxima and minima makes the normalized metric depend strongly on the number of bootstrap replicates and on sampling noise; please report sensitivity to the choice of anchors or use fixed baselines.","section":"§2.4"},{"comment":"The gray region is defined as 'within z standard errors of 0', but the choice of z = 2 versus z = 1.4 is only heuristically justified; please explain the relationship to Eq. (3) and to the actual standard errors used in the plots.","section":"Figures 3 and 4"},{"comment":"There is a typo in the first sentence: 'adpated' should be 'adapted'.","section":"§4"}],"recommendation":"major_revision","confidential_remarks":"The manuscript reads as a workshop-style demonstration, and the methodological novelty is limited. The main risk for a journal is that the empirical claims are based entirely on simulated data and the BHM simulation study is circular. I would be willing to reconsider after the authors add a correlated-data sensitivity analysis and redo the simulation study with fair priors."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read this paper. Short version: it is a clean demonstration of standard statistical methods for putting error bars on aggregate ML benchmark scores, and the simplex visualization is genuinely nice. But every empirical 'insight' about VTAB models is computed from simulated data that assume independent responses, and the paper's own correlation math undercuts the intervals it reports. The methodological core holds up; the application section does not.\n\nWhat is actually new: applying bootstrap and Bayesian hierarchical modeling to task-aggregate metrics for pretrained models, plus the ternary plots that shade the best-model region under different task weights while accounting for standard errors. The variance derivations are correct, the Goldstein-Spiegelhalter lineage is properly credited, and the paper is clear about the difference between bootstrap (conditional on fitted models) and BHM. The warning against interpreting overlapping 95% intervals as 'no difference', and the use of 83.4% intervals plus pairwise differences, is sound practice.\n\nThe soft spots are real. Section 3 says plainly that per-task responses are unavailable, so the authors simulate independent Bernoulli draws per model and task from the VTAB leaderboard accuracies. That imposes zero cross-model correlation. Equation (3) shows positive correlation shrinks the variance of the difference by roughly 30% when rho=0.5, yet the bootstrap and BHM intervals in Tables 2-5 ignore it entirely; the authors only apply a crude z-scaling in the simplex plots. So the 'no meaningful difference' and 'Rotation wins under structured weighting' results are not established for actual VTAB. They are deterministic consequences of the leaderboard accuracies used to generate the simulation. The authors are transparent about the simulation, but the abstract overstates by saying the techniques 'reveal insights' about the benchmark. I would not call it circular in the formal sense, but the empirical demonstration is illustrative, not evidential.\n\nTable 1 is internally inconsistent: the text gives N2=10000 and theta_B1=.525, while the table shows N2=1000, 5000 correct responses for Model A (impossible), and 115/200=.575. That one worked example, which is the main evidence that BHM beats bootstrap, is not reproducible as printed. The simulation study also uses priors centered on the desired conclusion (alpha2=2100, beta2=1900), so it shows what strong priors do, not that BHM is generally better.\n\nWho gets value: practitioners who want to add uncertainty to benchmark comparisons, and statisticians happy to see standard tools applied to model evaluation. It is a worthwhile workshop paper with a useful message. I would send it to serious peer review with major revision requested: fix Table 1, relabel the VTAB analysis as illustrative, add a paired or correlated sensitivity check, and tone down the abstract.","headline":"A well-written methodology demonstration whose VTAB 'insights' are not empirical findings, because the simulation ignores the cross-model correlation the paper itself flags.","tokens_in":20091,"tokens_out":4217,"would_cite":false,"duration_ms":39165,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62F40","62F15","62F07"],"pacs":[],"model":"deepseek-v4-flash","headline":"Benchmark rankings that ignore uncertainty can mislead: with task weighting and standard errors, a sixth-place model becomes the best on structured-image tasks.","keywords":["uncertainty quantification","machine learning benchmarks","bootstrap","Bayesian hierarchical model","task aggregation","VTAB","leaderboard rankings","task weighting"],"falsifier":"Obtain the actual per-task test responses for the 16 VTAB models and re-run the bootstrap, the hierarchical model, and the simplex weighting with the real data; if Rotation no longer occupies the structured-weighted winning region, or if the Sup-Rotation versus Sup-Exemplar difference becomes significant, the paper's empirical claims would be overturned.","tokens_in":19001,"feed_emoji":"📊","tokens_out":5851,"duration_ms":50782,"temperature":0.7,"pith_summary":"Machine learning leaderboards typically report a single number that averages a model's accuracy across many benchmark tasks, and that number ignores the sampling uncertainty in each task's test set. This paper argues that such aggregate scores need confidence and credibility intervals, and demonstrates two standard statistical tools for building them: bootstrapping the evaluation data and a beta-binomial Bayesian hierarchical model. Applied to the VTAB benchmark's 16 models, the intervals show that the top two models are statistically indistinguishable in overall average accuracy, and that a model ranked sixth overall can be the best when structured-image tasks are weighted heavily. The paper also provides simplex visualizations that show which model wins for every combination of task-category weights, with uncertainty shading that indicates when the top two models are too close to call. The payoff is a way for benchmark users to attach honest uncertainty to leaderboard rankings and to see how task weighting changes the conclusion.","feed_headline":"Structured-task weighting makes a low-ranked model the best","feed_subtitle":"Bootstraps and Bayesian intervals show which leaderboard rankings survive uncertainty, and which flip.","key_machinery":"The machinery is the weighted aggregate score $S_i = \\sum_j w_j \\hat{p}_{ij}$ with its variance decomposition $\\mathrm{Var}(S_i) = \\sum_j w_j^2 \\mathrm{Var}(\\hat{p}_{ij}) + 2\\sum_{j<j'} w_j w_{j'} \\mathrm{Cov}(\\hat{p}_{ij}, \\hat{p}_{ij'})$, which the paper uses to build standard errors for model differences on ternary simplex plots; the variance formula also shows that positive between-task correlation inflates the true variance relative to the independence assumption. Supporting this is the $\\beta$-binomial hierarchical model in which the number of correct responses $Y_{ij}$ follows $\\mathrm{Binomial}(\\theta_{ij}, N_j)$ with $\\theta_{ij} \\sim \\mathrm{Beta}(\\alpha_i, \\beta_i)$ and exponential hyperpriors, fitted by a Gibbs sampler with slice-sampling steps, which provides shrinkage for tasks with small test sets. The bootstrap serves as the nonparametric counterpart, resampling each task's test instances and recomputing the benchmark's own aggregation rule. Together these objects convert leaderboard point estimates into distributions over aggregate scores, differences, and ranks.","core_discovery":"On the paper's own terms, the central discovery is that accounting for uncertainty in aggregated benchmark metrics changes the conclusions that leaderboards invite: the two leading supervised models, Sup-Rotation-100% and Sup-Exemplar-100%, have overlapping 83.4% intervals for unweighted average accuracy, so their apparent ordering is not reliable, while under a normalized-accuracy scheme their ranking reverses with non-overlapping intervals. More strikingly, rank-based aggregation and task-weighting visualizations both suggest that the Rotation model, sixth by average accuracy, is the best model when structured-image tasks receive heavy weight. These findings are demonstrated through bootstrapped test data, a beta-binomial hierarchical model, and standard-error-aware simplex plots rather than through new statistical theory; the contribution is the demonstration that established statistical machinery can be transplanted onto model benchmarking.","pith_inferences":["The same toolkit transfers directly to LLM benchmarks that publish per-example responses: item-level resampling and a hierarchical model over tasks would let anyone reproduce these intervals without access to model training runs.","A natural next step the paper does not pursue is estimating the between-task and between-model covariance matrix from response-level data, which would replace the independence assumption and the $z = 2\\sqrt{1/2}$ heuristic with an empirical correlation adjustment.","The simplex plots could be inverted into a decision aid: a stakeholder who can only articulate qualitative priorities ('mostly structured images') gets a region of weights, not a single weighting, for which a model is the best within uncertainty.","If benchmark owners published test responses with item-level metadata, the bootstrap step would no longer need simulation, and the whole framework could be run exactly by any practitioner."],"forward_implications":["Practitioners can attach bootstrap or hierarchical-model intervals to any leaderboard that aggregates per-task accuracies, and should treat point-estimate orderings within those intervals as unresolved.","Task weighting can reverse model dominance: the same model can be worst overall yet best for a stakeholder who cares about one task category, as the Rotation model shows for structured images.","Normalizing task scores before aggregation changes rankings and should be reported alongside raw averages, since it rewards strength on harder tasks.","Rank aggregation methods that reward many small wins (average-rank schemes) can disagree sharply with mean-based rankings, so leaderboards should report more than one aggregation rule.","If tasks are positively correlated, the true uncertainty is larger than the independence-based intervals suggest, so the gray 'too close to call' regions in the plots would grow."],"supporting_citations":[{"why":"Supplies the VTAB benchmark, its 19 tasks, and the leaderboard accuracies that the analysis simulates.","marker":"[Zhai et al., 2019]"},{"why":"Establishes the bootstrap resampling procedure used to build confidence intervals for aggregate metrics.","marker":"[Efron and Tibshirani, 1994]"},{"why":"Provides the league-table template of bootstrapped intervals plus Bayesian hierarchical modeling that this paper adapts to model benchmarks.","marker":"[Goldstein and Spiegelhalter, 1996]"},{"why":"Justifies displaying 83.4% intervals so overlap corresponds to a 5% significance level.","marker":"[Goldstein and Healy, 1995]"},{"why":"Supplies slice sampling for the non-conjugate conditional distributions of the beta-binomial hierarchical model.","marker":"[Neal, 2003]"},{"why":"Provides the ternary plotting routine used for the task-weighting simplex visualizations.","marker":"[Smith, 2017]"},{"why":"Supplies the min-max normalization scheme used to rescale task accuracies.","marker":"[Srivastava et al., 2022]"}],"fun_headline_variants":["Uncertainty flips the ML leaderboard order","Bootstraps reveal hidden best model in benchmarks","A sixth-ranked model wins under uncertainty","Task weighting crowns an underdog model"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The VTAB demonstration relies on test responses simulated from the published leaderboard accuracies with independent Bernoulli draws per model and task; if the real test data contain cross-model correlations or item-level structure that this simulation omits, the reported intervals and the Rotation result may not transfer to the actual benchmark.","fun_headline_variants_meta":{"raw":{"variants":["Uncertainty flips the ML leaderboard order","Bootstraps reveal hidden best model in benchmarks","A sixth-ranked model wins under uncertainty","Task weighting crowns an underdog model"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000466,"raw_usage":{"total_tokens":2292,"prompt_tokens":877,"completion_tokens":1415,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":493,"completion_tokens_details":{"reasoning_tokens":1359}},"tokens_in":493,"tokens_out":1415,"duration_ms":12169,"temperature":1.0,"reasoning_tokens":1359,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:37:53.998551+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Obtain the actual per-task test responses for the 16 VTAB models and re-run the bootstrap, the hierarchical model, and the simplex weighting with the real data; if Rotation no longer occupies the structured-weighted winning region, or if the Sup-Rotation versus Sup-Exemplar difference becomes significant, the paper's empirical claims would be overturned.","supporting_citations":[{"cited_title":"League tables and their limitations: S tatistical issues in comparisons of institutional performance","cited_arxiv_id":null,"evidence_quote":"Provides the league-table template of bootstrapped intervals plus Bayesian hierarchical modeling that this paper adapts to model benchmarks."},{"cited_title":"The graphical presentation of a collection of means","cited_arxiv_id":null,"evidence_quote":"Justifies displaying 83.4% intervals so overlap corresponds to a 5% significance level."}],"review_version":1}