{"id":"9498ac57-71a1-4641-b325-fcc0f231de5a","arxiv_id":"2604.23099","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":4,"one_line_summary":"Bayesian deep learning method rankings are unreliable under data scarcity, reversing across datasets and sample sizes, and a hierarchical Bayesian framework with predictive detectability curves is needed to assess evaluation sufficiency.","lead":"The paper shows that rankings of Bayesian deep learning methods become unreliable when training data is scarce, with method orderings reversing across datasets and sample sizes. A smart generalist should read it because it provides practical tools to determine whether evaluation data is sufficient before claiming one method beats another.","discovery_kind":"unclear","skeptic_critique":{"model":"glm-5.2","headline":"The BHM's independence assumption across realizations is violated by substantial subsample overlap in small-pool datasets (Concrete n=500: ~69% pairwise overlap), potentially inflating the extreme posterior probabilities that anchor the central claim.","rationale":"The reader correctly identifies Equation 1's assumptions as the weak point but focuses on the Gaussian likelihood for NLL, which is secondary since CRPS is the primary metric and its Gaussian fit is verified via posterior predictive checks (Figure 5, Appendix B). The more pressing concern is the independence assumption across realizations: with finite pools and large n, subsamples overlap substantially, inducing positive correlation in metric values that the BHM ignores. This could inflate the extreme posterior probabilities (0.000, 1.000) that anchor the central claim. The concern is most acute for Concrete (pool=721, n=500: 69% overlap) and Yacht (pool=215: 47% overlap at n=100), which are exactly the datasets with the most borderline or extreme results. For Kin8nm (pool=5734, n=50: <0.5% overlap), the concern is negligible. The paper's stability check (R=30 vs R=50) does not address this because both subsets have the same overlap structure. However, the qualitative conclusion — that rankings are dataset-dependent and no universal threshold exists — would likely survive even with overlap correction, because the cross-dataset variation in posterior probabilities is large (0.00 to 1.00) and would persist even if individual probabilities shifted by 0.05–0.10. The SWAG exclusion (convergence rate < 0.80) is a separate data selection issue that affects SWAG comparisons but not the central MCD vs Ensemble claim. No code or data is shipped, limiting independent verification. The verdict remains CONDITIONAL: the paper is a legitimate contribution with a real phenomenon, but the specific posterior probabilities that form the quantitative core of the claim rest on an unvalidated independence assumption that is most strained exactly where the results are most interesting.","tokens_in":18188,"tokens_out":7393,"duration_ms":152776,"concrete_test":"For Concrete at n=500 (pool=721), recompute P(MCD≺Ensemble) using a BHM that accounts for subsample overlap — e.g., by fitting with a correlated likelihood where the correlation between realizations i,j is proportional to their training-set Jaccard overlap |S_i∩S_j|/|S_i∪S_j|, or by using only realizations with <20% pairwise overlap if feasible. If the posterior probability shifts by more than 0.05 from the reported 0.95, the overlap-induced bias materially affects the ranking claims.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim depends on posterior probabilities from the BHM in Equation 1, which treats the R=50 metric observations per cell as independent draws from a common distribution. However, the training subsamples are drawn from a finite pool, and for small datasets at large n, the expected pairwise overlap is substantial: Concrete (pool=721, n=500) has ~69% expected pairwise overlap (500×500/721 ≈ 347 shared samples out of 500), and Yacht (pool=215, n=100) has ~47%. When two training sets share a large fraction of data, the resulting metric values are positively correlated because the trained models are partially fit to the same examples. The BHM's independence assumption then underestimates the effective sample size, producing posterior probabilities that are more extreme (closer to 0 or 1) than warranted. This directly affects the headline numbers: P(MCD≺Ensemble)=0.95 on Concrete at n=500 — the weakest link in the 'no universal threshold' argument — and the extreme probabilities on Yacht (P=0.00 at n=50) could be inflated by this mechanism. The paper acknowledges exchangeability may fail for 'non-stationary distributions' (Section 6) but does not address the more basic issue of finite-pool subsample overlap. The stability check in Appendix B (R=30 vs R=50, changes <0.02) does not address this because both subsets suffer the same overlap structure. The MDD formula (Equation 4) inherits the same issue, as σ_pred depends on within-method variance s_m² estimated under the independence assumption. That said, the qualitative finding — rankings vary substantially across datasets — would likely survive overlap correction, since the cross-dataset differences are large enough that moderate inflation would not eliminate them. The concern is about the precision of specific posterior probabilities, not the existence of ranking instability.","agreement_with_reader":"partial"},"referee_report":{"model":"glm-5.2","summary":"This manuscript investigates whether standard evaluation metrics for Bayesian deep learning (BDL) methods are reliable when training data is scarce. The authors draw R=50 independent subsamples of size n from a fixed pool for each of five regression datasets, train six BDL methods from scratch on each subsample, and record CRPS, NLL, PICP, and MPIW. They fit a Bayesian hierarchical model (BHM) with method-specific variances to these metric observations, compute posterior probabilities of pairwise method superiority (Eq. 3), and define a predictive Minimum Detectable Difference (MDD) curve (Eq. 4) to assess whether an observed gap would be detectable in a new experiment. The central empirical finding is that method rankings are unstable and dataset-dependent at small n: the same comparison (MCD vs. Deep Ensembles on CRPS) yields P=1.000 at n=50 on one dataset but remains below 0.95 even at n=500 on another, and no universal sample-size threshold separates reliable from unreliable comparisons. The paper is clearly written and addresses a genuine gap in the BDL evaluation literature.","tokens_in":18680,"tokens_out":2091,"duration_ms":171548,"significance":"The paper makes a timely and practical contribution by formalizing the problem of metric unreliability under training-data scarcity, a regime that existing large-scale benchmark studies (which assume thousands of training samples) do not address. The BHM framework with method-specific variances, the posterior superiority probability, and the predictive MDD curve are useful, concrete tools for practitioners. The experimental design—50 realizations per cell, MCMC convergence diagnostics (R-hat < 1.01, ESS > 400), posterior predictive checks, and a stability check (R=30 vs. R=50)—is thorough by the standards of the subfield. The power-law variance decomposition (Table 7, Figure 7) and the cross-dataset replication (Table 2) strengthen the central claim. The falsifiable prediction that no universal n threshold exists is well-supported by the data presented.","major_comments":[{"comment":"Section 3.1, Eq. (1): The BHM treats the R=50 metric observations per cell as conditionally independent given the latent parameters. However, because subsamples are drawn without replacement from a finite pool, the training sets for different realizations overlap substantially at large n. For Concrete (pool=721, n=500), the expected pairwise overlap is approximately 500*500/721 ≈ 69%; for Yacht (pool=215, n=100), it is approximately 47%. This overlap induces positive correlation between metric values across realizations, because the trained models share a large fraction of training data. The BHM's independence assumption then underestimates the effective sample size, potentially producing posterior probabilities that are more extreme (closer to 0 or 1) than warranted. This directly affects the headline numbers: P(MCD≺Ensemble)=0.95 on Concrete at n=500 (Table 2), which is the weakest and","section":null},{"comment":"Section 3.1, Eq. (1): The BHM treats the R=50 metric observations per cell as conditionally independent given the latent parameters. However, because subsamples are drawn without replacement from a finite pool, the training sets for different realizations overlap substantially at large n. For Concrete (pool=721, n=500), the expected pairwise overlap is approximately 69%; for Yacht (pool=215, n=100), approximately 47%. This overlap induces positive correlation between metric values across realizations, violating the independence assumption and potentially inflating the extremity of posterior probabilities. The paper acknowledges exchangeability may fail for 'non-stationary distributions' (Section 6) but does not address finite-pool subsample overlap. The stability check in Appendix B (R=30 vs. R=50, changes <0.02) does not address this because both subsets share the same overlap structure","section":null},{"comment":"Section 5.1, Table 1 and Appendix A.5, Table 4: SWAG runs with convergence rates below 0.80 are excluded from the BHM fits (Appendix B). This is a post-hoc data exclusion that affects the SWAG comparisons. At n=30 on the synthetic dataset, the convergence rate is 0.58 (Table 4), meaning 42% of runs are excluded. The paper states this benchmarks 'the uncertainty quantification quality of successful SWAG fits rather than its convergence reliability,' but the excluded runs are not missing at random—they represent a systematic failure mode of SWAG at small n. The paper should explicitly discuss how this exclusion affects the generalizability of the SWAG comparisons and whether the same exclusion criterion was pre-registered or chosen after inspecting the data.","section":null},{"comment":"Section 3.1, Eq. (1) and Appendix B: The Gaussian working likelihood is acknowledged to be inadequate for NLL at small n due to heavy right skew from sigma collapse (Appendix B, Figure 5). The paper states that 'conclusions based on NLL below n=100 should be interpreted with caution,' but NLL results are still reported in the main text (Table 8) and used in the power-law analysis (Figure 1, right panel). If the Gaussian likelihood is substantially misspecified for NLL at small n, the posterior probabilities and MDD derived from the BHM for NLL could be biased. The paper should either (a) restrict all NLL-based posterior claims to n≥100 where the approximation is adequate, or (b) provide a sensitivity analysis using a heavy-tailed likelihood (e.g., Student-t) to show that the ranking conclusions are robust to the likelihood choice.","section":null},{"comment":"Section 3.3, Eq. (4): The MDD formula depends on σ_pred, which combines posterior uncertainty in the mean difference with method-specific variability. However, σ_pred is derived under the same independence and Gaussian assumptions as the BHM. If the within-method variance s_m² is underestimated due to subsample overlap (Major Comment 1), the MDD curve will be optimistically narrow, understating the sample size required for reliable detection. The paper should discuss this dependency and, ideally, provide a corrected MDD that accounts for finite-pool effects, or at minimum note that the MDD values are lower bounds on the true detectable difference.","section":null}],"minor_comments":[{"comment":"Abstract: The abstract mentions 'P(MCD≺Ensemble)=1.000 at n=50 on one dataset and remains below 0.95 even at n=500 on another,' but does not name the datasets. Adding 'on the synthetic dataset' and 'on Concrete' would improve clarity.","section":null},{"comment":"Table 2: The Energy dataset row shows '—' for n=500, but the reason (pool size 538 is too small for n=500) is only explained in Table 3. A footnote on Table 2 would help.","section":null},{"comment":"Figure 2 caption: The caption refers to 'three representative datasets' but the figure shows only two lines (Synthetic and Energy). Please clarify or correct.","section":null},{"comment":"Section 5.4: The text states 'Bayes By Backprop outperforms Monte Carlo Dropout on the Concrete dataset at n=200,' but Table 11 does not include a BBB vs. MCD comparison for Concrete. Please add this comparison or clarify the source of the claim.","section":null},{"comment":"Appendix B, Table 6: The R-hat table reports values for CRPS, PICP, and Interval Score, but not for NLL. Please add NLL convergence diagnostics.","section":null},{"comment":"Section 3.2: The Beta-Binomial model for PICP (Eq. 2) uses a logit link for p_k, but the precision parameter φ is not discussed. Please state the prior on φ and whether it was sensitivity-checked.","section":null},{"comment":"Table 1: SWAG CRPS values at n=30 and n=50 (1.399 and 1.287) are orders of magnitude larger than other methods. A note explaining that these reflect only converged runs (as stated in the caption) but are still highly variable would help readers contextualize these numbers.","section":null},{"comment":"References: Several arXiv preprints are cited without DOIs (e.g., Dehghani et al. 2021, Longjohn et al. 2025). Please add DOIs where available.","section":null},{"comment":"Section 6: The limitation about exchangeability mentions 'non-stationary distributions' but does not mention the finite-pool overlap issue (see Major Comment 1). This limitation should be acknowledged explicitly.","section":null}],"recommendation":"major_revision","confidential_remarks":"The stress-test concern about subsample overlap is, in my assessment, the most important issue for this paper. The overlap arithmetic is straightforward and the concern is valid: at n=500 on Concrete (pool=721), the 50 subsamples of size 500 drawn without replacement will share on average ~69% of their data pairwise. This is not a marginal effect. The BHM's conditional independence assumption is violated, and the direction of bias (inflated posterior extremity) is clear. The paper's central claim—that no universal threshold exists—may still hold even after correcting for this, because the dataset-dependence is qualitative, but the specific posterior probabilities that anchor the claim (e.g., P=0.95 on Concrete at n=500) could shift. I recommend requiring the authors to either (a) use sampling-with-replacement to eliminate overlap, (b) add a finite-population correction to the BHM, or (c) provide a sensitivity analysis showing the posterior probabilities are robust to the overlap. The SWAG exclusion and NLL likelihood issues are secondary but also need explicit discussion. The paper is well-motivated and the framework is useful; with these corrections it would be a solid contribution."},"author_rebuttal":{"model":"glm-5.2","summary":"We thank the referee for a careful and constructive report. The comments identify two genuine modeling limitations (finite-pool subsample overlap and NLL likelihood misspecification) and two presentation gaps (SWAG exclusion criteria and MDD dependency on BHM assumptions). We address each below and indicate where revisions will be made.","responses":[{"response":"We agree this is a legitimate concern that the manuscript does not currently address. We note two points in partial mitigation, followed by the revision we will make. First, the BHM already includes a realization-specific random intercept b_r (Eq. 1) that captures the paired structure across methods within each realization—i.e., the fact that all methods share the same training subsample in realization r. This term accounts for within-realization correlation but does not address between-realization overlap, which is the referee's point. Second, the overlap problem is most severe at large n, where our qualitative finding is that rankings stabilize. At small n—where the main findings of ranking instability and reversal are concentrated—the overlap is negligible (e.g., n=30 from a pool of 840 gives ~3.6% expected pairwise overlap). The specific result most affected is the borderline P(MCD≺Ensemble)=0.95 on Concrete at n=500, which is exactly where overlap is largest (~69%) and the posterior is closest to the decision threshold. We acknowledge that the independence assumption likely makes this posterior more extreme than warranted, and the corrected value could fall below 0.95. We will revise the manuscript to: (1) explicitly discuss finite-pool overlap as a source of between-realization correlation, (2) note that posterior probabilities at large n (especially on smaller-pool datasets like Concrete and Yacht) may be more extreme than warranted, (3) add a sensitivity analysis using a bootstrap estimate of the effective sample size correction (design-effect adjustment) for the Concrete n=500 case to assess robustness of the borderline result, and (4) note that this issue strengthens, rather than weakens, our central claim that practitioners should treat borderline posterior概率","revision_made":"no","referee_comment":"Section 3.1, Eq. (1): Subsample overlap violates conditional independence, potentially inflating posterior extremity. The BHM treats R=50 observations as independent given latent parameters, but without-replacement sampling from a finite pool induces positive correlation at large n (e.g., ~69% pairwise overlap for Concrete at n=500)."},{"response":"We agree that the exclusion criterion and its implications should be discussed more explicitly. To clarify the nature of the exclusion: the criterion is based on SWAG's optimization convergence (NaN loss or insufficient weight snapshots), not on the resulting metric values. Failed runs produce no meaningful posterior approximation, so including them would conflate SWAG's optimization reliability with its uncertainty quantification quality—two distinct properties. However, the referee is correct that this exclusion is post-hoc (chosen after observing convergence rates) and that the excluded runs represent a systematic failure mode of SWAG at small n, not random missingness. This means our SWAG results represent a best-case scenario: the UQ quality of SWAG conditional on successful convergence. We will revise the manuscript to: (1) state explicitly that the exclusion criterion was chosen after inspecting convergence rates and was not pre-registered, (2) add a sentence noting that the SWAG comparisons therefore characterize successful SWAG fits and should not be interpreted as statements about SWAG's overall reliability at small n, and (3) report SWAG convergence rates alongside all SWAG posterior probability entries in the results tables (currently only in Appendix A.5) so readers can assess the fraction of excluded runs for each cell. We note that the main findings of the paper do not depend on SWAG comparisons—the ranking instability and reversal findings are driven by MCD, Deep Ensembles, BBB, and MAP, which do not have convergence failures.","revision_made":"yes","referee_comment":"Section 5.1, Table 1 and Appendix A.5, Table 4: SWAG runs with convergence rates below 0.80 are excluded from BHM fits. This is post-hoc data exclusion affecting SWAG comparisons, and the excluded runs represent a systematic failure mode at small n."},{"response":"We agree and will adopt option (a) proposed by the referee: restrict all NLL-based posterior claims to n≥100, where the Gaussian approximation is adequate as confirmed by posterior predictive checks (Appendix B, Figure 5). Specifically, we will: (1) add an explicit caveat to Table 8 and the NLL panel of Figure 1 stating that NLL-based posterior probabilities and variance estimates below n=100 are unreliable due to likelihood misspecification, (2) remove or clearly flag any NLL-based posterior probability claims below n=100 in the main text, and (3) note in the power-law analysis (Figure 1, right panel) that the NLL variance estimates at n=30 and n=50 are affected by the Gaussian likelihood's failure to capture the heavy right tail. We considered option (b) (Student-t sensitivity analysis) but note that the NLL results are not central to the paper's main claims—CRPS is the primary metric throughout, and the ranking instability and reversal findings are based on CRPS. The NLL analysis serves mainly to motivate the choice of CRPS as the primary metric, and the qualitative conclusion (CRPS is more stable than NLL at small n) holds regardless of the likelihood specification because it is based on empirical standard deviations, not posterior probabilities.","revision_made":"yes","referee_comment":"Section 3.1 and Appendix B: Gaussian working likelihood is inadequate for NLL at small n due to heavy right skew from sigma collapse. NLL results are still reported in main text and used in power-law analysis. Posterior probabilities and MDD for NLL could be biased."},{"response":"The referee is correct that the MDD inherits the BHM's assumptions. We will address this in two ways. First, we will add an explicit statement that the MDD values are derived under the same independence and Gaussian assumptions as the BHM, and that in the presence of finite-pool overlap, the reported MDD values should be interpreted as lower bounds on the true minimum detectable difference (i.e., the true sample size required for reliable detection is at least as large as reported). Second, we note that the MDD analysis in the main text (Figure 4) is presented for the synthetic dataset at n=50, where the pool size is 840 and the expected pairwise overlap is approximately 50*50/840 ≈ 3%, so the overlap effect is minimal for this specific result. The general MDD scaling claim (MDD ∝ n^{-α/2}) is an empirical description of variance reduction rates and does not depend on the independence assumption—it is computed from empirical standard deviations across realizations. We will add a note clarifying which aspects of the MDD analysis are affected by the overlap assumption (the absolute MDD values at large n on small-pool datasets) and which are not (the scaling relationship and the small-n results on large-pool datasets).","revision_made":"yes","referee_comment":"Section 3.3, Eq. (4): MDD formula depends on σ_pred, which inherits the same independence and Gaussian assumptions as the BHM. If within-method variance s_m² is underestimated due to subsample overlap, the MDD curve is optimistically narrow."}],"tokens_in":18597,"tokens_out":2639,"duration_ms":89676,"standing_objections":[]},"desk_editor":{"model":"glm-5.2","letter":"The paper's central empirical finding is solid: BDL method rankings are not stable at small training set sizes, and the instability is dataset-dependent in ways that point estimates miss. The Bayesian hierarchical model with method-specific variances is a reasonable framework for this, and the MDD curve is a genuinely useful diagnostic that practitioners would benefit from. The cross-dataset comparison — Concrete stays inconclusive at n=500 while Kin8nm is nearly deterministic at n=50 — is the kind of result that should change how people run BDL benchmarks in low-data settings. Credit is earned here: the experimental design (50 realizations per cell, MCMC convergence diagnostics, posterior predictive checks) is competent, and the framework is self-contained without circularity. The power-law variance decomposition is descriptive but informative. The SWAG exclusion is handled transparently and is defensible — they benchmark UQ quality of successful fits, not convergence reliability, and they say so. The stress-test concern about finite-pool subsample overlap is real but overstated in its impact. On Concrete (pool=721, n=500), the ~69% pairwise overlap does mean the BHM's independence assumption underestimates effective sample size, which would inflate posterior probabilities toward extremes. This affects the precision of headline numbers like P(MCD≺Ensemble)=0.95 on Concrete at n=500. However, the qualitative finding — that rankings vary substantially across datasets and no universal threshold exists — would survive overlap correction. The cross-dataset differences are large enough that moderate inflation doesn't eliminate them. The paper's own stability check (R=30 vs R=50, changes <0.02) doesn't address this because both subsets share the same overlap structure, so that's a gap, but not a fatal one. The Gaussian working likelihood for NLL at small n is acknowledged honestly in Appendix B. Posterior predictive checks show adequate fit for CRPS, which is the primary metric. NLL conclusions at small n should be treated cautiously, which the paper says. No code or data is shipped, which limits reproducibility — this is the most actionable thing to flag to authors. This paper is for researchers working on uncertainty quantification evaluation and practitioners running BDL benchmarks in low-data regimes. It deserves a serious referee. The main things to push on: (1) acknowledge and ideally quantify the overlap effect on posterior probabilities, (2) ship code and data, (3) be more explicit about how the MDD formula inherits the independence assumption. None of these break the paper.","headline":"The paper shows that BDL method rankings are unstable under training-data scarcity, and the core finding is real. The stress-test concern about subsample overlap is technically valid but does not change the qualitative conclusion.","tokens_in":19035,"tokens_out":596,"would_cite":true,"duration_ms":74304,"reading_group":"no","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"glm-5.2","headline":"Rankings of Bayesian deep learning methods are unreliable when training data is scarce","keywords":["Bayesian deep learning","uncertainty quantification","evaluation reliability","small-sample statistics","Bayesian hierarchical model","method ranking stability","CRPS","minimum detectable difference"],"falsifier":"The paper's central empirical claim would be falsified if, across a broader set of datasets and methods, a universal sample size threshold were found that reliably separates stable from unstable method rankings, or if the ranking reversals documented here turned out to be artifacts of the specific small neural network architecture used rather than genuine properties of the metrics under data scarcity.","tokens_in":18462,"feed_emoji":"🎲","tokens_out":1222,"duration_ms":71502,"temperature":0.7,"pith_summary":"This paper argues that the standard practice of ranking Bayesian deep learning methods by point-estimate metrics breaks down when training data is limited. The authors show empirically that when training sets contain only tens to a few hundred samples, the variance of evaluation metrics like CRPS and NLL is large enough that method rankings reflect sampling noise as much as true performance differences. They demonstrate that the same pairwise comparison between methods can yield a posterior probability of superiority of 1.000 on one dataset while remaining below 0.95 even at 500 training samples on another, and that ranking reversals occur as training size increases in ways invisible to point estimates. To address this, the authors propose a Bayesian hierarchical model that treats evaluation metrics as random variables across independent training set draws, with method-specific variances, enabling posterior inference over pairwise method comparisons. They also introduce a predictive Minimum Detectable Difference curve that quantifies the smallest performance gap reliably detectable at a given training size. The central claim is that no universal sample size threshold separates reliable from unreliable method comparisons across datasets, making dataset-specific posterior inference necessary.","feed_headline":"Bayesian deep learning rankings are unreliable when training data is scarce","feed_subtitle":"Same method comparison yields posterior probability 1.000 on one dataset and below 0.95 at 500 samples on another; no universal threshold","key_machinery":"The paper's central machinery is a Bayesian hierarchical model (Equation 1) in which the metric value for method i on data realization r is modeled as m_ir = μ_i + α_i + γ_r + ε_ir, where μ_i is the latent true metric value, α_i captures within-method variance, γ_r is a realization-specific random intercept for paired structure, and ε_ir is residual noise. Method-specific variances are essential because methods differ substantially in metric variability at small n. Posterior ordering (Equation 3) computes the probability that method i outperforms method j by integrating over the joint posterior. The predictive Minimum Detectable Difference (Equation 4) gives the smallest true gap detectable ","core_discovery":"The paper's central discovery is that method rankings in Bayesian deep learning are not stable under data scarcity, and that this instability is dataset-dependent in ways that point-estimate comparisons cannot reveal. The authors demonstrate this through a systematic study of six methods across five regression datasets at five training set sizes, finding that the posterior probability that Monte Carlo Dropout outperforms Deep Ensembles on CRPS is 1.000 at n=50 on one dataset yet remains below 0.95 even at n=500 on another. They attribute this to the compounding of data variance (from finite training samples) with algorithm variance (from training randomness), which standard evaluation treats","pith_inferences":["If the Gaussian working likelihood is substantially misspecified for heavy-tailed metrics like NLL at small n, the posterior probabilities in Equation 3 could be biased in unknown directions. A robust likelihood or nonparametric model might yield different ranking conclusions for NLL-based comparisons, particularly below n=100.","The exchangeability assumption across data realizations may break if training sets are drawn from non-stationary distributions, which is common in time-series or streaming data settings; the framework's conclusions may not extend to such scenarios without modification.","The finding that no universal sample size threshold exists suggests that a meta-analytic approach pooling posterior evidence across multiple datasets could provide more reliable method rankings than any single-dataset evaluation, analogous to clinical trial meta-analysis."],"forward_implications":["Practitioners in low-data domains (drug discovery, medical imaging, environmental monitoring) cannot assume that benchmark rankings from large-scale studies transfer to their setting; they need dataset-specific posterior inference to determine whether their evaluation data supports any method-superiority claim.","Point-estimate leaderboards should be supplemented with uncertainty-aware comparison statistics and detectability diagnostics before drawing conclusions about method superiority, particularly when training data is limited.","The finding that CRPS is substantially more stable than NLL at small training sizes suggests that metric choice affects not only accuracy but statistical power for method comparison, with practical consequences for experimental design.","The compounding of data variance and algorithm variance implies that methods with higher training instability require disproportionately more data to achieve reliable metric estimates, a factor that current benchmarking practice does not account for."],"fun_headline_variants":["ProEval finds AI failures with 8-65x fewer test samples","Gaussian Process surrogate estimates model performance with 1% accuracy","Active input selection exposes diverse AI failures under strict budgets","Pre-trained GPs make generative AI evaluation unbiased and sample-efficient","Bayesian quadrature cuts generative AI evaluation cost by up to 65x"],"cache_read_input_tokens":0,"weakest_assumption_plain":"The Bayesian hierarchical model assumes that metric values across independent training set draws are exchangeable and approximately Gaussian. For CRPS this holds well, but for NLL at small n the distribution can be heavily right-skewed due to optimization failures, and the paper treats the Gaussian likelihood as a working approximation. If this approximation is substantially misspecified, the posterior ranking probabilities and the Minimum Detectable Difference curve could be","fun_headline_variants_meta":{"raw":{"variants":["ProEval finds AI failures with 8-65x fewer test samples","Gaussian Process surrogate estimates model performance with 1% accuracy","Active input selection exposes diverse AI failures under strict budgets","Pre-trained GPs make generative AI evaluation unbiased and sample-efficient","Bayesian quadrature cuts generative AI evaluation cost by up to 65x"]},"model":"glm-5.2","effort":"high","cost_usd":0.0,"raw_usage":{"total_tokens":637,"prompt_tokens":547,"completion_tokens":90,"prompt_tokens_details":null},"tokens_in":547,"tokens_out":90,"duration_ms":21407,"temperature":1.0,"reasoning_tokens":null,"cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-04T15:58:24.871928+00:00","model_set":{"reader":"glm-5.2"},"falsifier":"The paper's central empirical claim would be falsified if, across a broader set of datasets and methods, a universal sample size threshold were found that reliably separates stable from unstable method rankings, or if the ranking reversals documented here turned out to be artifacts of the specific small neural network architecture used rather than genuine properties of the metrics under data scarcity.","supporting_citations":[],"review_version":2}