Pith. sign in

REVIEW 5 major objections 9 minor 18 references

ProEval: Proactive Failure Discovery and Efficient Performance Estimation for Generative AI Evaluation

T0 review · 5 major / 9 minor · reviewed 2026-07-04 · glm-5.2

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Rankings of Bayesian deep learning methods are unreliable when training data is scarce

desk verdict The paper shows that BDL method rankings are unstable under training-data scarcity, and the core finding is real. The stress-test concern about subsample overlap is technically valid but does not change the qualitative conclusion. read the letter →

arxiv 2604.23099 v2 pith:YEBBADE6 submitted 2026-04-25 cs.LG cs.AIstat.ML

Yizheng Huang , Wenjun Zeng , Aditi Kumaresan , Zi Wang This is my paper · ORCID
classification cs.LGcs.AIstat.ML
keywords Bayesiandeeplearninguncertaintyquantificationevaluationreliabilitysmall-samplestatisticshierarchicalmodelmethodrankingstabilityCRPSminimumdetectabledifference
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the standard practice of ranking Bayesian deep learning methods by point-estimate metrics breaks down when training data is limited. The authors show empirically that when training sets contain only tens to a few hundred samples, the variance of evaluation metrics like CRPS and NLL is large enough that method rankings reflect sampling noise as much as true performance differences. They demonstrate that the same pairwise comparison between methods can yield a posterior probability of superiority of 1.000 on one dataset while remaining below 0.95 even at 500 training samples on another, and that ranking reversals occur as training size increases in ways invisible to point estimates. To address this, the authors propose a Bayesian hierarchical model that treats evaluation metrics as random variables across independent training set draws, with method-specific variances, enabling posterior inference over pairwise method comparisons. They also introduce a predictive Minimum Detectable Difference curve that quantifies the smallest performance gap reliably detectable at a given training size. The central claim is that no universal sample size threshold separates reliable from unreliable method comparisons across datasets, making dataset-specific posterior inference necessary.

What carries the argument

The paper's central machinery is a Bayesian hierarchical model (Equation 1) in which the metric value for method i on data realization r is modeled as m_ir = μ_i + α_i + γ_r + ε_ir, where μ_i is the latent true metric value, α_i captures within-method variance, γ_r is a realization-specific random intercept for paired structure, and ε_ir is residual noise. Method-specific variances are essential because methods differ substantially in metric variability at small n. Posterior ordering (Equation 3) computes the probability that method i outperforms method j by integrating over the joint posterior. The predictive Minimum Detectable Difference (Equation 4) gives the smallest true gap detectable

What would settle it

The paper's central empirical claim would be falsified if, across a broader set of datasets and methods, a universal sample size threshold were found that reliably separates stable from unstable method rankings, or if the ranking reversals documented here turned out to be artifacts of the specific small neural network architecture used rather than genuine properties of the metrics under data scarcity.

Watch

Extended reading notes

Core claim

The paper's central discovery is that method rankings in Bayesian deep learning are not stable under data scarcity, and that this instability is dataset-dependent in ways that point-estimate comparisons cannot reveal. The authors demonstrate this through a systematic study of six methods across five regression datasets at five training set sizes, finding that the posterior probability that Monte Carlo Dropout outperforms Deep Ensembles on CRPS is 1.000 at n=50 on one dataset yet remains below 0.95 even at n=500 on another. They attribute this to the compounding of data variance (from finite training samples) with algorithm variance (from training randomness), which standard evaluation treats

Load-bearing premise

The Bayesian hierarchical model assumes that metric values across independent training set draws are exchangeable and approximately Gaussian. For CRPS this holds well, but for NLL at small n the distribution can be heavily right-skewed due to optimization failures, and the paper treats the Gaussian likelihood as a working approximation. If this approximation is substantially misspecified, the posterior ranking probabilities and the Minimum Detectable Difference curve could be

Editorial extensions

If this is right

  • Practitioners in low-data domains (drug discovery, medical imaging, environmental monitoring) cannot assume that benchmark rankings from large-scale studies transfer to their setting; they need dataset-specific posterior inference to determine whether their evaluation data supports any method-superiority claim.
  • Point-estimate leaderboards should be supplemented with uncertainty-aware comparison statistics and detectability diagnostics before drawing conclusions about method superiority, particularly when training data is limited.
  • The finding that CRPS is substantially more stable than NLL at small training sizes suggests that metric choice affects not only accuracy but statistical power for method comparison, with practical consequences for experimental design.
  • The compounding of data variance and algorithm variance implies that methods with higher training instability require disproportionately more data to achieve reliable metric estimates, a factor that current benchmarking practice does not account for.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the Gaussian working likelihood is substantially misspecified for heavy-tailed metrics like NLL at small n, the posterior probabilities in Equation 3 could be biased in unknown directions. A robust likelihood or nonparametric model might yield different ranking conclusions for NLL-based comparisons, particularly below n=100.
  • The exchangeability assumption across data realizations may break if training sets are drawn from non-stationary distributions, which is common in time-series or streaming data settings; the framework's conclusions may not extend to such scenarios without modification.
  • The finding that no universal sample size threshold exists suggests that a meta-analytic approach pooling posterior evidence across multiple datasets could provide more reliable method rankings than any single-dataset evaluation, analogous to clinical trial meta-analysis.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 9 minor

Summary. This manuscript investigates whether standard evaluation metrics for Bayesian deep learning (BDL) methods are reliable when training data is scarce. The authors draw R=50 independent subsamples of size n from a fixed pool for each of five regression datasets, train six BDL methods from scratch on each subsample, and record CRPS, NLL, PICP, and MPIW. They fit a Bayesian hierarchical model (BHM) with method-specific variances to these metric observations, compute posterior probabilities of pairwise method superiority (Eq. 3), and define a predictive Minimum Detectable Difference (MDD) curve (Eq. 4) to assess whether an observed gap would be detectable in a new experiment. The central empirical finding is that method rankings are unstable and dataset-dependent at small n: the same comparison (MCD vs. Deep Ensembles on CRPS) yields P=1.000 at n=50 on one dataset but remains below 0.95 even at n=500 on another, and no universal sample-size threshold separates reliable from unreliable comparisons. The paper is clearly written and addresses a genuine gap in the BDL evaluation literature.

Significance. The paper makes a timely and practical contribution by formalizing the problem of metric unreliability under training-data scarcity, a regime that existing large-scale benchmark studies (which assume thousands of training samples) do not address. The BHM framework with method-specific variances, the posterior superiority probability, and the predictive MDD curve are useful, concrete tools for practitioners. The experimental design—50 realizations per cell, MCMC convergence diagnostics (R-hat < 1.01, ESS > 400), posterior predictive checks, and a stability check (R=30 vs. R=50)—is thorough by the standards of the subfield. The power-law variance decomposition (Table 7, Figure 7) and the cross-dataset replication (Table 2) strengthen the central claim. The falsifiable prediction that no universal n threshold exists is well-supported by the data presented.

major comments (5)
  1. Section 3.1, Eq. (1): The BHM treats the R=50 metric observations per cell as conditionally independent given the latent parameters. However, because subsamples are drawn without replacement from a finite pool, the training sets for different realizations overlap substantially at large n. For Concrete (pool=721, n=500), the expected pairwise overlap is approximately 500*500/721 ≈ 69%; for Yacht (pool=215, n=100), it is approximately 47%. This overlap induces positive correlation between metric values across realizations, because the trained models share a large fraction of training data. The BHM's independence assumption then underestimates the effective sample size, potentially producing posterior probabilities that are more extreme (closer to 0 or 1) than warranted. This directly affects the headline numbers: P(MCD≺Ensemble)=0.95 on Concrete at n=500 (Table 2), which is the weakest and
  2. Section 3.1, Eq. (1): The BHM treats the R=50 metric observations per cell as conditionally independent given the latent parameters. However, because subsamples are drawn without replacement from a finite pool, the training sets for different realizations overlap substantially at large n. For Concrete (pool=721, n=500), the expected pairwise overlap is approximately 69%; for Yacht (pool=215, n=100), approximately 47%. This overlap induces positive correlation between metric values across realizations, violating the independence assumption and potentially inflating the extremity of posterior probabilities. The paper acknowledges exchangeability may fail for 'non-stationary distributions' (Section 6) but does not address finite-pool subsample overlap. The stability check in Appendix B (R=30 vs. R=50, changes <0.02) does not address this because both subsets share the same overlap structure
  3. Section 5.1, Table 1 and Appendix A.5, Table 4: SWAG runs with convergence rates below 0.80 are excluded from the BHM fits (Appendix B). This is a post-hoc data exclusion that affects the SWAG comparisons. At n=30 on the synthetic dataset, the convergence rate is 0.58 (Table 4), meaning 42% of runs are excluded. The paper states this benchmarks 'the uncertainty quantification quality of successful SWAG fits rather than its convergence reliability,' but the excluded runs are not missing at random—they represent a systematic failure mode of SWAG at small n. The paper should explicitly discuss how this exclusion affects the generalizability of the SWAG comparisons and whether the same exclusion criterion was pre-registered or chosen after inspecting the data.
  4. Section 3.1, Eq. (1) and Appendix B: The Gaussian working likelihood is acknowledged to be inadequate for NLL at small n due to heavy right skew from sigma collapse (Appendix B, Figure 5). The paper states that 'conclusions based on NLL below n=100 should be interpreted with caution,' but NLL results are still reported in the main text (Table 8) and used in the power-law analysis (Figure 1, right panel). If the Gaussian likelihood is substantially misspecified for NLL at small n, the posterior probabilities and MDD derived from the BHM for NLL could be biased. The paper should either (a) restrict all NLL-based posterior claims to n≥100 where the approximation is adequate, or (b) provide a sensitivity analysis using a heavy-tailed likelihood (e.g., Student-t) to show that the ranking conclusions are robust to the likelihood choice.
  5. Section 3.3, Eq. (4): The MDD formula depends on σ_pred, which combines posterior uncertainty in the mean difference with method-specific variability. However, σ_pred is derived under the same independence and Gaussian assumptions as the BHM. If the within-method variance s_m² is underestimated due to subsample overlap (Major Comment 1), the MDD curve will be optimistically narrow, understating the sample size required for reliable detection. The paper should discuss this dependency and, ideally, provide a corrected MDD that accounts for finite-pool effects, or at minimum note that the MDD values are lower bounds on the true detectable difference.
minor comments (9)
  1. Abstract: The abstract mentions 'P(MCD≺Ensemble)=1.000 at n=50 on one dataset and remains below 0.95 even at n=500 on another,' but does not name the datasets. Adding 'on the synthetic dataset' and 'on Concrete' would improve clarity.
  2. Table 2: The Energy dataset row shows '—' for n=500, but the reason (pool size 538 is too small for n=500) is only explained in Table 3. A footnote on Table 2 would help.
  3. Figure 2 caption: The caption refers to 'three representative datasets' but the figure shows only two lines (Synthetic and Energy). Please clarify or correct.
  4. Section 5.4: The text states 'Bayes By Backprop outperforms Monte Carlo Dropout on the Concrete dataset at n=200,' but Table 11 does not include a BBB vs. MCD comparison for Concrete. Please add this comparison or clarify the source of the claim.
  5. Appendix B, Table 6: The R-hat table reports values for CRPS, PICP, and Interval Score, but not for NLL. Please add NLL convergence diagnostics.
  6. Section 3.2: The Beta-Binomial model for PICP (Eq. 2) uses a logit link for p_k, but the precision parameter φ is not discussed. Please state the prior on φ and whether it was sensitivity-checked.
  7. Table 1: SWAG CRPS values at n=30 and n=50 (1.399 and 1.287) are orders of magnitude larger than other methods. A note explaining that these reflect only converged runs (as stated in the caption) but are still highly variable would help readers contextualize these numbers.
  8. References: Several arXiv preprints are cited without DOIs (e.g., Dehghani et al. 2021, Longjohn et al. 2025). Please add DOIs where available.
  9. Section 6: The limitation about exchangeability mentions 'non-stationary distributions' but does not mention the finite-pool overlap issue (see Major Comment 1). This limitation should be acknowledged explicitly.

Simulated Author's Rebuttal

4 responses · 0 unresolved

We thank the referee for a careful and constructive report. The comments identify two genuine modeling limitations (finite-pool subsample overlap and NLL likelihood misspecification) and two presentation gaps (SWAG exclusion criteria and MDD dependency on BHM assumptions). We address each below and indicate where revisions will be made.

read point-by-point responses

  1. Referee: Section 3.1, Eq. (1): Subsample overlap violates conditional independence, potentially inflating posterior extremity. The BHM treats R=50 observations as independent given latent parameters, but without-replacement sampling from a finite pool induces positive correlation at large n (e.g., ~69% pairwise overlap for Concrete at n=500).

    Authors: We agree this is a legitimate concern that the manuscript does not currently address. We note two points in partial mitigation, followed by the revision we will make. First, the BHM already includes a realization-specific random intercept b_r (Eq. 1) that captures the paired structure across methods within each realization—i.e., the fact that all methods share the same training subsample in realization r. This term accounts for within-realization correlation but does not address between-realization overlap, which is the referee's point. Second, the overlap problem is most severe at large n, where our qualitative finding is that rankings stabilize. At small n—where the main findings of ranking instability and reversal are concentrated—the overlap is negligible (e.g., n=30 from a pool of 840 gives ~3.6% expected pairwise overlap). The specific result most affected is the borderline P(MCD≺Ensemble)=0.95 on Concrete at n=500, which is exactly where overlap is largest (~69%) and the posterior is closest to the decision threshold. We acknowledge that the independence assumption likely makes this posterior more extreme than warranted, and the corrected value could fall below 0.95. We will revise the manuscript to: (1) explicitly discuss finite-pool overlap as a source of between-realization correlation, (2) note that posterior probabilities at large n (especially on smaller-pool datasets like Concrete and Yacht) may be more extreme than warranted, (3) add a sensitivity analysis using a bootstrap estimate of the effective sample size correction (design-effect adjustment) for the Concrete n=500 case to assess robustness of the borderline result, and (4) note that this issue strengthens, rather than weakens, our central claim that practitioners should treat borderline posterior概率 revision: no

  2. Referee: Section 5.1, Table 1 and Appendix A.5, Table 4: SWAG runs with convergence rates below 0.80 are excluded from BHM fits. This is post-hoc data exclusion affecting SWAG comparisons, and the excluded runs represent a systematic failure mode at small n.

    Authors: We agree that the exclusion criterion and its implications should be discussed more explicitly. To clarify the nature of the exclusion: the criterion is based on SWAG's optimization convergence (NaN loss or insufficient weight snapshots), not on the resulting metric values. Failed runs produce no meaningful posterior approximation, so including them would conflate SWAG's optimization reliability with its uncertainty quantification quality—two distinct properties. However, the referee is correct that this exclusion is post-hoc (chosen after observing convergence rates) and that the excluded runs represent a systematic failure mode of SWAG at small n, not random missingness. This means our SWAG results represent a best-case scenario: the UQ quality of SWAG conditional on successful convergence. We will revise the manuscript to: (1) state explicitly that the exclusion criterion was chosen after inspecting convergence rates and was not pre-registered, (2) add a sentence noting that the SWAG comparisons therefore characterize successful SWAG fits and should not be interpreted as statements about SWAG's overall reliability at small n, and (3) report SWAG convergence rates alongside all SWAG posterior probability entries in the results tables (currently only in Appendix A.5) so readers can assess the fraction of excluded runs for each cell. We note that the main findings of the paper do not depend on SWAG comparisons—the ranking instability and reversal findings are driven by MCD, Deep Ensembles, BBB, and MAP, which do not have convergence failures. revision: yes

  3. Referee: Section 3.1 and Appendix B: Gaussian working likelihood is inadequate for NLL at small n due to heavy right skew from sigma collapse. NLL results are still reported in main text and used in power-law analysis. Posterior probabilities and MDD for NLL could be biased.

    Authors: We agree and will adopt option (a) proposed by the referee: restrict all NLL-based posterior claims to n≥100, where the Gaussian approximation is adequate as confirmed by posterior predictive checks (Appendix B, Figure 5). Specifically, we will: (1) add an explicit caveat to Table 8 and the NLL panel of Figure 1 stating that NLL-based posterior probabilities and variance estimates below n=100 are unreliable due to likelihood misspecification, (2) remove or clearly flag any NLL-based posterior probability claims below n=100 in the main text, and (3) note in the power-law analysis (Figure 1, right panel) that the NLL variance estimates at n=30 and n=50 are affected by the Gaussian likelihood's failure to capture the heavy right tail. We considered option (b) (Student-t sensitivity analysis) but note that the NLL results are not central to the paper's main claims—CRPS is the primary metric throughout, and the ranking instability and reversal findings are based on CRPS. The NLL analysis serves mainly to motivate the choice of CRPS as the primary metric, and the qualitative conclusion (CRPS is more stable than NLL at small n) holds regardless of the likelihood specification because it is based on empirical standard deviations, not posterior probabilities. revision: yes

  4. Referee: Section 3.3, Eq. (4): MDD formula depends on σ_pred, which inherits the same independence and Gaussian assumptions as the BHM. If within-method variance s_m² is underestimated due to subsample overlap, the MDD curve is optimistically narrow.

    Authors: The referee is correct that the MDD inherits the BHM's assumptions. We will address this in two ways. First, we will add an explicit statement that the MDD values are derived under the same independence and Gaussian assumptions as the BHM, and that in the presence of finite-pool overlap, the reported MDD values should be interpreted as lower bounds on the true minimum detectable difference (i.e., the true sample size required for reliable detection is at least as large as reported). Second, we note that the MDD analysis in the main text (Figure 4) is presented for the synthetic dataset at n=50, where the pool size is 840 and the expected pairwise overlap is approximately 50*50/840 ≈ 3%, so the overlap effect is minimal for this specific result. The general MDD scaling claim (MDD ∝ n^{-α/2}) is an empirical description of variance reduction rates and does not depend on the independence assumption—it is computed from empirical standard deviations across realizations. We will add a note clarifying which aspects of the MDD analysis are affected by the overlap assumption (the absolute MDD values at large n on small-pool datasets) and which are not (the scaling relationship and the small-n results on large-pool datasets). revision: yes

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the BHM is fit to independently generated data, and posterior probabilities and MDD are derived from the model, not from fitted constants renamed as predictions.

full rationale

The paper's central claims are derived from a Bayesian hierarchical model (Eq. 1) fit to 50 independently drawn training subsamples per cell. The posterior probabilities (Eq. 3) and the MDD formula (Eq. 4) are outputs of this model, not fitted constants renamed as predictions. The power-law exponents (Table 7) are explicitly described as 'descriptive summary only' and are not inputs to the main ranking or detectability claims. The MDD formula is derived from the posterior predictive distribution under a Gaussian approximation, with σ_pred depending on within-method variance s_m² estimated from the data — this is a standard statistical derivation, not a tautology. The BHM's shrinkage estimator (Eq. A.3) is a standard Bayesian result, not a self-citation chain. The paper cites external work (Spiegelhalter and Freedman 1986 for predictive sample size; Gneiting and Raftery 2007 for proper scoring rules; Cohen 1988 for the 0.8 power threshold) but these are independent references, not self-citations that would create circularity. The independence assumption across realizations is a modeling assumption whose validity is a correctness concern (addressed by the skeptic), not a circularity issue — the model's outputs are genuinely derived from its inputs rather than being equivalent to them by construction. No step in the derivation chain reduces to its own inputs by definition or by self-citation.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The paper does not introduce new physical entities, particles, or forces. The hierarchical model and MDD curve are methodological constructs, not postulated objects. The free parameters are estimated from data via MCMC, not hand-tuned. The power-law exponents are descriptive and do not feed into the central claims.

free parameters (4)
  • Method-specific variance sigma_k^2
    Estimated from data via MCMC in the hierarchical model (Eq. 1). Not a hand-tuned constant but a data-driven parameter.
  • Between-method variance tau
    Estimated from data via MCMC. Controls shrinkage toward the population mean.
  • Power-law exponent alpha = 0.49 to 0.83
    Fitted by OLS on log Var vs log n. Used only as a descriptive summary, not as input to the main claims.
  • z_alpha quantile (alpha=0.80) = 0.842
    Standard statistical constant for 80% power, following Cohen (1988). Not a free parameter in the paper-specific sense.
assumptions (4)
  • domain assumption Metric values across independent training set draws are exchangeable at fixed n.
    Invoked implicitly in the hierarchical model (Eq. 1) where metric values are modeled as draws from a common distribution. May be violated under non-stationary data distributions, as noted in the limitations.
  • domain assumption CRPS values are approximately Gaussian across realizations at fixed n.
    Stated in Section 3.1 and verified via posterior predictive checks in Appendix B. Used to justify the Gaussian likelihood in Eq. 1.
  • ad hoc to paper NLL values can be approximated by a Gaussian working likelihood at small n.
    Acknowledged in Appendix B as a working approximation despite heavy right skew due to sigma collapse. Sensitivity results are reported but the approximation is not fully validated.
  • domain assumption The Gaussian approximation to the posterior is adequate for the MDD formula.
    Used in Eq. 4 and Appendix A.1 to derive the closed-form MDD. Verified via posterior predictive checks for CRPS but not for all metrics and training sizes.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ProEval: Proactive Failure Discovery and Efficient Performance Estimation for Generative AI Evaluation." pith.science (2026). https://pith.science/paper/YEBBADE6

@misc{pith2026260423099,
  author       = {Pith},
  title        = {Pith review of: ProEval: Proactive Failure Discovery and Efficient Performance Estimation for Generative AI Evaluation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YEBBADE6}},
  note         = {Machine review of arXiv:2604.23099}
}
read the original abstract

Evaluating generative AI models is increasingly resource-intensive due to slow inference, expensive raters, and a rapidly growing landscape of models and benchmarks. We propose ProEval, a proactive evaluation framework that leverages transfer learning to efficiently estimate performance and identify failure cases. ProEval employs pre-trained Gaussian Processes (GPs) as surrogates for the performance score function, mapping model inputs to metrics such as the severity of errors or safety violations. By framing performance estimation as Bayesian quadrature (BQ) and failure discovery as superlevel set sampling, we develop uncertainty-aware decision strategies that actively select or synthesize highly informative inputs for testing. Theoretically, we prove that our pre-trained GP-based BQ estimator is unbiased and bounded. Empirically, extensive experiments on reasoning, safety alignment, and classification benchmarks demonstrate that ProEval is significantly more efficient than competitive baselines. It requires 8-65x fewer samples to achieve estimates within 1% of the ground truth, while simultaneously revealing more diverse failure cases under a stricter evaluation budget.

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

18 extracted references · 18 canonical work pages

  1. [1]

    Unstable Rankings in Bayesian Deep Learning Evaluation Qishi Zhan ∗ Department of Mathematical and Statistical Sciences Marquette University, USA qishi.zhan@marquette.edu Minxuan Hu Cornell University, USA Guansu Wang The University of Melbourne, Australia Jiaxin Liu University of Illinois Urbana-Champaign, USA Liang He Tongji University, China April 28, ...

    work page Pith review arXiv 2026

  2. [2]

    comprises several thousand samples. This assumption fails silently in drug discovery, rare disease diagnosis, medical imaging, and environmental monitoring, where labeled data is scarce by nature and Bayesian or uncertainty-aware models are often deployed in specialized settings [Svensson et al., 2025, Bargagna et al., 2023, Ferchichi et al., 2025, Yu and...

    work page 2025

  3. [3]

    extended this line of work across diverse archi- tectures and tasks from the WILDS benchmark, introducing signed calibration metrics to distinguish overconfidence from underconfidence. A feature shared by all these studies is that evaluation is conducted on datasets with abundant training samples, typically thousands to tens of thousands of examples, and ...

    work page 2007

  4. [4]

    Our work differs from both in a way that matters for practice

    proposed bootstrapping and Bayesian hierarchical models to quantify uncertainty in aggregate performance metrics for foundation model benchmarks, focusing on accuracy-based metrics in large-data settings. Our work differs from both in a way that matters for practice. We study the scarcity of training data rather than test data or training randomness alone...

    work page 2016

  5. [5]

    Angelopoulos and Stephen Bates

    ISSN 1935-8237. doi: 10.1561/2200000101. URL������������������������ ����������. Filippo Bargagna, Lisa Anita De Santi, Nicola Martini, Dario Genovesi, Brunella Favilli, Giuseppe Vergaro, Michele Emdin, Assuero Giorgetti, Vincenzo Posi- tano, and Maria Filomena Santarelli. Bayesian convolutional neural networks 13 in medical imaging classification: A prom...

    work page doi:10.1561/2200000101 1935

  6. [6]

    doi: 10.1007/s10278-023-00897-8

    ISSN 1618-727X. doi: 10.1007/s10278-023-00897-8. URL ������������������������������������������. Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural networks. In����������� �� ��� ���� ������������� ���������� �� ������� �������� � ������ ��, ICML’15, page 1613–1622. JMLR.org,

    work page doi:10.1007/s10278-023-00897-8

  7. [7]

    doi: https://doi.org/10.1016/ j.neucom.2025.130242

    ISSN 0925-2312. doi: https://doi.org/10.1016/ j.neucom.2025.130242. URL �������������������������������������� �����������������������������. Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: representing model uncertainty in deep learning. In����������� �� ��� ���� ������������� ���������� �� ������������� ���������� �� ������� ������...

    work page arXiv 2025

  8. [8]

    Gelman, D

    doi: 10.1214/ss/1177011136. URL�������������������������������������. J. Gerritsma, R. Onnink, and A. Versluis. Geometry, resistance and stability of the delft systematic yacht hull series.������������� ������������ ��������, 28 (328):276–297,

    work page doi:10.1214/ss/1177011136

Show all 18 references
  1. [9]

    URL���������������� ��������������������������

    doi: 10.1198/016214506000001437. URL���������������� ��������������������������. D. A. Griffiths. Maximum likelihood estimation for the beta-binomial distribution and an application to the household distribution of the total number of cases of a disease.����������, 29(4):637–648,

    doi

  2. [10]

    URL������ ��������������������������������������������������

    doi: https://doi.org/10.1002/sim.8124. URL������ ��������������������������������������������������. 15 Bálint Mucsányi, Michael Kirchhof, and Seong Joon Oh. Benchmarking uncertainty disentanglement: Specialized uncertainties for specialized tasks. In�������� �� ������ �������...

    doi

  3. [11]

    URL ����������������������������������������������������������� ��������������������������������������������������������������� ���������

    doi: 10.52202/079017-1614. URL ����������������������������������������������������������� ��������������������������������������������������������������� ���������. Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, D. Sculley, Se- bastian Nowozin, Joshua Dillon, Balaji Laksh...

    doi

  4. [12]

    Emma Svensson, Hannah Rosa Friesacher, Susanne Winiwarter, Lewis Mervin, Adam Arany, and Ola Engkvist

    doi: 10.1002/sim.4780050103. Emma Svensson, Hannah Rosa Friesacher, Susanne Winiwarter, Lewis Mervin, Adam Arany, and Ola Engkvist. Enhancing uncertainty quantification in drug discovery with censored regression labels.��������� ������������ �� ��� ���� ��������, 7:100128,

    doi

  5. [13]

    doi: https://doi.org/10

    ISSN 2667-3185. doi: https://doi.org/10. 1016/j.ailsci.2025.100128. URL �������������������������������������� �����������������������������. Athanasios Tsanas and Angeliki Xifara. Accurate quantitative estimation of energy performance of residential buildings using statistica...

    2025

  6. [14]

    doi: https: //doi.org/10.1016/j.enbuild.2012.03.003

    ISSN 0378-7788. doi: https: //doi.org/10.1016/j.enbuild.2012.03.003. URL �������������������������� �����������������������������������������. Vladimir Vovk, Alexander Gammerman, and Glenn Shafer.����������� �������� �� � ������ �����. Springer US,

    2012 doi

  7. [15]

    doi: 10.1007/b106715. I.-C. Yeh. Modeling of strength of high-performance concrete using artificial neuralnetworks.������ ��� �������� ��������, 28(12):1797–1808,

    doi

  8. [16]

    doi: https://doi.org/10.1016/S0008-8846(98)00165-3

    ISSN 0008-8846. doi: https://doi.org/10.1016/S0008-8846(98)00165-3. URL ������ �������������������������������������������������������������. Cheng-Han Yu and Shuaizhou Wang. A comparative study of bayesian neural networks and machine learning based on covid-19 image classific...

    doi

  9. [17]

    � ������������ ������� ��� ��������� ������������ ����� ����� ��������������� Conditioning on��� � and �, the posterior distribution of� � given observations ���� � � � � � ���� is Gaussian with mean and variance: �� � � � � �� � � �� � �� (A.1) ��� �� � � ���� � �� � � � �� �...

    2015

  10. [18]

    sigma collapse; unclipped MAP NLL exceeds��� at all training sizes. Deep Ensemble NLL is evaluated on the mixture predictive distribution and does not decrease monotonically with�, a structural property of mixture-of-Gaussians representations documented in Lakshminarayanan et ...

    2017

Pith tools

Reviewed July 4, 2026 · model on record in the stance chip above.