REVIEW 3 major objections 5 minor 68 references
Rethinking Likelihood distributions: Student's t Likelihood Boosts Bayesian Neural Network Performance
T0 review · 3 major / 5 minor · reviewed 2026-08-01 · deepseek-v4-flash
Pith's one-line read A location-scale Student's t likelihood with five degrees of freedom is a more robust default than Gaussian in Bayesian neural networks trained by variational inference, across tested noise families and MLP architectures.
desk verdict Solid empirical study with an oversized abstract: Student's t is a robust default under their moment-matched VI scheme, but not unconditionally better as claimed. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The likelihood parameters are set by moment matching from stochastic forward passes: the predictive mean and standard deviation are the sample moments of Monte Carlo weight samples, and the Student's t scale is derived to match that sample variance, with degrees of freedom fixed at five. This turns a heavy-tailed density into a one-line change to the ELBO loss, providing a robust training signal without extra learned parameters. The skew-normal version adds a third-moment shape correction but performs worse, isolating heavy tails as the operative feature.
What would settle it
Re-run the comparison with likelihood parameters optimized directly (for example, maximizing validation log-likelihood for each family) instead of moment-matched, keeping ν=5; if Student's t no longer dominates Gaussian, the paper's conclusion is an artifact of moment matching. Alternatively, on a large dataset with clean Gaussian noise, if Gaussian does not at least tie Student's t when both receive the same estimated scale, the claim that heavy tails help regardless of noise is refuted.
Extended reading notes
Core claim
The central claim is that the likelihood distribution in mean-field variational-inference BNNs need not match the data noise to be optimal; a Student's t with five degrees of freedom is generally a more robust choice than the Gaussian. The evidence comes from controlled experiments: when the noise is Gaussian and the sample is large enough, Student's t outperforms or ties the Gaussian; when the noise is heavy-tailed or skewed, Student's t dominates. Only for extreme lognormal right-skew does the Gaussian give slightly lower MSE, while Student's t still gives better CRPS. The paper's own conclusion softens the abstract's universal wording to 'generally a more robust choice'.
Load-bearing premise
The comparison hinges on the likelihood being parameterized by empirical moments of Monte Carlo weight samples; if that moment-matched parameterization is not a fair or consistent representation of the predictive distribution, the ranking between likelihood families could be an artifact of the estimator rather than the family.
Editorial extensions
If this is right
- Replacing the Gaussian likelihood with Student's t (ν=5) is a drop-in change in VI-trained MLP BNNs that improves or ties MSE and CRPS across tested noise families and architectures.
- Matching the likelihood family to the data noise is not necessary for good performance; tail robustness matters more than exact distributional match.
- Skew-normal likelihood, despite addressing asymmetry, is dominated; asymmetry alone does not drive the gains.
- In some real-world settings, Student's t reduces training epochs or wall-clock time, while predictive intervals stay similarly calibrated.
- The advantage grows with dataset size; with small samples and Gaussian noise, Gaussian remains competitive or better.
Reading between the lines
- The fixed ν=5 was chosen from a sensitivity analysis run on Gaussian noise; letting ν be learned per dataset might strengthen the result, but would weaken the 'no need to know the noise' message.
- Because moment matching fixes the scale from sample variance, the comparison partly conflates distribution family with tail weight; an exact optimization of likelihood parameters could change the ranking, so a direct re-test with optimized scales would isolate the true source of the gain.
- The result likely transfers to other ReLU-based Bayesian architectures, since the paper notes that node activations become heavy-tailed in deep layers; testing convolutional or transformer BNNs would be a natural extension.
- Even in cases where MSE ties or slightly loses, CRPS improves, so practitioners optimizing for probabilistic forecasts — such as energy load forecasting — should prefer Student's t even if point accuracy looks similar.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper studies the choice of likelihood distribution inside variational-inference Bayesian neural networks (VI-BNNs). For regression, it compares Gaussian, skew-normal, and Student's t likelihoods, where the likelihood parameters are computed from empirical moments of Monte Carlo weight samples. Experiments cover univariate (y=x^3+noise), multivariate (20-dimensional input, 5-dimensional output) synthetic data with five noise families, and two real-world regression tasks (POWER, ENTSO-E), across various MLP depths/widths and dataset sizes, with 30 seeds. The paper reports that Student's t (with fixed ν=5) is usually the best or competitive likelihood, especially on CRPS, and concludes that Student's t is a robust default; the abstract makes a stronger universal claim of superiority independent of data distribution and architecture.
Significance. If the finding is valid, the practical recommendation to replace the default Gaussian likelihood in VI-BNNs with a location-scale Student's t (ν=5) is useful and easily actionable. The paper's strengths are the breadth of experiments (30 seeds, several architectures, five noise families, two real datasets) and the availability of source code. The statistical significance testing against the Gaussian baseline adds rigor. However, the universal formulation in the abstract is inconsistent with the paper's own tables, and the moment-matched likelihood parameterization is not the standard ELBO likelihood, so the central claim is currently not established for conventional VI-BNN likelihoods. A reframed or extended version could be a solid empirical contribution.
major comments (3)
- [Abstract] The abstract's claim that Student's t is better 'independent of the data distribution and MLP architecture' is contradicted by the paper's own results. In Table 3, under lognormal noise with σ=3 and n=10,000, Gaussian is the significant best for MSE in all four architectures (G***); at n=50,000 it remains best in the 2×128 and 2×64 cells. In Table 4, Gaussian attains the best MSE with G*** for POWER 8×4 and for ENTSO-E 4×4 and 8×4. Table 2 with n=300 and Gaussian noise also favors Gaussian on MSE. The later text already softens to 'generally more robust'; the abstract and conclusion should be aligned with the actual scope, or the claim needs to be conditioned.
- [§4 vs. §3 Eq. (3.4)] The density actually evaluated is not the ELBO likelihood p(Y|X,W). Eq. (3.4) defines the ELBO with an expectation of log p(Y|X,W), but §4 parameterizes all candidate likelihoods with sample moments \hat{\mu} and \hat{\sigma} computed from T=10 stochastic forward passes (§5.5). The evaluated density is log p(y|\hat{\mu},\hat{\sigma}), not (1/T)\sum_t \log p(y|f_{W_t}(x)). This is a moment-matched approximation to the posterior predictive, not the likelihood appearing in the VI objective. The paper never compares against the standard parameterization in which likelihood parameters are direct network outputs or a learned/fixed noise variance, nor bounds the approximation gap. The observed Student-t advantage could therefore be an artifact of the moment-matched estimator rather than of the likelihood family. This issue must be addressed, either by ablating T, adding standard-parameterizatio
- [§5.7, Table 1] The degrees of freedom ν=5 is selected via a sensitivity analysis performed only on the univariate Gaussian-noise, n=3,000 case (Table 1) and then fixed across all noise families, architectures, and real datasets. The robustness and 'data-independent' wording therefore depend on a hyperparameter tuned on one specific condition. The authors should either report sensitivity of the main conclusions to ν across a broader set of conditions or explicitly state that ν=5 is a selected, fixed hyperparameter and discuss the risk of transfer.
minor comments (5)
- [§4, Eqs. (4.1)–(4.2)] The skew-normal parameterization is ambiguous: δ is introduced only as |δ|, but α is then defined as δ/sqrt(1-δ^2), which requires a signed δ. Specify δ = sign(\hat{\gamma}) |δ| or the equivalent, otherwise the skew-normal implementation is under-specified.
- [Table 3 caption] The caption says rows are grouped by noise type '(Gaussian, Gamma, Student's t)', but the table also contains Lognormal and Laplace rows. Update the caption to list all five noise families.
- [Figures 6 and 7] The figure captions label the model as '32x8' and '32x1', while the surrounding text describes depth 1 width 16 and depth 1 width 32. Please check the labels and keep them consistent.
- [§5.9] Typo: 'POWER ans ENTSO-E' should be 'and'. Please proofread the final text.
- [§5.6] The paper reports significance tests on training epochs and wall-clock time in Table 4. These are computational-cost measures, not predictive quality; it would help to state this explicitly and clarify whether the t-tests on time are adjusted for multiple comparisons or only used descriptively.
Circularity Check
No circularity: the Student-t advantage is an empirical, held-out result; ν=5 is hyperparameter tuning, not a fitted prediction.
full rationale
The paper's central claim is an empirical ranking of likelihood families within a fixed VI-BNN pipeline, not a derivation from first principles. The likelihood parameterization in Section 4 (Eqs. 4.1–4.3) maps sample moments of stochastic forward passes to location/scale parameters of Gaussian, skew-normal, and Student's t; the same moments are used for all three families, so the comparison is not forced by construction. The Student's t advantage is measured on held-out test sets via MSE/CRPS, not on the training fit. The only tuned parameter is ν=5, selected in Table 1 by a sensitivity analysis on one synthetic dataset; this is ordinary hyperparameter selection, not a fitted input later renamed a prediction, and the paper's conclusions are consistent even if ν=7 or 10 (nearly identical CRPS/MSE in Table 1). There is one self-citation ([1] torch-bayesian) but it is a software implementation reference and is not load-bearing. The skeptic's concern that the moment-matched density is a predictive approximation rather than the ELBO likelihood of Eq. (3.4) is a possible construct-validity threat, but it does not make the ranking equivalent to its inputs by definition; it is a methodological caveat outside circularity. No circular steps found.
Assumptions & free parameters
free parameters (2)
- Student's t degrees of freedom ν =
5
- Numerical stability constant ε =
1e-5
assumptions (5)
- domain assumption Mean-field Gaussian variational posterior with Bayes-by-Backprop yields a usable approximation to the BNN weight posterior
- ad hoc to paper The likelihood distribution can be parameterized by empirical moments (μhat, σhat) of Monte Carlo forward passes rather than by learned outputs
- ad hoc to paper Location-scale Student's t with ν=5 and variance-matched τ is a fair representative of heavy-tailed likelihoods
- domain assumption ReLU networks induce heavy-tailed pre-activations, motivating t likelihood
- standard math CRPS and unpaired t-tests across 30 seeds are valid evaluation choices
Cite this review
Pith. "Pith review of Rethinking Likelihood distributions: Student's t Likelihood Boosts Bayesian Neural Network Performance." pith.science (2026). https://pith.science/paper/BVE7K42I
@misc{pith2026260725376,
author = {Pith},
title = {Pith review of: Rethinking Likelihood distributions: Student's t Likelihood Boosts Bayesian Neural Network Performance},
year = {2026},
howpublished = {\url{https://pith.science/paper/BVE7K42I}},
note = {Machine review of arXiv:2607.25376}
}
read the original abstract
In Bayesian neural networks (BNNs), variational inference is a widely adopted framework for modeling uncertainty in a distributional way, with the evidence lower bound (ELBO) serving as the standard objective function. Several distributions contribute to the ELBO loss, such as the prior, approximated posterior, and likelihood distribution. Typically, these distributions are all approximated by a Gaussian distribution, since it is easy to compute, allows for reparameterized gradients, and provides a closed-form loss for training. However, several works have highlighted that this assumption may not generally hold, posing the risk of model misspecification. Alternative distributions have been proposed for the prior specifically, while the effect of distribution choice on the likelihood distribution remains unexplored. In this work, our aim is to close this gap by investigating whether alternative assumptions for the likelihood distribution can outperform the commonly used Gaussian. We compare several likelihood distribution assumptions, such as skewed or heavy-tailed, across regression tasks on both artificial and real-world datasets using standard multilayer perceptrons (MLPs). Our findings demonstrate that Student's t yields better predictive performance than a Gaussian likelihood distribution, independent of the data distribution and MLP architecture (depth and width). In some cases, Student's t can also lead to shorter training times, while still being easy to implement.
Reference graph
Works this paper leans on
-
[1]
doi:10.1137/140951758 , journal =
An Adaptive Shifted Power Method for Computing Generalized Tensor Eigenpairs , author =. doi:10.1137/140951758 , journal =
-
[2]
Nick Higham , title =
-
[3]
Kolda and Ali Pinar , eprint =
Chengbin Peng and Tamara G. Kolda and Ali Pinar , eprint =. Accelerating Community Detection by Using
-
[4]
and Zhang, Shanrong and Merritt, Matthew E
Woessner, Donald E. and Zhang, Shanrong and Merritt, Matthew E. and Sherry, A. Dean , title =. Magnetic Resonance in Medicine , doi =
-
[5]
Properties of Highly Clustered Networks , author =. 2003 , eid =. doi:10.1103/PhysRevE.68.026121 , journal =
-
[6]
Clawpack Software , author =
-
[7]
: A Document Preparation System
Leslie Lamport. : A Document Preparation System. 1986
1986
-
[8]
Frank Mittlebach and Michel Goossens , title =
Show all 68 references
-
[9]
and Van Loan, Charles F
Golub, Gene H. and Van Loan, Charles F. , title =
-
[10]
Paul Dawkins , title =
-
[11]
User's Guide for the
-
[12]
Michael Downes , title =
-
[13]
Christian Feuers\"anger , title =
-
[14]
Neurocomputing , volume=
Variational inference for Student-t MLP models , author=. Neurocomputing , volume=. 2010 , publisher=
2010
-
[15]
Proceedings of the National Academy of Sciences , volume=
Bayesian interpolation with deep linear networks , author=. Proceedings of the National Academy of Sciences , volume=. 2023 , publisher=
2023
-
[16]
arXiv preprint arXiv:1904.02063 , year=
Generalized variational inference: Three arguments for deriving new posteriors , author=. arXiv preprint arXiv:1904.02063 , year=
1904 arXiv
-
[17]
arXiv preprint arXiv:2501.11773 , year=
Can Bayesian Neural Networks Make Confident Predictions? , author=. arXiv preprint arXiv:2501.11773 , year=
-
[18]
Artificial Intelligence Review , volume=
Bayesian learning for neural networks: an algorithmic survey , author=. Artificial Intelligence Review , volume=. 2023 , publisher=
2023
-
[19]
Probabilistic Deep Learning with Generalised Variational Inference , author=
-
[20]
arXiv preprint arXiv:2111.11763 , year=
Uncertainty estimation under model misspecification in neural network regression , author=. arXiv preprint arXiv:2111.11763 , year=
-
[21]
International Conference on Machine Learning , pages=
Bayesian model selection, the marginal likelihood, and generalization , author=. International Conference on Machine Learning , pages=. 2022 , organization=
2022
-
[22]
Journal of the Royal Statistical Society Series B: Statistical Methodology , volume=
Robust generalised Bayesian inference for intractable likelihoods , author=. Journal of the Royal Statistical Society Series B: Statistical Methodology , volume=. 2022 , publisher=
2022
-
[23]
Artificial intelligence and statistics , pages=
Student-t processes as alternatives to Gaussian processes , author=. Artificial intelligence and statistics , pages=. 2014 , organization=
2014
-
[24]
Advances in neural information processing systems , volume=
Simple and scalable predictive uncertainty estimation using deep ensembles , author=. Advances in neural information processing systems , volume=
-
[25]
, author=
Robust Gaussian Process Regression with a Student-t Likelihood. , author=. Journal of Machine Learning Research , volume=
-
[26]
arXiv preprint arXiv:1912.07104 , year=
Robust inference and model criticism using bagged posteriors , author=. arXiv preprint arXiv:1912.07104 , year=
1912 arXiv
-
[27]
Entropy , volume=
The prior can often only be understood in the context of the likelihood , author=. Entropy , volume=. 2017 , publisher=
2017
-
[28]
arXiv preprint arXiv:2102.06571 , year=
Bayesian neural network priors revisited , author=. arXiv preprint arXiv:2102.06571 , year=
-
[29]
International conference on machine learning , pages=
Efficient and scalable bayesian neural nets with rank-1 factors , author=. International conference on machine learning , pages=. 2020 , organization=
2020
-
[30]
Journal of the American statistical Association , volume=
Variational inference: A review for statisticians , author=. Journal of the American statistical Association , volume=. 2017 , publisher=
2017
-
[31]
the Journal of machine Learning research , volume=
Stochastic variational inference , author=. the Journal of machine Learning research , volume=. 2013 , publisher=
2013
-
[32]
Proceedings of the sixth annual conference on Computational learning theory , pages=
Keeping the neural networks simple by minimizing the description length of the weights , author=. Proceedings of the sixth annual conference on Computational learning theory , pages=
-
[33]
International conference on machine learning , pages=
Weight uncertainty in neural network , author=. International conference on machine learning , pages=. 2015 , organization=
2015
-
[34]
International conference on machine learning , pages=
Fast and scalable bayesian deep learning by weight-perturbation in adam , author=. International conference on machine learning , pages=. 2018 , organization=
2018
-
[35]
Advances in neural information processing systems , volume=
Variational dropout and the local reparameterization trick , author=. Advances in neural information processing systems , volume=
-
[36]
The annals of mathematical statistics , volume=
On information and sufficiency , author=. The annals of mathematical statistics , volume=. 1951 , publisher=
1951
-
[37]
Machine learning , volume=
An introduction to variational methods for graphical models , author=. Machine learning , volume=. 1999 , publisher=
1999
-
[38]
International conference on machine learning , pages=
What are Bayesian neural network posteriors really like? , author=. International conference on machine learning , pages=. 2021 , organization=
2021
-
[39]
1992 , institution=
Bayesian training of backpropagation networks by the hybrid Monte Carlo method , author=. 1992 , institution=
1992
-
[40]
Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment , volume=
Bayesian neural networks and density networks , author=. Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment , volume=. 1995 , publisher=
1995
-
[41]
International Conference on Machine Learning , pages=
Understanding priors in Bayesian neural networks at the unit level , author=. International Conference on Machine Learning , pages=. 2019 , organization=
2019
-
[42]
Advances in Neural Information Processing Systems , volume=
Precise characterization of the prior predictive distribution of deep ReLU networks , author=. Advances in Neural Information Processing Systems , volume=
-
[43]
Journal of the American statistical Association , volume=
Strictly proper scoring rules, prediction, and estimation , author=. Journal of the American statistical Association , volume=. 2007 , publisher=
2007
-
[44]
2014 , howpublished =
Tfekci, Pnar and Kaya, Heysem , title =. 2014 , howpublished =
2014
-
[45]
Avalable: https://transparency
ENTSO-E transparency platform , author=. Avalable: https://transparency. entsoe. eu/dashboard/show , year=
-
[46]
Advances in neural information processing systems , volume=
Pytorch: An imperative style, high-performance deep learning library , author=. Advances in neural information processing systems , volume=
-
[47]
International conference on learning representations (ICLR) , volume=
A method for stochastic optimization , author=. International conference on learning representations (ICLR) , volume=. 2015 , organization=
2015
-
[48]
Advances in neural information processing systems , volume=
What uncertainties do we need in bayesian deep learning for computer vision? , author=. Advances in neural information processing systems , volume=
-
[49]
International Journal of Forecasting , volume=
Probabilistic electric load forecasting: A tutorial review , author=. International Journal of Forecasting , volume=. 2016 , publisher=
2016
-
[50]
Bulletin of the Seismological Society of America , volume=
A deep-learning phase picker with calibrated Bayesian-derived uncertainties for earthquakes in the Yellowstone volcanic region , author=. Bulletin of the Seismological Society of America , volume=. 2023 , publisher=
2023
-
[51]
Frontiers in Applied Mathematics and Statistics , volume=
Quantifying uncertainty of machine learning methods for loss given default , author=. Frontiers in Applied Mathematics and Statistics , volume=. 2022 , publisher=
2022
-
[52]
Scandinavian journal of statistics , pages=
A class of distributions which includes the normal ones , author=. Scandinavian journal of statistics , pages=. 1985 , publisher=
1985
-
[53]
2024 , url =
Regulation (EU) 2024/1689 of the European Parliament and of the Council of 13 June 2024 laying down harmonised rules on artificial intelligence (Artificial Intelligence Act) , howpublished =. 2024 , url =
2024
-
[54]
Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2 , pages=
Pytorch 2: Faster machine learning through dynamic python bytecode transformation and graph compilation , author=. Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2 , pages=
-
[55]
Artificial Intelligence and Statistics , pages=
State space methods for efficient inference in Student-t process regression , author=. Artificial Intelligence and Statistics , pages=. 2015 , organization=
2015
-
[56]
Journal of the American Statistical Association , volume=
Robust statistical modeling using the t distribution , author=. Journal of the American Statistical Association , volume=. 1989 , publisher=
1989
-
[57]
, urldate =
Hernández-Lobato, José Miguel and Adams, Ryan P. , urldate =. Probabilistic backpropagation for scalable learning of Bayesian neural networks , series =. Proceedings of the 32nd International Conference on International Conference on Machine Learning - Volume 37 , publisher =....
2015
-
[58]
Journal of Machine Learning Research , volume=
Model selection in Bayesian neural networks via horseshoe priors , author=. Journal of Machine Learning Research , volume=
-
[59]
Artificial intelligence and statistics , pages=
Handling sparsity via the horseshoe , author=. Artificial intelligence and statistics , pages=. 2009 , organization=
2009
-
[60]
IEEE transactions on pattern analysis and machine intelligence , volume=
Advances in variational inference , author=. IEEE transactions on pattern analysis and machine intelligence , volume=. 2018 , publisher=
2018
-
[61]
Learning in graphical models , pages=
A view of the EM algorithm that justifies incremental, sparse, and other variants , author=. Learning in graphical models , pages=. 1998 , publisher=
1998
-
[62]
torch bayesian - Easy Variational Inference , howpublished =
-
[63]
anonymous repository , howpublished =
-
[64]
Dynamic neural regression models , author=
-
[65]
arXiv preprint arXiv:2107.01408 , year=
Scale mixtures of neural network Gaussian processes , author=. arXiv preprint arXiv:2107.01408 , year=
-
[66]
Nature Communications , volume=
Scalable spatiotemporal prediction with Bayesian neural fields , author=. Nature Communications , volume=. 2024 , publisher=
2024
-
[67]
Neural networks , volume=
Bayesian approach for neural networks—review and case studies , author=. Neural networks , volume=. 2001 , publisher=
2001
-
[68]
International Conference on Artificial Intelligence and Statistics , pages=
Variational inference based on robust divergences , author=. International Conference on Artificial Intelligence and Statistics , pages=. 2018 , organization=
2018
Reviewed August 1, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.