REVIEW 4 major objections 5 minor 28 references
Analog Bayesian neural networks are insensitive to the shape of the weight distribution
T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read Bayesian neural networks are insensitive to the shape of the weight distribution: with matching weight means and variances, predictive distributions converge regardless of variational form.
desk verdict Plausible shape-insensitivity result, honestly bounded, but the evidence is thinner than the title suggests. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the per-layer central limit theorem for pre-activation sums in a feed-forward network. Because each neuron computes a sum over many independent weights, the argument runs, the distribution of that sum converges to a Gaussian with a mean and variance fixed by the weight means and variances alone; higher cumulants of the weight distribution wash out. Supporting machinery is a numerical toolkit for working with non-Gaussian variational families: a maximum-likelihood fit to measured device noise, custom Gaussian quadrature rules for expectations and KL divergences with that distribution, and a corrected inverse-transform sampler to generate device-noise samples.
What would settle it
Measure predictive distributions in the Section 4.1 energy-distance setup at fixed large width and growing depth, using a weight distribution with very high skewness or kurtosis; if the divergence from the Gaussian-weight predictive distribution does not shrink toward zero as width grows, or is visible in calibration curves at production widths, the per-layer CLT explanation is wrong.
Extended reading notes
Core claim
The central discovery is an empirical invariance: for deep mean-field Bayesian neural networks, the posterior predictive distribution depends on the first two moments of each variational weight distribution and is effectively independent of its higher shape. The authors establish this by training with Gaussian distributions, then replacing the Gaussian base distribution by a fitted real-device distribution and by a bimodal mixture at fixed $\mu$ and $\sigma$, and observing that regression fits, calibration curves, and predictive densities coincide as layer width grows. They attribute the invariance to the CLT within each layer: each pre-activation is a sum of many independent weighted inputs, so its distribution is determined by the mean and variance of the summed weights regardless of the individual weight-distribution shape.
Load-bearing premise
The load-bearing premise is that in each layer the sum over many independent weights becomes Gaussian fast enough that only the mean and variance of the weight noise matter, and that this already holds at the network widths used in practice.
Editorial extensions
If this is right
- Analog hardware designers can use memory devices whose noise has the right mean and variance but any shape; the predictive uncertainty of the deployed BNN will match the software-trained Gaussian BNN.
- A network can be trained in software with Gaussian variational weights, and its scale and shift parameters transferred directly to hardware sampling device noise.
- The invariance is quantitative in width: narrow layers show measurable shape dependence, but the divergence to the Gaussian predictive distribution falls quickly as width increases.
- A second device with qualitatively different noise shape, an ECRAM device, shows the same convergence in the energy-distance test, suggesting the result is not specific to one device.
Reading between the lines
- If the per-layer CLT explanation is right, the required width should scale with the tail-heaviness or skewness of the weight distribution; one testable prediction is that heavier-tailed device noise requires wider layers to reach a given divergence tolerance.
- The invariance suggests a practical calibration shortcut: hardware uncertainty metrics, such as expected calibration error, could be certified in software with Gaussian weights and then carried over to analog hardware without re-measuring the device-specific noise shape.
- The same argument should apply to other linear-in-weight architectures, such as convolutional layers, but may fail for attention blocks or normalization layers where pre-activations are not simple sums of independent weights; the paper's convolutional result hints at this, but the scope is untested.
Formalized claims in Lean
-
Claim #1: Predictive distributions from BNNs with the same weight means and variances converge to the same distribution regardless of variational shape.
/-- @claim 1 Predictive distributions from BNNs with the same weight means and variances converge to the same distribution regardless of variational shape. -/ noncomputable def shape_invariance_of_predictive_distributions : Prop :=
-
Claim #2: Analog device designers do not need to control the shape of device noise; only its mean and variance matter.
/-- @claim 2 Analog device designers do not need to control the shape of device noise; only its mean and variance matter. -/ noncomputable def analog_design_shape_irrelevance : Prop :=
-
Claim #3: Real-device-noise variational distributions can replace Gaussian variational distributions in MFVI when their means and variances match.
/-- @claim 3 Real-device-noise variational distributions can replace Gaussian variational distributions in MFVI when their means and variances match. -/ noncomputable def non_gaussian_mfvi_feasibility : Prop :=
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript proposes a mean-field variational inference (MFVI) framework in which the variational distribution is a fitted model of real analog device noise (Bayes-MTJ, with ECRAM in Appendix D) rather than a Gaussian. Its central claim is that the posterior predictive distributions of BNNs with matching weight means and variances converge to the same distribution regardless of the shape of the variational weight distribution. This is supported by comparing Gaussian, fitted-device, and bimodal weight distributions on an energy-distance task (Section 4.1), a 1D regression task (Section 4.2), and the UTKFACE benchmark (Section 4.3), with numerical tools for quadrature and inverse sampling developed in Section 3.
Significance. If the shape-invariance claim held as stated, the result would be practically significant: analog hardware designers could ignore the shape of device noise and match only the mean and variance of the variational distribution, enabling a train-in-software, deploy-in-hardware workflow. The paper also contributes useful numerical machinery, including custom quadratures for non-Gaussian device distributions and a corrected inverse-CDF sampler. However, the significance is currently bounded by the fact that the core claim is an empirical finite-width trend backed by an unquantified CLT heuristic, while the paper's own Section 5 restricts the result to sufficiently wide networks and defers narrow deep networks to future work.
major comments (4)
- [Section 4.1, Fig. 4] The headline energy-distance experiment tests only one input location, x=0, and reports KL divergences without repeated-seed or Monte Carlo error bars, so it is unclear whether the apparent convergence is real or an artifact of estimator noise. More importantly, Fig. 4 shows visible shape dependence at width=1, and Section 5 concedes that "sufficiently wide neural network layers are still necessary" and that narrow deep networks are future work. This directly contradicts the unqualified title and abstract claim; the evidence supports at most an asymptotic wide-network statement. The authors should either provide quantitative convergence results with uncertainties (repeated seeds, error bars, a stated width threshold) or restrict the claim accordingly.
- [Section 1, final paragraph; Section 5] The mechanism invoked for shape-insensitivity is the central limit theorem "within each layer," but no theorem or quantitative rate is given. In deep networks with ELU or ReLU activations, layer outputs are not independent sums of weights; nonlinearities can regenerate non-Gaussianity, so higher weight moments could matter at practical widths and depths. Because this CLT heuristic is the only mechanistic support for the invariance claim, the manuscript needs either a rigorous finite-width/depth bound or a much more comprehensive empirical stress test than the few configurations shown in Fig. 4.
- [Section 4.2 and Section 4.3, Figs. 5 and 6] The scalar regression and UTKFACE comparisons are qualitative. There are no numerical discrepancies between predictive distributions (e.g., Wasserstein or energy distance with confidence intervals), no repeated seeds, and no baseline showing what a meaningful difference would look like. The statement in Section 4.3 that predictions are "identical" is not supported by calibration-only evidence, because calibration curves can match while predictive densities differ substantially. Quantitative distributional discrepancy metrics with uncertainty are needed to substantiate the convergence claim.
- [Eq. (3), Fig. 1, Appendix D] The only non-Gaussian shapes tested are the Bayes-MTJ fit, which is near-Gaussian, and a symmetric two-component Gaussian mixture; the ECRAM shape in Appendix D is also close to Gaussian. These tests do not justify the phrase "regardless of the shape." Skewed or heavy-tailed device-like distributions (e.g., log-normal, exponentially tilted, or truncated Pareto) should be tested, or the claim should be explicitly limited to symmetric, light-tailed shapes that are close to Gaussian.
minor comments (5)
- [Eq. (1)] The optimization is written as min_alpha ELBO(alpha), but the ELBO is to be maximized; Eq. (2) minimizes the negative ELBO. Please fix the sign and notation for consistency.
- [Section 2] There are typos in the text: "can can appear" and "opreations" should be corrected. Also, "This approach is restricted to mean field variational inference" is a sentence fragment.
- [Fig. 4, right panel] The label "Divergence to Gaussian" is ambiguous. Please specify which divergence is plotted (KL, energy distance, etc.) and how it is estimated.
- [Section 4.1] The energy-distance experiment uses a single synthetic target distribution at x=0. The invariance claim would be more convincing if the comparison were repeated at multiple input points and for non-Gaussian target distributions.
- [Section 3.1.1] The parameterization of qD in Eq. (3) is clear, but the constraints imposed during maximum likelihood fitting (normalization, zero mean, and variance matching) are not fully described. Please state them explicitly.
Circularity Check
No significant circularity: the shape-insensitivity claim is tested on external benchmarks and synthetic tasks, and the self-cited device distributions are input data rather than fitted predictions.
full rationale
The paper's central claim is that predictive distributions from BNNs with matched weight means and variances converge regardless of the variational distribution's shape. This is tested by training with Gaussian weights and swapping in device and bimodal base distributions with identical first two moments, then evaluating on an energy-distance toy problem, 1D regression, and the UTKFACE benchmark. The device distribution parameters A, B, C in Eq. (3) are maximum-likelihood fits to hardware noise from the authors' prior work [9,12], but those fitted values are not reused as predictions; they are inputs to the comparison, and the invariance conclusion does not reduce to them. The self-citations supply the device noise model and hardware context, but the load-bearing evidence is the paired predictive-distribution comparison against external tasks, so the citations are not load-bearing. The CLT argument in Section 1 is standard external mathematics; its finite-width validity is a genuine limitation (the paper concedes wide layers are necessary and narrow deep networks are future work), but that is a correctness risk, not circularity. No equation or fitted parameter is equal by construction to the claimed result, so no circular step can be exhibited.
Assumptions & free parameters
free parameters (4)
- A (Bayes-MTJ device distribution fit) =
not reported in text
- B (Bayes-MTJ device distribution fit) =
not reported in text
- C (Bayes-MTJ device distribution fit) =
not reported in text
- A, B, C (ECRAM device distribution fit) =
not reported in text
assumptions (4)
- domain assumption Mean-field independence of weights
- domain assumption Shift-scale reparameterization
- domain assumption Gaussian likelihood with heteroscedastic noise
- ad hoc to paper Central limit theorem applies to finite-width pre-activations
Cite this review
Pith. "Pith review of Analog Bayesian neural networks are insensitive to the shape of the weight distribution." pith.science (2026). https://pith.science/paper/EIDDIUNN
@misc{pith2026250105564,
author = {Pith},
title = {Pith review of: Analog Bayesian neural networks are insensitive to the shape of the weight distribution},
year = {2026},
howpublished = {\url{https://pith.science/paper/EIDDIUNN}},
note = {Machine review of arXiv:2501.05564}
}
read the original abstract
Recent work has demonstrated that Bayesian neural networks (BNN's) trained with mean field variational inference (MFVI) can be implemented in analog hardware, promising orders of magnitude energy savings compared to the standard digital implementations. However, while Gaussians are typically used as the variational distribution in MFVI, it is difficult to precisely control the shape of the noise distributions produced by sampling analog devices. This paper introduces a method for MFVI training using real device noise as the variational distribution. Furthermore, we demonstrate empirically that the predictive distributions from BNN's with the same weight means and variances converge to the same distribution, regardless of the shape of the variational distribution. This result suggests that analog device designers do not need to consider the shape of the device noise distribution when hardware-implementing BNNs performing MFVI.
Figures
Figures from the paper (1 more)
Reference graph
Works this paper leans on
-
[1]
To trust or not to trust a classifier
Heinrich Jiang, Been Kim, Melody Guan, and Maya Gupta. To trust or not to trust a classifier. Advances in neural information processing systems, 31, 2018
work page 2018
-
[2]
A practical bayesian framework for backpropagation networks
David JC MacKay. A practical bayesian framework for backpropagation networks. Neural computation, 4(3):448–472, 1992
1992
-
[3]
Hands-on bayesian neural networks—a tutorial for deep learning users
Laurent Valentin Jospin, Hamid Laga, Farid Boussaid, Wray Buntine, and Mohammed Ben- namoun. Hands-on bayesian neural networks—a tutorial for deep learning users. IEEE Com- putational Intelligence Magazine, 17(2):29–48, 2022
work page 2022
-
[4]
Weight uncer- tainty in neural network
Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncer- tainty in neural network. In International conference on machine learning, pages 1613–1622, 2015
work page 2015
-
[5]
Variational inference: A review for statisticians
David M Blei, Alp Kucukelbir, and Jon D McAuliffe. Variational inference: A review for statisticians. Journal of the American statistical Association, 112(518):859–877, 2017
2017
-
[6]
Vibnn: Hardware acceleration of bayesian neural networks.ACM SIGPLAN Notices, 53(2):476–488, 2018
Ruizhe Cai, Ao Ren, Ning Liu, Caiwen Ding, Luhao Wang, Xuehai Qian, Massoud Pedram, and Yanzhi Wang. Vibnn: Hardware acceleration of bayesian neural networks.ACM SIGPLAN Notices, 53(2):476–488, 2018
work page 2018
-
[7]
Analog architectures for neural network acceleration based on non-volatile memory
T Patrick Xiao, Christopher H Bennett, Ben Feinberg, Sapan Agarwal, and Matthew J Marinella. Analog architectures for neural network acceleration based on non-volatile memory. Applied Physics Reviews, 7(3), 2020
work page 2020
-
[8]
Stochastic memory devices for security and computing
Roberto Carboni and Daniele Ielmini. Stochastic memory devices for security and computing. Advanced Electronic Materials, 5(9):1900198, 2019
work page 2019
Show all 28 references
-
[9]
Patrick Xiao, Jaesuk Kwon, Bert J
Samuel Liu, T. Patrick Xiao, Jaesuk Kwon, Bert J. Debusschere, Sapan Agarwal, Jean Anne C. Incorvia, and Christopher H. Bennett. Bayesian neural networks using magnetic tunnel junction-based probabilistic in-memory computing. Frontiers in Nanotechnology, 4, 2022
2022
-
[10]
Uncertainty quantification via a memristor bayesian deep neural network for risk-sensitive reinforcement learning
Yudeng Lin, Qingtian Zhang, Bin Gao, Jianshi Tang, Peng Yao, Chongxuan Li, Shiyu Huang, Zhengwu Liu, Ying Zhou, Yuyi Liu, et al. Uncertainty quantification via a memristor bayesian deep neural network for risk-sensitive reinforcement learning. Nature Machine Intelligence, 5(7)...
2023
-
[11]
Two-dimensional materials-based probabilistic synapses and reconfigurable neurons for measuring inference uncertainty using bayesian neural networks
Amritanand Sebastian, Rahul Pendurthi, Azimkhan Kozhakhmetov, Nicholas Trainor, Joshua A Robinson, Joan M Redwing, and Saptarshi Das. Two-dimensional materials-based probabilistic synapses and reconfigurable neurons for measuring inference uncertainty using bayesian neural net...
2022
-
[12]
S. Oh, T. P. Xiao, C. Bennett, A. J Weiss, S. R. Bishop, P. S. Finnegen, E. J. Fuller, S. Agarwal, and A. A. Talin. Bayesian neural network implemented by dynamically programmable noise in vanadium oxide. In International Electron Devices Meeting (IEDM), pages 1–4, 2023
2023
-
[13]
Ex situ transfer of bayesian neural networks to resistive memory-based inference hardware
Thomas Dalgaty, Eduardo Esmanhotto, Niccolo Castellani, Damien Querlioz, and Elisa Vianello. Ex situ transfer of bayesian neural networks to resistive memory-based inference hardware. Advanced Intelligent Systems, 3(8):2000103, 2021
2021
-
[14]
Noise tailoring in memristive filaments
Botond Santa, Zoltan Balogh, Laszlo Posa, David Krisztian, Timea Nora Torok, Daniel Mol- nar, Csaba Sinko, Roland Hauert, Miklos Csontos, and Andras Halbritter. Noise tailoring in memristive filaments. ACS applied materials & interfaces, 13(6):7453–7460, 2021
2021
-
[15]
Exploiting read current noise of tio x resistive memory by controlling forming conditions for probabilistic neural network hardware
Wooseok Choi, Wonjae Ji, Seongjae Heo, Donguk Lee, Kyungmi Noh, Chuljun Lee, Jiyong Woo, Seyoung Kim, and Hyunsang Hwang. Exploiting read current noise of tio x resistive memory by controlling forming conditions for probabilistic neural network hardware. IEEE Electron Device L...
2022
-
[16]
Kingma and Jimmy Ba
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv:1412.6980, 2017
2017 arXiv
-
[17]
Numerical recipes in FORTRAN: the art of scientific computing
William H Press. Numerical recipes in FORTRAN: the art of scientific computing. Cambridge University Press, 2005
2005
-
[18]
What uncertainties do we need in bayesian deep learning for computer vision? Advances in neural information processing systems, 30, 2017
Alex Kendall and Yarin Gal. What uncertainties do we need in bayesian deep learning for computer vision? Advances in neural information processing systems, 30, 2017
2017
-
[19]
Age progression/regression by conditional adver- sarial autoencoder
Zhifei Zhang, Yang Song, and Hairong Qi. Age progression/regression by conditional adver- sarial autoencoder. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2017
2017
-
[20]
Mart ´ın Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Good- fellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manj...
2015
-
[21]
Harris, K
Charles R. Harris, K. Jarrod Millman, St ´efan J. van der Walt, Ralf Gommers, Pauli Vir- tanen, David Cournapeau, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel J. Smith, Robert Kern, Matti Picus, Stephan Hoyer, Marten H. van Kerkwijk, Matthew Brett, Allan Hal- dane, Ja...
2020
-
[22]
Oliphant, Matt Haberland, Tyler Reddy, David Cour- napeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, St ´efan J
Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cour- napeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, St ´efan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nelso...
2020
-
[23]
Chaospy: An open source tool for designing methods of uncertainty quantification
Jonathan Feinberg and Hans Petter Langtangen. Chaospy: An open source tool for designing methods of uncertainty quantification. Journal of Computational Science, 11:46–57, 2015
2015
-
[24]
Smith, Mateusz Paprocki, Ond ˇrej ˇCert´ık, Sergey B
Aaron Meurer, Christopher P. Smith, Mateusz Paprocki, Ond ˇrej ˇCert´ık, Sergey B. Kirpichev, Matthew Rocklin, AMiT Kumar, Sergiu Ivanov, Jason K. Moore, Sartaj Singh, Thilina Rath- nayake, Sean Vig, Brian E. Granger, Richard P. Muller, Francesco Bonazzi, Harsh Gupta, Shivam V...
2017
-
[25]
GNU Parallel 2018
Ole Tange. GNU Parallel 2018. Ole Tange, March 2018
2018
-
[26]
Hyperbolic quadrature method of moments for the one-dimensional kinetic equation
Rodney O Fox and Fr ´ed´erique Laurent. Hyperbolic quadrature method of moments for the one-dimensional kinetic equation. SIAM Journal on Applied Mathematics , 82(2):750–771, 2022
2022
-
[27]
Modified moments and gaussian quadratures
John C Wheeler. Modified moments and gaussian quadratures. The Rocky Mountain Journal of Mathematics, 4(2):287–296, 1974
1974
-
[28]
Fast and accurate deep network learning by exponential linear units (elus)
Djork-Arn ´e Clevert. Fast and accurate deep network learning by exponential linear units (elus). arXiv:1511.07289, 2015. A Formula for P2 and ˆG2 In Section 3.1.3 we utilize the second order Taylor expansion of the device CDF, P2(−1 +x) = A + 2BCe 1 B (x + 1)2 e− 1 B 2B (11) ...
2015 arXiv
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.