Pith. sign in

REVIEW 1 major objections 4 minor 50 references

On the Expressiveness of Approximate Inference in Bayesian Neural Networks

T0 review · 1 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read For one-hidden-layer ReLU Bayesian neural networks, mean-field Gaussian variational inference and Monte Carlo dropout provably cannot represent the increased uncertainty between well-separated data clusters that exact Bayesian inference…

desk verdict First proof that mean-field Gaussian and MC dropout posteriors provably lack in-between uncertainty in shallow ReLU BNNs; a solid, citable negative result with one minor appendix overstatement. read the letter →

arxiv 1909.00719 v4 pith:O2HCSKFH submitted 2019-09-02 stat.ML cs.LG

Andrew Y. K. Foong , David R. Burt , Yingzhen Li , Richard E. Turner This is my paper · ORCID
classification stat.MLcs.LG MSC 62F1568T07
keywords Bayesianneuralnetworksvariationalinferencemean-fieldapproximationMonteCarlodropoutpredictiveuncertaintyin-betweenReLUactivelearning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper sets out to show that two of the most common approximate-inference methods for Bayesian neural networks — mean-field Gaussian variational inference and Monte Carlo dropout — can, even in the simplest one-hidden-layer ReLU network, fail to reproduce a basic qualitative feature of the exact Bayesian predictive distribution: the rise in uncertainty that should occur in the gap between two well-separated clusters of data. It proves this as a theorem about the weight-space structure of the approximating families, and it shows empirically that exact inference, represented by the infinite-width Gaussian process and Hamiltonian Monte Carlo, does exhibit that in-between uncertainty on the same problem. The paper also proves that the limitation is not fundamental to depth: with at least two hidden layers, both families contain distributions whose predictive mean and variance can approximate any continuous target functions. Yet when the usual variational objectives are optimized in deep networks, the same pathological confidence persists, and in an active-learning case study it leads MFVI and MCDO to choose points worse than random selection while the Gaussian-process reference gains a threefold improvement. The conclusion is that failures in BNN uncertainty estimates can be caused by the approximation family and the optimization objective, not by the model or prior.

What carries the argument

The load-bearing object is the decomposition of predictive variance by the law of total variance, conditioned on the first-layer parameters. Under the factorised families, the conditional variance of the output is $V[f(x) | \theta_{in}] = \sum_i V[w_i] \psi(a_i(x))^2 + V[b]$, where $\psi(a) = \max(0, a)$ is the ReLU; because $\psi^2$ is convex, this term is convex in $x$, which alone proves Theorem 2 and explains the non-convexity that could rescue Theorem 1. The remaining term for mean-field Gaussian, $V[\mathbb{E}[f | \theta_{in}]]$, is a positive linear combination of variances of rectified activations, and the appendix proves these satisfy reflection and monotonicity properties (sets $T_1$ and $T_2$) that force the line-segment bound of Theorem 1. Depth enters through a construction that splits the two units of the second hidden layer into a mean-control channel and a variance-control channel, using the deterministic universal approximation theorem for ReLU networks to argue that both the approximate mean and variance can match any continuous targets.

What would settle it

A single counterexample would settle the main claim: find a single-hidden-layer ReLU network whose weights follow a fully factorised Gaussian or an MC dropout distribution (inputs not dropped), pick two points $p$ and $q$ satisfying the conditions of Theorem 1, and observe strictly larger predictive variance at an interior point $r$ than the sum of the endpoint variances; the theorem says this is impossible. For the empirical half, one can repeat the two-cluster regression with three clusters arranged so the middle cluster is unobserved: the paper predicts Hamiltonian Monte Carlo and the infinite-width GP will show elevated uncertainty at the unobserved middle cluster while MFVI and MCDO will not, across all hyperparameter settings.

Watch

Extended reading notes

Core claim

For a single-hidden-layer fully connected ReLU network, any approximate posterior drawn from the fully factorised Gaussian family or from the MC dropout family (with inputs not dropped out) has a predictive variance that cannot be substantially larger between well-separated regions of low uncertainty. Theorem 1 proves that on line segments passing through the origin, or crossing an axis plane, the variance at an interior point is bounded by the sum of the variances at two flanking endpoints, $V[f(r)] \le V[f(p)] + V[f(q)]$; Theorem 2 proves the stronger statement that for MC dropout the variance function is convex in the input, so on any segment it is bounded by the maximum at the endpoints. The paper provides strong empirical evidence that the exact posterior does not share this pathology: the infinite-width Gaussian process and Hamiltonian Monte Carlo posteriors show visibly elevated uncertainty between two data clusters. For networks with at least two hidden layers, Theorem 3 states that the predictive mean and variance of both approximate families are universal approximators, yet ELBO-optimised deep MFVI and MCDO still exhibit the same overconfidence between clusters, and the active-learning experiment on the Naval dataset shows MFVI and MCDO degrading performance relative to random acquisition.

Load-bearing premise

The proofs assume the approximate posterior factorises across hidden neurons, so the output weights attached to different neurons are independent and carry no negative correlations; if correlations between those weights were allowed, the variance could become non-convex and the bounds in Theorems 1 and 2 would no longer follow.

Editorial extensions

If this is right

  • For single-hidden-layer ReLU BNNs, no choice of variational parameters, prior, or optimiser can make a fully factorised Gaussian or MC dropout posterior represent in-between uncertainty on the line segments covered by Theorems 1 and 2; the failure is a property of the approximating family itself.
  • Any inference method that uses the fully factorised Gaussian family — including diagonal Laplace and other variational schemes — inherits the same function-space limitation, because Theorem 1 depends only on the family.
  • The exact posterior of the BNN model does show increased uncertainty between well-separated data, so the lack of in-between uncertainty in the approximations is not a modelling defect; it is introduced by approximate inference.
  • In deep networks, theoretical expressiveness alone does not guarantee good approximate inference: the universality result is real, but ELBO-optimised MFVI and MCDO empirically remain overconfident between data clusters, even when initialised from a function-space fit that already has the correct uncertainty shape.
  • In the active-learning case study, the exact-inference GP reduces test RMSE by roughly a factor of three relative to random selection, whereas 1HL MFVI and MCDO increase RMSE and deeper variants mostly fail to beat random selection, showing a concrete practical cost of the pathology.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The core mechanism — independence of outgoing weights cancels the covariance terms that make variance non-convex — suggests the same obstruction applies to any factorised family whose activations have a convex squared non-linearity, and to deeper variants that use such factorisation in the final layer; a testable prediction is that a full-covariance Gaussian posterior with negative correlations be
  • A natural extension would be to measure the 'distance-to-origin bias' of MFVI predictive variance in high-dimensional input spaces: the Naval result implies that even where the theorem's precise line conditions are rare, the family still ranks uncertainty more by distance from the origin than by local data density, and this ranking could be quantified directly.
  • The deep-network result separates expressiveness from optimisation: Theorem 3 says good distributions exist in the family, while the ELBO experiments say they are not found. This suggests a testable diagnostic for new variational objectives: initialise from the squared-loss solution and check whether ELBO maximisation preserves or destroys in-between uncertainty.
  • If the pathology is caused by the factorised family and the ELBO landscape, then non-factorised approximations (for example, inducing correlations between output weights) or alternative objectives that directly match function-space moments should eliminate the overconfidence pattern in the same toy dataset; this is a direct, checkable implication.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

1 major / 4 minor

Summary. The paper studies the function-space expressiveness of mean-field Gaussian (QFFG) and Monte Carlo dropout (QMCDO) approximating families for Bayesian neural networks. The central theoretical result is that for single-hidden-layer ReLU BNNs, QFFG and QMCDO cannot represent 'in-between uncertainty' in the predictive variance: Theorem 1 bounds the variance at a point between two well-separated low-uncertainty regions on certain line segments, and Theorem 2 shows the MCDO variance is convex. The authors separate Criterion 1 (does the family contain good approximations to the posterior?) from Criterion 2 (does optimization find them?). Empirical comparisons against HMC and the infinite-width GP on a two-cluster regression task show that the exact posterior does exhibit in-between uncertainty, while MFVI and MCDO do not. For two-hidden-layer networks, Theorem 3 gives universality of predictive mean and variance for both families, but ELBO optimization in deeper networks still shows overconfidence between clusters (Section 4.2). A final active-learning case study on the Naval dataset shows GP-based selection succeeds while MFVI/MCDO-based selection degrades RMSE relative to random selection.

Significance. If the results hold, this is an important contribution to the BNN approximation literature: it gives a crisp, weight-space-to-function-space explanation of a known qualitative failure of two widely used variational families, and it carefully separates model quality from approximation quality using HMC and the infinite-width GP as references. The theorems are proven in detail with standard tools (law of total variance, convexity of rectified quadratics, Chebyshev and union-bound arguments), and the paper ships code for reproduction. The universality result for two-hidden-layer networks is a useful counterpoint showing that depth can in principle restore expressiveness, and the explicit constructive proofs for both QFFG and QMCDO are a strength. The empirical protocol—direct squared-loss minimization in function space, ELBO-based fitting, overconfidence-ratio boxplots, and active learning—triangulates the theoretical claims well. The main caveat is that the exact-posterior claim is empirical rather than proven, and the shallow-case theorems cover special line configurations; these limitations are acknowledged in the paper.

major comments (1)
  1. [Appendix B, Theorem 4; Appendix E.1, Theorem 8] Theorem 4 is stated for arbitrary conditional densities q_i(w_i|U,v) and q(b|U,v), but the proof's equation (5) treats E[w_{k,i}|U,v] and E[b_k|U,v] as constants independent of U,v. If the conditional densities are truly allowed to depend on U,v, then V[E[f|U,v]] = V[sum_i μ_{w_{k,i}}(U,v)ψ(a_i) + μ_{b_k}(U,v)] contains additional covariance terms that are not accounted for. The same gap appears in Theorem 8, where V[E[f(0)|U]]=0 is asserted although q_i(w_i|U) and q(b|U) are allowed to depend on U. Both gaps disappear for the actual QFFG and MCDO families, whose conditional means are constant, so Theorems 1 and 2 survive; nevertheless, the statements of Theorems 4 and 8 should be restricted (e.g., to q_i(w_i|U,v)=q_i(w_i) and q(b|U,v)=q(b)) or the proofs extended to handle the general case.
minor comments (4)
  1. [Section 4, paragraph before Theorem 3] The sentence 'this provides theoretical evidence that approximate inference in deep BNNs satisfies Criterion 1' is stronger than the theorem warrants: Criterion 1 is about containing good approximations to the actual posterior, whereas Theorem 3 only constructs elements of the family with prescribed first two output moments on a compact set. The caveats in the following paragraphs mitigate this, but the wording should be qualified.
  2. [Appendix F.1] The prior-scaling procedure is described by the list of weight standard deviations and the statement that prior standard deviations in function space are 'between 10 and 15'; a more precise statement of the target value or the validation criterion would improve reproducibility.
  3. [Section 5 and Appendix H.1] The active-learning experiment fixes the observation-noise standard deviation to 0.01 for all models and all depths; a brief sensitivity check for this choice would strengthen the comparison, since the GP reference and the BNNs share the same fixed noise.
  4. [Figure 5 caption; Appendix D] The Figure 5 caption contains a duplicated article ('and the the box extends'), and several displayed sup-norm inequalities in Appendix D contain duplicated vertical-bar characters (e.g., '|||...|||∞'); these formatting issues should be cleaned up.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the limitation theorems are proved from the defining structure of QFFG/QMCDO and checked against independent HMC/GP references.

full rationale

The central claims are derived from the definitions of the approximating families, not from the target conclusion. Theorem 1 and Theorem 2 use the conditional independence of outgoing weights (equation (2) and Appendix C.1 equation (6)) to expand the conditional variance, then apply non-trivial lemmas about rectified-Gaussian variances and convexity; the bound is therefore a consequence of the family's factorization, not an assumption of the result being proved. The claim that exact inference lacks the pathology is supported empirically by HMC and by the infinite-width GP, so no fitted parameter is relabelled as a prediction. The deep-network universality theorem is an existence construction using the external universal-approximation theorem (Leshno et al.) with explicit parameter choices, and the paper separately checks whether ELBO optimization actually finds such distributions (Criterion 2). Self-citations are present but not load-bearing: the GP-limit references include external independent results (Neal; Lee et al.), and the in-house citations are used only for initialization practice or as examples of methods using QFFG. One non-circular caveat is worth flagging: Appendix B's Theorem 4 is stated for arbitrary conditional densities q_i(w_i|U,v), but its proof implicitly treats E[w_i|U,v] as constant in U,v when deriving equation (5); for the fully-factorised Gaussian family used in the main text this condition holds, so Theorems 1 and 2 are unaffected. This is a correctness/generalization gap in a supporting statement, not a circular reduction of the paper's main argument.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

The theoretical claims rest on standard probability results and the universal approximation theorem. The empirical comparison to 'exact inference' uses the GP-limit and HMC as external references, which are well-supported but not proven in this paper.

assumptions (5)
  • standard math Universal approximation by deterministic ReLU networks (Leshno et al., 1993)
    Used in the proof of Theorem 3 to approximate the target mean g and variance h by the activations of a 2-hidden-layer network; see Appendix D, proof of Theorem 6.
  • standard math Law of total variance and conditional independence of output weights under factorized variational families
    Used in proofs of Theorems 1, 2, 4, and 5; see Section 3.1, equation (2), and Appendix C.1, equation (6).
  • standard math Variance formula for a rectified Gaussian unit (Frey and Hinton, 1999)
    Used in Lemma 2 to characterize the variance of a single neuron's output as a function of input; see Appendix C.2.
  • domain assumption Infinite-width BNNs converge to Gaussian processes and moderate-width BNNs resemble their GP limit under the chosen prior scaling
    Used as the reference for exact inference in the empirical comparisons (Figures 3, 5, 6); see Section 2.2.
  • domain assumption Hamiltonian Monte Carlo samples are representative of the exact posterior
    Used as a gold-standard reference for exact inference in the 1HL and 2HL cases; see Section 3.2 and Appendix F.

how reviews work

0 comments
Cite this review

Pith. "Pith review of On the Expressiveness of Approximate Inference in Bayesian Neural Networks." pith.science (2026). https://pith.science/paper/O2HCSKFH

@misc{pith2026190900719,
  author       = {Pith},
  title        = {Pith review of: On the Expressiveness of Approximate Inference in Bayesian Neural Networks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/O2HCSKFH}},
  note         = {Machine review of arXiv:1909.00719}
}
read the original abstract

While Bayesian neural networks (BNNs) hold the promise of being flexible, well-calibrated statistical models, inference often requires approximations whose consequences are poorly understood. We study the quality of common variational methods in approximating the Bayesian predictive distribution. For single-hidden layer ReLU BNNs, we prove a fundamental limitation in function-space of two of the most commonly used distributions defined in weight-space: mean-field Gaussian and Monte Carlo dropout. We find there are simple cases where neither method can have substantially increased uncertainty in between well-separated regions of low uncertainty. We provide strong empirical evidence that exact inference does not have this pathology, hence it is due to the approximation and not the model. In contrast, for deep networks, we prove a universality result showing that there exist approximate posteriors in the above classes which provide flexible uncertainty estimates. However, we find empirically that pathologies of a similar form as in the single-hidden layer case can persist when performing variational inference in deeper networks. Our results motivate careful consideration of the implications of approximate inference methods in BNNs.

Figures

Figures reproduced from arXiv: 1909.00719 by the authors.

Figure 1
Figure 1. Illustration of the bounded regions in theorem 1, showing the input domain of a 1HL [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Results of directly minimising the squared error in function space between V[f(x)] (for a single-hidden layer NN) and a target variance function. Left: FFG distribution, Right: MCDO distribution. The bound for FFG distributions (red) applies on [−1, 1] with p = −1, q = 1. The MCDO variance function is convex, and almost constant. The FFG and MCDO variance functions underestimate the target near the origin and overes… view at source ↗
Figure 3
Figure 3. Regression on a 2D synthetic dataset (red crosses). The colour plots show the standard [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (12 more)
Figure 4
Figure 4. Figure 4: Results of minimising the squared error in function space between [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Box and whisker plots of the overconfidence ratios of HMC, MFVI and MCDO relative [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Points chosen during active learning in the 1HL case. Colours denote distance from the [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Mean and 2 standard deviation bars of the predictive distribution on lines joining random [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]
Figure 8
Figure 8. Figure 8: Same experimental set-up as in figure 7 for the 3HL case. [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]
Figure 9
Figure 9. Figure 9: Schematic illustration of the bound in theorem 2, showing the input domain of a single [PITH_FULL_IMAGE:figures/full_fig_p028_9.png]
Figure 10
Figure 10. Figure 10: figure 10. With this prior both methods tend to be less over-confident between data clusters, but more [PITH_FULL_IMAGE:figures/full_fig_p030_10.png]
Figure 11
Figure 11. Figure 11: Plots of the overconfidence ratio against [PITH_FULL_IMAGE:figures/full_fig_p030_11.png]
Figure 12
Figure 12. Figure 12: Mean and error bars (± 2 standard deviations) for the GP and the BNN with each inference scheme, trained on the data shown by the red crosses. The inference algorithms were initialised by first minimising the squared error to the reference GP mean and variance, and th…
Figure 13
Figure 13. Figure 13: Points chosen during active learning in the 3HL case. Colours denote distance from the [PITH_FULL_IMAGE:figures/full_fig_p033_13.png]
Figure 14
Figure 14. Figure 14: Colours denote predictive uncertainties in the 1HL case, at the beginning of active learning. As the noise standard deviation was fixed to 0.01 for all models, changes in the predictive standard deviation reflect model uncertainty. Grey crosses (✕) denote the five poi…
Figure 15
Figure 15. Figure 15: Colours denote predictive uncertainties in the 1HL case, after 50 iterations of active learning. As the noise standard deviation was fixed to 0.01 for all models, changes in the predictive standard deviation reflect model uncertainty. Grey crosses (✕) denote the five …

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

50 extracted references · 43 canonical work pages

  1. [1]

    Handbook of mathematical functions: with formulas, graphs, and mathematical tables, volume 55

    Milton Abramowitz and Irene A Stegun. Handbook of mathematical functions: with formulas, graphs, and mathematical tables, volume 55. Courier Corporation, 1965

    work page 1965

  2. [2]

    Ensemble learning in Bayesian neural networks

    David Barber and Christopher M Bishop. Ensemble learning in Bayesian neural networks. Neural networks and machine learning, 168:215–237, 1998

    work page 1998

  3. [3]

    Variational algorithms for approximate Bayesian inference

    Matthew James Beal. Variational algorithms for approximate Bayesian inference. PhD thesis, University College London, 2003

    work page 2003

  4. [4]

    Chen, Martin Jankowiak, Fritz Obermeyer, Neeraj Pradhan, Theofanis Karaletsos, Rohit Singh, Paul Szerlip, Paul Horsfall, and Noah D

    Eli Bingham, Jonathan P. Chen, Martin Jankowiak, Fritz Obermeyer, Neeraj Pradhan, Theofanis Karaletsos, Rohit Singh, Paul Szerlip, Paul Horsfall, and Noah D. Goodman. Pyro: Deep universal probabilistic programming. Journal of Machine Learning Research (JMLR), 2018

    work page 2018

  5. [5]

    Variational inference: A review for statisticians

    David M Blei, Alp Kucukelbir, and Jon D McAuliffe. Variational inference: A review for statisticians. Journal of the American Statistical Association, 112(518):859–877, 2017

    work page 2017

  6. [6]

    Weight uncertainty in neural networks

    Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural networks. In International Conference on Machine Learning (ICML), 2015

    2015

  7. [7]

    Stochastic gradient Hamiltonian Monte Carlo

    Tianqi Chen, Emily Fox, and Carlos Guestrin. Stochastic gradient Hamiltonian Monte Carlo. In International Conference on Machine Learning, pages 1683–1691, 2014

    2014

  8. [8]

    Machine learning approaches for improving condition-based maintenance of naval propulsion plants

    Andrea Coraddu, Luca Oneto, Alessandro Ghio, Stefano Savio, Davide Anguita, and Massimo Figari. Machine learning approaches for improving condition-based maintenance of naval propulsion plants. Journal of Engineering for the Maritime Environment, 2014

    work page 2014

Show all 50 references
  1. [9]

    Transforming neural-net output levels to probability distribu- tions

    John S Denker and Yann Lecun. Transforming neural-net output levels to probability distribu- tions. In Advances in Neural Information Processing Systems (NIPS), 1991

    1991

  2. [10]

    Liberty or depth: Deep Bayesian neural nets do not need complex weight posterior approximations

    Sebastian Farquhar, Lewis Smith, and Yarin Gal. Liberty or depth: Deep Bayesian neural nets do not need complex weight posterior approximations. arXiv preprint arXiv:2002.03704, 2020

    2002 arXiv

  3. [11]

    Gomez, Tim G

    Angelos Filos, Sebastian Farquhar, Aidan N. Gomez, Tim G. J. Rudner, Zachary Kenton, Lewis Smith, Milad Alizadeh, Arnoud de Kroon, and Yarin Gal. Benchmarking Bayesian deep learning with diabetic retinopathy diagnosis. https://github.com/OATML/bdl-benchmarks, 2019

    2019

  4. [12]

    Variational learning in nonlinear Gaussian belief networks

    Brendan J Frey and Geoffrey E Hinton. Variational learning in nonlinear Gaussian belief networks. Neural Computation, 11(1):193–213, 1999. 10

    1999

  5. [13]

    Uncertainty in deep learning

    Yarin Gal. Uncertainty in deep learning. PhD thesis, University of Cambridge, 2016

    2016

  6. [14]

    Dropout as a Bayesian approximation: Representing model uncertainty in deep learning

    Yarin Gal and Zoubin Ghahramani. Dropout as a Bayesian approximation: Representing model uncertainty in deep learning. In International Conference on Machine Learning (ICML), 2016

    2016

  7. [15]

    Deep Bayesian active learning with image data

    Yarin Gal, Riashat Islam, and Zoubin Ghahramani. Deep Bayesian active learning with image data. In International Conference on Machine Learning (ICML), 2017

    2017

  8. [16]

    Probabilistic backpropagation for scalable learning of Bayesian neural networks

    José Miguel Hernández-Lobato and Ryan Adams. Probabilistic backpropagation for scalable learning of Bayesian neural networks. In International Conference on Machine Learning (ICML), 2015

    2015

  9. [17]

    Black-boxα-divergence minimization

    José Miguel Hernández-Lobato, Yingzhen Li, Mark Rowland, Thang Bui, Daniel Hernández- Lobato, and Richard Turner. Black-boxα-divergence minimization. In International Conference on Machine Learning (ICML), 2016

    2016

  10. [18]

    Keeping the neural networks simple by minimizing the description length of the weights

    Geoffrey E Hinton and Drew Van Camp. Keeping the neural networks simple by minimizing the description length of the weights. In Conference on Computational learning theory (COLT), 1993

    1993

  11. [19]

    The No-U-Turn sampler: adaptively setting path lengths in Hamiltonian Monte Carlo

    Matthew D Hoffman and Andrew Gelman. The No-U-Turn sampler: adaptively setting path lengths in Hamiltonian Monte Carlo. Journal of Machine Learning Research (JMLR), 15(1): 1593–1623, 2014

    2014

  12. [20]

    Variational Bayesian dropout: Pitfalls and fixes

    Jiri Hron, Alex Matthews, and Zoubin Ghahramani. Variational Bayesian dropout: Pitfalls and fixes. In International Conference on Machine Learning (ICML), 2018

    2018

  13. [21]

    Exact posterior distributions of wide Bayesian neural networks

    Jiri Hron, Yasaman Bahri, Roman Novak, Jeffrey Pennington, and Jascha Sohl-Dickstein. Exact posterior distributions of wide Bayesian neural networks. In Uncertainty in deep learning Workshop, ICML., 2020

    2020

  14. [22]

    An in- troduction to variational methods for graphical models

    Michael I Jordan, Zoubin Ghahramani, Tommi S Jaakkola, and Lawrence K Saul. An in- troduction to variational methods for graphical models. Machine Learning, 37(2):183–233, 1999

    1999

  15. [23]

    Fast and scalable Bayesian deep learning by weight-perturbation in Adam.International Conference on Machine Learning (ICML), 2018

    Mohammad Emtiyaz Khan, Didrik Nielsen, V oot Tangkaratt, Wu Lin, Yarin Gal, and Akash Sri- vastava. Fast and scalable Bayesian deep learning by weight-perturbation in Adam.International Conference on Machine Learning (ICML), 2018

    2018

  16. [24]

    Variational dropout and the local reparameter- ization trick

    Durk P Kingma, Tim Salimans, and Max Welling. Variational dropout and the local reparameter- ization trick. In Advances in Neural Information Processing Systems (NIPS), pages 2575–2583, 2015

    2015

  17. [25]

    Deep neural networks as Gaussian processes

    Jaehoon Lee, Jascha Sohl-Dickstein, Jeffrey Pennington, Roman Novak, Sam Schoenholz, and Yasaman Bahri. Deep neural networks as Gaussian processes. In International Conference on Learning Representations (ICLR), 2018

    2018

  18. [26]

    Multilayer feedforward networks with a nonpolynomial activation function can approximate any function

    Moshe Leshno, Vladimir Ya Lin, Allan Pinkus, and Shimon Schocken. Multilayer feedforward networks with a nonpolynomial activation function can approximate any function. Neural Networks, 6(6):861–867, 1993

    1993

  19. [27]

    Rényi divergence variational inference

    Yingzhen Li and Richard E Turner. Rényi divergence variational inference. In Advances in Neural Information Processing Systems (NIPS), pages 1073–1081, 2016

    2016

  20. [28]

    Stochastic expectation propagation

    Yingzhen Li, José Miguel Hernández-Lobato, and Richard E Turner. Stochastic expectation propagation. In Advances in Neural Information Processing Systems (NIPS), pages 2323–2331, 2015

    2015

  21. [29]

    Multiplicative normalizing flows for variational Bayesian neural networks

    Christos Louizos and Max Welling. Multiplicative normalizing flows for variational Bayesian neural networks. In International Conference on Machine Learning (ICML), 2017

    2017

  22. [30]

    David J. C. MacKay. A practical Bayesian framework for backpropagation networks. Neural Computation, 4(3):448–472, 1992. 11

    1992

  23. [31]

    Alexander G. de G. Matthews, Mark van der Wilk, Tom Nickson, Keisuke. Fujii, Alexis Boukouvalas, Pablo León-Villagrá, Zoubin Ghahramani, and James Hensman. GPflow: A Gaussian process library using TensorFlow. Journal of Machine Learning Research (JMLR), 18(40):1–6, 2017

    2017

  24. [32]

    Alexander G. de G. Matthews, Jiri Hron, Mark Rowland, Richard E. Turner, and Zoubin Ghahramani. Gaussian process behaviour in wide deep neural networks. In International Conference on Learning Representations (ICLR), 2018

    2018

  25. [33]

    On the importance of strong baselines in Bayesian deep learning

    Jishnu Mukhoti, Pontus Stenetorp, and Yarin Gal. On the importance of strong baselines in Bayesian deep learning. arXiv preprint arXiv:1811.09385, 2018

    2018 arXiv

  26. [34]

    Bayesian learning for neural networks

    Radford M Neal. Bayesian learning for neural networks. PhD thesis, University of Toronto, 1995

    1995

  27. [35]

    MCMC using Hamiltonian dynamics

    Radford M Neal et al. MCMC using Hamiltonian dynamics. Handbook of Markov chain Monte Carlo, 2(11):2, 2011

    2011

  28. [36]

    Randomized prior functions for deep rein- forcement learning

    Ian Osband, John Aslanides, and Albin Cassirer. Randomized prior functions for deep rein- forcement learning. In Advances in Neural Information Processing Systems (NeurIPS), pages 8617–8629, 2018

    2018

  29. [37]

    Can you trust your model’s uncertainty? Evaluating predictive uncertainty under dataset shift

    Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, D Sculley, Sebastian Nowozin, Joshua V Dillon, Balaji Lakshminarayanan, and Jasper Snoek. Can you trust your model’s uncertainty? Evaluating predictive uncertainty under dataset shift. In Advances in Neural Information Process...

    2019

  30. [38]

    A scalable Laplace approximation for neural networks

    Hippolyt Ritter, Aleksandar Botev, and David Barber. A scalable Laplace approximation for neural networks. In International Conference on Learning Representations (ICLR), 2018

    2018

  31. [39]

    Deep in- formation propagation

    Samuel S Schoenholz, Justin Gilmer, Surya Ganguli, and Jascha Sohl-Dickstein. Deep in- formation propagation. In International Conference on Learning Representations (ICLR) , 2017

    2017

  32. [40]

    Active learning literature survey

    Burr Settles. Active learning literature survey. Technical report, University of Wisconsin- Madison Department of Computer Sciences, 2009

    2009

  33. [41]

    Functional variational Bayesian neural networks

    Shengyang Sun, Guodong Zhang, Jiaxin Shi, and Roger Grosse. Functional variational Bayesian neural networks. In International Conference on Learning Representations (ICLR), 2019

    2019

  34. [42]

    Improving and understanding variational continual learning

    Siddharth Swaroop, Cuong V Nguyen, Thang D Bui, and Richard E Turner. Improving and understanding variational continual learning. arXiv preprint arXiv:1905.02099, 2019

    1905 arXiv

  35. [43]

    Neural network ensembles and variational inference revisited

    Marcin B Tomczak, Siddharth Swaroop, and Richard E Turner. Neural network ensembles and variational inference revisited. In 1st Symposium on Advances in Approximate Bayesian Inference (AABI), 2018

    2018

  36. [44]

    Visualizing data using t-SNE.Journal of Machine Learning Research (JMLR), 9:2579–2605, 2008

    Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-SNE.Journal of Machine Learning Research (JMLR), 9:2579–2605, 2008

    2008

  37. [45]

    Bayesian learning via stochastic gradient Langevin dynamics

    Max Welling and Yee W Teh. Bayesian learning via stochastic gradient Langevin dynamics. In Proceedings of the 28th international conference on machine learning (ICML-11) , pages 681–688, 2011

    2011

  38. [46]

    How good is the Bayes posterior in deep neural networks really? In International Conference on Machine Learning (ICML), 2020

    Florian Wenzel, Kevin Roth, Bastiaan S Veeling, Jakub´Swi ˛ atkowski, Linh Tran, Stephan Mandt, Jasper Snoek, Tim Salimans, Rodolphe Jenatton, and Sebastian Nowozin. How good is the Bayes posterior in deep neural networks really? In International Conference on Machine Learning...

    2020

  39. [47]

    Cyclical stochastic gradient MCMC for Bayesian deep learning

    Ruqi Zhang, Chunyuan Li, Jianyi Zhang, Changyou Chen, and Andrew Gordon Wilson. Cyclical stochastic gradient MCMC for Bayesian deep learning. InInternational Conference on Learning Representations (ICLR), 2020. 12 A In-between Uncertainty in Other Regions of Input Space In thi...

    2020

  40. [48]

    2 r+ √ r2+8/π ≤ √ π/2,

  41. [49]

    Plugging these into equation (10), I(r)≤− 1 + √ π/2√ 2π + 1√ 2πe + 1 2π =−1 2 + 1√ 2πe + 1 2π≈− 0.098< 0

    2r r+ √ r2+4≥ 0. Plugging these into equation (10), I(r)≤− 1 + √ π/2√ 2π + 1√ 2πe + 1 2π =−1 2 + 1√ 2πe + 1 2π≈− 0.098< 0. C.3 Proof of Lemma 3 Proof. Recall that T1 ={f≥ 0 :∀0≤b<a,f (a)≥f(−a) andf(b)≤f(a)} and T2 ={f≥ 0 :∀a<b ≤ 0,f (a)≥f(−a) andf(b)≤f(a)}. First, note thatT1,...

  42. [50]

    Finally, applying lemma 5, we have V[ψ(f (k) θ (x))]<ϵ for 1≤k≤K

    Then V[f (k) θ (x)]≤I 2 ϵ 2I 2 +σ′2 <ϵ for 1≤k≤K. Finally, applying lemma 5, we have V[ψ(f (k) θ (x))]<ϵ for 1≤k≤K. The following lemma states that by setting the standard deviation of the weights to be sufficiently small, we can with high probability make the sampled BNN outpu...

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.