Pith. sign in

REVIEW 3 major objections 4 minor 1 cited by

Convergence Rates of Variational Inference in Sparse Deep Learning

T0 review · 3 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Sparse spike-and-slab variational inference for deep ReLU networks converges at the near-minimax rate of exact Bayesian inference.

desk verdict First real frequentist bound for variational inference in deep networks, but Theorem 2's uniform-family proof has a genuine gap; the Gaussian appendix goes through, so the paper is repairable and worth refereeing. read the letter →

arxiv 1908.04847 v2 pith:RXGKAWIV submitted 2019-08-09 math.ST cs.LGstat.MLstat.TH

Badr-Eddine Chérief-Abdellatif This is my paper · ORCID
classification math.STcs.LGstat.MLstat.TH MSC 62G0862F1562G2068T07
keywords variationalinferencesparsespike-and-slabpriorsdeepReLUnetworksnonparametricregressionHöldersmoothfunctionsminimaxratesELBOmodelselectionposteriorconcentration
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Variational inference is the workhorse approximation when Bayesian posteriors are intractable, and this paper asks whether that approximation destroys the statistical guarantees of Bayesian deep learning. The author considers nonparametric regression with a deep ReLU network, a spike-and-slab prior, and a sparse spike-and-slab variational family, and proves an oracle inequality: the expected squared $L^2$ error of the variational approximation is at most the best approximation error of the network class plus a rate term $r_n^{S,L,D}$ that grows like $(S/n)\log(nL/S)$ plus $(SL/n)\log D$. For $\beta$-Hölder regression functions with $0<\beta

What carries the argument

The central object is the sparse spike-and-slab variational family $\mathcal{F}_{S,L,D}$: distributions that first pick a sparsity pattern with exactly $S$ active weights and then place independent slab distributions $U([l_t,u_t])$ on the active coordinates and point mass at $0$ on the rest, with $-1 \le l_t \le u_t \le 1$. The load-bearing step is the explicit $q_n^*$ built from the best network $\theta_*$, with slab width $s_n$ chosen so that the supremum-norm distance between $f_\theta$ and $f_{\theta_*}$ is controlled by $S/(2n)$; this makes the variational family rich enough to satisfy the same prior-mass condition that drives posterior concentration, and the rate $r_n^{S,L,D}$ emerges as the local covering entropy of the network class. The oracle inequality then turns that rate into a bound on the generalization error of the variational approximation.

What would settle it

Choose a target $f_0$ that is itself a ReLU network with one active weight $\theta_{*,t}=3/2$, all other weights zero, and $B=2$. The proof of Theorem 2 builds $q_n^*$ with an active slab $U([\theta_{*,t}-s_n,\theta_{*,t}+s_n])$, but every distribution in $\mathcal{F}_{S,L,D}$ has slabs contained in $[-1,1]$, so the constructed $q_n^*$ is not an admissible minimizer in the variational problem. Evaluating the right-hand side of the oracle inequality for this case, analytically or by direct numerical minimization of the ELBO, would settle whether the stated parameter range in Corollary 3 must be shrunk.

Watch

Extended reading notes

Core claim

At the paper's core is a nonasymptotic oracle inequality for the variational approximation $\tilde{\pi}_{n,\alpha}$ of the $\alpha$-tempered posterior. Under a 1-Lipschitz activation with $|\rho(x)| \le |x|$ and any network structure $(S,L,D)$, the generalization error satisfies $$\mathbb{E}\left[\int \lVert f_\$\theta$ - f_0\$rVert_2^{2}$ \,d\tilde{\pi}_{n,\$\alpha$}\right] \le \frac{2}{1-\$\alpha$}\inf_{\theta_*} \lVert f_{\theta_*} - f_0\$rVert_2^{2}$ + \frac{2}{1-\$\alpha$}\left(1+\frac{\$sigma^{2}$}{\$\alpha$}\right) $r_n^{{S,L,D}}$,$$ where $r_n^{S,L,D}$ is a sum of three covering-entropy terms. The proof constructs, inside the sparse spike-and-slab family, a distribution $q_n^*$ whose active slabs are centered at the best network's weights, with width chosen so that $\int \lVert f_\theta - f_{\theta_*}\rVert_2^2 \,dq_n^* \le S/(2n)$ and $\mathrm{KL}(q_n^* \| \pi) \le n r_n^{S,L,D}$; this extended prior-mass condition is what transfers posterior concentration to the variational projection. With the ReLU approximation result for Hölder functions and the architecture of Theorem 1, the bias term is of order $n^{-2\beta/(2\beta+d)}\log^2 n$ and the rate term matches it, giving the minimax rate in Corollary 3. The same machinery, augmented by an ELBO-gap term, yields the optimization-aware Theorem 4, and a penalized ELBO over models yields the adaptive Theorem 5.

Load-bearing premise

The load-bearing premise is that the variational family's active weights can be centered at any coefficient of the best approximating network, but the family only allows slab intervals inside $[-1,1]$ while network weights are only assumed bounded by $B \ge 2$, so if even one target weight lies outside $[-1,1]$ the proof's approximating distribution is not a member of the family it optimizes over.

Editorial extensions

If this is right

  • If Theorem 2 is correct, any 1-Lipschitz activation with $|\rho(x)| \le |x|$, in particular ReLU, inherits a nonasymptotic oracle bound, so the statistical guarantee for variational deep learning is not limited to asymptotic or exactly solvable cases.
  • Corollary 3 implies that a practitioner using the stated architecture can report uncertainty from the variational approximation with the same near-minimax frequentist guarantee as the exact tempered posterior, provided the optimization reaches the ELBO maximizer.
  • Theorem 4 shows that an optimization algorithm only needs to drive the expected ELBO gap below $O(S \log n)$ to preserve the ideal variational rate, separating statistical error from optimization error in a usable way.
  • Theorem 5 says the penalized ELBO selects depth, width, and sparsity adaptively, so the user does not need to know the smoothness level $\beta$ to obtain a near-minimax rate for Hölder targets.
  • The bias-variance split in the oracle inequality gives a concrete architecture-selection rule: minimize the right-hand side over $(S,L,D)$, which in the Hölder case recovers the network sizes used for the minimax rate.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The proof's restriction of variational slabs to $[-1,1]$ suggests that the Gaussian variant presented in an appendix is not merely a technical alternative but the natural repair: on unbounded parameter sets, target coefficients outside $[-1,1]$ do not push the constructed $q_n^*$ outside the variational family, so the oracle inequality is established for the stated $B\ge 2$ range only in the Gauss
  • A testable extension would be to turn the explicit constants in Theorem 2 into a finite-sample sample-size calculator for variational inference in deep networks, something the paper does not provide.
  • Because the ELBO selection result keeps the tempering parameter $\alpha$ fixed, an analogous adaptive choice of $\alpha$ would connect the architecture-selection framework to robustness-to-misspecification questions for tempered posteriors.
  • The bound's linear dependence on the number of layers $L$ and sparsity $S$ suggests a practical design rule: for a fixed target smoothness, increasing depth costs only linearly while width enters logarithmically, so deeper-but-sparser networks should be preferred; this prediction could be checked empirically against selected architectures.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper studies variational inference for sparse spike-and-slab approximations of tempered posteriors in deep ReLU networks, in a nonparametric regression model with Gaussian noise. The main results are: (Theorem 2) a PAC-Bayesian oracle inequality for the expected squared-L2 generalization error of the variational approximation, with the explicit rate r^{S,L,D}_n; (Corollary 3) consistency and near-minimax concentration for beta-Hölder regression functions when the architecture is chosen as in Rockova and Polson; (Theorem 4) a generalization bound that includes an ELBO optimization error; and (Theorem 5) an adaptive model-selection guarantee for an architecture chosen by penalized ELBO maximization. Proofs are deferred to appendices and rely on an extended prior-mass condition.

Significance. If established, the results would be a valuable contribution: they would provide the first nonasymptotic generalization bounds for variational inference in deep Bayesian networks, explicitly connect the bias-variance tradeoff to architecture choice, and show that ELBO-based model selection does not overfit. The paper is also useful in making the extended prior-mass condition explicit and in giving concrete rates with logarithmic factors. The proof strategy is transparent, and the Gaussian variant in Appendix G provides a useful robustness check. However, the central oracle inequality currently has a support-membership gap (see Major Comment 1), so the advertised claims are not yet proven for the stated variational family.

major comments (3)
  1. [Section 2.4, Appendix B (Steps 2–3), Eq. (3)] The variational family F_{S,L,D} is defined with slab intervals [l_t,u_t] constrained by -1 ≤ l_t ≤ u_t ≤ 1. In the proof of Theorem 2, the proposed q*_n places each active coefficient on U([θ*_t − s_n, θ*_t + s_n]), with θ*_t only known to satisfy |θ*_t| ≤ B for B ≥ 2. For any θ*_t with |θ*_t| > 1 − s_n, this interval is not within [-1,1], so q*_n is not in F_{S,L,D}; the bound (4) is therefore only proved for a strict subset of the θ* over which the infimum in (3) is taken. Moreover, if |θ*_t| is within s_n of B, the interval is not contained in the prior support [-B,B] and the KL term KL(q*_n||π) in (5) is infinite. Appendix G repairs the analogous construction only for the Gaussian spike-and-slab family; it does not establish Theorem 2 for the uniform family stated in Section 2.4. Since Theorem 2 is the load-bearing result used in Corollary 3 and Theorems 4–5, this gap affects the paper's main claims.
  2. [Appendix B, Step 3, Inequality (5)] The computation of KL(q*_n||π) assumes that the intervals [θ*_t − s_n, θ*_t + s_n] are contained in the prior support [-B,B]. This is not guaranteed for parameters with |θ*_t| close to B, and a clipped interval would change the mean of the slab, requiring a new estimate of ∫||fθ − fθ*||²₂ q*_n(dθ). The manuscript does not provide such an argument, so Inequality (5) is not established for the full parameter range stated in Theorem 2.
  3. [Section 4, Theorem 5] Theorem 5 is proven by applying the same q*_n construction inside each model F_{S,L,D}. Consequently, the support-membership problem in Theorem 2 also invalidates the current proof of the adaptive ELBO model-selection bound. A revision should either restrict the infimum in Theorem 2 to parameters with |θ*_t| ≤ 1 − s_n and |θ*_t| ≤ B − s_n, or change the variational family (for example, to the Gaussian family of Appendix G) and make that the main result.
minor comments (4)
  1. [Assumption 3.1] There are typos: '1-Lispchitz' should be '1-Lipschitz' and 'aboluste value' should be 'absolute value'.
  2. [Section 2.4] The notation S^S_T is used without a definition; it should be defined as the set of T-dimensional binary vectors with exactly S nonzero entries.
  3. [Appendix B, Step 2] The displayed definition writes 'q*_n(θ)' although q*_n is a probability distribution; the notation q*_n(dθ) would be more accurate.
  4. [Appendix C, Eq. (8)] The displayed inequality writes inf_{θ*} ||fθ* − f0||²_∞ directly after citing Theorem 2, whereas Theorem 2 has the L2 norm; the inequality is still valid because ||f||_2 ≤ ||f||_∞, but the step should be made explicit.

Circularity Check

1 steps flagged · score 2.0 of 10

Central variational-rate derivation is self-contained; only the ELBO model-selection theorem leans on the author's prior work.

  1. self citation load bearing [Appendix F, proof of Theorem 5 (architecture selection via ELBO maximization)]
    "We start from the last inequality obtained in the proof of Theorem 3 in Cherief-Abdellatif (2019) that provides an upper bound in α-Rényi divergence for the ELBO model selection framework."

    Theorem 5's model-selection oracle inequality is not derived in the present paper; its proof imports the key bound from the author's own earlier paper Cherief-Abdellatif (2019). The remaining steps only insert the r^{S,L,D}_n rate from Theorem 2. Hence the claim that ELBO architecture selection 'does not overfit and adaptively achieves the optimal rate' rests on a self-citation rather than a self-contained derivation. This is load-bearing for the model-selection contribution, although the cited inequality is a published, parameter-free model-selection bound rather than a fitted or target-equivalent input, so the circularity is modest.

full rationale

Score 2 reflects one modest self-citation point, not a fitted-input circularity. Theorem 2's oracle inequality and Corollary 3's minimax rate are obtained from an explicit PAC-Bayes/KL computation in Appendix B: q*_n is constructed, the bias term is controlled by a layer-wise Lipschitz induction, and KL(q*_n||π) ≤ n r_n is checked by direct calculation. No parameter is fitted to data, and the rate is computed from the prior and the variational family rather than read off from the conclusion. The support-range mismatch flagged by the reader — q*_n uses intervals [θ*_t − s_n, θ*_t + s_n] that can leave the variational family's allowed [-1,1] bounds when |θ*_t| > 1, while Θ_{S,L,D} only bounds coefficients by B ≥ 2 — is a proof gap or correctness risk for the stated parameter range, not a reduction of the conclusion to an input. The main self-citation appears in Appendix F, where Theorem 5 starts from 'the last inequality' of the author's Cherief-Abdellatif (2019); this is load-bearing for the architecture-selection claim. Because the cited inequality is a published, general model-selection bound rather than a restatement of the present result, and because the central rate results (Theorems 2, 4 and Corollary 3) stand independently of it, the paper is not circular in the stronger sense.

Assumptions & free parameters 0 free parameters · 6 assumptions · 0 invented entities

No data-fitted parameters; constants B, α, σ², C_D are model inputs or from cited theorems. No new entities. The main ledger entry is the restrictive variational family, which is an ad-hoc modeling choice that drives the proof gap.

assumptions (6)
  • standard math PAC-Bayes inequality for tempered posteriors in α-Rényi divergence (Alquier and Ridgway 2017)
    Used as the starting point of the proof of Theorem 2, Appendix B first step.
  • standard math Donsker-Varadhan variational formula
    Used in the proof of Theorem 4, Appendix E.
  • domain assumption Deep ReLU approximation rates for Hölder functions (Yarotsky 2016; Rockova and Polson 2018 Lemma 5.1)
    Basis for the bias term in Corollary 3 and Theorem 1; assumes f0 is β-Hölder with 0<β<d.
  • ad hoc to paper Spike-and-slab prior and variational family F_{S,L,D} with intervals constrained to [-1,1]
    Defines the model and creates the proof gap: θ* coefficients are only bounded by B ≥ 2, so the constructed q*_n may not lie in F_{S,L,D}.
  • domain assumption i.i.d. Gaussian noise with known variance σ² and uniform input distribution
    Statistical model in Section 2.1, used throughout for likelihood and KL/Rényi computations.
  • domain assumption Tempered posterior with α in (0,1)
    Needed for the PAC-Bayes inequality; robustness to misspecification.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Convergence Rates of Variational Inference in Sparse Deep Learning." pith.science (2026). https://pith.science/paper/RXGKAWIV

@misc{pith2026190804847,
  author       = {Pith},
  title        = {Pith review of: Convergence Rates of Variational Inference in Sparse Deep Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RXGKAWIV}},
  note         = {Machine review of arXiv:1908.04847}
}
read the original abstract

Variational inference is becoming more and more popular for approximating intractable posterior distributions in Bayesian statistics and machine learning. Meanwhile, a few recent works have provided theoretical justification and new insights on deep neural networks for estimating smooth functions in usual settings such as nonparametric regression. In this paper, we show that variational inference for sparse deep learning retains the same generalization properties than exact Bayesian inference. In particular, we highlight the connection between estimation and approximation theories via the classical bias-variance trade-off and show that it leads to near-minimax rates of convergence for H\"older smooth functions. Additionally, we show that the model selection framework over the neural network architecture via ELBO maximization does not overfit and adaptively achieves the optimal rate of convergence.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Can Bayesian Neural Networks Make Confident Predictions?

    stat.ML 2025-01 reject novelty 5.0 of 10

    Under a discrete hidden-layer prior, a Bayesian neural network's predictive distribution is a Gaussian mixture, and parameters with identical training error can produce distinct predictive modes, so unimodal approxima...

Reference graph

Works this paper leans on

78 extracted references · 49 canonical work pages · cited by 1 Pith paper

  1. [1]

    A convergence theory for deep learning via over-parameterization

    Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song. A convergence theory for deep learning via over-parameterization. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 242--252, Long Beach, California, USA, 09--15 Jun 2019. PM...

    work page 2019

  2. [2]

    Concentration of tempered posteriors and of their variational approximations

    P. Alquier and J. Ridgway. Concentration of tempered posteriors and of their variational approximations. arXiv preprint arXiv:1706.09293, 2017

    work page Pith review arXiv 2017

  3. [3]

    Alquier, J

    P. Alquier, J. Ridgway, and N. Chopin. On the properties of variational approximations of G ibbs posteriors. JMLR, 17 0 (239): 0 1--41, 2016

    work page 2016

  4. [4]

    Neural networks and principal component analysis: Learning from examples without local minima'', ne

    Pierre Baldi and Kurt Hornik. Neural networks and principal component analysis: Learning from examples without local minima'', ne. Neural Networks, 2: 0 53--58, 12 1989. doi:10.1016/0893-6080(89)90014-2

    doi:10.1016/0893-6080(89)90014-2 1989

  5. [5]

    Barron, a.e.: Universal approximation bounds for superpositions of a sigmoidal function

    Andrew Barron. Barron, a.e.: Universal approximation bounds for superpositions of a sigmoidal function. ieee trans. on information theory 39, 930-945. Information Theory, IEEE Transactions on, 39: 0 930 -- 945, 06 1993. doi:10.1109/18.256500

    doi:10.1109/18.256500 1993

  6. [6]

    Approximation and estimation bounds for artificial neural networks

    Andrew R Barron. Approximation and estimation bounds for artificial neural networks. Machine Learning, 14 0 (1): 0 115--133, 1994

    work page 1994

  7. [7]

    Spectrally-normalized margin bounds for neural networks

    Peter L Bartlett, Dylan J Foster, and Matus J Telgarsky. Spectrally-normalized margin bounds for neural networks. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, pages 6240--6249. Curran Associates, Inc., 2017. URL http://papers.nips.cc/paper/7204...

    work page 2017

  8. [8]

    Baum and Ted Petrie

    Leonard E. Baum and Ted Petrie. Statistical inference for probabilistic functions of finite state markov chains. Ann. Math. Statist., 37 0 (6): 0 1554--1563, 12 1966. doi:10.1214/aoms/1177699147. URL https://doi.org/10.1214/aoms/1177699147

    arXiv 1966

Show all 78 references
  1. [9]

    Behrens, N

    G. Behrens, N. Friel, and M. Hurn. Tuning tempered transitions. Statistics and computing, 22 0 (1): 0 65--78, 2012

    2012

  2. [10]

    Deep rewiring: Training very sparse deep networks

    Guillaume Bellec, David Kappel, Wolfgang Maass, and Robert Legenstein. Deep rewiring: Training very sparse deep networks. In International Conference on Learning Representations, 2018. URL https://openreview.net/forum?id=BJ_wN01C-

    2018

  3. [11]

    On the expressive power of deep architectures

    Yoshua Bengio and Olivier Delalleau. On the expressive power of deep architectures. In Proceedings of the 22Nd International Conference on Algorithmic Learning Theory, ALT'11, pages 18--36, Berlin, Heidelberg, 2011. Springer-Verlag. ISBN 978-3-642-24411-7. URL http://dl.acm.or...

    2011

  4. [12]

    Bhattacharya, D

    A. Bhattacharya, D. Pati, and Y. Yang. Bayesian fractional posteriors. arXiv preprint arXiv:1611.01125, to appear in the Annals of Statistics, 2016

    2016 arXiv

  5. [13]

    Bhattacharya, D

    A. Bhattacharya, D. Pati, and Y. Yang. On statistical optimality of variational Bayes . Proceedings of Machine Learning Research, 84 - AISTAT, 2018

    2018

  6. [14]

    Variational inference: A review for statisticians

    David M Blei, Alp Kucukelbir, and Jon D McAuliffe. Variational inference: A review for statisticians. Journal of the American Statistical Association, 112 0 (518): 0 859--877, 2017

    2017

  7. [15]

    Weight uncertainty in neural networks

    Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural networks. In Proceedings of the 32Nd International Conference on International Conference on Machine Learning - Volume 37, ICML'15, pages 1613--1622. JMLR.org, 2015. URL http...

    2015

  8. [16]

    Concentration inequalities using the entropy method

    Stéphane Boucheron, Gábor Lugosi, and Pascal Massart. Concentration inequalities using the entropy method. Ann. Probab., 31 0 (3): 0 1583--1614, 07 2003. doi:10.1214/aop/1055425791. URL https://doi.org/10.1214/aop/1055425791

    2003

  9. [17]

    Quasi- M onte C arlo variational inference

    Alexander Buchholz, Florian Wenzel, and Stephan Mandt. Quasi- M onte C arlo variational inference. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 668...

    2018

  10. [18]

    Universal boosting variational inference

    Trevor Campbell and Xinglong Li. Universal boosting variational inference. volume arXiv:1903.05220, 2019

    1903 arXiv

  11. [19]

    Bayesian linear regression with sparse priors

    Ismaël Castillo, Johannes Schmidt-Hieber, and Aad van der Vaart. Bayesian linear regression with sparse priors. Ann. Statist., 43 0 (5): 0 1986--2018, 10 2015. doi:10.1214/15-AOS1334. URL https://doi.org/10.1214/15-AOS1334

    1986 doi

  12. [20]

    O. Catoni. PAC - B ayesian supervised classification: the thermodynamics of statistical learning . Institute of Mathematical Statistics Lecture Notes---Monograph Series, 56. Institute of Mathematical Statistics, Beachwood, OH, 2007

    2007

  13. [21]

    Ch\'erief-Abdellatif and P

    B. Ch\'erief-Abdellatif and P. Alquier. Consistency of variational bayes inference for estimation and model selection in mixtures. Electronic Journal of Statistics, 12 0 (2): 0 2995--3035, 2018. ISSN 1935-7524. doi:10.1214/18-EJS1475

    2018 doi

  14. [22]

    Ch \'e rief-Abdellatif, P

    B.-E. Ch \'e rief-Abdellatif, P. Alquier, and M.E. Khan. A generalization bound for online variational inference. Preprint arXiv:1904.03920v1, 2019

    1904 arXiv

  15. [23]

    Consistency of elbo maximization for model selection

    Badr-Eddine Cherief-Abdellatif. Consistency of elbo maximization for model selection. In Francisco Ruiz, Cheng Zhang, Dawen Liang, and Thang Bui, editors, Proceedings of The 1st Symposium on Advances in Approximate Bayesian Inference, volume 96 of Proceedings of Machine Learni...

    2019

  16. [24]

    G. Cybenko. Approximation by superpositions of a sigmoidal function . Mathematics of Control, Signals, and Systems (MCSS), 2 0 (4): 0 303--314, December 1989. ISSN 0932-4194. doi:10.1007/BF02551274. URL http://dx.doi.org/10.1007/BF02551274

    1989 doi

  17. [25]

    Doucet and A

    A. Doucet and A. Johansen. A tutorial on particle filtering and smoothing: Fifteen years later. Handbook of Nonlinear Filtering, 12, 01 2009

    2009

  18. [26]

    Gradient descent finds global minima of deep neural networks

    Simon Du, Jason Lee, Haochuan Li, Liwei Wang, and Xiyu Zhai. Gradient descent finds global minima of deep neural networks. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of...

    2019

  19. [27]

    Uncertainty in Deep Learning

    Yarin Gal. Uncertainty in Deep Learning. PhD thesis, University of Cambridge, 2016

    2016

  20. [28]

    Ghosh, and Aad W

    Subhashis Ghosal, Jayanta K. Ghosh, and Aad W. van der Vaart. Convergence rates of posterior distributions. Ann. Statist., 28 0 (2): 0 500--531, 04 2000. doi:10.1214/aos/1016218228. URL https://doi.org/10.1214/aos/1016218228

  21. [29]

    Generative adversarial nets

    Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger, editors, Advances in Neural Information ...

    2014

  22. [30]

    Deep Learning

    Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press, 2016. http://www.deeplearningbook.org

    2016

  23. [31]

    Practical variational inference for neural networks

    Alex Graves. Practical variational inference for neural networks. In J. Shawe-Taylor, R. S. Zemel, P. L. Bartlett, F. Pereira, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems 24, pages 2348--2356. Curran Associates, Inc., 2011. URL http://paper...

    2011

  24. [32]

    Deep neural network approximation theory, 01 2019

    Philipp Grohs, Dmytro Perekrestenko, Dennis Elbrächter, and Helmut Bölcskei. Deep neural network approximation theory, 01 2019

    2019

  25. [33]

    P. D. Gr \"u nwald and T. Van Ommen. Inconsistency of B ayesian inference for misspecified linear models, and a proposal for repairing it. Bayesian Analysis, 12 0 (4): 0 1069--1103, 2017

    2017

  26. [34]

    B. Guedj. A primer on pac-bayesian learning. arXiv preprint arXiv:1901.05353, 2019

    1901 arXiv

  27. [35]

    On the minimax optimality and superiority of deep neural network learning over sparse parameter spaces

    Satoshi Hayakawa and Taiji Suzuki. On the minimax optimality and superiority of deep neural network learning over sparse parameter spaces. arXiv preprint arXiv:1905.09195, 2019

    1905 arXiv

  28. [36]

    Hinton and Drew van Camp

    Geoffrey E. Hinton and Drew van Camp. Keeping the neural networks simple by minimizing the description length of the weights. In Proceedings of the Sixth Annual Conference on Computational Learning Theory, COLT '93, pages 5--13, New York, NY, USA, 1993. ACM. ISBN 0-89791-611-5...

    1993

  29. [37]

    M. D. Hoffman, D. M. Blei, C. Wang, and J. Paisley. Stochastic variational inference. The Journal of Machine Learning Research, 14 0 (1): 0 1303--1347, 2013

    2013

  30. [38]

    Huggins, Trevor Campbell, Mikolaj Kasprzak, and Tamara Broderick

    Jonathan H. Huggins, Trevor Campbell, Mikolaj Kasprzak, and Tamara Broderick. Practical bounds on the error of bayesian posterior approximations: A nonasymptotic approach. ArXiv, abs/1809.09505, 2018

    2018 arXiv

  31. [39]

    Deep neural networks learn non-smooth functions effectively

    Masaaki Imaizumi and Kenji Fukumizu. Deep neural networks learn non-smooth functions effectively. In Kamalika Chaudhuri and Masashi Sugiyama, editors, Proceedings of Machine Learning Research, volume 89 of Proceedings of Machine Learning Research, pages 869--878. PMLR, 16--18 ...

    2019

  32. [40]

    Jaiswal, V

    P. Jaiswal, V. A. Rao, and H. Honnappa. Asymptotic consistency of -r\'enyi-approximate posteriors. Preprint arXiv:1902.01902, 2019 a

    1902 arXiv

  33. [41]

    Prateek Jaiswal, Harsha Honnappa, and Vinayak A. Rao. Risk-sensitive variational bayes: Formulations and bounds. volume arXiv:1906.01235, 2019 b

    1906 arXiv

  34. [42]

    M. I. Jordan, Z. Ghahramani, T. S. Jaakkola, and L. K. Saul. An introduction to variational methods for graphical models. Machine Learning, 37: 0 183--233, 1999

    1999

  35. [43]

    Deep learning without poor local minima

    Kenji Kawaguchi. Deep learning without poor local minima. In D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems 29, pages 586--594. Curran Associates, Inc., 2016. URL http://papers.nips.cc/paper/6112-deep...

    2016

  36. [44]

    Effect of depth and width on local minima in deep learning

    Kenji Kawaguchi, Jiaoyang Huang, and Leslie Pack Kaelbling. Effect of depth and width on local minima in deep learning. Neural Computation, 31 0 (6): 0 1462--1498, 2019

    2019

  37. [45]

    Fast and scalable B ayesian deep learning by weight-perturbation in A dam

    Mohammad Khan, Didrik Nielsen, Voot Tangkaratt, Wu Lin, Yarin Gal, and Akash Srivastava. Fast and scalable B ayesian deep learning by weight-perturbation in A dam. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning,...

    2018

  38. [46]

    Auto-encoding variational bayes

    Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013

    2013 arXiv

  39. [47]

    Gradient-based learning applied to document recognition

    Yann Lecun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. In Proceedings of the IEEE, pages 2278--2324, 1998

    1998

  40. [48]

    Deep learning

    Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature, 521 0 (7553): 0 436--444, 5 2015. ISSN 0028-0836. doi:10.1038/nature14539

    2015 doi

  41. [49]

    Christos Louizos, Max Welling, and Diederik P. Kingma. Learning sparse neural networks through l_0 -regularization. In International Conference on Learning Representations, 2018. URL https://openreview.net/forum?id=H1Y8hhg0b

    2018

  42. [50]

    David J. C. MacKay. A practical bayesian framework for backpropagation networks. Neural Computation, 4 0 (3): 0 448--472, 1992 a . doi:10.1162/neco.1992.4.3.448. URL https://doi.org/10.1162/neco.1992.4.3.448

    1992 doi

  43. [51]

    David J. C. MacKay. Bayesian methods for adaptive models. PhD thesis, California Institute of Technology, 1992 b

    1992

  44. [52]

    T. P. Minka. Expectation propagation for approximate bayesian inference. In Proceedings of the 17th Conference in Uncertainty in Artificial Intelligence, UAI '01, pages 362--369, San Francisco, CA, USA, 2001. Morgan Kaufmann Publishers Inc. ISBN 1-55860-800-1. URL http://dl.ac...

    2001

  45. [53]

    Slang: Fast structured covariance approximations for bayesian deep learning with natural gradient

    Aaron Mishkin, Frederik Kunstner, Didrik Nielsen, Mark Schmidt, and Mohammad Emtiyaz Khan. Slang: Fast structured covariance approximations for bayesian deep learning with natural gradient. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, e...

    2018

  46. [54]

    Radford. M. Neal. Bayesian learning for neural networks. PhD thesis, University of Toronto, 1995

    1995

  47. [55]

    A PAC -bayesian approach to spectrally-normalized margin bounds for neural networks

    Behnam Neyshabur, Srinadh Bhojanapalli, and Nathan Srebro. A PAC -bayesian approach to spectrally-normalized margin bounds for neural networks. In International Conference on Learning Representations, 2018. URL https://openreview.net/forum?id=Skz_WfbCZ

    2018

  48. [56]

    On the loss landscape of a class of deep neural networks with no bad local valleys

    Quynh Nguyen, Mahesh Chandra Mukkamala, and Matthias Hein. On the loss landscape of a class of deep neural networks with no bad local valleys. In International Conference on Learning Representations, 2019. URL https://openreview.net/forum?id=HJgXsjA5tQ

    2019

  49. [57]

    The variational gaussian approximation revisited

    Manfred Opper and Cedric Archambeau. The variational gaussian approximation revisited. Neural computation, 21: 0 786--92, 10 2008. doi:10.1162/neco.2008.08-07-592

    2008 doi

  50. [58]

    Turner, Rio Yokota, and Mohammad Emtiyaz Khan

    Kazuki Osawa, Siddharth Swaroop, Anirudh Jain, Runa Eschenhagen, Richard E. Turner, Rio Yokota, and Mohammad Emtiyaz Khan. Practical deep learning with bayesian principles, 2019. URL http://arxiv.org/abs/1906.02506. cite arxiv:1906.02506Comment: Under review

    2019 arXiv

  51. [59]

    Optimal approximation of piecewise smooth functions using deep relu neural networks

    Philipp Petersen and Felix Voigtländer. Optimal approximation of piecewise smooth functions using deep relu neural networks. Neural Networks, 09 2017. doi:10.1016/j.neunet.2018.08.019

    2017 doi

  52. [60]

    Posterior concentration for sparse deep learning

    Veronika Rockova and nicholas Polson. Posterior concentration for sparse deep learning. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems 31, pages 930--941. Curran Associates, Inc., 2018

    2018

  53. [61]

    The power of deeper networks for expressing natural functions

    David Rolnick and Max Tegmark. The power of deeper networks for expressing natural functions. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings , 2018. URL https://openreview.net/...

    2018

  54. [62]

    Rumelhart, Geoffrey E

    David E. Rumelhart, Geoffrey E. Hinton, and Ronald J. Williams. Learning Representations by Back-propagating Errors . Nature, 323 0 (6088): 0 533--536, 1986. doi:10.1038/323533a0. URL http://www.nature.com/articles/323533a0

    1986 doi

  55. [63]

    Nonparametric regression using deep neural networks with relu activation function

    Johannes Schmidt-Hieber. Nonparametric regression using deep neural networks with relu activation function. ArXiv, arxiv:1708.06633, 2017

    2017 arXiv

  56. [64]

    Excess risk bounds for the bayes risk using variational inference in latent gaussian models

    Rishit Sheth and Roni Khardon. Excess risk bounds for the bayes risk using variational inference in latent gaussian models. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems...

    2017

  57. [65]

    Mastering the game of go without human knowledge

    David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, Yutian Chen, Timothy Lillicrap, Fan Hui, Laurent Sifre, George van den Driessche, Thore Graepel, and Demis Hassabis. Masterin...

    2017 doi

  58. [66]

    No bad local minima: Data independent training error guarantees for multilayer neural networks

    Daniel Soudry and Yair Carmon. No bad local minima: Data independent training error guarantees for multilayer neural networks. 05 2016

    2016

  59. [67]

    Dropout: A simple way to prevent neural networks from overfitting

    Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15: 0 1929--1958, 2014. URL http://jmlr.org/papers/v15/srivastava14a.html

    1929

  60. [68]

    Stanford, K Giardina, G.A

    J.A. Stanford, K Giardina, G.A. Gerhardt, Kenji Fukumizu, and Shun-ichi Amari. Local minima and plateaus in hierarchical structures of multilayer perceptrons. Neural Networks, 13, 05 2000. doi:10.1016/S0893-6080(00)00009-5

    doi

  61. [69]

    Fast generalization error bound of deep learning from a kernel perspective

    Taiji Suzuki. Fast generalization error bound of deep learning from a kernel perspective. In Amos Storkey and Fernando Perez-Cruz, editors, Proceedings of the Twenty-First International Conference on Artificial Intelligence and Statistics, volume 84 of Proceedings of Machine L...

    2018

  62. [70]

    Adaptivity of deep re LU network for learning in besov and mixed smooth besov spaces: optimal rate and curse of dimensionality

    Taiji Suzuki. Adaptivity of deep re LU network for learning in besov and mixed smooth besov spaces: optimal rate and curse of dimensionality. In International Conference on Learning Representations, 2019. URL https://openreview.net/forum?id=H1ebTsActm

    2019

  63. [71]

    Titsias and Miguel L\' a zaro-Gredilla

    Michalis K. Titsias and Miguel L\' a zaro-Gredilla. Spike and slab variational inference for multi-task and multiple kernel learning. In J. Shawe-Taylor, R. S. Zemel, P. L. Bartlett, F. Pereira, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems 2...

    2011

  64. [72]

    Variational sparse coding, 2019

    Francesco Tonolini, Bjorn Sand Jensen, and Roderick Murray-Smith. Variational sparse coding, 2019. URL https://openreview.net/forum?id=SkeJ6iR9Km

    2019

  65. [73]

    Tsybakov

    Alexandre B. Tsybakov. Introduction to Nonparametric Estimation. Springer Publishing Company, Incorporated, 1st edition, 2008. ISBN 0387790519, 9780387790510

    2008

  66. [74]

    Understanding priors in B ayesian neural networks at the unit level

    Mariia Vladimirova, Jakob Verbeek, Pablo Mesejo, and Julyan Arbel. Understanding priors in B ayesian neural networks at the unit level. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of P...

    2019

  67. [75]

    Wang and D

    Y. Wang and D. M. Blei. Frequentist consistency of variational B ayes. Journal of the American Statistical Association (to appear), 2018

    2018

  68. [76]

    Error bounds for approximations with deep relu networks

    Dmitry Yarotsky. Error bounds for approximations with deep relu networks. Neural Networks, 94, 10 2016. doi:10.1016/j.neunet.2017.07.002

    2016 doi

  69. [77]

    Understanding deep learning requires rethinking generalization

    Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. 2017. URL https://arxiv.org/abs/1611.03530

    2017 arXiv

  70. [78]

    Zhang and C

    F. Zhang and C. Gao. Convergence rates of variational posterior distributions. arXiv preprint arXiv:1712.02519v1, 2017

    2017 arXiv

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.