{"id":"5d53ab25-d309-45a6-9c58-0ac159a26305","arxiv_id":"1908.04847","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Variational inference for sparse deep ReLU networks achieves near-minimax rates for Hölder regression functions, matching exact Bayesian inference.","lead":"This paper proves that variational approximations of Bayesian neural networks, with spike-and-slab sparsity, converge at the same near-minimax rates as the exact Bayesian posterior when estimating smooth functions. It also shows that selecting the network architecture by maximizing the ELBO does not overfit and reaches the optimal rate adaptively.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 2's oracle inequality is unproved for parameters with |θ*_t|>1 because the constructed q*_n uses intervals [θ*_t−s_n, θ*_t+s_n] that may fall outside the variational family's allowed [-1,1] bounds.","rationale":"The reader's identified mismatch is real and directly affects the proof of the main oracle inequality. Section 2.4 fixes the variational interval endpoints to [-1,1], while Section 2.2 allows network weights up to B ≥ 2. Appendix B constructs q*_n with intervals centered at θ*_t; for θ* with |θ*_t| > 1, q*_n is outside F_{S,L,D}. Because the infimum in (3) is over all θ* in Θ_{S,L,D}, the bound is not established for the stated theorem. This is an internal consistency issue in the proof, not a disagreement with the consensus. It is load-bearing because the paper's variational-inference guarantee rests on this oracle inequality; Corollary 3 inherits the problem unless the approximation theorem produces weights in [-1,1], which is not shown. The Gaussian variant in Appendix G sidesteps the issue by moving to unbounded parameter sets, confirming that a repair exists but is absent for the uniform family. The concrete check of q*_n membership for a weight 1.5 with B=2 would settle whether the concern lands. No objection to the paper's overall approach or novelty; conditional acceptance is appropriate pending this fix, so the reader's verdict should remain unchanged.","tokens_in":31150,"tokens_out":8268,"duration_ms":83047,"concrete_test":"Re-derive Appendix B Step 2 while tracking the constraint l_t,u_t ∈ [-1,1]. Concretely, set d=1, L=3, D=1, B=2, and take θ* with a single active weight θ*_1 = 1.5 and all other coefficients zero. Using the proof's definition of s_n, check whether [θ*_1 − s_n, θ*_1 + s_n] ⊆ [-1,1]. If not, then q*_n ∉ F_{S,L,D}, which settles that the oracle inequality (3) is not proven for this θ*. Repeat for θ*_1 = 1.9. If clipping the interval to [-1,1] instead is proposed, verify that the extra bias term in (4) is controlled at the same rate; otherwise Theorem 2 requires restatement with θ* restricted to [-1,1] or with the variational family enlarged to [-B,B].","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 2.4 defines the variational family F_{S,L,D} with intervals [l_t,u_t] satisfying -1 ≤ l_t ≤ u_t ≤ 1. In Appendix B, Step 2, the proof of Theorem 2 constructs q*_n by setting θ_t ~ γ*_t U([θ*_t − s_n, θ*_t + s_n]) + (1−γ*_t)δ_0. For q*_n to belong to F_{S,L,D}, every active interval must satisfy [θ*_t − s_n, θ*_t + s_n] ⊆ [-1,1]. However, θ* is only assumed to lie in Θ_{S,L,D}, whose coefficients are bounded by B ≥ 2 (Section 2.2). Thus, if any θ*_t has |θ*_t| > 1, the interval is not contained in [-1,1] and q*_n is not in the variational family. For example, with B=2 and a single active weight θ*_1 = 1.5, any s_n < 0.5 gives an interval extending beyond 1. The infimum in (3) is over all θ* ∈ Θ_{S,L,D}, so the proof does not establish the stated oracle inequality. Step 3 has a related support issue: if θ*_t is within s_n of ±B, q*_n charges values outside the support [-B,B] of the prior, making KL(q*_n‖π) infinite unless the interval is clipped; clipping would break the mean-zero bias control used to prove (4). Appendix G repairs the construction for Gaussian spike-and-slab variational families, but Theorem 2 as stated for the uniform family remains unproved.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies variational inference for sparse spike-and-slab approximations of tempered posteriors in deep ReLU networks, in a nonparametric regression model with Gaussian noise. The main results are: (Theorem 2) a PAC-Bayesian oracle inequality for the expected squared-L2 generalization error of the variational approximation, with the explicit rate r^{S,L,D}_n; (Corollary 3) consistency and near-minimax concentration for beta-Hölder regression functions when the architecture is chosen as in Rockova and Polson; (Theorem 4) a generalization bound that includes an ELBO optimization error; and (Theorem 5) an adaptive model-selection guarantee for an architecture chosen by penalized ELBO maximization. Proofs are deferred to appendices and rely on an extended prior-mass condition.","tokens_in":31529,"tokens_out":8441,"duration_ms":88971,"significance":"If established, the results would be a valuable contribution: they would provide the first nonasymptotic generalization bounds for variational inference in deep Bayesian networks, explicitly connect the bias-variance tradeoff to architecture choice, and show that ELBO-based model selection does not overfit. The paper is also useful in making the extended prior-mass condition explicit and in giving concrete rates with logarithmic factors. The proof strategy is transparent, and the Gaussian variant in Appendix G provides a useful robustness check. However, the central oracle inequality currently has a support-membership gap (see Major Comment 1), so the advertised claims are not yet proven for the stated variational family.","major_comments":[{"comment":"The variational family F_{S,L,D} is defined with slab intervals [l_t,u_t] constrained by -1 ≤ l_t ≤ u_t ≤ 1. In the proof of Theorem 2, the proposed q*_n places each active coefficient on U([θ*_t − s_n, θ*_t + s_n]), with θ*_t only known to satisfy |θ*_t| ≤ B for B ≥ 2. For any θ*_t with |θ*_t| > 1 − s_n, this interval is not within [-1,1], so q*_n is not in F_{S,L,D}; the bound (4) is therefore only proved for a strict subset of the θ* over which the infimum in (3) is taken. Moreover, if |θ*_t| is within s_n of B, the interval is not contained in the prior support [-B,B] and the KL term KL(q*_n||π) in (5) is infinite. Appendix G repairs the analogous construction only for the Gaussian spike-and-slab family; it does not establish Theorem 2 for the uniform family stated in Section 2.4. Since Theorem 2 is the load-bearing result used in Corollary 3 and Theorems 4–5, this gap affects the paper's main claims.","section":"Section 2.4, Appendix B (Steps 2–3), Eq. (3)"},{"comment":"The computation of KL(q*_n||π) assumes that the intervals [θ*_t − s_n, θ*_t + s_n] are contained in the prior support [-B,B]. This is not guaranteed for parameters with |θ*_t| close to B, and a clipped interval would change the mean of the slab, requiring a new estimate of ∫||fθ − fθ*||²₂ q*_n(dθ). The manuscript does not provide such an argument, so Inequality (5) is not established for the full parameter range stated in Theorem 2.","section":"Appendix B, Step 3, Inequality (5)"},{"comment":"Theorem 5 is proven by applying the same q*_n construction inside each model F_{S,L,D}. Consequently, the support-membership problem in Theorem 2 also invalidates the current proof of the adaptive ELBO model-selection bound. A revision should either restrict the infimum in Theorem 2 to parameters with |θ*_t| ≤ 1 − s_n and |θ*_t| ≤ B − s_n, or change the variational family (for example, to the Gaussian family of Appendix G) and make that the main result.","section":"Section 4, Theorem 5"}],"minor_comments":[{"comment":"There are typos: '1-Lispchitz' should be '1-Lipschitz' and 'aboluste value' should be 'absolute value'.","section":"Assumption 3.1"},{"comment":"The notation S^S_T is used without a definition; it should be defined as the set of T-dimensional binary vectors with exactly S nonzero entries.","section":"Section 2.4"},{"comment":"The displayed definition writes 'q*_n(θ)' although q*_n is a probability distribution; the notation q*_n(dθ) would be more accurate.","section":"Appendix B, Step 2"},{"comment":"The displayed inequality writes inf_{θ*} ||fθ* − f0||²_∞ directly after citing Theorem 2, whereas Theorem 2 has the L2 norm; the inequality is still valid because ||f||_2 ≤ ||f||_∞, but the step should be made explicit.","section":"Appendix C, Eq. (8)"}],"recommendation":"major_revision","confidential_remarks":"The main gap is substantial but appears repairable: the author could either prove the construction under an explicit margin condition on θ*, or switch the main result to the Gaussian spike-and-slab family for which the support issue is resolved. I do not think rejection is warranted at this stage, but the current version should not be accepted because the central oracle inequality is not established for the stated uniform variational family."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: this is the first paper I know to give a nonasymptotic generalization bound for variational inference in deep ReLU networks. The core idea is right, but Theorem 2, as stated for the uniform spike-and-slab family, is not proved. The Gaussian version in Appendix G goes through, so the program is very likely repairable and the main insight survives.\n\nThe genuinely new piece is the combination of the PAC-Bayes oracle inequality from Alquier and Ridgway with deep ReLU approximation machinery. That gives explicit rates in S, L, D and n, a clean bias-variance split, and a principled ELBO-based model selection bound. The treatment of optimization error in Theorem 4 is a useful addition, and the discussion of architecture selection is sensible. The citation pattern is fine; the self-citation for the ELBO selection inequality is legitimate because that result is imported from the author's own previous work rather than derived from the current argument.\n\nThe soft spot is exactly the one the stress-test flags. Step 2 of Appendix B constructs a variational distribution q*_n whose active intervals are [θ*_t − s_n, θ*_t + s_n], but the variational family F_{S,L,D} restricts intervals to [−1,1]. Since θ*_t is only bounded by B ≥ 2, q*_n need not belong to the family. There is also a related support problem for coefficients within s_n of ±B: q*_n charges values outside the prior's support, making the KL infinite unless the interval is clipped, and clipping breaks the bias control used to get (4). So the oracle inequality is not established for the stated parameter range. The Gaussian construction in Appendix G avoids both problems because the prior has full support, and that appendix looks correct. The fix is either to enlarge the uniform family to [−B,B] and handle the boundary issue, or to promote the Gaussian version to the main theorem.\n\nThis is not a fatal objection to the paper's program. The flaw is in the proof, not in the concept, and the Gaussian variant already delivers the same rates. But a referee should not let Theorem 2 through without repair. The paper deserves a serious refereeing round; the ELBO and optimization sections are useful even if the uniform-family result has to be revised.\n\nRecommendation: send it to an expert referee. It is the kind of paper where a conditional accept after a solid revision is realistic.","headline":"First real frequentist bound for variational inference in deep networks, but Theorem 2's uniform-family proof has a genuine gap; the Gaussian appendix goes through, so the paper is repairable and worth refereeing.","tokens_in":32052,"tokens_out":4504,"would_cite":true,"duration_ms":47725,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62G08","62F15","62G20","68T07"],"pacs":[],"model":"deepseek-v4-flash","headline":"Sparse spike-and-slab variational inference for deep ReLU networks converges at the near-minimax rate of exact Bayesian inference.","keywords":["variational inference","sparse spike-and-slab priors","deep ReLU networks","nonparametric regression","Hölder smooth functions","minimax rates","ELBO model selection","posterior concentration"],"falsifier":"Choose a target $f_0$ that is itself a ReLU network with one active weight $\\theta_{*,t}=3/2$, all other weights zero, and $B=2$. The proof of Theorem 2 builds $q_n^*$ with an active slab $U([\\theta_{*,t}-s_n,\\theta_{*,t}+s_n])$, but every distribution in $\\mathcal{F}_{S,L,D}$ has slabs contained in $[-1,1]$, so the constructed $q_n^*$ is not an admissible minimizer in the variational problem. Evaluating the right-hand side of the oracle inequality for this case, analytically or by direct numerical minimization of the ELBO, would settle whether the stated parameter range in Corollary 3 must be shrunk.","tokens_in":30917,"feed_emoji":"🧠","tokens_out":10459,"duration_ms":93283,"temperature":0.7,"pith_summary":"Variational inference is the workhorse approximation when Bayesian posteriors are intractable, and this paper asks whether that approximation destroys the statistical guarantees of Bayesian deep learning. The author considers nonparametric regression with a deep ReLU network, a spike-and-slab prior, and a sparse spike-and-slab variational family, and proves an oracle inequality: the expected squared $L^2$ error of the variational approximation is at most the best approximation error of the network class plus a rate term $r_n^{S,L,D}$ that grows like $(S/n)\\log(nL/S)$ plus $(SL/n)\\log D$. For $\\beta$-Hölder regression functions with $0<\\beta<d$, the chosen architecture makes the variational approximation concentrate at the minimax rate $n^{-2\\beta/(2\\beta+d)}$ up to a squared logarithmic factor. The paper also shows that choosing depth, width, and sparsity by penalized ELBO maximization is adaptive, achieving the optimal rate without knowing the smoothness level $\\beta$, and that an optimization error entering only through the ELBO gap does not change the rate once that gap is small. If these results hold, variational inference for deep networks inherits the optimality of exact Bayesian inference rather than trading it away.","feed_headline":"Sparse variational Bayes hits the minimax rate for deep ReLU nets","feed_subtitle":"A new oracle inequality shows the approximate posterior converges at the same near-optimal rate as exact Bayesian inference.","key_machinery":"The central object is the sparse spike-and-slab variational family $\\mathcal{F}_{S,L,D}$: distributions that first pick a sparsity pattern with exactly $S$ active weights and then place independent slab distributions $U([l_t,u_t])$ on the active coordinates and point mass at $0$ on the rest, with $-1 \\le l_t \\le u_t \\le 1$. The load-bearing step is the explicit $q_n^*$ built from the best network $\\theta_*$, with slab width $s_n$ chosen so that the supremum-norm distance between $f_\\theta$ and $f_{\\theta_*}$ is controlled by $S/(2n)$; this makes the variational family rich enough to satisfy the same prior-mass condition that drives posterior concentration, and the rate $r_n^{S,L,D}$ emerges as the local covering entropy of the network class. The oracle inequality then turns that rate into a bound on the generalization error of the variational approximation.","core_discovery":"At the paper's core is a nonasymptotic oracle inequality for the variational approximation $\\tilde{\\pi}_{n,\\alpha}$ of the $\\alpha$-tempered posterior. Under a 1-Lipschitz activation with $|\\rho(x)| \\le |x|$ and any network structure $(S,L,D)$, the generalization error satisfies $$\\mathbb{E}\\left[\\int \\lVert f_\\$\\theta$ - f_0\\$rVert_2^{2}$ \\,d\\tilde{\\pi}_{n,\\$\\alpha$}\\right] \\le \\frac{2}{1-\\$\\alpha$}\\inf_{\\theta_*} \\lVert f_{\\theta_*} - f_0\\$rVert_2^{2}$ + \\frac{2}{1-\\$\\alpha$}\\left(1+\\frac{\\$sigma^{2}$}{\\$\\alpha$}\\right) $r_n^{{S,L,D}}$,$$ where $r_n^{S,L,D}$ is a sum of three covering-entropy terms. The proof constructs, inside the sparse spike-and-slab family, a distribution $q_n^*$ whose active slabs are centered at the best network's weights, with width chosen so that $\\int \\lVert f_\\theta - f_{\\theta_*}\\rVert_2^2 \\,dq_n^* \\le S/(2n)$ and $\\mathrm{KL}(q_n^* \\| \\pi) \\le n r_n^{S,L,D}$; this extended prior-mass condition is what transfers posterior concentration to the variational projection. With the ReLU approximation result for Hölder functions and the architecture of Theorem 1, the bias term is of order $n^{-2\\beta/(2\\beta+d)}\\log^2 n$ and the rate term matches it, giving the minimax rate in Corollary 3. The same machinery, augmented by an ELBO-gap term, yields the optimization-aware Theorem 4, and a penalized ELBO over models yields the adaptive Theorem 5.","pith_inferences":["The proof's restriction of variational slabs to $[-1,1]$ suggests that the Gaussian variant presented in an appendix is not merely a technical alternative but the natural repair: on unbounded parameter sets, target coefficients outside $[-1,1]$ do not push the constructed $q_n^*$ outside the variational family, so the oracle inequality is established for the stated $B\\ge 2$ range only in the Gauss","A testable extension would be to turn the explicit constants in Theorem 2 into a finite-sample sample-size calculator for variational inference in deep networks, something the paper does not provide.","Because the ELBO selection result keeps the tempering parameter $\\alpha$ fixed, an analogous adaptive choice of $\\alpha$ would connect the architecture-selection framework to robustness-to-misspecification questions for tempered posteriors.","The bound's linear dependence on the number of layers $L$ and sparsity $S$ suggests a practical design rule: for a fixed target smoothness, increasing depth costs only linearly while width enters logarithmically, so deeper-but-sparser networks should be preferred; this prediction could be checked empirically against selected architectures."],"forward_implications":["If Theorem 2 is correct, any 1-Lipschitz activation with $|\\rho(x)| \\le |x|$, in particular ReLU, inherits a nonasymptotic oracle bound, so the statistical guarantee for variational deep learning is not limited to asymptotic or exactly solvable cases.","Corollary 3 implies that a practitioner using the stated architecture can report uncertainty from the variational approximation with the same near-minimax frequentist guarantee as the exact tempered posterior, provided the optimization reaches the ELBO maximizer.","Theorem 4 shows that an optimization algorithm only needs to drive the expected ELBO gap below $O(S \\log n)$ to preserve the ideal variational rate, separating statistical error from optimization error in a usable way.","Theorem 5 says the penalized ELBO selects depth, width, and sparsity adaptively, so the user does not need to know the smoothness level $\\beta$ to obtain a near-minimax rate for Hölder targets.","The bias-variance split in the oracle inequality gives a concrete architecture-selection rule: minimize the right-hand side over $(S,L,D)$, which in the Hölder case recovers the network sizes used for the minimax rate."],"supporting_citations":[{"why":"Supplies the deep-ReLU architecture and the $L_\\infty$ approximation bound for $\\beta$-Hölder functions used to control the bias term in Corollary 3.","marker":"Rockova and Polson (2018)"},{"why":"Provides the tempered-posterior oracle inequality in $\\alpha$-Rényi divergence from which Theorem 2 starts.","marker":"Alquier and Ridgway (2017)"},{"why":"Defines the prior-mass condition whose rate fixes $r_n$ and links concentration to consistency.","marker":"Ghosal et al. (2000)"},{"why":"Shows the prior-mass condition is sufficient for concentration of tempered posteriors, which Theorem 2 extends to variational approximations.","marker":"Bhattacharya et al. (2016)"},{"why":"Gives the covering-entropy bound for deep ReLU networks that identifies $r_n^{S,L,D}$ with the local entropy rate.","marker":"Schmidt-Hieber (2017)"},{"why":"Provides the matching deep-ReLU rate in nonparametric regression used to compare $r_n^{S,L,D}$ with the minimax rate.","marker":"Suzuki (2019)"},{"why":"Supplies the penalized-ELBO model-selection inequality that Theorem 5 builds on.","marker":"Cherief-Abdellatif (2019)"}],"fun_headline_variants":["VI matches exact Bayes rates in sparse deep nets","Sparse VI achieves near-optimal convergence for deep nets","ELBO model selection attains minimax rate adaptively","Oracle inequality shows VI near-minimax in deep learning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the variational family's active weights can be centered at any coefficient of the best approximating network, but the family only allows slab intervals inside $[-1,1]$ while network weights are only assumed bounded by $B \\ge 2$, so if even one target weight lies outside $[-1,1]$ the proof's approximating distribution is not a member of the family it optimizes over.","fun_headline_variants_meta":{"raw":{"variants":["VI matches exact Bayes rates in sparse deep nets","Sparse VI achieves near-optimal convergence for deep nets","ELBO model selection attains minimax rate adaptively","Oracle inequality shows VI near-minimax in deep learning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000612,"raw_usage":{"total_tokens":2900,"prompt_tokens":1053,"completion_tokens":1847,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":669,"completion_tokens_details":{"reasoning_tokens":1782}},"tokens_in":669,"tokens_out":1847,"duration_ms":14035,"temperature":1.0,"reasoning_tokens":1782,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T14:09:19.456672+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Choose a target $f_0$ that is itself a ReLU network with one active weight $\\theta_{*,t}=3/2$, all other weights zero, and $B=2$. The proof of Theorem 2 builds $q_n^*$ with an active slab $U([\\theta_{*,t}-s_n,\\theta_{*,t}+s_n])$, but every distribution in $\\mathcal{F}_{S,L,D}$ has slabs contained in $[-1,1]$, so the constructed $q_n^*$ is not an admissible minimizer in the variational problem. Evaluating the right-hand side of the oracle inequality for this case, analytically or by direct numerical minimization of the ELBO, would settle whether the stated parameter range in Corollary 3 must be shrunk.","supporting_citations":[{"cited_title":"Posterior concentration for sparse deep learning","cited_arxiv_id":null,"evidence_quote":"Supplies the deep-ReLU architecture and the $L_\\infty$ approximation bound for $\\beta$-Hölder functions used to control the bias term in Corollary 3."},{"cited_title":"Adaptivity of deep re LU network for learning in besov and mixed smooth besov spaces: optimal rate and curse of dimensionality","cited_arxiv_id":null,"evidence_quote":"Provides the matching deep-ReLU rate in nonparametric regression used to compare $r_n^{S,L,D}$ with the minimax rate."},{"cited_title":"Consistency of elbo maximization for model selection","cited_arxiv_id":null,"evidence_quote":"Supplies the penalized-ELBO model-selection inequality that Theorem 5 builds on."}],"review_version":1}