{"id":"5162c702-c43b-4e0d-af42-720c77e1d7b4","arxiv_id":"1909.00719","paper_version":4,"verdict":"ACCEPT","confidence":"MODERATE","novelty_score":8.0,"correctness_risk":"low","formal_verification":"none","parameter_count":0,"one_line_summary":"For single-hidden-layer ReLU Bayesian neural networks, mean-field Gaussian and Monte Carlo dropout posteriors provably cannot express higher predictive variance between well-separated low-variance regions, and this limitation persists empirically in deep networks despite a universality theorem.","lead":"This paper proves that mean-field variational inference and Monte Carlo dropout, two common ways to estimate uncertainty in Bayesian neural networks, cannot represent increased uncertainty between well-separated groups of data. It matters because these approximate methods are widely used in active learning and safety-critical decisions, where knowing what you do not know is essential.","discovery_kind":"first_principles","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No significant objection identified; the only caveat is a repairable overgeneralization in Appendix Theorem 4 that does not affect the QFFG/MCDO central claim.","rationale":"The paper's central claim is scoped precisely: for single-hidden-layer ReLU BNNs, mean-field Gaussian and MC-dropout (inputs retained) predictive variances cannot represent increased uncertainty between separated low-uncertainty regions. I checked the proof geometry: Theorem 1's V(r)<=V(p)+V(q) and Theorem 2's convexity follow from the conditional independence of output weights and activations, which is the defining feature of these families, so the factorization is not an unstated vulnerability. The empirical sections use two independent exact-inference references (HMC and the wide-limit GP) and direct function-space fitting, which supports the attribution of the pathology to the approximating family. The deep-network universality result is intricate but not needed for the shallow claim; its known limitation (no higher moments or calibration) is stated by the authors. The one concrete weakness I found is an overgeneralization in Appendix Theorem 4: the proof silently assumes conditional output-weight means are constants, which holds for QFFG but not for arbitrary q_i(w_i|U,v) in equation (3). This is repairable and does not alter the truth of Theorem 1 for QFFG, so the reader's ACCEPT verdict stands.","tokens_in":31789,"tokens_out":26005,"duration_ms":248217,"concrete_test":"Independently re-derive Theorem 4 starting from equation (3) with a conditional mean E[w_i|U,v] that depends on U,v (e.g., w_i|U_i is a point mass at U_i); if this yields a counterexample to the stated bound, restrict the theorem to distributions whose first-layer conditional output-weight means are constants and confirm that the QFFG proof of Theorem 1 remains valid.","verdict_should_be":"UNCHANGED","load_bearing_attack":"No load-bearing concern identified. Theorems 1 and 2 are correctly proved for the stated families: the factorized output weights in equation (2) and Appendix C.1 equation (6) are the defining property of QFFG and of MCDO with independent dropout masks, not a hidden assumption, and the law-of-total-variance argument in Appendix B.2 goes through for QFFG. The HMC/GP comparison in Figure 3 gives independent support for the claim that the missing in-between uncertainty is due to the approximation. The only caveat I found is non-central: Theorem 4 in Appendix B is stated for arbitrary conditional densities q_i(w_i|U,v), but its proof requires E[w_i|U,v] to be constant in U,v (as it is for QFFG); otherwise equation (5) omits covariance terms. This does not undermine Theorem 1 as used in the main text, but the statement of Theorem 4 should be restricted or its proof extended.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies the function-space expressiveness of mean-field Gaussian (QFFG) and Monte Carlo dropout (QMCDO) approximating families for Bayesian neural networks. The central theoretical result is that for single-hidden-layer ReLU BNNs, QFFG and QMCDO cannot represent 'in-between uncertainty' in the predictive variance: Theorem 1 bounds the variance at a point between two well-separated low-uncertainty regions on certain line segments, and Theorem 2 shows the MCDO variance is convex. The authors separate Criterion 1 (does the family contain good approximations to the posterior?) from Criterion 2 (does optimization find them?). Empirical comparisons against HMC and the infinite-width GP on a two-cluster regression task show that the exact posterior does exhibit in-between uncertainty, while MFVI and MCDO do not. For two-hidden-layer networks, Theorem 3 gives universality of predictive mean and variance for both families, but ELBO optimization in deeper networks still shows overconfidence between clusters (Section 4.2). A final active-learning case study on the Naval dataset shows GP-based selection succeeds while MFVI/MCDO-based selection degrades RMSE relative to random selection.","tokens_in":31810,"tokens_out":13641,"duration_ms":120285,"significance":"If the results hold, this is an important contribution to the BNN approximation literature: it gives a crisp, weight-space-to-function-space explanation of a known qualitative failure of two widely used variational families, and it carefully separates model quality from approximation quality using HMC and the infinite-width GP as references. The theorems are proven in detail with standard tools (law of total variance, convexity of rectified quadratics, Chebyshev and union-bound arguments), and the paper ships code for reproduction. The universality result for two-hidden-layer networks is a useful counterpoint showing that depth can in principle restore expressiveness, and the explicit constructive proofs for both QFFG and QMCDO are a strength. The empirical protocol—direct squared-loss minimization in function space, ELBO-based fitting, overconfidence-ratio boxplots, and active learning—triangulates the theoretical claims well. The main caveat is that the exact-posterior claim is empirical rather than proven, and the shallow-case theorems cover special line configurations; these limitations are acknowledged in the paper.","major_comments":[{"comment":"Theorem 4 is stated for arbitrary conditional densities q_i(w_i|U,v) and q(b|U,v), but the proof's equation (5) treats E[w_{k,i}|U,v] and E[b_k|U,v] as constants independent of U,v. If the conditional densities are truly allowed to depend on U,v, then V[E[f|U,v]] = V[sum_i μ_{w_{k,i}}(U,v)ψ(a_i) + μ_{b_k}(U,v)] contains additional covariance terms that are not accounted for. The same gap appears in Theorem 8, where V[E[f(0)|U]]=0 is asserted although q_i(w_i|U) and q(b|U) are allowed to depend on U. Both gaps disappear for the actual QFFG and MCDO families, whose conditional means are constant, so Theorems 1 and 2 survive; nevertheless, the statements of Theorems 4 and 8 should be restricted (e.g., to q_i(w_i|U,v)=q_i(w_i) and q(b|U,v)=q(b)) or the proofs extended to handle the general case.","section":"Appendix B, Theorem 4; Appendix E.1, Theorem 8"}],"minor_comments":[{"comment":"The sentence 'this provides theoretical evidence that approximate inference in deep BNNs satisfies Criterion 1' is stronger than the theorem warrants: Criterion 1 is about containing good approximations to the actual posterior, whereas Theorem 3 only constructs elements of the family with prescribed first two output moments on a compact set. The caveats in the following paragraphs mitigate this, but the wording should be qualified.","section":"Section 4, paragraph before Theorem 3"},{"comment":"The prior-scaling procedure is described by the list of weight standard deviations and the statement that prior standard deviations in function space are 'between 10 and 15'; a more precise statement of the target value or the validation criterion would improve reproducibility.","section":"Appendix F.1"},{"comment":"The active-learning experiment fixes the observation-noise standard deviation to 0.01 for all models and all depths; a brief sensitivity check for this choice would strengthen the comparison, since the GP reference and the BNNs share the same fixed noise.","section":"Section 5 and Appendix H.1"},{"comment":"The Figure 5 caption contains a duplicated article ('and the the box extends'), and several displayed sup-norm inequalities in Appendix D contain duplicated vertical-bar characters (e.g., '|||...|||∞'); these formatting issues should be cleaned up.","section":"Figure 5 caption; Appendix D"}],"recommendation":"minor_revision","confidential_remarks":"The technical overgeneralization in Theorems 4 and 8 is local and repairable, and it does not affect the main QFFG/MCDO claims. I recommend minor revision rather than major revision; the remaining caveats are expository or empirical and do not require new experiments."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Key thing to know: this is the first theoretical demonstration that two standard approximate inference families for BNNs—mean-field Gaussian and MC dropout—cannot represent a basic qualitative feature of the exact posterior: increased uncertainty between well-separated regions of low uncertainty. The result is real. The proof is not just a heuristic; Theorems 1 and 2 are rigorously proved for single-hidden-layer ReLU networks, and the mechanism (factorized output weights removing negative covariance terms) is clearly identified. The empirical work backs up the theory: HMC and the infinite-width GP show the exact posterior does have in-between uncertainty, while MFVI and MCDO do not. The separation of Criterion 1 (family expressiveness) from Criterion 2 (optimization quality) is the right framing and the paper handles it honestly.\n\nWhat is genuinely new: MacKay noted the qualitative issue, but there was no proof of a pathology before this. The deep-network universality result (Theorem 3) is also new and non-obvious, but the paper correctly cautions that universality in function space does not mean the ELBO will find good solutions; the deep experiments support that caution. The active learning case study is a useful practical consequence, showing a real dataset where approximate BNNs underperform random selection while the GP does well.\n\nSoft spots: the most notable is a minor overstatement in Appendix Theorem 4. It states a result for arbitrary conditional densities q_i(w_i|U,v), but the proof in B.2 requires the means E[w_i|U,v] to be constants—otherwise equation (5) omits covariance terms that arise. This does not affect Theorem 1 as used in the main text, since QFFG satisfies that condition, but the theorem statement should be restricted or the proof extended. The GP limit is used as an exact-inference proxy for networks deeper than two layers; HMC is only run for one and two layers. That is a reasonable compromise, but worth keeping in mind. The code is cited but not pinned to a specific commit, which is a minor reproducibility annoyance.\n\nOverall: the central claims hold up; the soft spots are repairable and do not undermine the main argument. This paper deserves serious refereeing and is a likely citation in any subsequent work on expressiveness of BNN approximations.","headline":"First proof that mean-field Gaussian and MC dropout posteriors provably lack in-between uncertainty in shallow ReLU BNNs; a solid, citable negative result with one minor appendix overstatement.","tokens_in":32470,"tokens_out":2673,"would_cite":true,"duration_ms":29959,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62F15","68T07"],"pacs":[],"model":"deepseek-v4-flash","headline":"For one-hidden-layer ReLU Bayesian neural networks, mean-field Gaussian variational inference and Monte Carlo dropout provably cannot represent the increased uncertainty between well-separated data clusters that exact Bayesian inference…","keywords":["Bayesian neural networks","variational inference","mean-field approximation","Monte Carlo dropout","predictive uncertainty","in-between uncertainty","ReLU networks","active learning"],"falsifier":"A single counterexample would settle the main claim: find a single-hidden-layer ReLU network whose weights follow a fully factorised Gaussian or an MC dropout distribution (inputs not dropped), pick two points $p$ and $q$ satisfying the conditions of Theorem 1, and observe strictly larger predictive variance at an interior point $r$ than the sum of the endpoint variances; the theorem says this is impossible. For the empirical half, one can repeat the two-cluster regression with three clusters arranged so the middle cluster is unobserved: the paper predicts Hamiltonian Monte Carlo and the infinite-width GP will show elevated uncertainty at the unobserved middle cluster while MFVI and MCDO will not, across all hyperparameter settings.","tokens_in":31467,"feed_emoji":"📉","tokens_out":8879,"duration_ms":76378,"temperature":0.7,"pith_summary":"This paper sets out to show that two of the most common approximate-inference methods for Bayesian neural networks — mean-field Gaussian variational inference and Monte Carlo dropout — can, even in the simplest one-hidden-layer ReLU network, fail to reproduce a basic qualitative feature of the exact Bayesian predictive distribution: the rise in uncertainty that should occur in the gap between two well-separated clusters of data. It proves this as a theorem about the weight-space structure of the approximating families, and it shows empirically that exact inference, represented by the infinite-width Gaussian process and Hamiltonian Monte Carlo, does exhibit that in-between uncertainty on the same problem. The paper also proves that the limitation is not fundamental to depth: with at least two hidden layers, both families contain distributions whose predictive mean and variance can approximate any continuous target functions. Yet when the usual variational objectives are optimized in deep networks, the same pathological confidence persists, and in an active-learning case study it leads MFVI and MCDO to choose points worse than random selection while the Gaussian-process reference gains a threefold improvement. The conclusion is that failures in BNN uncertainty estimates can be caused by the approximation family and the optimization objective, not by the model or prior.","feed_headline":"Proof: common BNN approximations cannot show mid-data uncertainty","feed_subtitle":"Mean-field and dropout posteriors provably miss the uncertainty bump between data clusters that exact Bayesian inference shows.","key_machinery":"The load-bearing object is the decomposition of predictive variance by the law of total variance, conditioned on the first-layer parameters. Under the factorised families, the conditional variance of the output is $V[f(x) | \\theta_{in}] = \\sum_i V[w_i] \\psi(a_i(x))^2 + V[b]$, where $\\psi(a) = \\max(0, a)$ is the ReLU; because $\\psi^2$ is convex, this term is convex in $x$, which alone proves Theorem 2 and explains the non-convexity that could rescue Theorem 1. The remaining term for mean-field Gaussian, $V[\\mathbb{E}[f | \\theta_{in}]]$, is a positive linear combination of variances of rectified activations, and the appendix proves these satisfy reflection and monotonicity properties (sets $T_1$ and $T_2$) that force the line-segment bound of Theorem 1. Depth enters through a construction that splits the two units of the second hidden layer into a mean-control channel and a variance-control channel, using the deterministic universal approximation theorem for ReLU networks to argue that both the approximate mean and variance can match any continuous targets.","core_discovery":"For a single-hidden-layer fully connected ReLU network, any approximate posterior drawn from the fully factorised Gaussian family or from the MC dropout family (with inputs not dropped out) has a predictive variance that cannot be substantially larger between well-separated regions of low uncertainty. Theorem 1 proves that on line segments passing through the origin, or crossing an axis plane, the variance at an interior point is bounded by the sum of the variances at two flanking endpoints, $V[f(r)] \\le V[f(p)] + V[f(q)]$; Theorem 2 proves the stronger statement that for MC dropout the variance function is convex in the input, so on any segment it is bounded by the maximum at the endpoints. The paper provides strong empirical evidence that the exact posterior does not share this pathology: the infinite-width Gaussian process and Hamiltonian Monte Carlo posteriors show visibly elevated uncertainty between two data clusters. For networks with at least two hidden layers, Theorem 3 states that the predictive mean and variance of both approximate families are universal approximators, yet ELBO-optimised deep MFVI and MCDO still exhibit the same overconfidence between clusters, and the active-learning experiment on the Naval dataset shows MFVI and MCDO degrading performance relative to random acquisition.","pith_inferences":["The core mechanism — independence of outgoing weights cancels the covariance terms that make variance non-convex — suggests the same obstruction applies to any factorised family whose activations have a convex squared non-linearity, and to deeper variants that use such factorisation in the final layer; a testable prediction is that a full-covariance Gaussian posterior with negative correlations be","A natural extension would be to measure the 'distance-to-origin bias' of MFVI predictive variance in high-dimensional input spaces: the Naval result implies that even where the theorem's precise line conditions are rare, the family still ranks uncertainty more by distance from the origin than by local data density, and this ranking could be quantified directly.","The deep-network result separates expressiveness from optimisation: Theorem 3 says good distributions exist in the family, while the ELBO experiments say they are not found. This suggests a testable diagnostic for new variational objectives: initialise from the squared-loss solution and check whether ELBO maximisation preserves or destroys in-between uncertainty.","If the pathology is caused by the factorised family and the ELBO landscape, then non-factorised approximations (for example, inducing correlations between output weights) or alternative objectives that directly match function-space moments should eliminate the overconfidence pattern in the same toy dataset; this is a direct, checkable implication."],"forward_implications":["For single-hidden-layer ReLU BNNs, no choice of variational parameters, prior, or optimiser can make a fully factorised Gaussian or MC dropout posterior represent in-between uncertainty on the line segments covered by Theorems 1 and 2; the failure is a property of the approximating family itself.","Any inference method that uses the fully factorised Gaussian family — including diagonal Laplace and other variational schemes — inherits the same function-space limitation, because Theorem 1 depends only on the family.","The exact posterior of the BNN model does show increased uncertainty between well-separated data, so the lack of in-between uncertainty in the approximations is not a modelling defect; it is introduced by approximate inference.","In deep networks, theoretical expressiveness alone does not guarantee good approximate inference: the universality result is real, but ELBO-optimised MFVI and MCDO empirically remain overconfident between data clusters, even when initialised from a function-space fit that already has the correct uncertainty shape.","In the active-learning case study, the exact-inference GP reduces test RMSE by roughly a factor of three relative to random selection, whereas 1HL MFVI and MCDO increase RMSE and deeper variants mostly fail to beat random selection, showing a concrete practical cost of the pathology."],"supporting_citations":[{"why":"Supplies the mean-field Gaussian variational family whose function-space variance Theorem 1 constrains.","marker":"[6]"},{"why":"Defines Monte Carlo dropout as an approximate posterior; Theorem 2 constrains its variance.","marker":"[14]"},{"why":"Provides the dropout-as-variational-inference interpretation and the KL-condition regulariser used in the experiments.","marker":"[13]"},{"why":"Sets the prior scaling and the infinite-width Gaussian-process limit that serves as the exact-inference reference.","marker":"[34]"},{"why":"Establishes that moderately wide BNNs track their infinite-width GP, justifying the overconfidence-ratio reference.","marker":"[32]"},{"why":"The deterministic ReLU universal approximation theorem used to construct the two-hidden-layer distributions in Theorem 3.","marker":"[26]"},{"why":"The Hamiltonian Monte Carlo sampler used as a gold-standard reference for the exact posterior predictive.","marker":"[19]"},{"why":"Defines the highest-predictive-variance acquisition rule used in the active-learning case study.","marker":"[16]"}],"fun_headline_variants":["Proof: MFVI and MC dropout miss mid-data uncertainty","Variational BNNs overconfident between data clusters","Exact BNN shows uncertainty approximations can't","Approximate BNN inference underestimates inter-cluster uncertainty","Mean-field and dropout BNNs fail between clusters"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The proofs assume the approximate posterior factorises across hidden neurons, so the output weights attached to different neurons are independent and carry no negative correlations; if correlations between those weights were allowed, the variance could become non-convex and the bounds in Theorems 1 and 2 would no longer follow.","fun_headline_variants_meta":{"raw":{"variants":["Proof: MFVI and MC dropout miss mid-data uncertainty","Variational BNNs overconfident between data clusters","Exact BNN shows uncertainty approximations can't","Approximate BNN inference underestimates inter-cluster uncertainty","Mean-field and dropout BNNs fail between clusters"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000536,"raw_usage":{"total_tokens":2591,"prompt_tokens":975,"completion_tokens":1616,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":591,"completion_tokens_details":{"reasoning_tokens":1537}},"tokens_in":591,"tokens_out":1616,"duration_ms":11771,"temperature":1.0,"reasoning_tokens":1537,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T05:39:16.429905+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A single counterexample would settle the main claim: find a single-hidden-layer ReLU network whose weights follow a fully factorised Gaussian or an MC dropout distribution (inputs not dropped), pick two points $p$ and $q$ satisfying the conditions of Theorem 1, and observe strictly larger predictive variance at an interior point $r$ than the sum of the endpoint variances; the theorem says this is impossible. For the empirical half, one can repeat the two-cluster regression with three clusters arranged so the middle cluster is unobserved: the paper predicts Hamiltonian Monte Carlo and the infinite-width GP will show elevated uncertainty at the unobserved middle cluster while MFVI and MCDO will not, across all hyperparameter settings.","supporting_citations":[{"cited_title":"Dropout as a Bayesian approximation: Representing model uncertainty in deep learning","cited_arxiv_id":null,"evidence_quote":"Defines Monte Carlo dropout as an approximate posterior; Theorem 2 constrains its variance."},{"cited_title":"Bayesian learning for neural networks","cited_arxiv_id":null,"evidence_quote":"Sets the prior scaling and the infinite-width Gaussian-process limit that serves as the exact-inference reference."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes that moderately wide BNNs track their infinite-width GP, justifying the overconfidence-ratio reference."},{"cited_title":"Multilayer feedforward networks with a nonpolynomial activation function can approximate any function","cited_arxiv_id":null,"evidence_quote":"The deterministic ReLU universal approximation theorem used to construct the two-hidden-layer distributions in Theorem 3."},{"cited_title":"The No-U-Turn sampler: adaptively setting path lengths in Hamiltonian Monte Carlo","cited_arxiv_id":null,"evidence_quote":"The Hamiltonian Monte Carlo sampler used as a gold-standard reference for the exact posterior predictive."},{"cited_title":"Probabilistic backpropagation for scalable learning of Bayesian neural networks","cited_arxiv_id":null,"evidence_quote":"Defines the highest-predictive-variance acquisition rule used in the active-learning case study."}],"review_version":1}