{"id":"2b384526-5b11-4e85-a59f-bfc24a1772f6","arxiv_id":"2411.18425","paper_version":4,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Single-pass analytic predictive distributions in Bayesian deep learning via local linearization and Gaussian moment propagation.","lead":"This paper replaces Monte Carlo sampling in Bayesian deep learning with an analytic single-pass approximation, by locally linearizing activations and propagating Gaussians through each layer. It reports competitive accuracy and uncertainty on MLPs, vision transformers, and GPT-2, and releases code as the SUQ library.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline NLPD/ECE gains may be driven by a validation-set-fitted variance scaling factor applied only to Ours, so the uncertainty comparison is not controlled.","rationale":"The reader's verdict is CONDITIONAL, with the weakest assumption identified as the independence and Gaussianity assumptions in the covariance propagation. I agree those are genuine limitations, and the paper itself acknowledges them in the limitations section without providing error bounds. However, the single most load-bearing issue for the paper's headline claim is the validation-set-fitted variance scaling factor. NLPD and ECE are exactly the quantities that this scalar can move, and it is applied only to the proposed method. Tuning a scalar on the validation set to minimize NLPD and then reporting NLPD as a headline result is a confound that the current experimental protocol does not control for. This does not invalidate the method as a computational tool, and the open-source code and broad experimental coverage are real strengths, but the empirical superiority claim is not yet established. The reader did mention the scaling factor in the rationale, so there is partial agreement, but the reader did not make it the primary weakest assumption. A controlled ablation would settle the question; without it, the CONDITIONAL verdict remains appropriate rather than a full rejection.","tokens_in":30329,"tokens_out":4387,"duration_ms":42365,"concrete_test":"Re-run the main NLPD/ECE tables with the variance scaling factor fixed to 1 for Ours, and separately apply the same validation-set grid-search procedure to each baseline (a single scalar multiplier on predictive variance, chosen by validation NLPD). If Ours no longer achieves better or on-par NLPD and ECE in either setting, the headline comparison is confounded by the free scaling parameter.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central empirical claim rests on NLPD and ECE comparisons in Tables 1, 2, 4, and 5. In Sec. 4 (Posterior approximations), the authors state: 'For our method, we fit an additional scaling factor on the predictive variance by minimising the NLPD on a validation set.' This scalar is tuned on the validation set to optimize exactly the metric used for comparison, and ECE is a direct function of predictive variance. The sampling and GLM baselines receive no analogous variance calibration. Consequently, 'better or on-par NLPD and ECE' may reflect the validation-tuned scalar rather than the local-linearisation and local-Gaussian propagation itself. Table 5 further undercuts the blanket claim: on GPT-2, LA Ours has worse NLPD than both baselines on BoolQ and MRPC (0.665 vs 0.656/0.633; 0.706 vs 0.658/0.609), and ACC drops to 0.416 on MRPC. The limitations section acknowledges the scaling-factor fitting but does not quantify its effect on the reported comparisons. Assumption 3.1 and the local Gaussian approximation are real limitations, but the validation-tuned scalar is more load-bearing because it is a free parameter selected on the evaluation metric and applied asymmetrically.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a single-forward-pass method for approximate posterior predictive inference in Bayesian deep learning. The method propagates mean and covariance through locally linearised activation functions and local Gaussian approximations at linear layers, supporting full, diagonal, and KFAC posterior covariance structures. It is applied to MLPs, Vision Transformers, and GPT-2, and evaluated on UCI regression, MNIST-like classification, and several large-scale classification and language-understanding tasks. The central claim is that this analytic propagation achieves better or on-par NLPD and ECE compared to Monte Carlo sampling and the global linearised Laplace approximation, with little or no runtime overhead.","tokens_in":30638,"tokens_out":4581,"duration_ms":41390,"significance":"If the empirical comparisons are valid, the contribution is practically useful: it removes the need for MC sampling at prediction time, provides a closed-form route to OOD detection and input-sensitivity analysis, and is supported by an open-source library and detailed derivations. The derivation of moment propagation, including the KFAC covariance block retrieval, is technically sound under the stated assumptions. However, the paper's headline uncertainty comparisons are confounded by a validation-set-fitted variance scaling factor applied only to the proposed method, so the significance of the empirical claims is currently not established.","major_comments":[{"comment":"The predictive variance for 'Ours' is rescaled by a scalar fitted by minimising NLPD on a validation set, while the MC sampling and GLM baselines receive no analogous calibration. Since NLPD and ECE both depend directly on predictive variance, the comparisons in Tables 1, 2, and 4 are not controlled for this free parameter; the Limitations section acknowledges the fitting but does not quantify its effect. Please provide an ablation without the scaling factor, report the fitted scaling values, or apply an equivalent validation-based calibration to the baselines, so that the reported gains can be attributed to the propagation method itself.","section":"Sec. 4, Posterior approximations paragraph; Sec. 5 Limitations"},{"comment":"On BoolQ, LA Ours has NLPD 0.665 vs 0.656 for LA Sampling and 0.633 for LA GLM; on MRPC, LA Ours has NLPD 0.706 vs 0.658 and 0.609, and ACC drops to 0.416 vs 0.665 and 0.720 for the baselines. The text states that 'our method systematically outperforms sampling,' which is contradicted by these rows. Please revise this claim and discuss the failure modes on these language-understanding tasks.","section":"Table 5"},{"comment":"On the CAC regression data set, LA Ours has NLPD 2.662±1.096, which is substantially worse than LA Sampling (1.848±0.055) and GLM (1.281±0.069), with a large standard error. The general claim in Sec. 4.1 that 'our method achieves better NLPD in general' and the Table 1 caption 'better or matching performance' are not supported by this row. Please address this failure case or qualify the claim accordingly.","section":"Table 1, CAC row"},{"comment":"The propagation relies on Assumption 3.1 (independence between activations and parameters) and Assumption A1 (local Gaussianity of each product a_i W_ki). The appendix itself shows the approximation fails to capture skewness for large-magnitude weights, and the manuscript provides no formal error bound or practical diagnostic for when activation variances are large or the posterior is heavy-tailed. Please add quantitative diagnostics, such as comparing propagated moments against MC samples on the actual posteriors used for the ViT/GPT-2 experiments, or a bounded-error statement.","section":"Assumption 3.1 and Assumption A1; Sec. A.2"}],"minor_comments":[{"comment":"The sentence 'the covariance between the kth and the jth hidden unit' uses 'jth' but Eq. (3) defines the covariance between the kth and lth hidden unit; please correct the index notation.","section":"Sec. 3.1, first paragraph"},{"comment":"Treating the query and key as deterministic is a strong approximation that ignores their posterior covariance; the text gives only a brief 'computational reasons' justification. Please state clearly that this can underestimate the attention output variance and discuss the potential impact on uncertainty calibration.","section":"Sec. 3.2, Eq. (7)"},{"comment":"The approximation (B⊗A+λ²I)^{-1} ≈ (B+λI)^{-1}⊗(A+λI)^{-1} is stated without a source or empirical check; please add a reference or a small numerical verification of its accuracy for the models used.","section":"Sec. 3.3, Eq. (10)"},{"comment":"ECE values are reported without standard errors or confidence intervals; please state whether these are averages over folds/seeds and how they were aggregated.","section":"Tables 2 and 4"},{"comment":"The acknowledgment mentions 'Jonas Vestergaard for finding a bug in our code' without indicating whether the bug affected any reported results; please clarify in the manuscript whether the experiments were rerun after the fix.","section":"Acknowledgments"}],"recommendation":"major_revision","confidential_remarks":"The central concern is the asymmetric validation-set tuning of the variance scalar, which directly affects the metrics used for the headline comparison. If the authors provide an uncalibrated comparison or calibrate the baselines equivalently, the paper's empirical claims would be much stronger. The acknowledgment of a code bug without stating its impact also deserves an explicit check. The paper's scope fits the journal well, and the derivations and open-source code are valuable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things you should know about this paper. It gives a clean recipe for turning any Gaussian approximate posterior into a single-pass predictive distribution: locally linearise activations, propagate moments through the linear layers, and handle full/diagonal/KFAC covariances along the way. That is genuinely useful, and the attention-layer extension is new for this line of work. But the headline uncertainty gains are not as clean as the sales line. The method gets an extra validation-tuned scalar on the predictive variance—fit by minimising NLPD on a validation set—and the MC and GLM baselines do not. Since NLPD and ECE are exactly the metrics that move, the comparison is not controlled.\n\nWhat is actually new: the combination of local linearisation with layer-wise Gaussian moment propagation, applied to transformer attention (Q/K treated deterministically, V propagated), plus the covariance-structure variants. The derivations in the appendix are careful, including the error analysis for the independence assumption. Code is open source, and the runtime numbers are honest: essentially the same cost as a single forward pass. That part is real value.\n\nWhere it gets soft. The scaling factor is the load-bearing weakness. The paper acknowledges it in the limitations section but never quantifies how much of the reported NLPD/ECE improvement comes from it. That needs an ablation: run Ours without scaling, and/or give the baselines the same validation-set calibration. The independence assumption (Assumption 3.1) and the local Gaussian product approximation are real, but they are standard for this kind of moment propagation; the paper includes a small MC check but no error bound. There are also specific failures that the 'better or on-par' abstract glosses over: CAC regression NLPD is 2.662 vs 1.848 for sampling, and on BoolQ/MRPC the method loses on NLPD and collapses on MRPC accuracy. Those don't sink the method, but they end the blanket claim.\n\nBottom line: the core mechanism is sound, and the paper is a solid extension of known local-linearisation ideas. It deserves a serious referee, but the empirical section as written overstates the uncertainty quality. The fix is straightforward—ablate the scaling factor and apply the same calibration to baselines. As it stands, I'd send it to review with the request for that ablation, and I'd bring it to the reading group for a discussion on how much of the gain is calibration versus propagation.","headline":"A useful single-pass recipe for BDL predictions, but the headline uncertainty gains are inflated by a validation-tuned variance scalar applied only to the proposed method.","tokens_in":31075,"tokens_out":2852,"would_cite":true,"duration_ms":27840,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T07","62F15"],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that the posterior predictive distribution of a Bayesian neural network can be computed analytically in one forward pass by locally linearising activations and Gaussian-approximating each layer, matching or beating Monte…","keywords":["Bayesian deep learning","posterior predictive distribution","local linearisation","local Gaussian approximation","single forward pass","uncertainty quantification","Laplace approximation","mean-field variational inference"],"falsifier":"Train an MLP whose weight posterior is made heavy-tailed (for instance, by increasing the posterior temperature), then compare the analytic predictive entropy from a single forward pass against a very large Monte Carlo estimate (e.g., 100,000 samples) on an out-of-distribution test set; if the analytic entropy is much lower or higher while sampling is stable, the Gaussian-independence approximation is the cause and the claim of calibration parity fails.","tokens_in":30122,"feed_emoji":"⚡","tokens_out":3871,"duration_ms":34747,"temperature":0.7,"pith_summary":"This paper aims to make Bayesian deep learning predictions cheap and practical by replacing Monte Carlo sampling with an analytic single forward pass. The authors show that by locally linearising activation functions and applying local Gaussian approximations at linear layers, the posterior predictive distribution can be propagated layer-by-layer in closed form. They demonstrate that this streamlined prediction matches or improves on the predictive quality of sampling-based baselines for neural networks including MLPs, Vision Transformers, and GPT-2, while adding little computational overhead. The result matters because it separates the cost of estimating a posterior from the cost of making predictions, which is a bottleneck for real-world Bayesian deep learning.","feed_headline":"Bayesian deep learning: single-pass predictive without sampling","feed_subtitle":"Local linearisation yields closed-form predictive distributions that match Monte Carlo at lower cost.","key_machinery":"The central mechanism is the simultaneous use of local linearisation and local Gaussian approximation at every layer. For a pre-activation $h^{(m)} = W^{(m)} a^{(m-1)} + b^{(m)}$, the method assumes activations of the previous layer and parameters of the current layer are independent (Assumption 3.1) and that each product $a_i^{(m-1)} W_{ki}^{(m)}$ is Gaussian; this yields closed-form mean and covariance for $h^{(m)}$ (Eqs. 2-4). For a nonlinear activation $g$, a first-order Taylor expansion $g(h) \\approx g(\\mathbb{E}[h]) + J_g|_{h=\\mathbb{E}[h]}(h - \\mathbb{E}[h])$ propagates the Gaussian through the activation, giving $a \\sim \\mathcal{N}(g(\\mathbb{E}[h]), J^\\top \\Sigma_h J)$. In attention layers, the query and key are treated deterministically at their means, while only the value projection carries the propagated uncertainty, keeping the path tractable.","core_discovery":"The central claim is that the posterior predictive distribution of a Bayesian neural network admits an accurate and tractable approximation through a single deterministic forward pass. The method recursively computes the mean and covariance of activations at every layer: for a linear layer, the pre-activation distribution is obtained from the weight posterior and the previous activation distribution under a local Gaussian assumption; for a nonlinear activation, a first-order Taylor expansion at the pre-activation mean maps the Gaussian distribution to a Gaussian. For classification, a probit approximation yields the predictive class probabilities. Empirical results show that this analytic predictive achieves lower or equal negative log predictive density and expected calibration error compared to Monte Carlo sampling and to the global linearised Laplace approximation, across regression and classification tasks, with near-zero runtime overhead relative to a standard forward pass.","pith_inferences":["This propagation scheme implies that the predictive uncertainty can be decomposed into contributions from weight uncertainty, bias uncertainty, and input (aleatoric) uncertainty at each layer, which the paper does not explicitly use but would be a natural extension for interpretable uncertainty attribution.","If the independence assumption between activations and weights degrades in deeper networks with correlated activations, the analytic covariance will underestimate the true uncertainty; a testable extension would be to compare against sampling on networks trained with increasing depth and width.","The method opens the possibility of real-time Bayesian prediction in latency-sensitive applications such as autonomous driving or medical diagnostics, where the cost of many forward passes is currently prohibitive.","A similar layer-wise moment propagation could be applied to other posterior families beyond Gaussians (e.g., Student-t or alpha-stable), provided the local linearisation is adjusted accordingly, since the paper already exploits stability under linear maps."],"forward_implications":["Predictions from a Bayesian neural network no longer require sampling: a single forward pass produces both the predictive mean and uncertainty, which is a direct consequence of the closed-form propagation.","The method applies to modern architectures including Vision Transformers and GPT-2, and to both Laplace and mean-field variational posteriors, making it compatible with large pre-trained models.","The analytic predictive distribution is better calibrated than Monte Carlo sampling in most experiments, with lower NLPD and ECE while maintaining equal accuracy.","Because the input distribution can be propagated analytically, sensitivity analysis with respect to inputs becomes tractable in a single pass, as demonstrated by the pixel-sensitivity maps in Sec. 4.3.","The method's cost scales with the number of parameters and the posterior covariance structure, allowing further speedups through diagonal or Kronecker-factored approximations."],"supporting_citations":[{"why":"Provides the foundation for stable distribution propagation and the result that local linearisation gives the optimal Gaussian approximation for ReLU activations.","marker":"Petersen et al., 2024"},{"why":"Introduces the linearised Laplace approximation (GLM), the main baseline that the paper compares against in the experiments.","marker":"Immer et al., 2021b"},{"why":"Supplies the probit approximation used to turn the analytic Gaussian predictive into classification probabilities.","marker":"MacKay, 1992b"},{"why":"Provides the IVON optimiser for mean-field variational inference, which the paper uses to obtain one of the posterior approximations.","marker":"Shen et al., 2024"},{"why":"Supplies the Kronecker-factored Laplace approximation and the predictive variance scaling factor that the method adopts.","marker":"Ritter et al., 2018"},{"why":"Provides the practical Laplace approximation toolbox and the subnetwork approach that underpin the posterior estimation.","marker":"Daxberger et al., 2021a"}],"fun_headline_variants":["Bayesian deep learning: one pass to predict, no sampling","Single forward pass: Bayesian predictions without MC sampling","Streamlining BDL: analytic predictive in one deterministic pass","Bayesian nets: closed-form predictions from a single forward pass"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the propagated uncertainty remains accurate under the twin assumptions that each weight-activation product is Gaussian and that activations and weights are independent, which can fail when activation variances are large or the posterior is heavy-tailed.","fun_headline_variants_meta":{"raw":{"variants":["Bayesian deep learning: one pass to predict, no sampling","Single forward pass: Bayesian predictions without MC sampling","Streamlining BDL: analytic predictive in one deterministic pass","Bayesian nets: closed-form predictions from a single forward pass"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000125,"raw_usage":{"total_tokens":1033,"prompt_tokens":798,"completion_tokens":235,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":414,"completion_tokens_details":{"reasoning_tokens":168}},"tokens_in":414,"tokens_out":235,"duration_ms":3037,"temperature":1.0,"reasoning_tokens":168,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T11:12:33.199609+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train an MLP whose weight posterior is made heavy-tailed (for instance, by increasing the posterior temperature), then compare the analytic predictive entropy from a single forward pass against a very large Monte Carlo estimate (e.g., 100,000 samples) on an out-of-distribution test set; if the analytic entropy is much lower or higher while sampling is stable, the Gaussian-independence approximation is the cause and the claim of calibration parity fails.","supporting_citations":[{"cited_title":"Uncertainty quantification via stable distribution propagation","cited_arxiv_id":null,"evidence_quote":"Provides the foundation for stable distribution propagation and the result that local linearisation gives the optimal Gaussian approximation for ReLU activations."},{"cited_title":"Variational learning is effective for large deep networks","cited_arxiv_id":null,"evidence_quote":"Provides the IVON optimiser for mean-field variational inference, which the paper uses to obtain one of the posterior approximations."},{"cited_title":"A scalable L aplace approximation for neural networks","cited_arxiv_id":null,"evidence_quote":"Supplies the Kronecker-factored Laplace approximation and the predictive variance scaling factor that the method adopts."}],"review_version":1}