{"id":"b2c94460-904f-46a3-b900-184e0a09bb17","arxiv_id":"2605.08446","paper_version":4,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Training a Gaussian last layer by direct minimization of a shared-cavity predictive log-loss yields single-pass NLL/calibration parity-or-better with tuned and ensembled baselines on 7/8 UCI and 4/5 large/deep benchmarks.","lead":"This paper trains Bayesian neural networks by minimizing a Bethe-free-energy-style objective in which each data point is scored by its own predictive density, with the last-layer belief parameters trained freely instead of being pinned to a closed-form posterior. The resulting single-pass method (SCROLL) reports test NLL and calibration equal to or better than validation-tuned and ensembled baselines on most UCI and large-scale benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Proposition 3's free-route representability of V* is unproven: the feasible variance profiles {σ²+ψ_nᵀΣψ_n} form a low-dimensional PSD cone, and the proof only shows this family contains the closed route, not the heteroscedastic optimum.","rationale":"The reader's weakest assumption is exactly the load-bearing point. I read Proposition 3 charitably as a relaxation inequality, but the abstract, introduction, and Section 5 all assert the stronger representability conclusion: the free route 'reaches' the heteroscedastic optimum. The Appendix's numerical confirmation (Fig. 4) uses a well-specified linear–Gaussian model with scalar isotropic prior and fixed features; it does not probe whether a generic V* lies in the feasible PSD-profile family. Thus the central 'predictive-optimal interior' claim is unsupported as stated. I do not see grounds to reject the empirical contribution: the protocol is careful (validation selection, seeds, significance tests, honest limitations), and even if the free family merely approximates V*, SCROLL may still be useful and competitive. But the central theoretical claim should be qualified or proved, and the current wording overstates the result. The reader's CONDITIONAL verdict already captures this, so no verdict change is needed. The only additional concerns (union-of-variants headline, missing code/data) are secondary to this representability gap.","tokens_in":25916,"tokens_out":6314,"duration_ms":77109,"concrete_test":"Run a synthetic representability check with fixed features and known heteroscedastic noise. Take H=50 random Gaussian features ψ_n, N=5000 inputs x_n, and a fixed μ; draw y_n ∼ N(μᵀψ_n, s_n²) with s_n² = exp(βᵀ z_n), where z_n is independent of ψ_n (e.g., another random projection of x_n), so V* is outside the PSD-profile span with probability 1. (i) Optimize L_reg (Eq. 11) over Σ⪰0 and σ_obs² with μ fixed. (ii) Compute the oracle NLL using V_n = s_n². If the optimized NLL exceeds the oracle by a non-negligible amount (e.g., >0.01 nats/point averaged over 20 seeds), the free route does not reach V*, and Proposition 3's representability claim fails. Repeat with features from the actually trained backbone to see whether the gap persists in the paper's real setting. A negligible gap would support the proposition; a positive gap refutes the 'predictive-optimal' wording.","verdict_should_be":"UNCHANGED","load_bearing_attack":"At the heart of the paper is Proposition 3 (§4.1): for a fixed mean, the shared-cavity data term is minimized in population at V*(x) = E[(y−μᵀψ(x))²|x], and the free route reaches that optimum while the closed route cannot. The proof in Appendix C.2 is insufficient. It shows that the free-route feasible set V_free = {σ_obs² + ψ_nᵀΣψ_n : Σ⪰0, σ_obs²≥0} contains the closed-route profile, so inf_{V_free} D ≤ inf_{V_closed} D. But containing the closed route says nothing about whether V* itself lies in V_free. For N observed points and H features, the vector of predictive variances lies in an affine cone of dimension at most 1 + H(H+1)/2 (the span of symmetric products ψ_iψ_j); for N > H(H+1)/2 — the paper's own data-rich regime — generic heteroscedastic V* is not representable by any PSD Σ. Moreover the objective includes the prior term −log Z_w (Eq. 8), which depends on Σ and biases the variance away from V* at finite N. The claim that 'the gap is exactly the residual heteroscedasticity' therefore overstates what has been established. This is load-bearing because SCROLL's novelty relative to a heteroscedastic NLL head (MVN) is precisely that its structured covariance reaches the predictive optimum; without representability, the empirical gains may reflect a larger variance family or regularization rather than the claimed predictive-optimality.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes training Bayesian neural networks by direct gradient minimization of the Bethe free energy, using a 'shared cavity' that makes the data term a per-plate predictive log-score, and 'free routing' that treats belief parameters as free variables rather than pinning them to a conjugate posterior. The framework is instantiated as SCROLL, a Gaussian last-layer model over a deterministic backbone. The paper claims that exact inference (the neural-linear marginal likelihood) appears as a known closed-routed corner (Theorem 1), that the shared-cavity data term is a strictly proper scoring rule (Proposition 2), and that the free-routed interior is 'predictive-optimal' in that it reaches the conditional residual variance V*(x) while the closed route cannot (Proposition 3). This predictive-optimality claim is used to explain the reported NLL and calibration gains over evidence-based methods. Extensive experiments compare SCROLL with MAP, Laplace, VBLL, MVN, Deep Ensembles, and MC Dropout on UCI, large tabular, and frozen deep-embedding benchmarks.","tokens_in":26485,"tokens_out":4301,"duration_ms":50792,"significance":"If the central predictive-optimality claim were established, this would be a significant contribution: a single-pass, batchable BNN with implicit empirical Bayes and a principled proper-scoring objective would be a genuinely useful alternative to ELBO-based variational methods and to ensembles. The paper has real strengths: Propositions 1 and 2 are elementary and correct; Theorem 1 is a standard telescoping identity; the empirical study is extensive and includes useful controls (design-space attribution, per-variant counts, architecture sensitivity, mini-batch behavior). The main theoretical novelty, however, rests on Proposition 3, and that proposition as stated and proved does not support the strong conclusions drawn from it. The manuscript currently overclaims its central result, and the gap is load-bearing because the reported gains are interpreted through that result.","major_comments":[{"comment":"The proof of Proposition 3 is incomplete at the load-bearing step. It shows that the free-route feasible set V_free = {σ_obs^2 + ψ_n^T Σ ψ_n : Σ ⪰ 0, σ_obs^2 ≥ 0} contains the closed-route profile, and therefore inf over V_free ≤ inf over V_closed. But the proposition claims that the free route 'represents' the predictive optimum V*(x) = E[(y−μ^Tψ)^2 | x]. Containing the closed route says nothing about whether V* lies in V_free. Since V_free is an affine image of the PSD cone of dimension at most 1+H(H+1)/2, for N exceeding this dimension a generic heteroscedastic V* is not representable by any PSD Σ. The paper's own data-rich regime (N ≫ H, e.g., Year with N=515k and H=100) is precisely where this obstruction is strongest. Moreover, the objective includes the prior term −log Z_w (Eq. 8), which depends on Σ and biases the variance away from V* at finite N. Thus the statements 'the free r","section":"§4.1, Proposition 3 and Appendix C.2"},{"comment":"The manuscript defines the shared-cavity loss F_SC as a factor-wise sum of local log-partition functions, with the statement that 'belief-entropy corrections that couple the plates are dropped (they cancel exactly for deterministic factors, Appendix B).' This is not the Bethe free energy as usually defined, which contains explicit entropy terms. The cancellation is shown only for deterministic factors; for general factors the dropped terms do not cancel, and the relation between F_SC and the Bethe free energy is therefore unclear. Since the paper's title and framing present this as a Bethe free energy minimization, the authors need to state precisely what objective is being minimized, what approximation the entropy dropping introduces, and why minimizing this surrogate is justified beyond the Gaussian last-layer instantiation. As written, the framework is an ad-hoc proper-scoring objecti","section":"§3, Eq. (4) and the definition of F_SC"},{"comment":"Proposition 3 fixes the mean function μ^Tψ and then considers the variance. The experiments train μ, Σ, σ_obs, α, and the backbone jointly, so the learned mean changes with the variance. The paper acknowledges this in the RMSE discussion (Appendix F), but the central explanatory claim—that SCROLL's NLL/calibration gains over the exact corner are 'exactly the residual heteroscedasticity'—is not supported by a fixed-mean theorem when the mean is not fixed. The observed gains could be due to a different mean, a different effective prior, or the larger variance family, rather than to the mechanism in Proposition 3. A controlled experiment that isolates the variance effect (e.g., fixing the mean at the closed-route optimum and then fitting only Σ, σ_obs) would be needed to support the attribution made in Section 5(ii).","section":"§4.1, Proposition 3 vs. joint training in §5"},{"comment":"The headline claim 'a fixed SCROLL variant is best-or-tied on 7/8' is based on fielding three fixed covariance variants and counting a win if at least one of them wins. The appendix provides per-variant counts (SCROLL-Full alone is 6/8), which is helpful, but the main text should be more careful: with three variants, the probability that at least one appears best by chance is inflated. The per-variant result should be moved into the main text or the claim should be rephrased. This does not affect the broader empirical conclusion but it does affect the precision of the central empirical summary.","section":"§5 and Table 2, headline count"}],"minor_comments":[{"comment":"The notation V_n is used both for the per-plate predictive variance and, in the population statement of Proposition 3, for a function V(x). Please distinguish the empirical and population objects, and state whether V(x) is restricted to the parametric family σ_obs^2 + ψ(x)^T Σ ψ(x) throughout.","section":"§4.1, Eq. (11)"},{"comment":"The caption says 'shaded: ±2 predictive sd' but the text above describes a variance band. Please make the units consistent (standard deviation or variance).","section":"Figure 1 caption"},{"comment":"The caption contains 'gd/elbo', which appears to be a typo for 'closed/ELBO'. Please fix.","section":"Appendix D, Table 5 caption"},{"comment":"In the proof of the closed-route limitation, the phrase 'the closed profiles form a two-parameter family of residual-independent leverage curves' is imprecise: the closed covariance depends on α and σ_obs, but also on the feature matrix Ψ, so the family is not simply two-dimensional in the space of variance profiles. Please rephrase.","section":"Appendix C.2"},{"comment":"SCROLL is selected by validation NLL while MAP-based baselines are selected by validation MSE. The paper reports an oracle comparison showing the criterion has little effect, which is good. Still, the sentence 'the criterion neither favours SCROLL nor systematically penalises the MSE-selected baselines' is stronger than what a median/mean difference of 0.09 nats establishes; please report the distribution or worst-case difference.","section":"Appendix A, validation criterion"}],"recommendation":"major_revision","confidential_remarks":"The paper is promising and the empirical work is substantial, but the central theoretical claim (predictive-optimality of the free route) is not established: the proof of Proposition 3 only shows containment of one feasible set in another, not that the claimed target lies in the free family. This is fixable by either proving representability under stated conditions, adding a controlled experiment that isolates the variance mechanism, or reframing the contribution as a variational-family optimization with a proper-scoring objective. I would not reject, but the manuscript should not be accepted in its current form."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague — quick take on arXiv:2605.08446. The paper is worth reading, but don't take the 'predictive-optimal' framing at face value.\n\nWhat's new: the SCROLL objective — a shared-cavity predictive log-loss over a Gaussian last layer with free-routed covariance and joint empirical Bayes — plus the cavity×routing table that organizes existing last-layer methods (neural-linear, VBLL, SNGP) into a coherent design space. That framing is genuinely useful. The observation that the sequential-cavity Bethe objective recovers the neural-linear marginal likelihood is a nice unifying result, though it's a known telescoping identity. The empirical work is careful: validation selection per seed, significance tests, honest reporting of the naval failure mode and the depth-two erosion. Credit where due.\n\nThe soft spot is exactly where the reader put it. Proposition 3 claims the free route reaches V*(x)=E[(y-μ^Tψ)^2|x], but the proof only shows the free family contains the closed family, so inf over free ≤ inf over closed. That doesn't establish V* is representable by σ²+ψ^TΣψ for some PSD Σ. With N > H(H+1)/2, the feasible variance profiles form a low-dimensional cone; generic heteroscedastic V* won't lie in it. The prior term −log Z_w also biases the variance at finite N. So 'predictive-optimal by construction' overreaches. The empirical gains may just come from a larger variance family plus regularization — still useful, but not the advertised guarantee.\n\nAlso, calling this 'Bethe free energy minimization' is a stretch: Equation (4) drops the belief-entropy terms, so it's really a composite predictive likelihood, not the Bethe objective. And there's no code or data released, which matters for a paper whose contribution is largely empirical.\n\nThat said, the central mechanism — free routing a structured covariance under a proper scoring rule — is plausible and the experiments are honest. The paper admits its own limitations, which makes me trust the positive results more. It deserves peer review: a good referee will push on Proposition 3 and the terminology, but the core idea is solid enough to engage with.\n\nBottom line: worth a serious referee. I'd bring it to reading group and would cite the design-space taxonomy.","headline":"A genuinely useful design-space taxonomy and a new single-pass last-layer objective, but the 'predictive-optimal' claim rests on an unproven representability assumption.","tokens_in":26843,"tokens_out":2504,"would_cite":true,"duration_ms":27134,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T07","62F15"],"pacs":[],"model":"deepseek-v4-flash","headline":"Training a Bayesian neural network by minimizing the Bethe free energy with free-routed posterior beliefs makes the data term a strictly proper predictive score; in the Gaussian last layer this yields SCROLL, a single-pass model that matche","keywords":["Bethe free energy","Bayesian neural networks","free routing","shared cavity","strictly proper scoring rule","last-layer Gaussian","heteroscedastic regression","single-pass calibration"],"falsifier":"On a dataset with known input-dependent noise, train SCROLL-Full and compare its predictive variance σ²_obs+ψᵀΣψ to the true conditional residual variance at held-out inputs as N grows with H fixed: if the fitted profile cannot match V*(x) because the feasible set of profiles is a low-dimensional cone when N > H, then Proposition 3's representability claim fails and the claimed single-pass predictive optimality is unsupported.","tokens_in":25839,"feed_emoji":"🎯","tokens_out":6577,"duration_ms":61906,"temperature":0.7,"pith_summary":"This paper argues that Bayesian neural networks should not be trained by maximizing the evidence lower bound, because that objective pays a per-observation Jensen gap. It proposes direct gradient minimization of the Bethe free energy instead, with posterior beliefs treated as free parameters rather than pinned to a conjugate closed form. In this objective each observation is scored by its own predictive density, a strictly proper scoring rule, so the population optimum is the true conditional distribution. For a Gaussian last layer, exact Bayesian inference (the neural-linear marginal likelihood) turns out to be one corner of this objective that is evidence-optimal but not predictive-optimal; the shared-cavity free-routed interior can represent input-dependent residual variance that the exact corner cannot, yielding better NLL and calibration in one forward pass and matching or beating validation-tuned and ensembled references on most benchmarks.","feed_headline":"Free routing beats exact Bayesian inference on predictive density","feed_subtitle":"Minimizing Bethe free energy with free-routed beliefs gives calibrated predictions in one forward pass, matching ensembles and tuned baselin","key_machinery":"The machine is the factor-wise decomposition of the Bethe free energy into local log-partition terms −log Z, combined with two design axes: cavity (which data each plate's scoring belief sees) and routing (whether the belief parameters are pinned to the conjugate posterior or trained as free parameters). The named objects are Proposition 1, which shows the Bethe data term is never larger than the ELBO's expected-log-loss term, and Proposition 2, which shows the data term is a strictly proper score for any likelihood with a tractable predictive convolution. In the Gaussian last layer, the closed-form convolution turns each plate into the predictive log-loss (y−µᵀψ)²/(2V)+½log V with V=σ²_obs+","core_discovery":"The central claim is that the data term of the Bethe free energy is a strictly proper predictive score (Proposition 2), so training beliefs to minimize it is consistent for the conditional distribution itself, and that in the Gaussian last-layer instance the classic exact marginal likelihood is a stationary closed/sequential corner that is evidence-optimal rather than predictive-optimal (Theorem 1, Proposition 3). The free-routed interior, SCROLL, optimizes the marginal predictive log-loss with predictive variance σ²_obs + ψᵀΣψ, where Σ is trained as a free parameter; because this family is larger than the leverage-variance family of the exact posterior, it can fit heteroscedastic residual v","pith_inferences":["If the representability assumption is met, the paper's logic implies a general recipe: in any factor graph with a tractable predictive convolution, replacing the pointwise loss with the marginal predictive log-loss should improve calibration—a testable design rule for non-Gaussian or deeper Bayesian layers beyond the last-layer instance shown here.","Proposition 3 predicts a sharp, checkable dichotomy: on homoscedastic data free and closed routing should coincide, while on heteroscedastic data the free route should strictly dominate; this could serve as a cheap diagnostic for whether a dataset's noise is input-dependent.","The observed trade-off between NLL/calibration and OOD AUROC suggests that no single covariance can simultaneously track residual noise and distance-to-support; a natural extension is two variance channels (one aleatoric, one epistemic), which the paper does not explore."],"forward_implications":["A single training run and forward pass can replace grid-searched regularization and 5–50× ensembles: SCROLL is best-or-tied on NLL and calibration on 7/8 standard regression benchmarks and best on 4/5 large-scale or deep-feature benchmarks.","The exact neural-linear marginal likelihood, the usual gold-standard last-layer posterior, is evidence-optimal not predictive-optimal; users optimizing predictive density should leave that corner when noise is heteroscedastic.","The strict properness result is graph-agnostic, so the same shared-cavity free-routed objective applies to non-Gaussian likelihoods (probit, Poisson) and to multiple probabilistic layers, where no exact corner exists.","Because the free route trades leverage variance for predictive variance, it weakens out-of-distribution detection; the closed corner recovers leverage-based OOD, so the two behaviors are two ends of one design axis rather than separate methods.","Near-noiseless data is the documented failure regime: observation noise collapses and prior precision becomes non-identified (Corollary 1); there the exact corner is preferable to free routing."],"fun_headline_variants":["Free-routed beliefs beat exact Bayes on predictive density","One-pass SCROLL BNN matches or beats tuned baselines","Proper scoring via Bethe free energy improves BNN calibration","Unconstrained beliefs reach predictive optimum exact Bayes misses"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The paper's central claim that free routing reaches the predictive optimum rests on the assumption that one positive-semidefinite covariance matrix can represent the input-dependent target residual variance V*(x) at every input; the proof shows the free family contains the closed family, not that the target profile lies in it, and the headline experiments train mean and backbone jointly, outside the fixed-mean theorem in which the optimality claim is proven.","fun_headline_variants_meta":{"raw":{"variants":["Free-routed beliefs beat exact Bayes on predictive density","One-pass SCROLL BNN matches or beats tuned baselines","Proper scoring via Bethe free energy improves BNN calibration","Unconstrained beliefs reach predictive optimum exact Bayes misses"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00044,"raw_usage":{"total_tokens":2116,"prompt_tokens":836,"completion_tokens":1280,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":580,"completion_tokens_details":{"reasoning_tokens":1215}},"tokens_in":580,"tokens_out":1280,"duration_ms":14960,"temperature":1.0,"reasoning_tokens":1215,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T14:33:44.257008+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a dataset with known input-dependent noise, train SCROLL-Full and compare its predictive variance σ²_obs+ψᵀΣψ to the true conditional residual variance at held-out inputs as N grows with H fixed: if the fitted profile cannot match V*(x) because the feasible set of profiles is a low-dimensional cone when N > H, then Proposition 3's representability claim fails and the claimed single-pass predictive optimality is unsupported.","supporting_citations":[],"review_version":4}