{"id":"7b58d63f-0a57-4d45-b366-aefc3c30198b","arxiv_id":"2501.05564","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Predictions from a Bayesian neural network depend mainly on the means and variances of its weight distributions, not their shape, allowing analog hardware to use device noise as-is.","lead":"This paper shows that when Bayesian neural networks are trained with mean-field variational inference, the shape of the random noise on each weight matters much less than its mean and variance, at least for sufficiently wide networks. The practical payoff is that analog chip designers could use the natural noise of memory devices for Bayesian inference instead of engineering a precise Gaussian distribution.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Shape-insensitivity rests on an unproven finite-width CLT and is tested only with mild, symmetric distributions; a skewed/heavy-tailed shape could still matter at practical widths.","rationale":"Working in good faith, the paper does make a plausible empirical case: the three distributions used are quite different, and the convergence trend in Fig. 4 makes sense via CLT heuristics. The quadrature/inverse-CDF machinery is a solid contribution and not the weak point. The load-bearing issue is the gap between the asymptotic CLT story and the unconditional word 'insensitive' in the title/abstract. For a hardware designer, the question is not whether convergence happens at infinite width but whether width-64 or width-16 layers are enough, and whether a device noise distribution with large skewness or heavy tails would invalidate the shortcut. The manuscript currently provides no error bars, no rate, and no shape stress test, so the conditionality in the reader's verdict is appropriate. If the proposed test closes the gap, the practical recommendation stands; if not, the claim must be narrowed.","tokens_in":9819,"tokens_out":6560,"duration_ms":71725,"concrete_test":"Re-run the Sec. 4.1 energy-distance comparison with a standardized lognormal (markedly skewed) and a Student-t(4) (heavy-tailed) weight distribution matched to the Gaussian in mean and variance, at depths 1 and 16 and widths 16/32/64/128, using at least 20 initialization/seed replicates and matched MC sample sizes. Report the KL/energy distance to the Gaussian-weight predictive distribution with bootstrap confidence intervals, and also evaluate at several x values, not only x=0. If the distance does not decrease to within MC noise at width 64, the finite-width shape-insensitivity claim fails; if it does, the claim is supported beyond the shapes tested.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that, with matching means and variances, the weight-shape has no effect on predictive distributions. The only mechanistic support is the CLT argument in Section 1, but that is asymptotic: it does not bound the residual dependence on higher moments at the widths used here, especially in deep nonlinear networks where non-Gaussianity can be regenerated by ELU/ReLU layers. The experiments do not stress the claim: the energy-distance study (Sec. 4.1) examines only one input point (x=0), uses a near-Gaussian device fit and a symmetric bimodal mixture, and reports KL values without repeated-seed error bars or MC uncertainty, so Fig. 4's decreasing curves could be dominated by estimator noise. Fig. 4 also shows visible shape dependence at width 1, and Sec. 5 concedes that 'sufficiently wide neural network layers are still necessary' and narrow/deep networks are future work. Thus the title's 'insensitive to the shape' is stronger than what has been demonstrated; the claim may hold asymptotically but is not established for practical finite-width BNNs, nor for skewed or heavy-tailed device noise.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes a mean-field variational inference (MFVI) framework in which the variational distribution is a fitted model of real analog device noise (Bayes-MTJ, with ECRAM in Appendix D) rather than a Gaussian. Its central claim is that the posterior predictive distributions of BNNs with matching weight means and variances converge to the same distribution regardless of the shape of the variational weight distribution. This is supported by comparing Gaussian, fitted-device, and bimodal weight distributions on an energy-distance task (Section 4.1), a 1D regression task (Section 4.2), and the UTKFACE benchmark (Section 4.3), with numerical tools for quadrature and inverse sampling developed in Section 3.","tokens_in":10072,"tokens_out":4537,"duration_ms":46806,"significance":"If the shape-invariance claim held as stated, the result would be practically significant: analog hardware designers could ignore the shape of device noise and match only the mean and variance of the variational distribution, enabling a train-in-software, deploy-in-hardware workflow. The paper also contributes useful numerical machinery, including custom quadratures for non-Gaussian device distributions and a corrected inverse-CDF sampler. However, the significance is currently bounded by the fact that the core claim is an empirical finite-width trend backed by an unquantified CLT heuristic, while the paper's own Section 5 restricts the result to sufficiently wide networks and defers narrow deep networks to future work.","major_comments":[{"comment":"The headline energy-distance experiment tests only one input location, x=0, and reports KL divergences without repeated-seed or Monte Carlo error bars, so it is unclear whether the apparent convergence is real or an artifact of estimator noise. More importantly, Fig. 4 shows visible shape dependence at width=1, and Section 5 concedes that \"sufficiently wide neural network layers are still necessary\" and that narrow deep networks are future work. This directly contradicts the unqualified title and abstract claim; the evidence supports at most an asymptotic wide-network statement. The authors should either provide quantitative convergence results with uncertainties (repeated seeds, error bars, a stated width threshold) or restrict the claim accordingly.","section":"Section 4.1, Fig. 4"},{"comment":"The mechanism invoked for shape-insensitivity is the central limit theorem \"within each layer,\" but no theorem or quantitative rate is given. In deep networks with ELU or ReLU activations, layer outputs are not independent sums of weights; nonlinearities can regenerate non-Gaussianity, so higher weight moments could matter at practical widths and depths. Because this CLT heuristic is the only mechanistic support for the invariance claim, the manuscript needs either a rigorous finite-width/depth bound or a much more comprehensive empirical stress test than the few configurations shown in Fig. 4.","section":"Section 1, final paragraph; Section 5"},{"comment":"The scalar regression and UTKFACE comparisons are qualitative. There are no numerical discrepancies between predictive distributions (e.g., Wasserstein or energy distance with confidence intervals), no repeated seeds, and no baseline showing what a meaningful difference would look like. The statement in Section 4.3 that predictions are \"identical\" is not supported by calibration-only evidence, because calibration curves can match while predictive densities differ substantially. Quantitative distributional discrepancy metrics with uncertainty are needed to substantiate the convergence claim.","section":"Section 4.2 and Section 4.3, Figs. 5 and 6"},{"comment":"The only non-Gaussian shapes tested are the Bayes-MTJ fit, which is near-Gaussian, and a symmetric two-component Gaussian mixture; the ECRAM shape in Appendix D is also close to Gaussian. These tests do not justify the phrase \"regardless of the shape.\" Skewed or heavy-tailed device-like distributions (e.g., log-normal, exponentially tilted, or truncated Pareto) should be tested, or the claim should be explicitly limited to symmetric, light-tailed shapes that are close to Gaussian.","section":"Eq. (3), Fig. 1, Appendix D"}],"minor_comments":[{"comment":"The optimization is written as min_alpha ELBO(alpha), but the ELBO is to be maximized; Eq. (2) minimizes the negative ELBO. Please fix the sign and notation for consistency.","section":"Eq. (1)"},{"comment":"There are typos in the text: \"can can appear\" and \"opreations\" should be corrected. Also, \"This approach is restricted to mean field variational inference\" is a sentence fragment.","section":"Section 2"},{"comment":"The label \"Divergence to Gaussian\" is ambiguous. Please specify which divergence is plotted (KL, energy distance, etc.) and how it is estimated.","section":"Fig. 4, right panel"},{"comment":"The energy-distance experiment uses a single synthetic target distribution at x=0. The invariance claim would be more convincing if the comparison were repeated at multiple input points and for non-Gaussian target distributions.","section":"Section 4.1"},{"comment":"The parameterization of qD in Eq. (3) is clear, but the constraints imposed during maximum likelihood fitting (normalization, zero mean, and variance matching) are not fully described. Please state them explicitly.","section":"Section 3.1.1"}],"recommendation":"major_revision","confidential_remarks":"This is a workshop-scale empirical study whose title and abstract claim more than the experiments establish. The numerical infrastructure is solid, and the invariance idea is attractive, but the core claim needs either a rigorous bound or a substantially stronger empirical demonstration with error bars, additional input locations, and a wider range of distribution shapes. If the authors can provide that, a revised version could be suitable; as it stands, the paper would not support the advertised conclusion."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper's main empirical result—that predictive distributions from BNNs with matched weight means and variances converge as width grows—holds up in the experiments shown, but the title oversells it. The honest reading is that the paper demonstrates convergence trends, not full insensitivity at practical widths.\n\nWhat's genuinely new: the proposal to use fitted device noise as the variational base distribution, with careful numerical machinery (custom Gaussian quadrature, inverse CDF with singularity correction). That is a real contribution for anyone trying to train BNNs on analog arrays. The energy-distance study in Fig. 4 is the strongest evidence, showing KL divergence to the Gaussian predictive decreasing with width for both device and bimodal distributions. The paper also does a second device (ECRAM) as a sanity check. Credit where due: the authors are transparent about the width requirement and list narrow/deep networks as future work.\n\nSoft spots, in rough order of severity. First, the proposed training method is not actually exercised in the main experiments: Sections 4.2 and 4.3 train with a Gaussian and swap the base distribution only at inference. So the 'method for MFVI training using real device noise' is implemented in code but never validated end-to-end. That is a gap. Second, the CLT argument is informal and asymptotic; the paper doesn't bound the error at finite width. Third, the stress test is mild: the device fits are near-Gaussian and the bimodal is symmetric. Skewed or heavy-tailed shapes could still matter, especially in deep ReLU/ELU networks that can regenerate non-Gaussianity. Fourth, no repeated-seed error bars; the Fig. 4 KL curves could be partly estimator noise.\n\nThese are addressable, not fatal. The regression and UTKFACE results are consistent with the claim but qualitative. The paper deserves a serious referee; I'd recommend conditional acceptance with a request for repeated-seed statistics, a true device-training experiment, and at least one skewed/heavy-tailed distribution. Who is it for: analog hardware folks and BNN practitioners wanting to know whether device noise shape matters. Worth a read, but don't take the title literally.","headline":"Plausible shape-insensitivity result, honestly bounded, but the evidence is thinner than the title suggests.","tokens_in":10560,"tokens_out":2473,"would_cite":true,"duration_ms":23545,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Bayesian neural networks are insensitive to the shape of the weight distribution: with matching weight means and variances, predictive distributions converge regardless of variational form.","keywords":["Bayesian neural networks","mean-field variational inference","analog in-memory computing","device noise","central limit theorem","predictive distributions","uncertainty calibration","weight-distribution shape"],"falsifier":"Measure predictive distributions in the Section 4.1 energy-distance setup at fixed large width and growing depth, using a weight distribution with very high skewness or kurtosis; if the divergence from the Gaussian-weight predictive distribution does not shrink toward zero as width grows, or is visible in calibration curves at production widths, the per-layer CLT explanation is wrong.","tokens_in":9630,"feed_emoji":"⚡","tokens_out":4152,"duration_ms":39680,"temperature":0.7,"pith_summary":"This paper asks whether the shape of the weight distribution matters to a Bayesian neural network trained with mean-field variational inference. The authors train networks with Gaussian variational weights and then swap in two non-Gaussian distributions with the same means and variances: the measured noise of a magnetic tunnel junction device and a deliberately bimodal mixture. Across energy-distance, scalar-regression, and image-age tasks, the predictive distributions are nearly unchanged by the swap, provided the network layers are reasonably wide. The paper reads this as the per-layer central limit theorem erasing the higher moments of the weight distribution, so a hardware designer only needs to match the mean and variance of device noise, not its shape.","feed_headline":"Device noise shape is irrelevant for Bayesian neural nets","feed_subtitle":"With matching weight means and variances, Gaussian and analog-device noise produce the same predictions.","key_machinery":"The load-bearing object is the per-layer central limit theorem for pre-activation sums in a feed-forward network. Because each neuron computes a sum over many independent weights, the argument runs, the distribution of that sum converges to a Gaussian with a mean and variance fixed by the weight means and variances alone; higher cumulants of the weight distribution wash out. Supporting machinery is a numerical toolkit for working with non-Gaussian variational families: a maximum-likelihood fit to measured device noise, custom Gaussian quadrature rules for expectations and KL divergences with that distribution, and a corrected inverse-transform sampler to generate device-noise samples.","core_discovery":"The central discovery is an empirical invariance: for deep mean-field Bayesian neural networks, the posterior predictive distribution depends on the first two moments of each variational weight distribution and is effectively independent of its higher shape. The authors establish this by training with Gaussian distributions, then replacing the Gaussian base distribution by a fitted real-device distribution and by a bimodal mixture at fixed $\\mu$ and $\\sigma$, and observing that regression fits, calibration curves, and predictive densities coincide as layer width grows. They attribute the invariance to the CLT within each layer: each pre-activation is a sum of many independent weighted inputs, so its distribution is determined by the mean and variance of the summed weights regardless of the individual weight-distribution shape.","pith_inferences":["If the per-layer CLT explanation is right, the required width should scale with the tail-heaviness or skewness of the weight distribution; one testable prediction is that heavier-tailed device noise requires wider layers to reach a given divergence tolerance.","The invariance suggests a practical calibration shortcut: hardware uncertainty metrics, such as expected calibration error, could be certified in software with Gaussian weights and then carried over to analog hardware without re-measuring the device-specific noise shape.","The same argument should apply to other linear-in-weight architectures, such as convolutional layers, but may fail for attention blocks or normalization layers where pre-activations are not simple sums of independent weights; the paper's convolutional result hints at this, but the scope is untested."],"forward_implications":["Analog hardware designers can use memory devices whose noise has the right mean and variance but any shape; the predictive uncertainty of the deployed BNN will match the software-trained Gaussian BNN.","A network can be trained in software with Gaussian variational weights, and its scale and shift parameters transferred directly to hardware sampling device noise.","The invariance is quantitative in width: narrow layers show measurable shape dependence, but the divergence to the Gaussian predictive distribution falls quickly as width increases.","A second device with qualitatively different noise shape, an ECRAM device, shows the same convergence in the energy-distance test, suggesting the result is not specific to one device."],"supporting_citations":[{"why":"Supplies the Bayes-MTJ stochastic device whose measured noise is the non-Gaussian variational distribution used throughout the main experiments.","marker":"[9]"},{"why":"Provides the heteroscedastic regression model used for the scalar-regression and UTKFACE predictive-distribution comparisons.","marker":"[18]"},{"why":"Supplies the ECRAM device with a different noise shape used in the second-device check in Appendix D.","marker":"[12]"},{"why":"Formulates variational inference and the ELBO/KL objective that the training method extends to non-Gaussian device distributions.","marker":"[5]"},{"why":"Is the UTKFACE face-age dataset on which the calibration-curve comparison is run.","marker":"[19]"}],"fun_headline_variants":["Weight shape doesn't affect Bayesian neural net predictions","Analog BNNs: shape of noise distribution is a non-issue","Only weight mean and variance shape Bayesian predictions","Noise shape ignored by analog Bayesian neural networks","CLT spares BNNs from weight distribution shape concerns"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that in each layer the sum over many independent weights becomes Gaussian fast enough that only the mean and variance of the weight noise matter, and that this already holds at the network widths used in practice.","fun_headline_variants_meta":{"raw":{"variants":["Weight shape doesn't affect Bayesian neural net predictions","Analog BNNs: shape of noise distribution is a non-issue","Only weight mean and variance shape Bayesian predictions","Noise shape ignored by analog Bayesian neural networks","CLT spares BNNs from weight distribution shape concerns"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000592,"raw_usage":{"total_tokens":2707,"prompt_tokens":809,"completion_tokens":1898,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":425,"completion_tokens_details":{"reasoning_tokens":1820}},"tokens_in":425,"tokens_out":1898,"duration_ms":14675,"temperature":1.0,"reasoning_tokens":1820,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:14:05.496176+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure predictive distributions in the Section 4.1 energy-distance setup at fixed large width and growing depth, using a weight distribution with very high skewness or kurtosis; if the divergence from the Gaussian-weight predictive distribution does not shrink toward zero as width grows, or is visible in calibration curves at production widths, the per-layer CLT explanation is wrong.","supporting_citations":[{"cited_title":"Patrick Xiao, Jaesuk Kwon, Bert J","cited_arxiv_id":null,"evidence_quote":"Supplies the Bayes-MTJ stochastic device whose measured noise is the non-Gaussian variational distribution used throughout the main experiments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the ECRAM device with a different noise shape used in the second-device check in Appendix D."},{"cited_title":"Age progression/regression by conditional adver- sarial autoencoder","cited_arxiv_id":null,"evidence_quote":"Is the UTKFACE face-age dataset on which the calibration-curve comparison is run."}],"review_version":1}