{"id":"cfab71fe-6563-4a65-a31d-488967998989","arxiv_id":"2501.18665","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"BARNN converts autoregressive and recurrent networks into Bayesian versions via time-dependent variational dropout and a temporal aggregated-posterior prior, yielding calibrated uncertainty on PDE and molecule-generation tasks.","lead":"A new training method lets any autoregressive neural network, for weather, molecules, or PDEs, output calibrated uncertainty estimates with its predictions. BARNN uses time-dependent variational dropout plus a self-referential 'prior' built from the model's own training outputs, and shows improved calibration on PDE and SMILES benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The tVAMP prior in Eq. (8) is computed from the current batch's α_t, so Eq. (9)'s KL is a self-referential within-batch regularizer that vanishes at N=1 or when α_t is homogeneous; if it does no calibration work, BARNN's Bayesian claim reduces to input-dependent MC dropout.","rationale":"The paper's central claim is that BARNN, via tVAMP, turns any autoregressive or recurrent model into a Bayesian one with calibrated uncertainty. For that claim to hold, Eq. (4) must be a genuine ELBO with a fixed prior. The weakest point is the transition from Eq. (7) to Eq. (8): the optimal prior is the data-aggregated posterior, but its practical implementation uses the current mini-batch's α statistics. Since α_t comes from Eψ and the batch itself, the KL in Eq. (9) is a function of q and the data, not a fixed Bayesian regularizer. The batch-size-1 degeneracy makes this concrete. This matches the reader's weakest assumption. I do not think this makes the experiments invalid: the model may still provide good MC-dropout-style uncertainty, and the public code allows direct checks. But the strongest claim—'principled Bayesian'—is not established unless the KL term is shown to matter and to correspond to a proper prior. Hence my verdict is unchanged: CONDITIONAL, pending the ablation in concrete_test.","tokens_in":24582,"tokens_out":5865,"duration_ms":57270,"concrete_test":"Use the public code to train the Burgers FNO under four conditions: (i) the current tVAMP objective; (ii) the same objective with the KL term of Eq. (9) removed; (iii) the full objective with batch size 1, where the tVAMP KL is identically zero; (iv) the full objective with batch size 32 but all α_t within each layer clamped to their batch mean β_t so that the KL is again zero. Compare 320-step NLL and ECE from Table 2. If conditions (ii)–(iv) match (i) within seed noise, the tVAMP prior is not doing the calibration work and the Bayesian interpretation fails; if calibration degrades, the concern is settled in the paper's favor. As a secondary diagnostic, log the KL value and its gradient norm during training: values near zero throughout indicate the prior is vacuous.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that BARNN is a principled Bayesian retrofit via the ELBO in Eq. (4). The tVAMP prior is the load-bearing piece: Eq. (7) defines the optimal prior as the aggregated variational posterior over the data distribution, and Eq. (8) replaces that expectation with statistics β_t, γ_t of the current mini-batch. Because β_t and γ_t are computed from the same α_t values that define the posterior, the KL in Eq. (9) is not D_KL[q||p] for a fixed prior; it is a penalty on within-batch deviations of α_t from its own mean. For batch size 1, β_t = γ_t = α_t and the KL is exactly 0. For any batch where α_t is identical across samples, the KL is also 0. Thus the tVAMP term provides no absolute constraint on the scale of the dropout coefficients or on the predictive variance; it only discourages within-batch heterogeneity. Appendix A.2 optimizes over p(ω_t) with q held fixed, but training re-estimates p every step from q, so the resulting objective is not the ELBO of a fixed Bayesian model. If this regularizer is negligible in practice, the reported calibrated NLL/ECE gains may be produced by Monte Carlo averaging over an input-dependent dropout mask whose variance is tuned by the likelihood alone—that is, standard deterministic training plus MC dropout—and the central 'principled Bayesian' claim is not supported by the experiments. This is not a dispute with the empirical results; it is a gap between the derivation and the implemented objective.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces BARNN, a variational Bayesian framework for autoregressive and recurrent models. The authors define a joint distribution over states and time-indexed network weights, derive a temporal ELBO, and approximate the posterior with a variational dropout parameterization in which per-layer dropout coefficients alpha_t are produced by an encoder. A new 'temporal VAMP' prior is defined as the aggregated posterior in time and estimated from minibatch statistics. Experiments on a synthetic time series, on Burgers/KS/KdV PDE forecasting with Fourier Neural Operators, and on SMILES-based molecule generation with LSTMs report competitive or better accuracy and better calibration (NLL, ECE) than MC dropout, ARD, input perturbation, PDE Refiner, and dropout LSTM baselines.","tokens_in":25012,"tokens_out":12384,"duration_ms":114303,"significance":"If the framework is sound, BARNN is a broadly applicable retrofit: it adds a small encoder and MC sampling to any autoregressive/recurrent model, with public code and a complete set of derivations in the appendix. The empirical evaluation is substantial and uses standard UQ metrics over two very different domains, which gives the headline accuracy and calibration claims some support. The main reservation is that the Bayesian interpretation of the method is not currently supported by the implemented objective: the tVAMP prior is re-estimated from the same minibatch as the posterior, so the KL term in Eq. (9) is a within-batch regularizer rather than a divergence to a fixed prior. This does not automatically invalidate the empirical results, but it changes what the method is and requires a fix or a reframing before the 'principled Bayesian' claim can be accepted.","major_comments":[{"comment":"The tVAMP prior is not a fixed prior. Eq (8) defines beta_t and gamma_t as averages of the alpha_t values produced by the encoder on the current mini-batch, and Eq (9) then measures each sample's posterior against those same batch statistics. For a batch of size 1, beta_t = gamma_t = alpha_t and the KL is exactly 0; for any batch in which all alpha_t are equal, the KL is also 0. Consequently the KL term imposes no absolute constraint on the scale of alpha_t, and the objective in Eq (4) is not the ELBO of the generative model in Eq (2). Appendix A.2 maximizes the prior with q held fixed, but Algorithm 2 re-estimates the prior from the current batch at every gradient step, so the conditions of that proof are not met. The authors should replace the batch-estimated prior by a proper VAMP-style pseudo-input prior, or by a moving average or held-out estimate of beta_t and gamma_t, and should verify that the resulting objective is still a lower bound or an unbiased stochastic estimate of one.","section":"§3.3, Eqs (7)-(9)"},{"comment":"The implemented loss is inconsistent with the stated model and the sign in the algorithm appears wrong. The text says the objective is the minimization of the one-step MSE plus the negative KL divergence, i.e., MSE + D_KL as a loss, which matches maximizing Eq (4). Algorithm 2, however, writes L = ||y_hat_t^k - y_t||^2 - D_KL(alpha_t) and then descends the gradient; minimizing this would drive the KL upward, which is not the ELBO. In addition, the Gaussian observation model in Eq (10) contains a fixed observation noise sigma_t that is said not to be learned, but its value is never reported; if the likelihood term is to be MSE/(2 sigma_t^2), the relative weight of the KL regularization depends on an unreported hyperparameter. Please correct the sign and report sigma_t, or state explicitly that Algorithm 2 is the intended objective and adjust the model equation and the derivation of the ELBO accordingly.","section":"§3.4 and Appendix D, Algorithm 2"}],"minor_comments":[{"comment":"The predictive distribution should be E_{omega_t ~ q_phi}[p(y_t | y_{0:t-1}, omega_t)]; the displayed formula contains an extra q_phi factor inside the expectation, which would make the expression dimensionally inconsistent with a probability distribution.","section":"Appendix A.4, Eq (27)"},{"comment":"The column header in Table 1 says 'MSE (↓)' while the text in Section 4.1 refers to RMSE; please align the notation.","section":"Table 1 and Section 4.1"},{"comment":"The captions for Figures 12 and 13 appear to be swapped: Figure 12 is captioned as a t-SNE plot but the text refers to it as temperature versus validity, and Figure 13's caption describes density distributions while the figure shows temperature versus uniqueness.","section":"Figures 12 and 13"},{"comment":"The 'MAP estimate' curve is not defined in the text; please specify how the MAP weights are extracted from the variational posterior (for example, alpha_t = 1) and whether the same encoder is used at inference.","section":"Figure 4"},{"comment":"The claim that BARNN excels at modelling long-range dependencies would be strengthened by a control experiment that varies sequence length or ring count while holding the architecture fixed; the current ring-closure analysis is suggestive but indirect.","section":"Section 4.3"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is within the scope of the venue and the experimental assets are valuable. The central issue is fixable: the authors should either switch to a proper VAMP-style prior with pseudo-inputs or rigorously justify the batch-estimated prior as a stochastic regularization scheme, and they should correct the sign in Algorithm 2 and report the observation noise. If the Bayesian framing cannot be maintained, the empirical contribution could still be published as a dropout-based approach, but the claims in the abstract and Section 1 would need to be substantially softened."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is a workable, well-tested extension of variational dropout to time-varying weights, with a temporal aggregated-posterior prior (tVAMP) and an encoder that outputs per-layer dropout rates. The writing is clear, the derivations in Appendices A.1-A.3 check out, and the empirical work is solid: PDE forecasting with FNO backbones and SMILES generation with LSTMs, compared against sensible baselines (MC dropout, ARD, input perturbation, PDE Refiner). Calibration metrics (NLL, ECE) are reported, code is public, and the ensemble convergence plots are a nice touch. That's real work.\n\nThe main soft spot is the tVAMP prior. Eq. (8) estimates the prior from the same batch that defines the posterior: beta and gamma are batch means of alpha. So the KL in Eq. (9) is not a distance to a fixed prior; it vanishes for batch size 1 and whenever alpha is homogeneous within the batch. It is a within-batch regularizer, not an absolute constraint. The paper frames Eq. (4) as an ELBO with a principled Bayesian prior, but the implemented objective is closer to 'deterministic training plus input-dependent MC dropout, regularized by a variance penalty on alpha.' That is not fatal to the empirical results—the method still gives calibrated uncertainty in their tests—but it means the central 'principled Bayesian' claim is oversold. The derivation in Appendix A.2 optimizes p with q held fixed, but training re-estimates p every step from q, so the fixed-point interpretation is shaky.\n\nMinor issues: the posterior is per-layer scalar, so uncertainty is coarse; the observation noise in the PDE experiments is not learned, which limits the aleatoric component; and the priority claim ('first to turn any AR/RNN into Bayesian') ignores existing Bayesian RNN work, though they do cite the closest.\n\nFor a reader: this is useful for anyone doing UQ in neural PDE solvers or generative chemistry who wants a drop-in stochastic version of an existing architecture. The empirical gains are plausible but not re-run by me; the lack of a deep-ensemble baseline is a gap, since deep ensembles are the standard strong baseline for calibration.\n\nRecommendation: send to peer review. The method is novel enough and the experiments are reproducible enough to deserve referee time, even though the circular prior construction needs to be addressed in revision.","headline":"A useful, empirically solid UQ method whose 'Bayesian' prior is actually a within-batch regularizer; worth reviewing, but the principled claim needs rework.","tokens_in":25486,"tokens_out":1831,"would_cite":true,"duration_ms":16867,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"BARNN claims a general recipe for turning any autoregressive or recurrent network into a Bayesian one: evolve the weights alongside the states, sample per-timestep weights from a variational posterior, and use a temporal…","keywords":["Bayesian deep learning","variational dropout","autoregressive models","recurrent neural networks","uncertainty quantification","neural PDE solvers","molecular generation","variational mixture of posteriors"],"falsifier":"Train BARNN on the PDE or molecule task with batch size 1, where the tVAMP prior statistics coincide with the single sample and the KL term in equation (9) vanishes; if the ECE and NLL remain as good as with full batches, the calibration gain comes from the input-dependent Monte Carlo sampling rather than from the proposed variational prior.","tokens_in":24341,"feed_emoji":"🎲","tokens_out":6851,"duration_ms":57911,"temperature":0.7,"pith_summary":"BARNN proposes a general recipe for turning any autoregressive or recurrent neural network into a Bayesian one. The paper's central claim is that by letting the network weights evolve alongside the observable states, sampling new weights at every time step from a variational posterior, one obtains predictive distributions with calibrated uncertainty estimates. The method derives a temporal evidence lower bound, reuses variational dropout for scalability, and introduces a time-dependent mixture-of-posteriors prior (tVAMP) to keep inference efficient. On PDE forecasting (Burgers, Kuramoto-Sivashinsky, Korteweg-de Vries) and SMILES-based molecule generation, the paper reports that the resulting model is at least as accurate as deterministic baselines and better calibrated, with a particular gain in long-range dependencies such as ring closure in molecules. A sympathetic reader comes away with the promise that any existing autoregressive or recurrent architecture can be retrofitted with Bayesian uncertainty by adding a small encoder and doing Monte Carlo sampling at inference.","feed_headline":"One small encoder turns any recurrent net Bayesian and calibrated","feed_subtitle":"Time-dependent variational dropout plus a temporal prior cuts NLL and ECE on PDE rollouts and SMILES generation.","key_machinery":"The load-bearing mechanism is the time-indexed reparametrization of the weights, ω_t^l = α_t^l Ω^l (1+ε) with ε ∼ N(0,I), where Ω are the static neural-network weights, α_t are positive per-layer dropout coefficients output by a small encoder E_ψ that reads previous states (and, for RNNs, the hidden state), and ε provides stochasticity. This makes sampling practical for large networks via the local reparametrization trick, which samples activations rather than full weight matrices. The companion tVAMP prior p(ω_t) = ∫ p(y_{0:t−1}) q(ω_t | y_{0:t−1}) dy_{0:t−1} is approximated using batch statistics β_t and γ_t; because both posterior and prior are normal with means proportional to the same Ω, the KL divergence in the ELBO becomes independent of Ω, turning Bayesian inference into a standard log-likelihood (or MSE/cross-entropy) loss plus a cheap KL regularizer per layer. That regularizer is what the paper credits for calibrated, well-shaped predictive uncertainties, and the encoder plus Monte Carlo sampling is what transfers the recipe to arbitrary autoregressive or recurrent architectures.","core_discovery":"The central object is a joint distribution over states and weights, p(y_0:T, ω_1:T)=∏_t p(y_t | y_<t, ω_t) p(ω_t), in which the weights are refreshed at every time step instead of being fixed after training. The paper derives a variational lower bound whose per-step form resembles a VAE ELBO, and then parametrizes the posterior over weights using variational dropout: each layer's weights are reparametrized as ω_t^l = α_t^l Ω^l (1+ε), with static base weights Ω, time-dependent scalar dropout coefficients α_t produced by a small encoder, and Gaussian noise ε. For the prior it introduces the tVAMP prior, the aggregate of variational posteriors over past states, whose batch-estimated statistics make the KL term independent of Ω and hence cheap to compute. The paper's claim is that this combination—time-varying weights, the temporal ELBO, and the tVAMP prior—yields a Bayesian autoregressive or recurrent model that matches or beats deterministic accuracy while providing calibrated and sharp uncertainties. Empirically, it reports lower NLL and ECE than Monte Carlo dropout, input perturbation, ARD dropout, and PDE-Refiner baselines on the three PDEs, and better validity, uniqueness, novelty, and molecular property statistics in SMILES generation.","pith_inferences":["My inference: if the KL regularizer is genuinely active, the same retrofit should transfer to transformer-style language models, giving token-level calibrated probabilities; the reported molecule results are a proxy for that, not a demonstration on transformers.","My inference: since the tVAMP prior is re-estimated from the current batch at every step, the method is best understood as amortized, input-dependent variational dropout; whether it is a fully Bayesian posterior over weights depends on the prior being held fixed after training, which the batch-estimated form does not strictly do.","My inference: the convergence of RMSE, NLL, and ECE at about 30 ensemble members suggests a practical testable recipe: set ensemble size by monitoring ECE during rollout rather than using a fixed 100-member ensemble.","My inference: if the gains persist when the encoder is applied to the hidden states of a large pretrained model, this could become a lightweight calibration layer for deployed systems; a direct comparison against temperature scaling and conformal prediction would clarify what the Bayesian component adds."],"forward_implications":["Any existing neural PDE solver or RNN language model can be turned Bayesian with a small encoder and Monte Carlo resampling at inference, without redesigning the backbone.","Calibrated uncertainty is available at roughly 30 stochastic forward passes, and if uncertainty is not needed, the maximum-a-posteriori weights give deterministic-speed predictions with the same reported RMSE.","Long-range dependencies improve: in SMILES generation, ring-closure errors drop by about 30% over a dropout LSTM and 50% over a plain LSTM, with larger gains as the number of rings grows.","On Burgers, Kuramoto-Sivashinsky, and Korteweg-de Vries, BARNN achieves lower negative log-likelihood and expected calibration error than Monte Carlo dropout, input perturbation, ARD dropout, and PDE-Refiner over 320-step unrolls.","The method supplies a decomposition of predictive variance into epistemic and aleatoric components via the law of total variance."],"supporting_citations":[{"why":"Supplies variational dropout and the local reparametrization trick used to sample time-dependent weights efficiently.","marker":"[Kingma et al., 2015]"},{"why":"Provides the dropout-as-Bayesian-approximation baseline and the Monte Carlo sampling interpretation that BARNN extends.","marker":"[Gal & Ghahramani, 2016]"},{"why":"Introduces the VAMP prior whose temporal version, tVAMP, is used as the prior over weights.","marker":"[Tomczak & Welling, 2018]"},{"why":"Gives the VAE/ELBO machinery from which the temporal variational lower bound is derived.","marker":"[Kingma & Welling, 2014]"},{"why":"Supplies the autoregressive neural PDE solver framework and the training and evaluation setting for the PDE experiments.","marker":"[Brandstetter et al., 2022b]"},{"why":"Provides the input perturbation baseline used for comparison on long PDE rollouts.","marker":"[Lam et al., 2022]"},{"why":"Provides PDE-Refiner, the strong UQ baseline for neural PDE solvers that BARNN is compared against.","marker":"[Lippe et al., 2024]"},{"why":"Contributes the ChEMBL SMILES dataset and the chemical language modeling context for molecule generation.","marker":"[Özçelik et al., 2024]"},{"why":"Defines the SMILES LSTM baseline and the molecular property metrics used in the evaluation.","marker":"[Segler et al., 2018]"}],"fun_headline_variants":["Time-varying weights bring Bayesian rigor to recurrent nets","A small encoder makes any recurrent net Bayesian","Calibrated uncertainty for long-range dependencies in RNNs","BARNN: Variational dropout that adapts over time","Refreshing weights each step yields calibrated Bayesian RNNs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the KL regularizer in the BARNN objective is a genuine constraint: because the tVAMP prior is estimated from the same batch statistics as the posterior, that term can shrink toward zero, and the paper's calibrated uncertainties are only attributable to the Bayesian machinery if the KL term is actually active during training.","fun_headline_variants_meta":{"raw":{"variants":["Time-varying weights bring Bayesian rigor to recurrent nets","A small encoder makes any recurrent net Bayesian","Calibrated uncertainty for long-range dependencies in RNNs","BARNN: Variational dropout that adapts over time","Refreshing weights each step yields calibrated Bayesian RNNs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000498,"raw_usage":{"total_tokens":2464,"prompt_tokens":992,"completion_tokens":1472,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":608,"completion_tokens_details":{"reasoning_tokens":1394}},"tokens_in":608,"tokens_out":1472,"duration_ms":11200,"temperature":1.0,"reasoning_tokens":1394,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T23:31:30.305078+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train BARNN on the PDE or molecule task with batch size 1, where the tVAMP prior statistics coincide with the single sample and the KL term in equation (9) vanishes; if the ECE and NLL remain as good as with full batches, the calibration gain comes from the input-dependent Monte Carlo sampling rather than from the proposed variational prior.","supporting_citations":[{"cited_title":"P., Salimans, T., and Welling, M","cited_arxiv_id":null,"evidence_quote":"Supplies variational dropout and the local reparametrization trick used to sample time-dependent weights efficiently."},{"cited_title":"and Ghahramani, Z","cited_arxiv_id":null,"evidence_quote":"Provides the dropout-as-Bayesian-approximation baseline and the Monte Carlo sampling interpretation that BARNN extends."},{"cited_title":"and Welling, M","cited_arxiv_id":null,"evidence_quote":"Introduces the VAMP prior whose temporal version, tVAMP, is used as the prior over weights."},{"cited_title":"PDE-Refiner: Achieving Accurate Long Rollouts with Neural PDE Solvers","cited_arxiv_id":null,"evidence_quote":"Provides PDE-Refiner, the strong UQ baseline for neural PDE solvers that BARNN is compared against."},{"cited_title":"H., Kogej, T., Tyrchan, C., and Waller, M","cited_arxiv_id":null,"evidence_quote":"Defines the SMILES LSTM baseline and the molecular property metrics used in the evaluation."}],"review_version":1}