REVIEW 4 major objections 5 minor 34 references
Improving the Noise Estimation of Latent Neural Stochastic Differential Equations
T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read Latent neural SDEs underestimate the noise in stochastic time series, and adding a penalty on the diffusion term to the loss restores the correct noise level for constant-diffusion systems.
desk verdict A clear diagnosis of why latent neural SDEs underestimate noise, plus a simple global penalty that can match path statistics for constant-diffusion systems; the abstract overclaims, but the core analysis is worth refereeing. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the noise penalty term $L_G = \int_0^T \|g_\theta(u(t),t)\|\, dt$ added to the latent neural SDE objective, giving $\max_{\theta,\phi}\mathbb{E}_{x_{\text{data}}}(L_E - \beta L_{KL} + \gamma L_G)$. It is a scalar, computed by integrating the Euclidean norm of the diffusion network's output along the posterior trajectory, so it taxes the total amount of noise the model injects. In the loss, it counteracts the likelihood's pressure toward zero diffusion, and its strength is controlled by the hyperparameter $\gamma$, which the paper tunes by matching summary statistics such as transition rates.
What would settle it
Use the method on data generated by an SDE with linear state-dependent diffusion, such as $dT = (a_1 + a_2 \tanh(T-T_0) - a_3 T^4)dt + 0.135 T \, dB_t$, and compare the recovered diffusion function with the true one. The paper's own appendix reports that the recovered diffusion is nearly constant while the truth is linear, which marks exactly where the central claim stops holding.
Extended reading notes
Core claim
The paper claims that the previously observed noise underestimation in latent neural SDEs is not an incidental training failure but a structural consequence of the loss function, and that it can be corrected by a simple additive term. In the objective $\max_{\theta,\phi}\mathbb{E}_{x_{\text{data}}}(L_E - \beta L_{KL})$, the diffusion $g_\theta$ is pulled in two directions: the likelihood term favors a small diffusion because it lets the posterior track the data closely, while the KL term favors a larger diffusion proportional to the discrepancy between posterior and prior drifts. The trained diffusion therefore reflects this balance point, not the data's noise. Adding $\gamma L_G$ with $L_G = \int_0^T \|g_\theta(u(t),t)\|\, dt$ shifts this balance so that, for constant-diffusion data, the prior SDE's diffusion size matches the data noise. The paper supports this by replacing the trained diffusion with constant values and showing the optimum of the loss components aligns with the final trained diffusion, and by demonstrating matching transition rates and Kramers-Moyal coefficients on several test systems. It also reports that the same penalty fails for multiplicative noise because it only adjusts the global diffusion magnitude.
Load-bearing premise
The data's diffusion is constant (or effectively constant) across state space, so one global number $\gamma$ can correct the estimated noise; the paper itself notes the result is only accurate for constant-diffusion data and shows the penalty fails for linear noise.
Editorial extensions
If this is right
- With a correctly tuned $\gamma$, the prior SDE reproduces the data's transition rates and marginal distributions on the bistable energy balance model, not just the mean path.
- The first two Kramers-Moyal coefficients (the estimated drift and diffusion) of generated trajectories match those of the data for constant-diffusion systems, so the generative model can be used as a simulator of the stochastic dynamics.
- The penalty can be tuned from summary statistics of the data alone, without knowing the underlying drift or diffusion.
- The method transfers to a two-dimensional FitzHugh-Nagumo system observed only in one dimension, to an Ornstein-Uhlenbeck process, and to a triple-well model.
- For multiplicative noise the global penalty cannot recover the diffusion function, so constant diffusion is a boundary of the method's validity.
Reading between the lines
- If the loss-balance explanation is right, the same underestimation should appear in any variational latent SDE with shared prior and posterior diffusion, and a state-dependent penalty $\int \gamma(u)\|g_\theta\|\, dt$ would be a natural next step to handle multiplicative noise.
- The transition-rate mismatch between data and prior SDE could serve as a cheap diagnostic for noise underestimation in applied settings, before any hyperparameter tuning.
- Because the data are normalized before training, the constant-diffusion assumption applies to the normalized coordinates; a system with homogeneous noise in physical units may acquire state-dependent effective diffusion once normalized, which would narrow the method's practical scope.
- An automatic selection of $\gamma$, for example by targeting a summary statistic, would remove the manual tuning and make the method more usable; the paper leaves this as future work.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper studies the systematic underestimation of the diffusion (noise) term in latent neural stochastic differential equations (SDEs). It first reproduces this underestimation on a bistable energy balance model (EBM) and analyzes the trade-off between log-likelihood and KL divergence in the loss. The authors then propose adding a noise penalty L_G = ∫ ||gθ(u(t), t)|| dt to the training objective, controlled by a hyperparameter γ, and show that for the EBM and several other constant-diffusion systems, a suitably chosen γ brings the prior SDE's transition rate, marginals, and finite-Δt Kramers-Moyal coefficients closer to the data. The paper claims that this enables the latent neural SDE to accurately capture the diffusion component of the data, and it discloses that the method is only accurate for constant diffusion and fails for multiplicative noise.
Significance. If the central claim held, the proposed penalty would be a simple, practical remedy for a known limitation of latent neural SDEs, with direct application to stochastic climate and energy-balance models. The paper is clearly written, makes its code and data publicly available, and includes several illustrative examples (EBM, FitzHugh-Nagumo, Ornstein-Uhlenbeck, triple-well). However, the evidence as presented does not support the strong claim of 'accurately capturing the diffusion component': the headline metric (transition rate) is used both to select γ and to evaluate the fit, and the paper's own direct comparison of the learned diffusion function with the true diffusion (Fig. 11) shows substantial disagreement. The contribution is better characterized as a one-parameter global noise dial that can match selected summary statistics in constant-diffusion systems, which is useful but weaker than the abstract's claim.
major comments (4)
- [Sec. VI, Fig. 7] The hyperparameter γ is selected by inspecting transition rates on the same data set that is later used for evaluation ('Having found a seemingly reasonable value γ = 200 by looking at the transition rates'). The reported transition-rate match is therefore a calibration outcome, not an independent prediction. The abstract's claim that the model 'accurately captures the diffusion component' is not supported by this metric, since the selection criterion and the reported statistic coincide. Please either select γ on a validation split and evaluate on a held-out set, or explicitly frame the reported match as calibration and temper the claim accordingly.
- [Sec. VI, Fig. 11] The paper's own direct comparison shows that at γ = 200 the learned drift and diffusion functions are 'significantly different' from the EBM's, even though the finite-Δt KM coefficients (Fig. 10) look similar. The KM coefficients are computed at Δt = 0.01 without extrapolation to Δt → 0, so the second KM coefficient includes finite-step and drift corrections and is not the true diffusion function. Consequently, matching finite-Δt KM coefficients does not demonstrate that the diffusion component is accurately captured. Either provide Δt → 0 extrapolated KM coefficients or an alternative direct estimate of the diffusion, or revise the central claim to say that the method matches finite-step effective statistics rather than the diffusion function.
- [Sec. VI, Appendix F] The paper states that 'the hyperparameter γ only fits diffusion terms globally in its current formulation' and that the result 'is thus only accurate for data sets with constant diffusion.' Appendix F confirms that the method fails for linear (multiplicative) noise. This is an honest and important limitation, but it directly contradicts the abstract's unqualified claim that the model 'accurately captures the diffusion component of the data.' The contribution should be framed as a global noise penalty for constant-diffusion systems, not as a general diffusion estimator.
- [Sec. IV and Sec. VI] All reported results, including the β-sweep in Fig. 3 and the γ-sweep in Fig. 7, appear to be single training runs without multiple seeds or error bars. Since the paper makes quantitative claims about matching transition rates and Wasserstein distances, it is important to show that the results are stable across random initializations; otherwise, the observed matches could be fortuitous. Please add at least a small number of independent runs and report the spread, or clearly state that the results are single-seed demonstrations.
minor comments (5)
- [Abstract / Introduction] The phrase 'W e' appears in the Introduction with a stray space; please correct the typo.
- [Fig. 4 caption] The caption reads 'Comparison of Wasserstein distane...'; 'distane' should be 'distance'.
- [Sec. IV D] The sentence 'The first two Kramers-Moyal coefficients are quantification of an SDE's drift and diffusion terms given data output by the system' is grammatically awkward; consider rephrasing.
- [Sec. V] The reference to 'GitHub issues of packages that implement them' is vague; a specific issue link or repository would be helpful.
- [Fig. 6 caption] The caption 'Replaced actual trained latent neural SDE's diffusion with constant value σlatent...' is hard to parse; consider rewriting for clarity.
Circularity Check
Headline evidence is a calibration outcome: γ is tuned on the transition rate (and in places the KM coefficients) that is then reported as the fit, so the central 'captures the diffusion component' claim is not independently predicted.
-
fitted input called prediction
[Section VI, 'Injecting Noise into Latent Neural SDEs', discussion of Fig. 7b and the paragraph following it.]
"In Fig. 7b, we observe that increasing γ will increase the transition rate. ... The latent neural SDE trained with γ = 200 matches the transition rate and Wasserstein distances quite well. ... Having found a seemingly reasonable value γ = 200 for the noise penalty by looking at the transition rates, we now investigate how well the trained latent neural SDE with β = 10 and γ = 200 fits the EBM."
The added penalty LG = ∫ ||g(u(t),t)|| dt directly raises the global diffusion magnitude, and in this bistable system the transition rate is monotone in that magnitude. The paper selects γ by inspecting the transition rate on the same test set later used for evaluation, so the reported transition-rate agreement is the optimization target, not an independent prediction. The main headline statistic is therefore forced by the choice of γ, and the abstract's claim that the model 'accurately captures the diffusion component of the data' is not established by this evidence.
-
fitted input called prediction
[Appendices D, E, and F ('EBM With Rarer Tipping', 'Triple-Well Model', 'EBM With Linear Diffusion').]
"We searched for γ as in Sec. VI and found that for γ = 150, the transition rate is similar (47.8 for the trained prior SDE and 50.5 for the noisy EBM). ... By looking at the KM coefficients, we select γ = 205. ... We select γ = 150 by analysing summary statistics and plot the KM coefficients for γ = 0 and γ = 150 in Fig. 18."
In each appendix the metric used to select γ (transition rate or KM diffusion coefficient) is the same metric later reported as validation of the fitted model, so the agreement is a calibration outcome. Appendix F is especially telling: with multiplicative noise the fitted diffusion stays nearly constant while the data diffusion is linear, showing that the global penalty matches only the chosen summary statistic, not the diffusion function itself. This directly undercuts the abstract's wording that the method 'accurately captures the diffusion component'.
full rationale
The derivation chain reduces at its decisive point: the new objective is max(LE − βLKL + γLG), where LG = ∫||g||dt is a global diffusion-size penalty. The paper then tunes γ so that the trained prior reproduces user-chosen summary statistics. Because the transition rate (and, in the appendices, the KM diffusion coefficient) is both the selection criterion and the reported success metric, the headline agreement is fitted input presented as prediction. The paper is transparent about this: it states that 'any metric can be chosen to optimize the noise penalty' and that γ is found 'by looking at the transition rates.' There is no self-citation or imported uniqueness theorem making the result circular in the other senses; the latent-SDE framework and KL objective are cited to independent prior work. The Wasserstein/marginal comparison in the main EBM experiment is not the tuning target and provides some independent support, which is why the circularity is partial rather than total. However, the paper's own Fig. 11 shows the learned drift and diffusion are 'significantly different' from the EBM's even when the KM coefficients look similar, confirming that the reported statistics do not identify the diffusion component. The honest limitation statement that γ 'only fits diffusion terms globally' and that the method fails for multiplicative noise (Appendix F) further narrows the central claim. Net assessment: partial circularity, score 6.
Assumptions & free parameters
free parameters (3)
- KL weighting beta =
10 (swept 0.01 to 10^4 in the EBM study)
- Noise penalty weight gamma =
EBM 200, FHN 500, OU 650, rare-tipping EBM 150, triple well 205, linear-noise EBM 150
- Observation model variance =
0.01
assumptions (3)
- domain assumption The data-generating diffusion is constant (or effectively constant) in the normalized coordinates.
- ad hoc to paper Increasing the noise penalty gamma monotonically increases the effective diffusion and does not destroy drift fidelity.
- standard math The path-space KL formula between prior and posterior is valid because the two SDEs share the same diffusion term.
Cite this review
Pith. "Pith review of Improving the Noise Estimation of Latent Neural Stochastic Differential Equations." pith.science (2026). https://pith.science/paper/4O2SKALL
@misc{pith2026241217499,
author = {Pith},
title = {Pith review of: Improving the Noise Estimation of Latent Neural Stochastic Differential Equations},
year = {2026},
howpublished = {\url{https://pith.science/paper/4O2SKALL}},
note = {Machine review of arXiv:2412.17499}
}
read the original abstract
Latent neural stochastic differential equations (SDEs) have recently emerged as a promising approach for learning generative models from stochastic time series data. However, they systematically underestimate the noise level inherent in such data, limiting their ability to capture stochastic dynamics accurately. We investigate this underestimation in detail and propose a straightforward solution: by including an explicit additional noise regularization in the loss function, we are able to learn a model that accurately captures the diffusion component of the data. We demonstrate our results on a conceptual model system that highlights the improved latent neural SDE's capability to model stochastic bistable dynamics.
Figures
Figures from the paper (13 more)
Reference graph
Works this paper leans on
-
[1]
merlin.mbs aipnum4-1.bst 2010-07-25 4.21a (PWD, AO, DPC) hacked
FUNCTION id.bst "merlin.mbs aipnum4-1.bst 2010-07-25 4.21a (PWD, AO, DPC) hacked" ENTRY address archive archivePrefix author bookaddress booktitle chapter collaboration doi edition editor eid eprint howpublished institution isbn issn journal key language month note number organization pages primaryClass publisher school SLACcitation series title translati...
2010
-
[2]
Hasselmann ,\ title title Stochastic climate models part i
author author K. Hasselmann ,\ title title Stochastic climate models part i. Theory , \ 10.3402/tellusa.v28i6.11316 journal journal Tellus \ volume 28 ,\ pages 473--485 ( year 1976 ) NoStop
-
[3]
author author C. L. E. \ Franzke , author T. J. \ O'Kane , author J. Berner , author P. D. \ Williams , \ and\ author V. Lucarini ,\ title title Stochastic climate theory and modeling , \ doi.org/10.1002/wcc.318 journal journal WIREs Climate Change \ volume 6 ,\ pages 63--78 ( year 2015 ) NoStop
-
[4]
author author J. K. \ M ller , author M. Zugno , \ and\ author H. Madsen ,\ title title Probabilistic forecasts of wind power generation by stochastic differential equation models , \ doi.org/10.1002/for.2367 journal journal Journal of Forecasting \ volume 35 ,\ pages 189--205 ( year 2016 ) NoStop
-
[5]
author author J. Dong , author M. M. \ Olama , author T. Kuruganti , author A. M. \ Melin , author S. M. \ Djouadi , author Y. Zhang , \ and\ author Y. Xue ,\ title title Novel stochastic methods to predict short-term solar radiation and photovoltaic power , \ 10.1016/j.renene.2019.05.073 journal journal Renewable Energy \ volume 145 ,\ pages 333--346 ( y...
-
[6]
author author J. Fagin , author J. W. \ Park , author H. Best , \ and\ author M. O'Dowd ,\ title title Latent stochastic differential equations for modeling quasar variability and inferring black hole properties , \ in\ @noop booktitle ICLR 2023 Workshop on Physics for Machine Learning \ ( year 2023 ) NoStop
work page 2023
-
[7]
author author F. Black \ and\ author M. Scholes ,\ title title The pricing of options and corporate liabilities , \ 10.1086/260062 journal journal Journal of Political Economy \ volume 81 ,\ pages 637--654 ( year 1973 ) NoStop
doi:10.1086/260062 1973
-
[8]
author author X. Li , author T.-K. L. \ Wong , author R. T. \ Chen , \ and\ author D. K. \ Duvenaud ,\ title title Scalable gradients and variational inference for stochastic differential equations , \ in\ @noop booktitle Symposium on Advances in Approximate Bayesian Inference \ ( organization PMLR ,\ year 2020 )\ pp.\ pages 1--28 NoStop
work page 2020
Show all 34 references
-
[9]
Kidger ,\ title On neural differential equations ,\ @noop Ph.D
author author P. Kidger ,\ title On neural differential equations ,\ @noop Ph.D. thesis ,\ school University of Oxford ( year 2021 ) NoStop
2021
-
[10]
Kidger , author J
author author P. Kidger , author J. Foster , author X. Li , \ and\ author T. J. \ Lyons ,\ title title Neural SDEs as infinite-dimensional GANs , \ in\ @noop booktitle Proceedings of the 38th International Conference on Machine Learning ,\ series Proceedings of Machine Learnin...
2021
-
[11]
Kidger , author J
author author P. Kidger , author J. Foster , author X. C. \ Li , \ and\ author T. Lyons ,\ title title Efficient and accurate gradients for neural SDEs , \ in\ @noop booktitle Advances in Neural Information Processing Systems ,\ Vol. volume 34 \ ( publisher Curran Associates, ...
2021
-
[12]
Zeng , author F
author author S. Zeng , author F. Graf , \ and\ author R. Kwitt ,\ title title Latent SDEs on homogeneous spaces , \ in\ @noop booktitle NeurIPS \ ( year 2023 ) NoStop
2023
-
[13]
note The artifact is available on Zenodo: https://doi.org/10.5281/zenodo.14534738 . Stop
-
[14]
author author C. K. \ Williams \ and\ author C. E. \ Rasmussen ,\ @noop title Gaussian processes for machine learning \ ( publisher MIT press Cambridge, MA ,\ year 2006 ) NoStop
2006
-
[15]
Xu , author R
author author W. Xu , author R. T. Q. \ Chen , author X. Li , \ and\ author D. Duvenaud ,\ title title Infinitely deep Bayesian neural networks with stochastic differential equations , \ in\ @noop booktitle Proceedings of The 25th International Conference on Artificial Intelli...
2022
-
[16]
Hasan , author J
author author A. Hasan , author J. M. \ Pereira , author S. Farsiu , \ and\ author V. Tarokh ,\ title title Identifying latent stochastic differential equations , \ @noop journal journal IEEE Transactions on Signal Processing \ volume 70 ,\ pages 89--104 ( year 2021 ) NoStop
2021
-
[17]
Jia \ and\ author A
author author J. Jia \ and\ author A. R. \ Benson ,\ @noop title Neural jump stochastic differential equations , \ ( year 2020 ),\ http://arxiv.org/abs/1905.10403 arXiv:1905.10403 [cs.LG] NoStop
2020 arXiv
-
[18]
Herrera , author F
author author C. Herrera , author F. Krach , \ and\ author J. Teichmann ,\ title title Neural jump ordinary differential equations: Consistent continuous-time prediction and filtering , \ in\ @noop booktitle International Conference on Learning Representations \ ( year 2021 ) NoStop
2021
-
[19]
Djeumou , author C
author author F. Djeumou , author C. Neary , \ and\ author U. Topcu ,\ title title How to learn and generalize from three minutes of data: Physics-constrained and uncertainty-aware neural stochastic differential equations , \ in\ @noop booktitle CoRL ,\ series Proceedings of M...
2023
-
[20]
author author M. A. \ Finzi , author A. Boral , author A. G. \ Wilson , author F. Sha , \ and\ author L. Zepeda-N \'u \ n ez ,\ title title User-defined event sampling and uncertainty quantification in diffusion models for physical dynamical systems , \ in\ @noop booktitle Int...
2023
-
[21]
author author D. J. \ Gagne , author H. M. \ Christensen , author A. C. \ Subramanian , \ and\ author A. H. \ Monahan ,\ title title Machine learning for stochastic parameterization: Generative adversarial networks in the lorenz'96 model , \ @noop journal journal Journal of Ad...
2020
-
[22]
Jinno , author T
author author T. Jinno , author T. Mitsui , author K. Nakai , author Y. Saiki , \ and\ author T. Yoneda ,\ title title Long-term prediction of el ni \ no-southern oscillation using reservoir computing with data-driven realtime filter , \ @noop journal journal arXiv preprint ar...
2025 arXiv
-
[23]
Tzen \ and\ author M
author author B. Tzen \ and\ author M. Raginsky ,\ title title Neural stochastic differential equations: Deep latent Gaussian models in the diffusion limit , \ @noop journal journal preprint, arXiv:1905.09883 \ ( year 2019 ) NoStop
1905 arXiv
-
[24]
author author D. P. \ Kingma \ and\ author M. Welling ,\ title title Auto-encoding variational Bayes , \ @noop journal journal preprint, arXiv:1312.6114 \ ( year 2013 ) NoStop
2013 arXiv
-
[25]
Higgins , author L
author author I. Higgins , author L. Matthey , author A. Pal , author C. Burgess , author X. Glorot , author M. Botvinick , author S. Mohamed , \ and\ author A. Lerchner ,\ title title beta-vae: Learning basic visual concepts with a constrained variational framework , \ in\ @n...
2016
-
[26]
author author K. Fraedrich ,\ title title Catastrophes and resilience of a zero-dimensional climate system with ice-albedo and greenhouse feedback , \ @noop journal journal Quarterly Journal of the Royal Meteorological Society \ volume 105 ,\ pages 147--167 ( year 1979 ) NoStop
1979
-
[27]
Ghil ,\ title title Climate stability for a sellers-type model , \ @noop journal journal Journal of Atmospheric Sciences \ volume 33 ,\ pages 3--20 ( year 1976 ) NoStop
author author M. Ghil ,\ title title Climate stability for a sellers-type model , \ @noop journal journal Journal of Atmospheric Sciences \ volume 33 ,\ pages 3--20 ( year 1976 ) NoStop
1976
-
[28]
author author A. Sutera ,\ title title On stochastic perturbation and long-term climate behaviour , \ @noop journal journal Quarterly Journal of the Royal Meteorological Society \ volume 107 ,\ pages 137--151 ( year 1981 ) NoStop
1981
-
[29]
Fu , author C
author author H. Fu , author C. Li , author X. Liu , author J. Gao , author A. Celikyilmaz , \ and\ author L. Carin ,\ title title Cyclical annealing schedule: A simple approach to mitigating KL vanishing , \ @noop journal journal preprint, arXiv:1903.10145 \ ( year 2019 ) NoStop
1903 arXiv
-
[30]
author author D. P. \ Kingma ,\ title title Adam: A method for stochastic optimization , \ @noop journal journal arXiv preprint arXiv:1412.6980 \ ( year 2014 ) NoStop
2014 arXiv
-
[31]
author author V. M. \ Panaretos \ and\ author Y. Zemel ,\ title title Statistical aspects of Wasserstein distances , \ @noop journal journal Annual review of statistics and its application \ volume 6 ,\ pages 405--431 ( year 2019 ) NoStop
2019
-
[32]
author author L. R. \ Gorjão \ and\ author F. Meirinhos ,\ title title kramersmoyal: Kramers -- Moyal coefficients for stochastic processes , \ 10.21105/joss.01693 journal journal Journal of Open Source Software \ volume 4 ,\ pages 1693 ( year 2019 ) NoStop
-
[33]
Lohmann \ and\ author P
author author J. Lohmann \ and\ author P. D. \ Ditlevsen ,\ title title A consistent statistical model selection for abrupt glacial climate changes , \ @noop journal journal Climate dynamics \ volume 52 ,\ pages 6411--6426 ( year 2019 ) NoStop
2019
-
[34]
author author R. FitzHugh ,\ title title Impulses and physiological states in theoretical models of nerve membrane , \ @noop journal journal Biophysical journal \ volume 1 ,\ pages 445--466 ( year 1961 ) NoStop
1961
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.