REVIEW 4 major objections 5 minor 1 cited by
Providing Machine Learning Potentials with High Quality Uncertainty Estimates
T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read A single Bayesian neural network can replace a nine-member ensemble for uncertainty-aware molecular energy prediction, producing error bars that cover the model's mistakes.
desk verdict A useful, honest engineering test of a no-KL BNN shortcut for ANI potentials, but the central coverage claim lacks the quantitative calibration evidence needed to support the title. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is MAUL, a Bayesian neural network built on the ANI architecture, in which every weight and bias is described by a Gaussian distribution with its own mean and variance. During a forward pass, random samples of the weights are drawn via the flipout reparameterization, so repeated passes yield a distribution of atomic energies and hence a molecular energy with a standard deviation. The model is initialized from a trained deterministic network, with variance weights set to 20 percent of the mean weights, and trained by alternating updates: first the variance weights with the means frozen, then the means with the variances frozen, using plain mean-squared error without the KL divergence term of formal variational inference. This training scheme, together with the weight distributions, is what lets a single network produce an uncertainty estimate that the paper finds covers its own errors.
What would settle it
Run MAUL on a large new test set of molecules with diverse functional groups and element compositions, and count how often the true DFT energy falls outside the reported uncertainty interval; the central claim fails if the observed miss rate is systematically larger than the nominal confidence level (for example, more than about five percent outside two standard deviations).
Extended reading notes
Core claim
The central claim is that MAUL's Bayesian uncertainty is high-quality in the specific sense that it bounds the error: for the 318 new molecules tested, nearly every true DFT energy lies within the MAUL uncertainty interval, whereas the spread of the nine-member DNN ensemble underestimates the actual error. Accuracy, measured by RMSE, is comparable between the two approaches (1.71 eV versus 1.76 eV for single-point energies, and 0.16 eV versus 0.21 eV for optimized geometries), so the difference is not that MAUL predicts better but that its uncertainty is informative. The paper therefore proposes MAUL as a more principled and cheaper route to uncertainty-aware neural potentials, and demonstrates the uncertainty rising in transition-state regions and with molecular size, enabling a hybrid workflow in which expensive DFT is called in only where the network's uncertainty is large.
Load-bearing premise
The calibrated coverage claim depends on the assumption that optimizing the variance weights with ordinary mean-squared error, without the Bayesian KL term, still produces an uncertainty estimate that reliably covers the model's errors on molecules it has never seen.
Editorial extensions
If this is right
- A single MAUL network supplies both a mean energy and a standard deviation, so uncertainty-aware evaluation no longer requires training and maintaining a full ensemble of separate networks.
- The fact that MAUL's uncertainty stays above its error lets a user treat the interval as a conservative error bar and decide automatically when to escalate a prediction to full DFT.
- Uncertainty grows in transition-state regions and with molecular size, so the same output can flag exactly where the neural potential is least trustworthy.
- Geometry optimizations using MAUL stay within reliable RMSD bounds while carrying correlated uncertainties, making the approach usable for structure searches.
- Equilibrium molecular dynamics with MAUL reproduces ensemble-averaged behavior, pointing toward uncertainty-aware dynamics simulations at lower cost.
Reading between the lines
- Inference: The same Bayesian treatment should transfer to force and stress predictions, where an error-bounding uncertainty estimate is arguably more valuable than for energies and could be tested on the existing ANI-1x force labels.
- Inference: Because the paper observes uncertainty rising with molecular size, MAUL's spread could serve as a cheap out-of-distribution detector for molecules larger or more complex than the training distribution, a use the authors mention only in passing.
- Inference: Since the KL term is omitted, the approach is not a full variational Bayesian treatment; a head-to-head comparison against the formal Bayes-by-Backprop loss on a small benchmark would reveal how much of the calibration comes from the initialization scheme rather than the Bayesian prior.
- Inference: If the calibration holds on broader chemical space, industrial screening could run the cheap potential over huge libraries and reserve DFT for the small fraction of molecules where the uncertainty is high, which is the hybrid workflow the paper sketches.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper presents MAUL, a Bayesian neural network variant of the ANI-1x neural network potential in which each weight and bias is represented by a Gaussian distribution and sampled using the flipout reparameterization. The authors compare a single MAUL model with a nine-member DNN ensemble on single-point energies and optimized geometries for 318 molecules from PubChem, finding similar RMSEs and claiming that MAUL's uncertainties are larger than the prediction errors in most cases, while the ensemble's are not. They further report transition-state energies, geometry optimizations, and 2 ps NVT molecular dynamics simulations for one molecule. The stated contribution is a single-network uncertainty quantification method that can replace ensembles and enable hybrid DFT/ML workflows.
Significance. Replacing a nine-member ensemble with a single network that provides calibrated energy uncertainties would be a practical and computational advance for neural potentials, and the paper provides useful public assets: code, model weights, a composition-based data split, and external validation on PubChem molecules. The central comparison is plausible, but the calibration evidence is currently qualitative: no coverage fraction, calibration curve, expected calibration error, or sharpness measure is reported, and the training procedure explicitly omits the KL term of variational Bayes. If the authors add quantitative calibration metrics and temper the Bayesian terminology, the contribution could be valuable; in its present form the 'high quality uncertainty estimates' claim is not yet demonstrated.
major comments (4)
- [Section 2.1, Figure 3] The central claim that MAUL's 'uncertainties are higher than the errors, covering a secure range for the predictions' is based on visual inspection of Figure 3 rather than on a quantitative measure. The manuscript reports no fraction of the 318 molecules for which |error| <= sigma or |error| <= 2sigma, no coverage probability, and no calibration curve. This is load-bearing because the reported RMSE values (1.71 and 1.76 eV) are much larger than the example MAUL uncertainty quoted in the text (0.65 eV), so the statement 'almost all cases' is not obvious and may be misleading. Please report per-molecule error and predictive standard deviation, coverage at 68% and 95%, and an expected calibration error or reliability diagram.
- [Section 4.3] The training scheme omits the KL divergence term and optimizes variance weights with plain MSE under alternating freezing of means and variances. The paper itself states that 'the lack of the KL term means this diverges from the formal variational Bayesian treatment as the influence of the prior is only through the initial weights.' Consequently, the learned variance is not a posterior variance, and a pattern of 'uncertainty greater than error' could be produced by systematically inflated variances rather than by calibrated uncertainty. To support the 'principled' and 'well-founded' descriptions used in the abstract and Section 1, the authors should either include the KL term (or another proper prior-dependent objective) or provide quantitative evidence that the simplified objective still yields calibrated predictive distributions, for example by comparing against a full Bayes-by-backprop baseline.
- [Section 4.3 and Sections 2.1-2.4] The methods state that early stopping used 'validation MSE and the uncertain calibration metrics within the uncertainty toolbox', but no such calibration metrics are reported anywhere in the results. Because the title and abstract claim high-quality uncertainty estimates, the absence of any reported reliability diagram, expected calibration error, or coverage metric is a missing central result rather than a presentation issue.
- [Section 2.4 and Section 4.4] The molecular dynamics evidence is a single molecule, three temperatures, and 2 ps trajectories per temperature. This is insufficient to support the conclusion that MAUL provides 'a viable alternative approach for ab initio MD simulations', and no uncertainty calibration along the trajectories is reported. At a minimum, the authors should report the empirical coverage of the energy uncertainty over time and, ideally, use longer or multiple trajectories and multiple molecules.
minor comments (5)
- [Captions and text] There are several typographical errors, including 'T able 1' in the Table 1 caption and 'optimised geomtery' in the Figure 2 caption; these should be corrected.
- [Section 4.1.1] The relationship between the BNN and the nine-fold DNN ensemble is unclear: the text says a '10th ensemble' was trained for the BNN using all components except the last 315 groups, but Section 4.3 says the BNN was initialised from 'a trained DNN model with no validation set'. Please clarify which DNN weights initialize the BNN and how this affects the cross-validation comparison.
- [Section 2.3, Figure 5] The text states that MAUL uncertainty correlates with geometry RMSD, but no correlation coefficient or confidence interval is reported; because the data are colored by heavy-atom count, the apparent trend may be confounded by molecular size. Report Pearson or Spearman correlations or a simple regression.
- [References and notation] Reference [24] appears to cite a Bayesian optimization paper rather than a Bayesian neural network method; also, the index m in Equations (1) and (3) is not explicitly defined as ranging over the (eta, Rs) parameter sets, and the vector notation for AEVs is inconsistent. These should be fixed.
- [Section 2.1 and Section 5] The paper claims resource efficiency for a single BNN relative to an ensemble but reports no wall-clock time or parameter-count comparison; a simple runtime measurement would substantiate the claim, and the repository URL in Section 5 should use https and a versioned commit hash for reproducibility.
Circularity Check
No circularity: MAUL's uncertainty/error coverage is an empirical claim on held-out molecules, not an identity or fitted quantity.
full rationale
The central claim that MAUL's uncertainty exceeds prediction error is not derived from, or defined as, the error itself. Section 4.3 describes training variance weights with plain MSE from an initialization at 20% of pretrained DNN weights, with early stopping against validation MSE and calibration metrics; the reported coverage is then evaluated on 318 PubChem molecules (and MD/TS cases) that were not part of training or validation. No equation in the paper makes sigma equal to or a function of |E_pred - E_DFT| for those molecules, so the coverage observation is not forced by construction. The BNN framework follows Blundell/Wen and Bayesian Torch, not a self-citation chain; the only self-citations (TopSearch, coarse-graining parameterization) are incidental tools, not load-bearing premises. The explicit caveat in Section 4.3 that omitting the KL term 'diverges from the formal variational Bayesian treatment' is a stated limitation about the method's Bayesian credentials, not evidence that the empirical coverage claim reduces to its inputs.
Assumptions & free parameters
free parameters (4)
- Initial variance scale =
20% of trained DNN mean weights
- Monte Carlo sample count =
20
- KL divergence weight =
0 (term omitted)
- AEV symmetry function parameters =
ANI-1x defaults
assumptions (6)
- domain assumption Atomic environment vectors from Behler-Parrinello symmetry functions provide a sufficient representation of the local chemical environment for energy prediction.
- domain assumption The ANI-1x dataset's ωB97x/6-31G* DFT energies are accurate ground truth, and PSI4 recalculations with the same functional provide a valid test reference.
- ad hoc to paper Sampling Gaussian weight distributions via the reparameterization trick gives a useful posterior approximation even when the KL term is omitted.
- domain assumption Ensemble standard deviation is an appropriate baseline uncertainty for comparison.
- domain assumption The 318 PubChem molecules and the single 2 ps MD trajectory are representative of out-of-distribution use.
- standard math The reparameterization trick and backpropagation provide correct gradients for training the Gaussian weight distributions.
Cite this review
Pith. "Pith review of Providing Machine Learning Potentials with High Quality Uncertainty Estimates." pith.science (2026). https://pith.science/paper/U7ZZL5GK
@misc{pith2026250105250,
author = {Pith},
title = {Pith review of: Providing Machine Learning Potentials with High Quality Uncertainty Estimates},
year = {2026},
howpublished = {\url{https://pith.science/paper/U7ZZL5GK}},
note = {Machine review of arXiv:2501.05250}
}
read the original abstract
Computational chemistry has come a long way over the course of several decades, enabling subatomic level calculations particularly with the development of Density Functional Theory (DFT). Recently, machine-learned potentials (MLP) have provided a way to overcome the prevalent time and length scale constraints in such calculations. Unfortunately, these models utilise complex and high dimensional representations, making it challenging for users to intuit performance from chemical structure, which has motivated the development of methods for uncertainty quantification. One of the most common methods is to introduce an ensemble of models and employ an averaging approach to determine the uncertainty. In this work, we introduced Bayesian Neural Networks (BNNs) for uncertainty aware energy evaluation as a more principled and resource efficient method to achieve this goal. The richness of our uncertainty quantification enables a new type of hybrid workflow where calculations can be offloaded to a MLP in a principled manner.
Forward citations
Cited by 1 Pith paper
-
Bayesian Neural Networks versus deep ensembles for uncertainty quantification in machine learning interatomic potentials
Deep ensembles outperform variational Bayesian neural networks in accuracy and uncertainty calibration on machine-learned TiO2 potentials, with some low-data exceptions when Bayesian methods are better calibrated.
Reference graph
Works this paper leans on
-
[1]
Kocer, E., Ko, T. W. & Behler, J. Neural network potentials: A concise overview of methods. Annual review of physical chemistry 73 (1), 163–186 (2022)
work page 2022
- [2]
-
[3]
Sholl, D. S. & Steckel, J. A. Density functional theory: a practical introduction (John Wiley & Sons, 2022)
work page 2022
-
[4]
Practical Aspects of Computational Chemistry-Methods, Concepts & Applications (Springer, 2022)
Leszczynski, J. Practical Aspects of Computational Chemistry-Methods, Concepts & Applications (Springer, 2022)
work page 2022
-
[5]
Bai, Q. et al. Application advances of deep learning methods for de novo drug design and molecular dynamics simulation. Wiley Interdisciplinary Reviews: Computational Molecular Science 12 (3), e1581 (2022)
work page 2022
-
[6]
ˇRez´ aˇ c, J. & Stewart, J. How well do semiempirical qm methods describe the structure of proteins? The Journal of Chemical Physics 158 (4) (2023)
work page 2023
-
[7]
Unke, O. T. et al. Machine Learning Force Fields. Chemical Reviews 121 (16), 10142–10186 (2021). URL https://pubs.acs.org/doi/10.1021/ acs.chemrev.0c01111. https://doi.org/10.1021/acs.chemrev.0c01111 . 16 Providing Machine Learning Potentials with High Quality Uncertainty Estimates
-
[8]
Anstine, D. M. & Isayev, O. Machine learning interatomic potentials and long-range physics. The Journal of Physical Chemistry A 127 (11), 2417–2431 (2023)
work page 2023
Show all 34 references
-
[9]
& Parrinello, M
Behler, J. & Parrinello, M. Generalized Neural-Network Representation of High-Dimensional Potential-Energy Surfaces. Physical Review Letters 98 (14), 146401 (2007). https://doi.org/10.1103/PhysRevLett.98.146401
2007 doi
-
[10]
L., Shkurti, A., Bray, D
McDonagh, J. L., Shkurti, A., Bray, D. J., Anderson, R. L. & Pyzer- Knapp, E. O. Utilizing machine learning for efficient parameterization of coarse grained molecular force fields. Journal of chemical information and modeling 59 (10), 4278–4288 (2019)
2019
-
[11]
P., Payne, M
Bart´ ok, A. P., Payne, M. C., Kondor, R. & Cs´ anyi, G. Gaussian approx- imation potentials: The accuracy of quantum mechanics, without the electrons. Physical review letters 104 (13), 136403 (2010)
2010
-
[12]
Deringer, V. L. et al. Gaussian process regression for materials and molecules. Chemical Reviews 121 (16), 10073–10141 (2021)
2021
-
[13]
C., Rohrer, J., Albe, K
Erhard, L. C., Rohrer, J., Albe, K. & Deringer, V. L. A machine-learned interatomic potential for silica and its relation to empirical models. npj Computational Materials 8 (1), 1–12 (2022)
2022
-
[14]
S., Isayev, O
Smith, J. S., Isayev, O. & Roitberg, A. E. Ani-1: an extensible neural network potential with dft accuracy at force field computational cost. Chemical science 8 (4), 3192–3203 (2017)
2017
-
[15]
Four generations of high-dimensional neural network potentials
Behler, J. Four generations of high-dimensional neural network potentials. Chemical Reviews 121 (16), 10037–10072 (2021)
2021
-
[16]
T., Sauceda, H
Sch¨ utt, K. T., Sauceda, H. E., Kindermans, P.-J., Tkatchenko, A. & M¨ uller, K.-R. Schnet–a deep learning architecture for molecules and materials. The Journal of Chemical Physics 148 (24), 241722 (2018)
2018
-
[17]
S., Nebgen, B., Lubbers, N., Isayev, O
Smith, J. S., Nebgen, B., Lubbers, N., Isayev, O. & Roitberg, A. E. Less is more: Sampling chemical space with active learning. The Journal of chemical physics 148 (24) (2018)
2018
-
[18]
& Tadmor, E
Wen, M. & Tadmor, E. B. Uncertainty quantification in molecular simula- tions with dropout neural network potentials.npj computational materials 6 (1), 124 (2020)
2020
-
[19]
& Lucia, S
Fiedler, F. & Lucia, S. Improved uncertainty quantification for neural networks with bayesian last layer. IEEE Access (2023) . Providing Machine Learning Potentials with High Quality Uncertainty Estimates 17
2023
-
[20]
Smith, J. S. et al. The ani-1ccx and ani-1x data sets, coupled-cluster and density functional theory properties for molecules. Scientific data 7 (1), 1–10 (2020)
2020
-
[21]
& Neely, A
Pearce, T., Leibfried, F., Brintrup, A., Zaki, M. & Neely, A. Uncertainty in neural networks: Approximately bayesian ensembling. arXiv preprint arXiv:1810.05546 (2018)
2018 arXiv
-
[22]
de Souza, D. A. et al. Thin and deep gaussian processes. Advances in Neural Information Processing Systems 36 (2024)
2024
-
[23]
Lee, J. et al. Deep neural networks as gaussian processes. arXiv preprint arXiv:1711.00165 (2017)
2017 arXiv
-
[24]
& Adams, R
Snoek, J., Larochelle, H. & Adams, R. P. Practical bayesian optimiza- tion of machine learning algorithms. Advances in neural information processing systems 25 (2012)
2012
-
[25]
TopSearch (2024)
Dicks, L. TopSearch (2024). URL https://github.com/IBM/ topography-searcher
2024
-
[26]
E., Jordan, K
Dicks, L., Graff, D. E., Jordan, K. E., Coley, C. W. & Pyzer-Knapp, E. O. A physics-inspired approach to the understanding of molecular representations and models. Molecular Systems Design & Engineering 9 (5), 449–455 (2024)
2024
-
[27]
& Schulten, K
Humphrey, W., Dalke, A. & Schulten, K. Vmd: visual molecular dynamics. Journal of molecular graphics 14 (1), 33–38 (1996)
1996
-
[28]
& Liu, T.-Y
Wang, T., He, X., Li, M., Shao, B. & Liu, T.-Y. Aimd-chig: Exploring the conformational space of a 166-atom protein chignolin with ab initio molecular dynamics. Scientific Data 10 (1), 549 (2023)
2023
-
[29]
Gao, X., Ramezanghorbani, F., Isayev, O., Smith, J. S. & Roitberg, A. E. TorchANI: A Free and Open Source PyTorch-Based Deep Learn- ing Implementation of the ANI Neural Network Potentials. Journal of Chemical Information and Modeling 60 (7), 3408–3415 (2020). URL https://pubs....
2020 doi
-
[30]
& Wierstra, D
Blundell, C., Cornebise, J., Kavukcuoglu, K. & Wierstra, D. Bach, F. & Blei, D. (eds) Weight uncertainty in neural network . (eds Bach, F. & Blei, D.) Proceedings of the 32nd International Conference on Machine Learning, Vol. 37 of Proceedings of Machine Learning Research , 16...
2015
-
[31]
& Grosse, R
Wen, Y., Vicol, P., Ba, J., Tran, D. & Grosse, R. Flipout: Efficient pseudo-independent weight perturbations on mini-batches. arXiv preprint arXiv:1803.04386 (2018)
2018 arXiv
-
[32]
& Subedar, M
Krishnan, R., Esposito, P. & Subedar, M. Bayesian-torch: Bayesian neural network layers for uncertainty estimation. https://github.com/IntelLabs/ bayesian-torch (2022). URL https://doi.org/10.5281/zenodo.5908307
2022 doi
-
[33]
Larsen, A. H. et al. The atomic simulation environment—a python library for working with atoms. Journal of Physics: Condensed Matter 29 (27), 273002 (2017)
2017
-
[34]
& Neiswanger, W
Chung, Y., Char, I., Guo, H., Schneider, J. & Neiswanger, W. Uncertainty toolbox: an open-source library for assessing, visualizing, and improving uncertainty quantification. arXiv preprint arXiv:2109.10254 (2021)
2021 arXiv
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.