REVIEW 3 major objections 4 minor 40 references
A single-model uncertainty signal selects the few training labels that match full-data accuracy in machine-learned force fields, and it flags a foundation model hallucinating before any labels are computed.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-02 01:55 UTC pith:K2SAHSXC
load-bearing objection Solid empirical case for LLPR-based active learning in MLFFs; the water/ice section is the real contribution, the electrolyte story needs clearer caveats. the 3 major comments →
Full-data accuracy with fewer labels for training and fine-tuning machine-learning force fields
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
Last-layer-projection regression (LLPR) uncertainty, defined per atom as the squared distance of the model's last-layer feature vector from the training-set covariance, is a quantitatively calibrated estimator of prediction error rather than just a ranking. This lets a single trained model drive active learning. In a train-from-scratch water/ice benchmark, energy- and force-LLPR pickers reach the full-data ceiling with K=100 labelled frames, about 2.5% of the 3955-frame full pool, while random sampling stays several-fold above the ceiling at the same budget and remains a factor of two to five away in the foundation fine-tuning setting. In the electrolyte case, the per-atom signal flags a hal
What carries the argument
The carrying object is the last-layer-projection-regression (LLPR) leverage u_a(x) = h_a(x)^T A^{-1} h_a(x), where h_a(x) is the per-atom activation vector of the final network layer and A is the regularized covariance of those activations over the training/reference set. It measures how far a configuration's internal representation lies from the training distribution. Built on this are two aggregates: an energy signal (sum-pooled frame feature) and a top-5 force signal (mean of the five largest per-atom leverages), plus a calibration factor fitted on held-out picks to convert the dimensionless leverage into absolute force uncertainty in eV/angstrom. This same quantity does three jobs: ranki
Load-bearing premise
The whole workflow rests on the premise that a model's feature-space distance to its training data is a trustworthy measure of when its predictions are wrong—trustworthy enough to select labels and to flag a hallucinated contact before labels exist—and, in the electrolyte test, that the model used as labelling oracle is itself accurate enough to be ground truth.
What would settle it
The paper's most distinctive prediction is that, on a foundation-MD trajectory that is stable but structurally wrong, the pathological atoms appear in the extreme tail of the per-atom LLPR distribution before any labels are computed. A reader can test this by taking any foundation model on a new dense electrolyte, running a short stable MD, and computing the per-atom leverage from the model's own features and training covariance; if the atoms involved in the unphysical short contact are not the tail, the pre-label hallucination detector does not work.
If this is right
- On the water/ice benchmark, the LLPR picker reaches the full-data ceiling at K=100 labels, against a 3955-frame full pool; random picking remains several-fold above the ceiling at the same budget.
- In foundation fine-tuning, the picker effect transfers: LLPR picks on the smaller foundation close the gap to the full-pool ceiling at K=100, while random picks leave a factor-of-two-to-five gap; which frames are picked matters more than which foundation is used.
- In the electrolyte case, about 50 labels across eight active-learning rounds lift the hallucinated Li-O potential, restore the physical radial distribution, and recover an equilibrium density consistent with experiment and reference.
- The same uncertainty signal detects the pathological short Li-O contact before any labelling, because the affected atoms sit in the extreme tail of the per-atom leverage distribution.
- An absolute per-atom force-error threshold (100 meV/angstrom, set by the thermal-force scale) provides a self-terminating stopping criterion for the active-learning loop.
Where Pith is reading between the lines
- Inference: the same per-atom feature leverage could act as a live alarm during production MD, flagging trajectories that drift into unexplored regions even when energy conservation and density look normal.
- Inference: if the training-set covariance is updated continuously, the method may extend to continual learning or domain-adaptive retraining where the target distribution shifts over time, rather than only fixed benchmark pools.
- Inference: the 100 meV/angstrom exit threshold is calibrated to the 300-K thermal scale on a roughly 1500-atom cell; at other temperatures, pressures, or cell sizes a user would need to re-derive the threshold, which could be done self-consistently from the LLPR calibration factor.
- Inference: a practical pre-screening recipe suggested by the paper is to run a short stable MD with a candidate foundation, inspect the per-atom LLPR tail, and only spend labels if the tail is elevated; this could be tested on other foundation-target pairs.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents an active-learning workflow based on last-layer projection regression (LLPR), a single-forward-pass uncertainty estimator computed from the model's last-layer features and training-set covariance. The workflow is applied in three settings: a train-from-scratch DeePMD model on a water/ice dataset, one-shot fine-tuning of MACE-MP-0 on the same water/ice pool, and iterative fine-tuning of MACE-OMAT-small on a 21 molal LiTFSI water-in-salt electrolyte. The central claims are that LLPR-selected labels reach near-full-data accuracy with substantially fewer labels than random selection, that LLPR is competitive with a model committee at much lower cost, that it detects a hallucinated Li–O contact in the electrolyte before labels are committed, and that it provides an absolute force-error threshold enabling automatic termination of the active-learning loop.
Significance. If the claims hold, this is a practically important result: model-committee uncertainty is often impractical for foundation-scale fine-tuning, and a cheap, deterministic, single-model uncertainty signal that drives data selection, pathology detection, and stopping would be a useful advance. The water/ice train-from-scratch benchmark is carefully designed: the same candidate pool, identical retraining recipe, three seeds, random and committee baselines, and a full-data ceiling are used, and the LLPR calibration curves against actual prediction errors are convincing. The main unresolved issues are the absence of a random+FPS control in the water/ice comparison, and the use of an MLFF pseudo-oracle rather than electronic-structure labels in the electrolyte section. These are fixable within the manuscript's scope, but they currently leave the headline label-efficiency claim partially underdetermined.
major comments (3)
- [Validating LLPR... / Methods: Train-from-scratch active learning on water/ice] The K-sweep comparison does not isolate the LLPR uncertainty signal from the diversity post-processing. Results states that 'within the K picks of each strategy, frames are diversified by SOAP-FPS on top of the uncertainty score,' but Methods defines the random picker simply as selecting K frames from the candidate pool, with no mention of an analogous FPS step. Because rare dense-ice phases (ice VIII, ice XV) dominate the worst-1% error and are unlikely to appear in 50–100 random draws, the low-K random baseline may be degraded by lack of diversity rather than by absence of an uncertainty ranking. Please add a random+FPS (and ideally a pure-FPS) arm, or explicitly state that the random picker also receives SOAP-FPS diversification and describe how.
- [Detecting and resolving foundation MD hallucination / Methods: Iterative LLPR-based active learning] The electrolyte demonstration does not use electronic-structure labels. In Methods, the labels are produced by the matpes-r2SCAN-OMAT-FT foundation, an MLFF, and this is described as a 'pseudo-oracle.' However, the Abstract claims recovery of full-data accuracy using 'a small fraction of electronic-structure labels' and the Introduction frames the detection as occurring 'before DFT labelling.' The absolute force-error threshold in Eq. (2) is calibrated against this pseudo-oracle, not against DFT. The electrolyte arm therefore demonstrates repair relative to a specific reference MLFF, not a reduction in DFT-labelling cost. Please rephrase the relevant claims and state the pseudo-oracle limitation explicitly.
- [Methods: Eq. (2) calibration] The calibration factor C_geom is estimated from the K=10 picks of round k, which are described as an 'unbiased held-out set.' These picks are selected by the LLPR signal itself, so they are not unbiased with respect to the uncertainty distribution; they preferentially sample high-uncertainty configurations. If the e^2/u ratio is not constant across the uncertainty range, this can bias the absolute units used for the exit threshold. With only 10 points per round, the calibration is also noisy. Please provide calibration diagnostics on a random sample or report the sensitivity of the stopping round to the calibration procedure.
minor comments (4)
- [Fig. 3a] The energy LLPR calibration uses a 'constant-offset-corrected ground-truth squared energy error.' Please state how the offset is determined and whether the offset correction affects the ranking used for selection. As presented, the reader cannot tell whether the excellent y=x agreement is partly a consequence of this correction.
- [Fig. 5c] The vertical axis is labelled 'uncertainty error' and reported in meV/Å, but the Methods define the stopping criterion using the 99th percentile of the calibrated σ_F. Please clarify whether Fig. 5c plots the calibrated σ_F, the raw LLPR leverage, or a different quantity, and how it relates to the 100 meV/Å threshold.
- [Data and code availability] The statement that data and code are available 'from the authors on reasonable request' and 'will be released open-source on Zenodo upon publication' is weaker than the reproducibility standard common for this type of computational study. At minimum, please provide the exact version of the code/checkpoints used and a timeline.
- [Abstract / Introduction] The phrase 'full-data accuracy' is used throughout, but the reported numbers at K=100 in the train-from-scratch case are 1.7 meV/atom versus a 1.30 meV/atom ceiling (31% above) and worst-1% errors within a factor of two. Consider using 'near-full-data accuracy' or defining the statistical criterion quantitatively to avoid overstatement.
Circularity Check
No significant circularity: LLPR selection, validation, and stopping are independent of the labels being selected; the only fitted calibration constant is disclosed and used only for thresholding.
full rationale
Walked the claimed derivation chain. Eq. 1 defines LLPR leverage u_a from last-layer features and the training-set covariance; it does not use the labels being selected, so the ranking signal is not derived from the target quantities it is used to select. Validation in Fig. 1 and Fig. 3 compares LLPR to held-out squared errors and to retrieval of worst-K frames, so the agreement is empirical rather than forced. The active-learning K-sweep compares picker strategies on a held-out test set; the absence of a SOAP-FPS-diversified random baseline is a possible confound between diversity and uncertainty, but it does not make the result equivalent to its inputs by construction. The C_geom calibration in Eq. 2 is fitted to the previous round's held-out picks, but it is a single disclosed scaling constant used only to express the stopping threshold in absolute units; it does not rank frames and does not determine the central label-efficiency comparison. The electrolyte pseudo-oracle is an independently benchmarked external reference [39], and final density and Li-O RDF are checked against experiment, so the recovery is not circular. Self-citations ([25] implementation, [30] prior LLPR calibration) are not load-bearing: the paper independently re-derives and validates the LLPR signal on held-out data. The Discussion notes a scope limitation (demonstrated on the MACE foundation family), which is a generality caveat rather than a circular step. No load-bearing step reduces by construction to a fit or to a self-citation chain.
Axiom & Free-Parameter Ledger
free parameters (5)
- Tikhonov ridge sigma_F =
5e-3
- C_geom calibration factor per round =
C_geom = exp(<log(e_a^2/u_a)>_a)
- Energy-offset correction in Fig. 3a =
not specified
- Per-element atomic-energy re-alignment for MACE heads =
per-element linear regression on 200 liquid-anchor frames
- Exit threshold for active learning =
100 meV/A
axioms (4)
- domain assumption LLPR uncertainty, computed from last-layer features and training-set covariance, is a valid per-configuration epistemic uncertainty estimate for MLFFs.
- domain assumption matpes-r2SCAN-OMAT-FT is an accurate reference for 21 m LiTFSI, including density, diffusion, X-ray structure factor, and the Li-O PES.
- domain assumption The OMAT24 pretraining distribution lacks Li-O contacts below ~1.5 A, causing the hallucinated attractive well.
- domain assumption The multihead fine-tuning recipe with replay buffer preserves the foundation's capabilities while fitting the target set.
read the original abstract
Machine-learning force fields (MLFFs) are reliable only near their training distribution, making efficient construction of diverse training sets a major bottleneck for both train-from-scratch and foundation fine-tuning workflows. Active learning can reduce this cost, but standard model-committee uncertainty is impractical for foundation MLFFs because each committee member requires a separate fine-tuning run. We present an active-learning workflow based on last-layer-projection regression (LLPR), a forward-pass-cheap per-configuration uncertainty estimator. Across molecular, condensed-phase, and electrolyte systems, LLPR identifies compact, high-value training sets that recover full-data accuracy using only a small fraction of electronic-structure labels. In foundation-model fine-tuning, LLPR-selected configurations reach the full-pool fine-tuning ceiling with substantially fewer labels than random selection. In iterative electrolyte fine-tuning, LLPR detects unphysical local coordination before DFT labelling, provides an absolute force-error threshold, and enables automatic termination of the learning loop. The resulting models reproduce reference density and ion-coordination structure, providing a scalable uncertainty-quantification strategy across MLFF training regimes.
Reference graph
Works this paper leans on
-
[1]
T.et al.Machine Learning Force Fields.Chemical Reviews121, 10142–10186 (2021)
Unke, O. T.et al.Machine Learning Force Fields.Chemical Reviews121, 10142–10186 (2021)
2021
-
[2]
L., Caro, M
Deringer, V. L., Caro, M. A. & Cs´ anyi, G. Machine Learning Interatomic Potentials as Emerging Tools for Materials Science.Advanced Materials31 (2019)
2019
-
[3]
Perspective: Machine learning potentials for atomistic simulations.The Journal of Chemical Physics145(2016)
Behler, J. Perspective: Machine learning potentials for atomistic simulations.The Journal of Chemical Physics145(2016)
2016
-
[4]
& Lordi, V
Schwalbe-Koda, D., Hamel, S., Sadigh, B., Zhou, F. & Lordi, V. Model-free esti- mation of completeness, uncertainties, and outliers in atomistic machine learning using information theory.Nature Communications16(2025)
2025
-
[5]
Perez, D., Subramanyam, A. P. A., Maliyov, I. & Swinburne, T. D. Uncer- tainty quantification for misspecified machine learned interatomic potentials.npj Computational Materials11(2025). 20
2025
-
[6]
Montes de Oca Zapiain, D.et al.Training data selection for accuracy and transferability of interatomic potentials.npj Computational Materials8(2022)
2022
-
[7]
P. A. Subramanyam, A. & Perez, D. Information-entropy-driven generation of material-agnostic datasets for machine-learning interatomic potentials.npj Computational Materials11(2025)
2025
-
[8]
Deng, B.et al.CHGNet as a pretrained universal neural network potential for charge-informed atomistic modelling.Nature Machine Intelligence5, 1031–1041 (2023)
2023
-
[9]
Barroso-Luque, L.et al.The Open Materials 2024 (OMat24) inorganic materials dataset and models.Nature Computational Science6, 642–652 (2026)
2024
-
[10]
Batatia, I.et al.A foundation model for atomistic materials chemistry.The Journal of Chemical Physics163(2025)
2025
-
[11]
Zhang, D.et al.DPA-2: a large atomic model as a multi-task learner.npj Computational Materials10(2024)
2024
-
[12]
G., Allen, C
Radova, M., Stark, W. G., Allen, C. S., Maurer, R. J. & Bart´ ok, A. P. Fine- tuning foundation models of materials interatomic potentials with frozen transfer learning.npj Computational Materials11(2025)
2025
-
[13]
Allen, A. E. A.et al.Learning together: Towards foundation models for machine learning interatomic potentials with meta-learning.npj Computational Materials 10(2024)
2024
-
[14]
Deng, B.et al.Systematic softening in universal machine learning interatomic potentials.npj Computational Materials11(2025)
2025
-
[15]
Tokita, A. M. & Behler, J. How to train a neural network potential.The Journal of Chemical Physics159(2023)
2023
-
[16]
& Tadmor, E
Wen, M. & Tadmor, E. B. Uncertainty quantification in molecular simulations with dropout neural network potentials.npj Computational Materials6(2020)
2020
-
[17]
R., Urata, S., Goldman, S., Dietschreit, J
Tan, A. R., Urata, S., Goldman, S., Dietschreit, J. C. B. & G´ omez-Bombarelli, R. Single-model uncertainty quantification in neural network potentials does not consistently outperform model ensembles.npj Computational Materials9(2023)
2023
-
[18]
S., Nebgen, B., Lubbers, N., Isayev, O
Smith, J. S., Nebgen, B., Lubbers, N., Isayev, O. & Roitberg, A. E. Less is more: Sampling chemical space with active learning.The Journal of Chemical Physics 148(2018)
2018
-
[19]
Zhang, L., Lin, D.-Y., Wang, H., Car, R. & E, W. Active learning of uni- formly accurate interatomic potentials for materials simulation.Physical Review Materials3(2019). 21
2019
-
[20]
Podryabinkin, E. V. & Shapeev, A. V. Active learning of linearly parametrized interatomic potentials.Computational Materials Science140, 171–180 (2017)
2017
-
[21]
& Marsalek, O
Schran, C., Brezina, K. & Marsalek, O. Committee neural network potentials control generalization errors and enable active learning.The Journal of Chemical Physics153(2020)
2020
-
[22]
Vandermause, J.et al.On-the-fly active learning of interpretable Bayesian force fields for atomistic rare events.npj Computational Materials6(2020)
2020
-
[23]
S., Gubaev, K., Podryabinkin, E
Novikov, I. S., Gubaev, K., Podryabinkin, E. V. & Shapeev, A. V. The MLIP package: moment tensor potentials with MPI and active learning.Machine Learning: Science and Technology2, 025002 (2021)
2021
-
[24]
Zhang, Y.et al.DP-GEN: A concurrent learning platform for the genera- tion of reliable deep learning based potential energy models.Computer Physics Communications253, 107206 (2020)
2020
-
[25]
ai2-kit: artificial intelligence x ab initio toolkit for complex chemistry systems research
Cheng Group. ai2-kit: artificial intelligence x ab initio toolkit for complex chemistry systems research. https://github.com/chenggroup/ai2-kit (2023). MIT-licensed software repository; accessed 2026-06-26
2023
-
[26]
S., Shamir, E
Freund, Y., Seung, H. S., Shamir, E. & Tishby, N. Selective Sampling Using the Query by Committee Algorithm.Machine Learning28, 133–168 (1997)
1997
-
[27]
URL https: //arxiv.org/abs/2504.06231
Rhodes, B.et al.Orb-v3: atomistic simulation at scale (2025). URL https: //arxiv.org/abs/2504.06231. arXiv:2504.06231
Pith/arXiv arXiv 2025
-
[28]
Liu, K.et al.Heterogeneous ensemble enables a universal uncertainty metric for atomistic foundation models.npj Computational Materials12(2025)
2025
-
[29]
A., Firoz, J
Bilbrey, J. A., Firoz, J. S., Lee, M.-S. & Choudhury, S. Uncertainty quantification for neural network potential foundation models.npj Computational Materials11 (2025)
2025
-
[30]
H., Ortner, C
Ho, C. H., Ortner, C. & Wang, Y. Flexible uncertainty calibration for machine- learned interatomic potentials.npj Computational Materials12(2026)
2026
-
[31]
& Grasselli, F
Bigi, F., Chong, S., Ceriotti, M. & Grasselli, F. A prediction rigidity formalism for low-cost uncertainties in trained neural networks.Machine Learning: Science and Technology5, 045018 (2024)
2024
-
[32]
& Behler, J
Morawietz, T., Singraber, A., Dellago, C. & Behler, J. How van der Waals inter- actions determine the unique properties of water.Proceedings of the National Academy of Sciences113, 8368–8373 (2016)
2016
-
[33]
Morawietz, T. & Behler, J. HDNNP training data set for H2O (2019). URL https://zenodo.org/record/2634098. 22
arXiv 2019
-
[34]
URL https://openreview.net/forum?id= A8pqQipwkt
Fu, X.et al.Forces are not Enough: Benchmark and Critical Evaluation for Machine Learning Force Fields with Molecular Simulations.Transactions on Machine Learning Research(2023). URL https://openreview.net/forum?id= A8pqQipwkt
2023
-
[35]
Imbalzano, G.et al.Automatic selection of atomic fingerprints and reference configurations for machine-learning potentials.The Journal of Chemical Physics 148(2018)
2018
-
[36]
Water-in-salt
Suo, L.et al.“Water-in-salt” electrolyte enables high-voltage aqueous lithium-ion chemistries.Science350, 938–943 (2015)
2015
-
[37]
Batatia, I., Kov´ acs, D. P., Simm, G. N. C., Ortner, C. & Cs´ anyi, G. MACE: Higher Order Equivariant Message Passing Neural Networks for Fast and Accu- rate Force Fields (2022). URL https://arxiv.org/abs/2206.07697. Advances in Neural Information Processing Systems 35 (NeurIPS 2022), arXiv:2206.07697
Pith/arXiv arXiv 2022
-
[38]
Gilbert, W. J. R.et al.Density, Viscosity, and Vapor Pressure Measurements of Water + Lithium Bis(trifluoromethylsulfonyl)imide Solutions.Journal of Chemical & Engineering Data62, 2056–2066 (2017)
2056
-
[39]
Brugnoli, L., Salanne, M., Saitta, A. M., Serva, A. & France-Lanord, A. Over- coming sampling limitations using machine-learned interatomic potentials: the case of water-in-salt electrolytes.arXiv preprint arXiv:2603.22099(2026). URL https://arxiv.org/abs/2603.22099
arXiv 2026
-
[40]
& Lee Holian, B
Melchionna, S., Ciccotti, G. & Lee Holian, B. Hoover NPT dynamics for systems varying in shape and size.Molecular Physics78, 533–544 (1993). 23
1993
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.