Pith. sign in

REVIEW 3 major objections 4 minor 40 references

A single-model uncertainty signal selects the few training labels that match full-data accuracy in machine-learned force fields, and it flags a foundation model hallucinating before any labels are computed.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

T0 review · deepseek-v4-flash

2026-08-02 01:55 UTC pith:K2SAHSXC

load-bearing objection Solid empirical case for LLPR-based active learning in MLFFs; the water/ice section is the real contribution, the electrolyte story needs clearer caveats. the 3 major comments →

arxiv 2607.14486 v1 pith:K2SAHSXC submitted 2026-07-16 physics.chem-ph cond-mat.mtrl-scics.LG

Full-data accuracy with fewer labels for training and fine-tuning machine-learning force fields

classification physics.chem-ph cond-mat.mtrl-scics.LG
keywords machine-learning force fieldsactive learninguncertainty quantificationfoundation model fine-tuninglast-layer projection regressiondata selectionwater-in-salt electrolytemolecular dynamics
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that a cheap uncertainty signal computed from a machine-learned force field's own last-layer features can replace expensive model-committee uncertainty for active learning, in both train-from-scratch and foundation fine-tuning. The central claim is that picking training configurations by this signal reaches full-data accuracy with roughly an order of magnitude fewer labels than random selection, and close to the ceiling with about 100 labels on the tests shown. It further claims the same signal detects a foundation model's unphysical molecular-dynamics configurations before any quantum labels are computed, and can stop the learning loop automatically in absolute error units. A sympathetic reader would care because it lowers the labelling bottleneck and catches failures that standard validation diagnostics miss.

Core claim

Last-layer-projection regression (LLPR) uncertainty, defined per atom as the squared distance of the model's last-layer feature vector from the training-set covariance, is a quantitatively calibrated estimator of prediction error rather than just a ranking. This lets a single trained model drive active learning. In a train-from-scratch water/ice benchmark, energy- and force-LLPR pickers reach the full-data ceiling with K=100 labelled frames, about 2.5% of the 3955-frame full pool, while random sampling stays several-fold above the ceiling at the same budget and remains a factor of two to five away in the foundation fine-tuning setting. In the electrolyte case, the per-atom signal flags a hal

What carries the argument

The carrying object is the last-layer-projection-regression (LLPR) leverage u_a(x) = h_a(x)^T A^{-1} h_a(x), where h_a(x) is the per-atom activation vector of the final network layer and A is the regularized covariance of those activations over the training/reference set. It measures how far a configuration's internal representation lies from the training distribution. Built on this are two aggregates: an energy signal (sum-pooled frame feature) and a top-5 force signal (mean of the five largest per-atom leverages), plus a calibration factor fitted on held-out picks to convert the dimensionless leverage into absolute force uncertainty in eV/angstrom. This same quantity does three jobs: ranki

Load-bearing premise

The whole workflow rests on the premise that a model's feature-space distance to its training data is a trustworthy measure of when its predictions are wrong—trustworthy enough to select labels and to flag a hallucinated contact before labels exist—and, in the electrolyte test, that the model used as labelling oracle is itself accurate enough to be ground truth.

What would settle it

The paper's most distinctive prediction is that, on a foundation-MD trajectory that is stable but structurally wrong, the pathological atoms appear in the extreme tail of the per-atom LLPR distribution before any labels are computed. A reader can test this by taking any foundation model on a new dense electrolyte, running a short stable MD, and computing the per-atom leverage from the model's own features and training covariance; if the atoms involved in the unphysical short contact are not the tail, the pre-label hallucination detector does not work.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • On the water/ice benchmark, the LLPR picker reaches the full-data ceiling at K=100 labels, against a 3955-frame full pool; random picking remains several-fold above the ceiling at the same budget.
  • In foundation fine-tuning, the picker effect transfers: LLPR picks on the smaller foundation close the gap to the full-pool ceiling at K=100, while random picks leave a factor-of-two-to-five gap; which frames are picked matters more than which foundation is used.
  • In the electrolyte case, about 50 labels across eight active-learning rounds lift the hallucinated Li-O potential, restore the physical radial distribution, and recover an equilibrium density consistent with experiment and reference.
  • The same uncertainty signal detects the pathological short Li-O contact before any labelling, because the affected atoms sit in the extreme tail of the per-atom leverage distribution.
  • An absolute per-atom force-error threshold (100 meV/angstrom, set by the thermal-force scale) provides a self-terminating stopping criterion for the active-learning loop.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Inference: the same per-atom feature leverage could act as a live alarm during production MD, flagging trajectories that drift into unexplored regions even when energy conservation and density look normal.
  • Inference: if the training-set covariance is updated continuously, the method may extend to continual learning or domain-adaptive retraining where the target distribution shifts over time, rather than only fixed benchmark pools.
  • Inference: the 100 meV/angstrom exit threshold is calibrated to the 300-K thermal scale on a roughly 1500-atom cell; at other temperatures, pressures, or cell sizes a user would need to re-derive the threshold, which could be done self-consistently from the LLPR calibration factor.
  • Inference: a practical pre-screening recipe suggested by the paper is to run a short stable MD with a candidate foundation, inspect the per-atom LLPR tail, and only spend labels if the tail is elevated; this could be tested on other foundation-target pairs.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper presents an active-learning workflow based on last-layer projection regression (LLPR), a single-forward-pass uncertainty estimator computed from the model's last-layer features and training-set covariance. The workflow is applied in three settings: a train-from-scratch DeePMD model on a water/ice dataset, one-shot fine-tuning of MACE-MP-0 on the same water/ice pool, and iterative fine-tuning of MACE-OMAT-small on a 21 molal LiTFSI water-in-salt electrolyte. The central claims are that LLPR-selected labels reach near-full-data accuracy with substantially fewer labels than random selection, that LLPR is competitive with a model committee at much lower cost, that it detects a hallucinated Li–O contact in the electrolyte before labels are committed, and that it provides an absolute force-error threshold enabling automatic termination of the active-learning loop.

Significance. If the claims hold, this is a practically important result: model-committee uncertainty is often impractical for foundation-scale fine-tuning, and a cheap, deterministic, single-model uncertainty signal that drives data selection, pathology detection, and stopping would be a useful advance. The water/ice train-from-scratch benchmark is carefully designed: the same candidate pool, identical retraining recipe, three seeds, random and committee baselines, and a full-data ceiling are used, and the LLPR calibration curves against actual prediction errors are convincing. The main unresolved issues are the absence of a random+FPS control in the water/ice comparison, and the use of an MLFF pseudo-oracle rather than electronic-structure labels in the electrolyte section. These are fixable within the manuscript's scope, but they currently leave the headline label-efficiency claim partially underdetermined.

major comments (3)
  1. [Validating LLPR... / Methods: Train-from-scratch active learning on water/ice] The K-sweep comparison does not isolate the LLPR uncertainty signal from the diversity post-processing. Results states that 'within the K picks of each strategy, frames are diversified by SOAP-FPS on top of the uncertainty score,' but Methods defines the random picker simply as selecting K frames from the candidate pool, with no mention of an analogous FPS step. Because rare dense-ice phases (ice VIII, ice XV) dominate the worst-1% error and are unlikely to appear in 50–100 random draws, the low-K random baseline may be degraded by lack of diversity rather than by absence of an uncertainty ranking. Please add a random+FPS (and ideally a pure-FPS) arm, or explicitly state that the random picker also receives SOAP-FPS diversification and describe how.
  2. [Detecting and resolving foundation MD hallucination / Methods: Iterative LLPR-based active learning] The electrolyte demonstration does not use electronic-structure labels. In Methods, the labels are produced by the matpes-r2SCAN-OMAT-FT foundation, an MLFF, and this is described as a 'pseudo-oracle.' However, the Abstract claims recovery of full-data accuracy using 'a small fraction of electronic-structure labels' and the Introduction frames the detection as occurring 'before DFT labelling.' The absolute force-error threshold in Eq. (2) is calibrated against this pseudo-oracle, not against DFT. The electrolyte arm therefore demonstrates repair relative to a specific reference MLFF, not a reduction in DFT-labelling cost. Please rephrase the relevant claims and state the pseudo-oracle limitation explicitly.
  3. [Methods: Eq. (2) calibration] The calibration factor C_geom is estimated from the K=10 picks of round k, which are described as an 'unbiased held-out set.' These picks are selected by the LLPR signal itself, so they are not unbiased with respect to the uncertainty distribution; they preferentially sample high-uncertainty configurations. If the e^2/u ratio is not constant across the uncertainty range, this can bias the absolute units used for the exit threshold. With only 10 points per round, the calibration is also noisy. Please provide calibration diagnostics on a random sample or report the sensitivity of the stopping round to the calibration procedure.
minor comments (4)
  1. [Fig. 3a] The energy LLPR calibration uses a 'constant-offset-corrected ground-truth squared energy error.' Please state how the offset is determined and whether the offset correction affects the ranking used for selection. As presented, the reader cannot tell whether the excellent y=x agreement is partly a consequence of this correction.
  2. [Fig. 5c] The vertical axis is labelled 'uncertainty error' and reported in meV/Å, but the Methods define the stopping criterion using the 99th percentile of the calibrated σ_F. Please clarify whether Fig. 5c plots the calibrated σ_F, the raw LLPR leverage, or a different quantity, and how it relates to the 100 meV/Å threshold.
  3. [Data and code availability] The statement that data and code are available 'from the authors on reasonable request' and 'will be released open-source on Zenodo upon publication' is weaker than the reproducibility standard common for this type of computational study. At minimum, please provide the exact version of the code/checkpoints used and a timeline.
  4. [Abstract / Introduction] The phrase 'full-data accuracy' is used throughout, but the reported numbers at K=100 in the train-from-scratch case are 1.7 meV/atom versus a 1.30 meV/atom ceiling (31% above) and worst-1% errors within a factor of two. Consider using 'near-full-data accuracy' or defining the statistical criterion quantitatively to avoid overstatement.

Circularity Check

0 steps flagged

No significant circularity: LLPR selection, validation, and stopping are independent of the labels being selected; the only fitted calibration constant is disclosed and used only for thresholding.

full rationale

Walked the claimed derivation chain. Eq. 1 defines LLPR leverage u_a from last-layer features and the training-set covariance; it does not use the labels being selected, so the ranking signal is not derived from the target quantities it is used to select. Validation in Fig. 1 and Fig. 3 compares LLPR to held-out squared errors and to retrieval of worst-K frames, so the agreement is empirical rather than forced. The active-learning K-sweep compares picker strategies on a held-out test set; the absence of a SOAP-FPS-diversified random baseline is a possible confound between diversity and uncertainty, but it does not make the result equivalent to its inputs by construction. The C_geom calibration in Eq. 2 is fitted to the previous round's held-out picks, but it is a single disclosed scaling constant used only to express the stopping threshold in absolute units; it does not rank frames and does not determine the central label-efficiency comparison. The electrolyte pseudo-oracle is an independently benchmarked external reference [39], and final density and Li-O RDF are checked against experiment, so the recovery is not circular. Self-citations ([25] implementation, [30] prior LLPR calibration) are not load-bearing: the paper independently re-derives and validates the LLPR signal on held-out data. The Discussion notes a scope limitation (demonstrated on the MACE foundation family), which is a generality caveat rather than a circular step. No load-bearing step reduces by construction to a fit or to a self-citation chain.

Axiom & Free-Parameter Ledger

5 free parameters · 4 axioms · 0 invented entities

The paper introduces no new physical entities or conservation laws, but the workflow depends on the LLPR reliability premise and on the pseudo-oracle accuracy premise. It also contains several fitted calibration parameters (ridge, C_geom, energy-offset correction, atomic-energy re-alignment, exit threshold) that should be tracked when interpreting the quantitative claims.

free parameters (5)
  • Tikhonov ridge sigma_F = 5e-3
    Eq. 1 uses sigma_F = 5e-3 in the covariance inverse A = (1/N)Sum + sigma_F^2 I. Chosen by hand; affects LLPR magnitudes and which frames are selected.
  • C_geom calibration factor per round = C_geom = exp(<log(e_a^2/u_a)>_a)
    Eq. 2 fits C_geom on the K=10 held-out picks of each round to convert LLPR leverage into absolute force-error units. Used for the stopping threshold.
  • Energy-offset correction in Fig. 3a = not specified
    The caption says ground-truth squared energy error is 'constant-offset-corrected'; this is a fitted offset that affects the claimed y=x calibration.
  • Per-element atomic-energy re-alignment for MACE heads = per-element linear regression on 200 liquid-anchor frames
    The MACE-MP-0 H/O atomic-energy references are re-aligned to RPBE-D3 by a linear fit; this is a standard but data-fitted transformation.
  • Exit threshold for active learning = 100 meV/A
    User-chosen threshold 'matching the 300-K thermal-force scale on a 1500-atom cell'; controls self-termination.
axioms (4)
  • domain assumption LLPR uncertainty, computed from last-layer features and training-set covariance, is a valid per-configuration epistemic uncertainty estimate for MLFFs.
    Assumed throughout (Eq. 1 and all figures); relies on ref. [31] and is not re-derived in this paper.
  • domain assumption matpes-r2SCAN-OMAT-FT is an accurate reference for 21 m LiTFSI, including density, diffusion, X-ray structure factor, and the Li-O PES.
    Used as the labelling pseudo-oracle in the electrolyte active-learning loop; supported only by citation to ref. [39], not independently verified here.
  • domain assumption The OMAT24 pretraining distribution lacks Li-O contacts below ~1.5 A, causing the hallucinated attractive well.
    Proposed as an explanation for the OMAT-small PES pathology (Results/Discussion); plausible but not directly verified against OMAT24 sampling statistics.
  • domain assumption The multihead fine-tuning recipe with replay buffer preserves the foundation's capabilities while fitting the target set.
    Implied by the fine-tuning experiments; no ablation of the replay-buffer size or loss weights is provided.

pith-pipeline@v1.3.0-alltime-deepseek · 15167 in / 10523 out tokens · 107351 ms · 2026-08-02T01:55:55.973291+00:00 · methodology

0 comments
read the original abstract

Machine-learning force fields (MLFFs) are reliable only near their training distribution, making efficient construction of diverse training sets a major bottleneck for both train-from-scratch and foundation fine-tuning workflows. Active learning can reduce this cost, but standard model-committee uncertainty is impractical for foundation MLFFs because each committee member requires a separate fine-tuning run. We present an active-learning workflow based on last-layer-projection regression (LLPR), a forward-pass-cheap per-configuration uncertainty estimator. Across molecular, condensed-phase, and electrolyte systems, LLPR identifies compact, high-value training sets that recover full-data accuracy using only a small fraction of electronic-structure labels. In foundation-model fine-tuning, LLPR-selected configurations reach the full-pool fine-tuning ceiling with substantially fewer labels than random selection. In iterative electrolyte fine-tuning, LLPR detects unphysical local coordination before DFT labelling, provides an absolute force-error threshold, and enables automatic termination of the learning loop. The resulting models reproduce reference density and ion-coordination structure, providing a scalable uncertainty-quantification strategy across MLFF training regimes.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

40 extracted references · 2 linked inside Pith

  1. [1]

    T.et al.Machine Learning Force Fields.Chemical Reviews121, 10142–10186 (2021)

    Unke, O. T.et al.Machine Learning Force Fields.Chemical Reviews121, 10142–10186 (2021)

  2. [2]

    L., Caro, M

    Deringer, V. L., Caro, M. A. & Cs´ anyi, G. Machine Learning Interatomic Potentials as Emerging Tools for Materials Science.Advanced Materials31 (2019)

  3. [3]

    Perspective: Machine learning potentials for atomistic simulations.The Journal of Chemical Physics145(2016)

    Behler, J. Perspective: Machine learning potentials for atomistic simulations.The Journal of Chemical Physics145(2016)

  4. [4]

    & Lordi, V

    Schwalbe-Koda, D., Hamel, S., Sadigh, B., Zhou, F. & Lordi, V. Model-free esti- mation of completeness, uncertainties, and outliers in atomistic machine learning using information theory.Nature Communications16(2025)

  5. [5]

    Perez, D., Subramanyam, A. P. A., Maliyov, I. & Swinburne, T. D. Uncer- tainty quantification for misspecified machine learned interatomic potentials.npj Computational Materials11(2025). 20

  6. [6]

    Montes de Oca Zapiain, D.et al.Training data selection for accuracy and transferability of interatomic potentials.npj Computational Materials8(2022)

  7. [7]

    P. A. Subramanyam, A. & Perez, D. Information-entropy-driven generation of material-agnostic datasets for machine-learning interatomic potentials.npj Computational Materials11(2025)

  8. [8]

    Deng, B.et al.CHGNet as a pretrained universal neural network potential for charge-informed atomistic modelling.Nature Machine Intelligence5, 1031–1041 (2023)

  9. [9]

    Barroso-Luque, L.et al.The Open Materials 2024 (OMat24) inorganic materials dataset and models.Nature Computational Science6, 642–652 (2026)

  10. [10]

    Batatia, I.et al.A foundation model for atomistic materials chemistry.The Journal of Chemical Physics163(2025)

  11. [11]

    Zhang, D.et al.DPA-2: a large atomic model as a multi-task learner.npj Computational Materials10(2024)

  12. [12]

    G., Allen, C

    Radova, M., Stark, W. G., Allen, C. S., Maurer, R. J. & Bart´ ok, A. P. Fine- tuning foundation models of materials interatomic potentials with frozen transfer learning.npj Computational Materials11(2025)

  13. [13]

    Allen, A. E. A.et al.Learning together: Towards foundation models for machine learning interatomic potentials with meta-learning.npj Computational Materials 10(2024)

  14. [14]

    Deng, B.et al.Systematic softening in universal machine learning interatomic potentials.npj Computational Materials11(2025)

  15. [15]

    Tokita, A. M. & Behler, J. How to train a neural network potential.The Journal of Chemical Physics159(2023)

  16. [16]

    & Tadmor, E

    Wen, M. & Tadmor, E. B. Uncertainty quantification in molecular simulations with dropout neural network potentials.npj Computational Materials6(2020)

  17. [17]

    R., Urata, S., Goldman, S., Dietschreit, J

    Tan, A. R., Urata, S., Goldman, S., Dietschreit, J. C. B. & G´ omez-Bombarelli, R. Single-model uncertainty quantification in neural network potentials does not consistently outperform model ensembles.npj Computational Materials9(2023)

  18. [18]

    S., Nebgen, B., Lubbers, N., Isayev, O

    Smith, J. S., Nebgen, B., Lubbers, N., Isayev, O. & Roitberg, A. E. Less is more: Sampling chemical space with active learning.The Journal of Chemical Physics 148(2018)

  19. [19]

    Zhang, L., Lin, D.-Y., Wang, H., Car, R. & E, W. Active learning of uni- formly accurate interatomic potentials for materials simulation.Physical Review Materials3(2019). 21

  20. [20]

    Podryabinkin, E. V. & Shapeev, A. V. Active learning of linearly parametrized interatomic potentials.Computational Materials Science140, 171–180 (2017)

  21. [21]

    & Marsalek, O

    Schran, C., Brezina, K. & Marsalek, O. Committee neural network potentials control generalization errors and enable active learning.The Journal of Chemical Physics153(2020)

  22. [22]

    Vandermause, J.et al.On-the-fly active learning of interpretable Bayesian force fields for atomistic rare events.npj Computational Materials6(2020)

  23. [23]

    S., Gubaev, K., Podryabinkin, E

    Novikov, I. S., Gubaev, K., Podryabinkin, E. V. & Shapeev, A. V. The MLIP package: moment tensor potentials with MPI and active learning.Machine Learning: Science and Technology2, 025002 (2021)

  24. [24]

    Zhang, Y.et al.DP-GEN: A concurrent learning platform for the genera- tion of reliable deep learning based potential energy models.Computer Physics Communications253, 107206 (2020)

  25. [25]

    ai2-kit: artificial intelligence x ab initio toolkit for complex chemistry systems research

    Cheng Group. ai2-kit: artificial intelligence x ab initio toolkit for complex chemistry systems research. https://github.com/chenggroup/ai2-kit (2023). MIT-licensed software repository; accessed 2026-06-26

  26. [26]

    S., Shamir, E

    Freund, Y., Seung, H. S., Shamir, E. & Tishby, N. Selective Sampling Using the Query by Committee Algorithm.Machine Learning28, 133–168 (1997)

  27. [27]

    URL https: //arxiv.org/abs/2504.06231

    Rhodes, B.et al.Orb-v3: atomistic simulation at scale (2025). URL https: //arxiv.org/abs/2504.06231. arXiv:2504.06231

  28. [28]

    Liu, K.et al.Heterogeneous ensemble enables a universal uncertainty metric for atomistic foundation models.npj Computational Materials12(2025)

  29. [29]

    A., Firoz, J

    Bilbrey, J. A., Firoz, J. S., Lee, M.-S. & Choudhury, S. Uncertainty quantification for neural network potential foundation models.npj Computational Materials11 (2025)

  30. [30]

    H., Ortner, C

    Ho, C. H., Ortner, C. & Wang, Y. Flexible uncertainty calibration for machine- learned interatomic potentials.npj Computational Materials12(2026)

  31. [31]

    & Grasselli, F

    Bigi, F., Chong, S., Ceriotti, M. & Grasselli, F. A prediction rigidity formalism for low-cost uncertainties in trained neural networks.Machine Learning: Science and Technology5, 045018 (2024)

  32. [32]

    & Behler, J

    Morawietz, T., Singraber, A., Dellago, C. & Behler, J. How van der Waals inter- actions determine the unique properties of water.Proceedings of the National Academy of Sciences113, 8368–8373 (2016)

  33. [33]

    & Behler, J

    Morawietz, T. & Behler, J. HDNNP training data set for H2O (2019). URL https://zenodo.org/record/2634098. 22

  34. [34]

    URL https://openreview.net/forum?id= A8pqQipwkt

    Fu, X.et al.Forces are not Enough: Benchmark and Critical Evaluation for Machine Learning Force Fields with Molecular Simulations.Transactions on Machine Learning Research(2023). URL https://openreview.net/forum?id= A8pqQipwkt

  35. [35]

    Imbalzano, G.et al.Automatic selection of atomic fingerprints and reference configurations for machine-learning potentials.The Journal of Chemical Physics 148(2018)

  36. [36]

    Water-in-salt

    Suo, L.et al.“Water-in-salt” electrolyte enables high-voltage aqueous lithium-ion chemistries.Science350, 938–943 (2015)

  37. [37]

    P., Simm, G

    Batatia, I., Kov´ acs, D. P., Simm, G. N. C., Ortner, C. & Cs´ anyi, G. MACE: Higher Order Equivariant Message Passing Neural Networks for Fast and Accu- rate Force Fields (2022). URL https://arxiv.org/abs/2206.07697. Advances in Neural Information Processing Systems 35 (NeurIPS 2022), arXiv:2206.07697

  38. [38]

    Gilbert, W. J. R.et al.Density, Viscosity, and Vapor Pressure Measurements of Water + Lithium Bis(trifluoromethylsulfonyl)imide Solutions.Journal of Chemical & Engineering Data62, 2056–2066 (2017)

  39. [39]

    M., Serva, A

    Brugnoli, L., Salanne, M., Saitta, A. M., Serva, A. & France-Lanord, A. Over- coming sampling limitations using machine-learned interatomic potentials: the case of water-in-salt electrolytes.arXiv preprint arXiv:2603.22099(2026). URL https://arxiv.org/abs/2603.22099

  40. [40]

    & Lee Holian, B

    Melchionna, S., Ciccotti, G. & Lee Holian, B. Hoover NPT dynamics for systems varying in shape and size.Molecular Physics78, 533–544 (1993). 23