Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

Uncertainty Quantification in Graph Neural Networks with Shallow Ensembles

T0 review · 4 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash

Pith's one-line read Shallow ensembles of 64 SchNet heads can flag out-of-domain molecules and materials through prediction variance alone, when the domain shift is large.

desk verdict A light but honest empirical study of DPOSE on SchNet; the composition-shift results are convincing, but the lack of calibration and baselines keeps it a preliminary exploration. read the letter →

arxiv 2504.12627 v1 pith:GIVQPRCW submitted 2025-04-17 cs.LG physics.comp-ph

classification cs.LGphysics.comp-ph
keywords uncertaintyquantificationshallowensemblesgraphneuralnetworksSchNetout-of-domaindetectionmachine-learnedpotentialsnegativelog-likelihoodloss
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that a lightweight uncertainty-quantification method, Direct Propagation of Shallow Ensembles (DPOSE), can make graph neural network potentials tell users when they are extrapolating. The authors attach 64 shared-weight output heads to SchNet, train them with a negative log-likelihood loss, and use the spread of the heads' energy predictions as the uncertainty estimate. They report that the spread stays low for in-domain inputs and rises sharply for unseen elements, larger molecules, and chemically different slabs, which matters because materials discovery depends on knowing when a prediction is unreliable. The paper also reports a clear limit: the same method did not separate amorphous gold from bulk gold, so it flags compositional novelty more readily than same-element structural novelty.

What carries the argument

The central object is DPOSE, a shallow-ensemble method in which the model's last layer is replaced by 64 parallel output heads that share all earlier weights; the mean of the heads is the prediction and their variance is the uncertainty. The heads are trained jointly with the negative log-likelihood loss $\text{NLL}(\Delta y,\sigma) = \frac{1}{2}\left(\frac{\Delta y^2}{\sigma^2} + \ln(2\pi\sigma^2)\right)$, which couples each head's squared error to the ensemble variance. This mechanism lets one SchNet pass produce both an energy estimate and an uncertainty estimate, with a computational cost the paper compares to adding an extra hidden layer.

What would settle it

Take the QM9 substitution experiments and build a reliability diagram: bin molecules by DPOSE variance and plot the mean absolute prediction error in each bin; if the highest-variance molecules such as CCl4 and SiCl4 do not also show the largest errors, or if variance and error rankings diverge across the full set, the variance-as-uncertainty interpretation would fail even though the plotted variance trends remain.

Watch

Extended reading notes

Core claim

On its own terms, the paper claims that DPOSE-equipped SchNet produces useful out-of-domain signals without the cost of deep ensembles. The variance across 64 output heads increases from 0.09 for CF4 to 62 for CCl4, from 0.09 for CH4 to 2.7 for SiH4, and from 0.00022 eV/atom for gold to 0.613 eV/atom for silver, while alkane uncertainties grow from 0.09 for methane to 0.66 for octadecane as chain length leaves the training range. In OC20, non-metal slabs show median variance near 0.03 while intermetallic slabs stay below 0.01. The authors conclude that DPOSE distinguishes in-domain from out-of-domain data when the two are far apart, but that structurally similar systems made of the same element, such as amorphous versus bulk gold, are not separated by variance.

Load-bearing premise

The load-bearing premise is that the variance among the 64 NLL-trained heads is a trustworthy stand-in for predictive error on out-of-domain inputs, which enters when the paper reads higher variance as lower confidence and is never checked against calibration.

Editorial extensions

If this is right

  • A single SchNet model with 64 heads can emit an uncertainty flag for every energy prediction, so out-of-domain detection does not require training several full models.
  • Compositional shifts, such as substituting an unseen element into a known molecule, produce variance jumps large enough to separate in-domain from out-of-domain molecules in QM9 and in gold-versus-silver comparisons.
  • Uncertainty tracks physical extrapolation distance, rising as carbon-carbon bonds are stretched or compressed and as alkane or alcohol chains grow beyond the training lengths.
  • The method's failure to separate amorphous from bulk gold implies that same-element structural diversity is not reliably flagged, so the practical alarm is for chemistry novelty, not morphology novelty.
  • Because the added cost is comparable to one extra hidden layer, the approach is light enough to be coupled with active learning loops that acquire new training data where variance is high.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Inference: the observed contrast suggests DPOSE variance encodes compositional novelty more strongly than geometric novelty; a testable extension is to correlate variance with element-embedding distance rather than with structural descriptors.
  • Inference: since the 64 heads share all weights except the last layer, the ensemble diversity is limited to the final projection, so the variance may reflect last-layer sensitivity more than full epistemic uncertainty; comparing head counts (e.g., 8, 16, 64) would show how much of the signal comes from width alone.
  • Inference: the paper does not calibrate variance against prediction error; a natural extension is to check whether the reported variance ordering matches the actual error ordering on the same out-of-domain molecules, which would turn a qualitative trend into a usable uncertainty score.
  • Inference: for active learning, a DPOSE-variance acquisition function would likely prioritize novel elements and long molecules first, while missing amorphous phases of known elements; pairing variance with a latent-distance or energy-based score could cover that blind spot.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper integrates Direct Propagation of Shallow Ensembles (DPOSE) into the SchNet graph neural network by replacing the final layer with 64 output heads and training with the negative log-likelihood loss of Eq. (1). The resulting variance of the heads is used as an uncertainty estimate. The model is evaluated on three datasets: QM9 (equilibrium vs. non-equilibrium geometries, known vs. unknown elements, small vs. large molecules), OC20 (intermetallic vs. non-metal slabs, volume compression/expansion), and a Gold MD dataset (bulk vs. amorphous, Au vs. Ag). The reported results show large variance gaps for compositional shifts (e.g., CF4 0.09 vs. CCl4 62; Au 0.00022 vs. Ag 0.613 eV/atom), a gradual variance increase with molecular size, higher variance for non-metal slabs, and essentially no variance difference between bulk and amorphous gold. The authors conclude that DPOSE distinguishes in-domain from out-of-domain data when the domains are far apart, while acknowledging difficulty for structurally similar systems.

Significance. If head variance is a reliable proxy for predictive error, this would be a useful lightweight UQ method for GNN potentials, particularly for active learning. The study has strengths: it tests three diverse datasets, examines both compositional and structural shifts, and is transparent about a negative result (bulk vs. amorphous). However, the central claim of 'reliable uncertainty estimates' is not quantitatively validated, and the reported variance differences are aggregate and potentially scale-confounded. The contribution is an exploratory empirical study; its value depends on a calibration check and comparison with existing baselines.

major comments (4)
  1. [§2.2, §3 (general)] The central claim that DPOSE provides reliable uncertainty estimates requires demonstrating that the head variance from Eq. (1) is predictive of predictive error on individual out-of-domain samples. All reported evidence consists of aggregate variance differences between hand-picked groups (Tables 1–2, Figures 3–9); there is no calibration metric such as a reliability diagram, Spearman rank correlation between variance and absolute error, or an error-vs-variance plot for OOD samples. Without such a check, the large average gaps could be a feature-scale artifact rather than evidence of usable uncertainty.
  2. [§3.1.3, Table 2] The claim that uncertainty increases with molecular size is confounded by the extensivity of total energy. The QM9 variance values are reported without units or per-atom normalization, and the trend is non-monotonic (dodecane 0.62, tridecane 0.82, tetradecane 0.61). Since larger molecules have larger total energies, a naive variance of total energy may scale with system size even when per-atom uncertainty is constant. The authors should report normalized per-atom variances or otherwise show that the size trend is not a scale effect.
  3. [§3.3.1, Figure 6(c), Conclusion] The bulk-vs-amorphous result contradicts the stated expectation of lower uncertainty for in-domain bulk systems; the text acknowledges 'the model seems to be confident for out-of-domain systems too' and later that DPOSE 'struggled to distinguish between in-domain and out-of-domain configurations.' This is a central negative result for structural shifts. The abstract and conclusion currently over-generalize by saying DPOSE 'often demonstrates' or 'successfully distinguishes' without clearly carving out that the supported success is limited to compositional shifts and large domain distances. The paper should explicitly state in the abstract that structural OOD detection failed for Au bulk vs. amorphous.
  4. [§2, §3 (no baseline comparison)] The paper motivates DPOSE as a computationally efficient alternative to deep ensembles but never compares against any baseline UQ method, such as deep ensembles, MC dropout, or latent-distance uncertainty. Without a baseline, the strong wording about 'reliable' and 'better-calibrated' uncertainty is unsupported. At minimum, the authors should include one standard baseline with a calibration metric on the same OOD tasks.
minor comments (6)
  1. [§2, Eq. (1)] Equation (1) is presented without derivation or an explicit connection to the original DPOSE paper [19]; please cite the source in the text and define all symbols (e.g., whether sigma is per-output or shared).
  2. [§2.1] The name 'SchNET' is inconsistent with the standard 'SchNet' used elsewhere; correct the typo.
  3. [Tables 1–2, §3.1] The QM9 variance values are reported without units and without stating whether they are per atom or total. Please specify units (e.g., eV^2, (kcal/mol)^2) and clarify normalization.
  4. [Figure 5, caption] The caption refers to a 'red line' while the text and figure show a red dotted line; make the reference consistent.
  5. [§3.2, consistent terminology] The terms 'inter-metallic' and 'intermetallic' are used inconsistently (e.g., §3.2.1 'inter-metallic slabs' vs. §2.2.2 'intermetallic slab subset'); choose one spelling.
  6. [Figure 6(c), §3.3.1] The text says 'the mean variance was slightly higher for bulk systems' while the caption says 'Variance estimates showing slightly higher uncertainty for bulk'; clarify whether the box plot displays means, medians, or distributions.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the OOD uncertainty evaluations are external held-out tests, not constructions from the training objective.

full rationale

The paper's derivation chain is not circular. DPOSE's variance is produced by 64 output heads trained with the NLL loss (Eq. 1) on in-domain data, but every central OOD claim is evaluated on data excluded from that training: QM9 molecules with unseen elements (Si, S, P, Cl) or larger chains (C10-C18), OC20 non-metal slabs after fine-tuning on intermetallic slabs, and silver systems for an Au-only model. The reported variance gaps (e.g., CF4 0.09 vs CCl4 62; gold 0.00022 vs silver 0.613 eV/atom) are model outputs on held-out groups, not parameters fitted to produce those group differences. The paper even reports a contrary result for bulk vs amorphous gold ('the mean variance was slightly higher for bulk systems, contrary to the expectation'), which confirms the evaluation is not rigged by construction. The self-citations (refs 9, 16, 24 include Kitchin as coauthor) are background and data-source citations and are not load-bearing; the DPOSE method itself is attributed to Kellner and Ceriotti (ref 19), not to the authors. No uniqueness theorem or ansatz is imported from the authors' prior work to force the conclusion. The absence of an error-vs-variance calibration check is a scientific validation gap, not a circularity.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The paper's evaluation rests on treating the DPOSE variance as epistemic uncertainty, on the chosen OOD test sets being representative shifts, and on DFT labels as ground truth. There are no invented entities and no data-derived constants beyond the usual fitted model weights and the chosen 64-head architecture.

free parameters (1)
  • Number of ensemble heads = 64
    Architecture choice for the shallow ensemble; no sensitivity analysis is reported, and all variance conclusions depend on this number.
assumptions (3)
  • domain assumption The variance of the 64 head predictions, trained under NLL loss, is a valid measure of predictive uncertainty for out-of-domain inputs.
    Section 2.2 and Equation 1. The central evaluation treats this variance as epistemic uncertainty without calibration checks.
  • domain assumption The chosen out-of-domain test sets (non-metal slabs, silver systems, larger molecules) represent meaningful distribution shifts.
    Sections 2.2.1 to 2.2.3 define out-of-domain by absence from fine-tuning; the paper assumes this absence produces a detectable shift.
  • domain assumption DFT reference energies used for training and evaluation are accurate ground truths.
    The paper uses DFT labels as ground truth without quantifying DFT error, which is standard practice but still an assumption.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Uncertainty Quantification in Graph Neural Networks with Shallow Ensembles." pith.science (2026). https://pith.science/paper/GIVQPRCW

@misc{pith2026250412627,
  author       = {Pith},
  title        = {Pith review of: Uncertainty Quantification in Graph Neural Networks with Shallow Ensembles},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GIVQPRCW}},
  note         = {Machine review of arXiv:2504.12627}
}
read the original abstract

Machine-learned potentials (MLPs) have revolutionized materials discovery by providing accurate and efficient predictions of molecular and material properties. Graph Neural Networks (GNNs) have emerged as a state-of-the-art approach due to their ability to capture complex atomic interactions. However, GNNs often produce unreliable predictions when encountering out-of-domain data and it is difficult to identify when that happens. To address this challenge, we explore Uncertainty Quantification (UQ) techniques, focusing on Direct Propagation of Shallow Ensembles (DPOSE) as a computationally efficient alternative to deep ensembles. By integrating DPOSE into the SchNet model, we assess its ability to provide reliable uncertainty estimates across diverse Density Functional Theory datasets, including QM9, OC20, and Gold Molecular Dynamics. Our findings often demonstrate that DPOSE successfully distinguishes between in-domain and out-of-domain samples, exhibiting higher uncertainty for unobserved molecule and material classes. This work highlights the potential of lightweight UQ methods in improving the robustness of GNN-based materials modeling and lays the foundation for future integration with active learning strategies.

Figures

Figures reproduced from arXiv: 2504.12627 by the authors.

Figure 1
Figure 1. Comparison of the Original Architecture and the DPOSE Architecture. The [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Internal energy predictions and uncertainty estimates for C [PITH_FULL_IMAGE:figures/full_fig_p011_2.png] view at source ↗
Figure 3
Figure 3. (a) Parity plot for inter-metallic slabs showing high accuracy. (b) Parity [PITH_FULL_IMAGE:figures/full_fig_p014_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Change in volume per atom for three test cases in inter-metallic slabs: (a) [PITH_FULL_IMAGE:figures/full_fig_p015_4.png]
Figure 5
Figure 5. Figure 5: Change in volume per atom for three nonmetal slab systems: (a) Pb [PITH_FULL_IMAGE:figures/full_fig_p015_5.png]
Figure 6
Figure 6. Figure 6: (a) Parity plot for bulk systems showing high accuracy. (b) Parity plot for [PITH_FULL_IMAGE:figures/full_fig_p016_6.png]
Figure 7
Figure 7. Figure 7: Energy-Variance-Error analysis: Energy, variance, and mean absolute error [PITH_FULL_IMAGE:figures/full_fig_p017_7.png]
Figure 8
Figure 8. Figure 8: Energy-Variance-Error analysis: Energy, variance, and mean absolute error [PITH_FULL_IMAGE:figures/full_fig_p018_8.png]
Figure 9
Figure 9. Figure 9: Box plot comparing the variance (σ/atom) of the AuMD-SE model on in￾domain (gold) and out-of-domain (silver) datasets. The gold dataset exhibits low variance, indicating high confidence due to fine-tuning on gold systems. In contrast, the silver dataset shows significa…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Geometric deep learning assists protein engineering. Opportunities and Challenges

    q-bio.QM 2025-06 conditional novelty 3.0 of 10

    A perspective synthesizing geometric deep learning applications in protein engineering and proposing an explainable, structure-aware design pipeline.

Reference graph

Works this paper leans on

28 extracted references · 15 canonical work pages · cited by 1 Pith paper

  1. [1]

    Fang J, Xie M, He X, Zhang J, Hu J, Chen Y, Yang Y and Jin Q 2022 Materials Today Communications 33 104900 ISSN 2352-4928 URL https://www.sciencedirect.com/science/ article/pii/S235249282201741X

  2. [2]

    Merchant A, Batzner S, Schoenholz S S, Aykol M, Cheon G and Cubuk E D 2023 Nature 624 80–85 ISSN 1476-4687 publisher: Nature Publishing Group URL https://www.nature.com/ articles/s41586-023-06735-9

  3. [3]

    Xia W, Tang L, Sun H, Zhang C, Ho K M, Viswanathan G, Kovnir K and Wang C Z 2023 Journal of Materials Chemistry A 11 25973–25982 ISSN 2050-7496 publisher: The Royal Society of Chemistry URL https://pubs.rsc.org/en/content/articlelanding/2023/ta/d3ta03771a

  4. [4]

    Dragoni D, Daff T D, Cs´ anyi G and Marzari N 2018Physical Review Materials 2 013808 publisher: American Physical Society URL https://link.aps.org/doi/10.1103/PhysRevMaterials.2. 013808

  5. [5]

    Goeminne R, Vanduyfhuys L, Van Speybroeck V and Verstraelen T 2023 Journal of Chemical Theory and Computation 19 6313–6325 ISSN 1549-9618 publisher: American Chemical Society URL https://doi.org/10.1021/acs.jctc.3c00495

  6. [6]

    1088/2515-7639/ab084b

    Schleder G R, Padilha A C M, Acosta C M, Costa M and Fazzio A 2019 Journal of Physics: Materials 2 032001 ISSN 2515-7639 publisher: IOP Publishing URL https://dx.doi.org/10. 1088/2515-7639/ab084b

  7. [7]

    Fung V, Zhang J, Juarez E and Sumpter B G 2021 npj Computational Materials 7 1–8 ISSN 2057-3960 publisher: Nature Publishing Group URL https://www.nature.com/articles/ Uncertainty Quantification in Graph Neural Networks with Shallow Ensembles 22 s41524-021-00554-0

  8. [8]

    Louis S Y, Zhao Y, Nasiri A, Wang X, Song Y, Liu F and Hu J 2020 Physical Chemistry Chemical Physics 22 18141–18148 ISSN 1463-9084 publisher: The Royal Society of Chemistry URL https://pubs.rsc.org/en/content/articlelanding/2020/cp/d0cp01474e

Show all 28 references
  1. [9]

    Sunshine E M, Shuaibi M, Ulissi Z W and Kitchin J R 2023 The Journal of Physical Chemistry C 127 23459–23466 ISSN 1932-7447 publisher: American Chemical Society URL https: //doi.org/10.1021/acs.jpcc.3c06157

  2. [10]

    Varivoda D, Dong R, Omee S S and Hu J 2022 Materials Property Prediction with Uncertainty Quantification: A Benchmark Study arXiv:2211.02235 [cond-mat] URL http://arxiv.org/ abs/2211.02235

  3. [11]

    Peterson A A, Christensen R and Khorshidi A 2017 Physical Chemistry Chemical Physics 19 10978–10985 ISSN 1463-9084 publisher: The Royal Society of Chemistry URL https: //pubs.rsc.org/en/content/articlelanding/2017/cp/c7cp00375g

  4. [12]

    Singh A and Li Y 2021 Uncertainty Management and Reduction of Machine Learning Potential AIAA Scitech 2021 Forum AIAA SciTech Forum (American Institute of Aeronautics and Astronautics) URL https://arc.aiaa.org/doi/10.2514/6.2021-1962

  5. [13]

    Xin R, Siriwardane E M D, Song Y, Zhao Y, Louis S Y, Nasiri A and Hu J 2021 The Journal of Physical Chemistry C 125 16118–16128 ISSN 1932-7447 publisher: American Chemical Society URL https://doi.org/10.1021/acs.jpcc.1c02438

  6. [14]

    Egele R, Maulik R, Raghavan K, Lusch B, Guyon I and Balaprakash P 2022 AutoDEUQ: Automated Deep Ensemble with Uncertainty Quantification arXiv:2110.13511 [cs] URL http: //arxiv.org/abs/2110.13511

  7. [15]

    Rahaman R and Thiery A H 2021 Uncertainty Quantification and Deep Ensembles arXiv:2007.08792 [stat] URL http://arxiv.org/abs/2007.08792

  8. [16]

    Musielewicz J, Lan J, Uyttendaele M and Kitchin J R 2024 The Journal of Physical Chemistry C 128 20799–20810 ISSN 1932-7447 publisher: American Chemical Society URL https: //doi.org/10.1021/acs.jpcc.4c04972

  9. [17]

    Olivier A, Shields M D and Graham-Brady L 2021 Computer Methods in Applied Mechanics and Engineering 386 114079 ISSN 0045-7825 URL https://www.sciencedirect.com/science/ article/pii/S0045782521004102

  10. [18]

    Gal Y and Ghahramani Z 2016 Dropout as a Bayesian Approximation: Representing Model Uncertainty Quantification in Graph Neural Networks with Shallow Ensembles 23 Uncertainty in Deep Learning arXiv:1506.02142 [stat] URL http://arxiv.org/abs/1506. 02142

  11. [19]

    Kellner M and Ceriotti M 2024 Machine Learning: Science and Technology 5 035006 ISSN 2632- 2153 publisher: IOP Publishing URL https://dx.doi.org/10.1088/2632-2153/ad594a

  12. [20]

    Sch¨ utt K T, Sauceda H E, Kindermans P J, Tkatchenko A and M¨ uller K R 2018The Journal of Chemical Physics 148 241722 ISSN 0021-9606 URL https://doi.org/10.1063/1.5019779

  13. [21]

    Chanussot L, Das A, Goyal S, Lavril T, Shuaibi M, Riviere M, Tran K, Heras-Domingo J, Ho C, Hu W, Palizhati A, Sriram A, Wood B, Yoon J, Parikh D, Zitnick C L and Ulissi Z 2021 ACS Catalysis 11 6059–6072 publisher: American Chemical Society URL https: //doi.org/10.1021/acscata...

  14. [22]

    Ramakrishnan R, Dral P O, Rupp M and von Lilienfeld O A 2014 Scientific Data 1 140022 ISSN 2052-4463 publisher: Nature Publishing Group URL https://www.nature.com/articles/ sdata201422

  15. [23]

    Ruddigkeit L, van Deursen R, Blum L C and Reymond J L 2012 Journal of Chemical Information and Modeling 52 2864–2875 ISSN 1549-9596 publisher: American Chemical Society URL https://doi.org/10.1021/ci300415d

  16. [24]

    wiley.com/doi/abs/10.1002/qua.25115

    Boes J R, Groenenboom M C, Keith J A and Kitchin J R 2016 Interna- tional Journal of Quantum Chemistry 116 979–987 ISSN 1097-461X eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1002/qua.25115 URL https://onlinelibrary. wiley.com/doi/abs/10.1002/qua.25115

  17. [25]

    Gasteiger J, Giri S, Margraf J T and G¨ unnemann S 2022 Fast and Uncertainty-Aware Directional Message Passing for Non-Equilibrium Molecules arXiv:2011.14115 [cs] URLhttp://arxiv.org/ abs/2011.14115

  18. [26]

    Gasteiger J, Groß J and G¨ unnemann S 2022 Directional Message Passing for Molecular Graphs arXiv:2003.03123 [cs] URL http://arxiv.org/abs/2003.03123

  19. [27]

    Liao Y L and Smidt T 2023 Equiformer: Equivariant Graph Attention Transformer for 3D Atomistic Graphs arXiv:2206.11990 [cs] URL http://arxiv.org/abs/2206.11990

  20. [28]

    Liao Y L, Wood B, Das A and Smidt T 2024 EquiformerV2: Improved Equivariant Transformer for Scaling to Higher-Degree Representations arXiv:2306.12059 [cs] URL http://arxiv.org/ abs/2306.12059

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.