REVIEW 4 major objections 6 minor 1 cited by
Uncertainty Quantification in Graph Neural Networks with Shallow Ensembles
T0 review · 4 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash
Pith's one-line read Shallow ensembles of 64 SchNet heads can flag out-of-domain molecules and materials through prediction variance alone, when the domain shift is large.
desk verdict A light but honest empirical study of DPOSE on SchNet; the composition-shift results are convincing, but the lack of calibration and baselines keeps it a preliminary exploration. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is DPOSE, a shallow-ensemble method in which the model's last layer is replaced by 64 parallel output heads that share all earlier weights; the mean of the heads is the prediction and their variance is the uncertainty. The heads are trained jointly with the negative log-likelihood loss $\text{NLL}(\Delta y,\sigma) = \frac{1}{2}\left(\frac{\Delta y^2}{\sigma^2} + \ln(2\pi\sigma^2)\right)$, which couples each head's squared error to the ensemble variance. This mechanism lets one SchNet pass produce both an energy estimate and an uncertainty estimate, with a computational cost the paper compares to adding an extra hidden layer.
What would settle it
Take the QM9 substitution experiments and build a reliability diagram: bin molecules by DPOSE variance and plot the mean absolute prediction error in each bin; if the highest-variance molecules such as CCl4 and SiCl4 do not also show the largest errors, or if variance and error rankings diverge across the full set, the variance-as-uncertainty interpretation would fail even though the plotted variance trends remain.
Extended reading notes
Core claim
On its own terms, the paper claims that DPOSE-equipped SchNet produces useful out-of-domain signals without the cost of deep ensembles. The variance across 64 output heads increases from 0.09 for CF4 to 62 for CCl4, from 0.09 for CH4 to 2.7 for SiH4, and from 0.00022 eV/atom for gold to 0.613 eV/atom for silver, while alkane uncertainties grow from 0.09 for methane to 0.66 for octadecane as chain length leaves the training range. In OC20, non-metal slabs show median variance near 0.03 while intermetallic slabs stay below 0.01. The authors conclude that DPOSE distinguishes in-domain from out-of-domain data when the two are far apart, but that structurally similar systems made of the same element, such as amorphous versus bulk gold, are not separated by variance.
Load-bearing premise
The load-bearing premise is that the variance among the 64 NLL-trained heads is a trustworthy stand-in for predictive error on out-of-domain inputs, which enters when the paper reads higher variance as lower confidence and is never checked against calibration.
Editorial extensions
If this is right
- A single SchNet model with 64 heads can emit an uncertainty flag for every energy prediction, so out-of-domain detection does not require training several full models.
- Compositional shifts, such as substituting an unseen element into a known molecule, produce variance jumps large enough to separate in-domain from out-of-domain molecules in QM9 and in gold-versus-silver comparisons.
- Uncertainty tracks physical extrapolation distance, rising as carbon-carbon bonds are stretched or compressed and as alkane or alcohol chains grow beyond the training lengths.
- The method's failure to separate amorphous from bulk gold implies that same-element structural diversity is not reliably flagged, so the practical alarm is for chemistry novelty, not morphology novelty.
- Because the added cost is comparable to one extra hidden layer, the approach is light enough to be coupled with active learning loops that acquire new training data where variance is high.
Reading between the lines
- Inference: the observed contrast suggests DPOSE variance encodes compositional novelty more strongly than geometric novelty; a testable extension is to correlate variance with element-embedding distance rather than with structural descriptors.
- Inference: since the 64 heads share all weights except the last layer, the ensemble diversity is limited to the final projection, so the variance may reflect last-layer sensitivity more than full epistemic uncertainty; comparing head counts (e.g., 8, 16, 64) would show how much of the signal comes from width alone.
- Inference: the paper does not calibrate variance against prediction error; a natural extension is to check whether the reported variance ordering matches the actual error ordering on the same out-of-domain molecules, which would turn a qualitative trend into a usable uncertainty score.
- Inference: for active learning, a DPOSE-variance acquisition function would likely prioritize novel elements and long molecules first, while missing amorphous phases of known elements; pairing variance with a latent-distance or energy-based score could cover that blind spot.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper integrates Direct Propagation of Shallow Ensembles (DPOSE) into the SchNet graph neural network by replacing the final layer with 64 output heads and training with the negative log-likelihood loss of Eq. (1). The resulting variance of the heads is used as an uncertainty estimate. The model is evaluated on three datasets: QM9 (equilibrium vs. non-equilibrium geometries, known vs. unknown elements, small vs. large molecules), OC20 (intermetallic vs. non-metal slabs, volume compression/expansion), and a Gold MD dataset (bulk vs. amorphous, Au vs. Ag). The reported results show large variance gaps for compositional shifts (e.g., CF4 0.09 vs. CCl4 62; Au 0.00022 vs. Ag 0.613 eV/atom), a gradual variance increase with molecular size, higher variance for non-metal slabs, and essentially no variance difference between bulk and amorphous gold. The authors conclude that DPOSE distinguishes in-domain from out-of-domain data when the domains are far apart, while acknowledging difficulty for structurally similar systems.
Significance. If head variance is a reliable proxy for predictive error, this would be a useful lightweight UQ method for GNN potentials, particularly for active learning. The study has strengths: it tests three diverse datasets, examines both compositional and structural shifts, and is transparent about a negative result (bulk vs. amorphous). However, the central claim of 'reliable uncertainty estimates' is not quantitatively validated, and the reported variance differences are aggregate and potentially scale-confounded. The contribution is an exploratory empirical study; its value depends on a calibration check and comparison with existing baselines.
major comments (4)
- [§2.2, §3 (general)] The central claim that DPOSE provides reliable uncertainty estimates requires demonstrating that the head variance from Eq. (1) is predictive of predictive error on individual out-of-domain samples. All reported evidence consists of aggregate variance differences between hand-picked groups (Tables 1–2, Figures 3–9); there is no calibration metric such as a reliability diagram, Spearman rank correlation between variance and absolute error, or an error-vs-variance plot for OOD samples. Without such a check, the large average gaps could be a feature-scale artifact rather than evidence of usable uncertainty.
- [§3.1.3, Table 2] The claim that uncertainty increases with molecular size is confounded by the extensivity of total energy. The QM9 variance values are reported without units or per-atom normalization, and the trend is non-monotonic (dodecane 0.62, tridecane 0.82, tetradecane 0.61). Since larger molecules have larger total energies, a naive variance of total energy may scale with system size even when per-atom uncertainty is constant. The authors should report normalized per-atom variances or otherwise show that the size trend is not a scale effect.
- [§3.3.1, Figure 6(c), Conclusion] The bulk-vs-amorphous result contradicts the stated expectation of lower uncertainty for in-domain bulk systems; the text acknowledges 'the model seems to be confident for out-of-domain systems too' and later that DPOSE 'struggled to distinguish between in-domain and out-of-domain configurations.' This is a central negative result for structural shifts. The abstract and conclusion currently over-generalize by saying DPOSE 'often demonstrates' or 'successfully distinguishes' without clearly carving out that the supported success is limited to compositional shifts and large domain distances. The paper should explicitly state in the abstract that structural OOD detection failed for Au bulk vs. amorphous.
- [§2, §3 (no baseline comparison)] The paper motivates DPOSE as a computationally efficient alternative to deep ensembles but never compares against any baseline UQ method, such as deep ensembles, MC dropout, or latent-distance uncertainty. Without a baseline, the strong wording about 'reliable' and 'better-calibrated' uncertainty is unsupported. At minimum, the authors should include one standard baseline with a calibration metric on the same OOD tasks.
minor comments (6)
- [§2, Eq. (1)] Equation (1) is presented without derivation or an explicit connection to the original DPOSE paper [19]; please cite the source in the text and define all symbols (e.g., whether sigma is per-output or shared).
- [§2.1] The name 'SchNET' is inconsistent with the standard 'SchNet' used elsewhere; correct the typo.
- [Tables 1–2, §3.1] The QM9 variance values are reported without units and without stating whether they are per atom or total. Please specify units (e.g., eV^2, (kcal/mol)^2) and clarify normalization.
- [Figure 5, caption] The caption refers to a 'red line' while the text and figure show a red dotted line; make the reference consistent.
- [§3.2, consistent terminology] The terms 'inter-metallic' and 'intermetallic' are used inconsistently (e.g., §3.2.1 'inter-metallic slabs' vs. §2.2.2 'intermetallic slab subset'); choose one spelling.
- [Figure 6(c), §3.3.1] The text says 'the mean variance was slightly higher for bulk systems' while the caption says 'Variance estimates showing slightly higher uncertainty for bulk'; clarify whether the box plot displays means, medians, or distributions.
Circularity Check
No significant circularity: the OOD uncertainty evaluations are external held-out tests, not constructions from the training objective.
full rationale
The paper's derivation chain is not circular. DPOSE's variance is produced by 64 output heads trained with the NLL loss (Eq. 1) on in-domain data, but every central OOD claim is evaluated on data excluded from that training: QM9 molecules with unseen elements (Si, S, P, Cl) or larger chains (C10-C18), OC20 non-metal slabs after fine-tuning on intermetallic slabs, and silver systems for an Au-only model. The reported variance gaps (e.g., CF4 0.09 vs CCl4 62; gold 0.00022 vs silver 0.613 eV/atom) are model outputs on held-out groups, not parameters fitted to produce those group differences. The paper even reports a contrary result for bulk vs amorphous gold ('the mean variance was slightly higher for bulk systems, contrary to the expectation'), which confirms the evaluation is not rigged by construction. The self-citations (refs 9, 16, 24 include Kitchin as coauthor) are background and data-source citations and are not load-bearing; the DPOSE method itself is attributed to Kellner and Ceriotti (ref 19), not to the authors. No uniqueness theorem or ansatz is imported from the authors' prior work to force the conclusion. The absence of an error-vs-variance calibration check is a scientific validation gap, not a circularity.
Assumptions & free parameters
free parameters (1)
- Number of ensemble heads =
64
assumptions (3)
- domain assumption The variance of the 64 head predictions, trained under NLL loss, is a valid measure of predictive uncertainty for out-of-domain inputs.
- domain assumption The chosen out-of-domain test sets (non-metal slabs, silver systems, larger molecules) represent meaningful distribution shifts.
- domain assumption DFT reference energies used for training and evaluation are accurate ground truths.
Cite this review
Pith. "Pith review of Uncertainty Quantification in Graph Neural Networks with Shallow Ensembles." pith.science (2026). https://pith.science/paper/GIVQPRCW
@misc{pith2026250412627,
author = {Pith},
title = {Pith review of: Uncertainty Quantification in Graph Neural Networks with Shallow Ensembles},
year = {2026},
howpublished = {\url{https://pith.science/paper/GIVQPRCW}},
note = {Machine review of arXiv:2504.12627}
}
read the original abstract
Machine-learned potentials (MLPs) have revolutionized materials discovery by providing accurate and efficient predictions of molecular and material properties. Graph Neural Networks (GNNs) have emerged as a state-of-the-art approach due to their ability to capture complex atomic interactions. However, GNNs often produce unreliable predictions when encountering out-of-domain data and it is difficult to identify when that happens. To address this challenge, we explore Uncertainty Quantification (UQ) techniques, focusing on Direct Propagation of Shallow Ensembles (DPOSE) as a computationally efficient alternative to deep ensembles. By integrating DPOSE into the SchNet model, we assess its ability to provide reliable uncertainty estimates across diverse Density Functional Theory datasets, including QM9, OC20, and Gold Molecular Dynamics. Our findings often demonstrate that DPOSE successfully distinguishes between in-domain and out-of-domain samples, exhibiting higher uncertainty for unobserved molecule and material classes. This work highlights the potential of lightweight UQ methods in improving the robustness of GNN-based materials modeling and lays the foundation for future integration with active learning strategies.
Figures
Figures from the paper (6 more)
Forward citations
Cited by 1 Pith paper
-
Geometric deep learning assists protein engineering. Opportunities and Challenges
A perspective synthesizing geometric deep learning applications in protein engineering and proposing an explainable, structure-aware design pipeline.
Reference graph
Works this paper leans on
-
[1]
Fang J, Xie M, He X, Zhang J, Hu J, Chen Y, Yang Y and Jin Q 2022 Materials Today Communications 33 104900 ISSN 2352-4928 URL https://www.sciencedirect.com/science/ article/pii/S235249282201741X
work page 2022
-
[2]
Merchant A, Batzner S, Schoenholz S S, Aykol M, Cheon G and Cubuk E D 2023 Nature 624 80–85 ISSN 1476-4687 publisher: Nature Publishing Group URL https://www.nature.com/ articles/s41586-023-06735-9
work page 2023
-
[3]
Xia W, Tang L, Sun H, Zhang C, Ho K M, Viswanathan G, Kovnir K and Wang C Z 2023 Journal of Materials Chemistry A 11 25973–25982 ISSN 2050-7496 publisher: The Royal Society of Chemistry URL https://pubs.rsc.org/en/content/articlelanding/2023/ta/d3ta03771a
work page 2023
-
[4]
Dragoni D, Daff T D, Cs´ anyi G and Marzari N 2018Physical Review Materials 2 013808 publisher: American Physical Society URL https://link.aps.org/doi/10.1103/PhysRevMaterials.2. 013808
-
[5]
Goeminne R, Vanduyfhuys L, Van Speybroeck V and Verstraelen T 2023 Journal of Chemical Theory and Computation 19 6313–6325 ISSN 1549-9618 publisher: American Chemical Society URL https://doi.org/10.1021/acs.jctc.3c00495
-
[6]
Schleder G R, Padilha A C M, Acosta C M, Costa M and Fazzio A 2019 Journal of Physics: Materials 2 032001 ISSN 2515-7639 publisher: IOP Publishing URL https://dx.doi.org/10. 1088/2515-7639/ab084b
work page 2019
-
[7]
Fung V, Zhang J, Juarez E and Sumpter B G 2021 npj Computational Materials 7 1–8 ISSN 2057-3960 publisher: Nature Publishing Group URL https://www.nature.com/articles/ Uncertainty Quantification in Graph Neural Networks with Shallow Ensembles 22 s41524-021-00554-0
work page 2021
-
[8]
Louis S Y, Zhao Y, Nasiri A, Wang X, Song Y, Liu F and Hu J 2020 Physical Chemistry Chemical Physics 22 18141–18148 ISSN 1463-9084 publisher: The Royal Society of Chemistry URL https://pubs.rsc.org/en/content/articlelanding/2020/cp/d0cp01474e
work page 2020
Show all 28 references
-
[9]
Sunshine E M, Shuaibi M, Ulissi Z W and Kitchin J R 2023 The Journal of Physical Chemistry C 127 23459–23466 ISSN 1932-7447 publisher: American Chemical Society URL https: //doi.org/10.1021/acs.jpcc.3c06157
2023 doi
-
[10]
Varivoda D, Dong R, Omee S S and Hu J 2022 Materials Property Prediction with Uncertainty Quantification: A Benchmark Study arXiv:2211.02235 [cond-mat] URL http://arxiv.org/ abs/2211.02235
2022 arXiv
-
[11]
Peterson A A, Christensen R and Khorshidi A 2017 Physical Chemistry Chemical Physics 19 10978–10985 ISSN 1463-9084 publisher: The Royal Society of Chemistry URL https: //pubs.rsc.org/en/content/articlelanding/2017/cp/c7cp00375g
2017
-
[12]
Singh A and Li Y 2021 Uncertainty Management and Reduction of Machine Learning Potential AIAA Scitech 2021 Forum AIAA SciTech Forum (American Institute of Aeronautics and Astronautics) URL https://arc.aiaa.org/doi/10.2514/6.2021-1962
2021 doi
-
[13]
Xin R, Siriwardane E M D, Song Y, Zhao Y, Louis S Y, Nasiri A and Hu J 2021 The Journal of Physical Chemistry C 125 16118–16128 ISSN 1932-7447 publisher: American Chemical Society URL https://doi.org/10.1021/acs.jpcc.1c02438
2021 doi
-
[14]
Egele R, Maulik R, Raghavan K, Lusch B, Guyon I and Balaprakash P 2022 AutoDEUQ: Automated Deep Ensemble with Uncertainty Quantification arXiv:2110.13511 [cs] URL http: //arxiv.org/abs/2110.13511
2022 arXiv
-
[15]
Rahaman R and Thiery A H 2021 Uncertainty Quantification and Deep Ensembles arXiv:2007.08792 [stat] URL http://arxiv.org/abs/2007.08792
2021 arXiv
-
[16]
Musielewicz J, Lan J, Uyttendaele M and Kitchin J R 2024 The Journal of Physical Chemistry C 128 20799–20810 ISSN 1932-7447 publisher: American Chemical Society URL https: //doi.org/10.1021/acs.jpcc.4c04972
2024 doi
-
[17]
Olivier A, Shields M D and Graham-Brady L 2021 Computer Methods in Applied Mechanics and Engineering 386 114079 ISSN 0045-7825 URL https://www.sciencedirect.com/science/ article/pii/S0045782521004102
2021
-
[18]
Gal Y and Ghahramani Z 2016 Dropout as a Bayesian Approximation: Representing Model Uncertainty Quantification in Graph Neural Networks with Shallow Ensembles 23 Uncertainty in Deep Learning arXiv:1506.02142 [stat] URL http://arxiv.org/abs/1506. 02142
2016 arXiv
-
[19]
Kellner M and Ceriotti M 2024 Machine Learning: Science and Technology 5 035006 ISSN 2632- 2153 publisher: IOP Publishing URL https://dx.doi.org/10.1088/2632-2153/ad594a
2024 doi
-
[20]
Sch¨ utt K T, Sauceda H E, Kindermans P J, Tkatchenko A and M¨ uller K R 2018The Journal of Chemical Physics 148 241722 ISSN 0021-9606 URL https://doi.org/10.1063/1.5019779
-
[21]
Chanussot L, Das A, Goyal S, Lavril T, Shuaibi M, Riviere M, Tran K, Heras-Domingo J, Ho C, Hu W, Palizhati A, Sriram A, Wood B, Yoon J, Parikh D, Zitnick C L and Ulissi Z 2021 ACS Catalysis 11 6059–6072 publisher: American Chemical Society URL https: //doi.org/10.1021/acscata...
2021 doi
-
[22]
Ramakrishnan R, Dral P O, Rupp M and von Lilienfeld O A 2014 Scientific Data 1 140022 ISSN 2052-4463 publisher: Nature Publishing Group URL https://www.nature.com/articles/ sdata201422
2014
-
[23]
Ruddigkeit L, van Deursen R, Blum L C and Reymond J L 2012 Journal of Chemical Information and Modeling 52 2864–2875 ISSN 1549-9596 publisher: American Chemical Society URL https://doi.org/10.1021/ci300415d
2012 doi
-
[24]
wiley.com/doi/abs/10.1002/qua.25115
Boes J R, Groenenboom M C, Keith J A and Kitchin J R 2016 Interna- tional Journal of Quantum Chemistry 116 979–987 ISSN 1097-461X eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1002/qua.25115 URL https://onlinelibrary. wiley.com/doi/abs/10.1002/qua.25115
2016 doi
-
[25]
Gasteiger J, Giri S, Margraf J T and G¨ unnemann S 2022 Fast and Uncertainty-Aware Directional Message Passing for Non-Equilibrium Molecules arXiv:2011.14115 [cs] URLhttp://arxiv.org/ abs/2011.14115
2022 arXiv
-
[26]
Gasteiger J, Groß J and G¨ unnemann S 2022 Directional Message Passing for Molecular Graphs arXiv:2003.03123 [cs] URL http://arxiv.org/abs/2003.03123
2022 arXiv
-
[27]
Liao Y L and Smidt T 2023 Equiformer: Equivariant Graph Attention Transformer for 3D Atomistic Graphs arXiv:2206.11990 [cs] URL http://arxiv.org/abs/2206.11990
2023 arXiv
-
[28]
Liao Y L, Wood B, Das A and Smidt T 2024 EquiformerV2: Improved Equivariant Transformer for Scaling to Higher-Degree Representations arXiv:2306.12059 [cs] URL http://arxiv.org/ abs/2306.12059
2024 arXiv
Reviewed August 16, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.