REVIEW 4 major objections 5 minor 27 references
GNN's Uncertainty Quantification using Self-Distillation
T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read A single GNN, trained by self-distillation, can output uncertainty estimates that are cheaper than ensemble methods and more precise than the standard disagreement metric.
desk verdict A plausible efficiency win for GNN UQ via self-distillation, but the precision claim rests on an untested teacher-as-oracle assumption. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The machinery is the self-distillation training objective paired with a depth-weighted JSD disagreement metric. Self-distillation here means a GNN with one classifier after each hidden layer; the deepest classifier is the teacher, and the training loss averages over layers $(1 - \alpha)$ cross-entropy with the true label plus $\alpha$ KL divergence between the student's soft outputs and the teacher's soft outputs, together with a weighted $\ell^2$ penalty on the difference between shallower-layer features and teacher features ($\alpha$ and $\lambda$ are set to zero for the teacher). The uncertainty metric compares each student's soft output to the teacher's via JSD, multiplies by a monotone depth weight, and normalizes by the theoretical upper bound, so values are comparable across networks with different numbers of layers. The weight function is the load-bearing object: it converts a depth-only distance into a disagreement signal that fires only when a classifier's predicted class conflicts with the teacher's.
What would settle it
Take a test point whose true label differs from the teacher's most confident prediction, and on which all students (having mimicked the teacher) agree with the teacher; the proposed normalized metric returns a value near zero, while a correctness-aware measure such as true-class probability or disagreement among independently trained models would show high uncertainty.
Extended reading notes
Core claim
The central discovery is that the diversity inside a single self-distilled network is usable uncertainty information that the standard disagreement metric throws away. The proposed score is $\mathrm{UC} = \sum_{l=1}^{m} W(l) \cdot \mathrm{JSD}(P_l \| P_{\mathrm{teacher}})$, where $m$ is the number of classifiers, the deepest classifier (the teacher) defines the reference distribution, and $W(l) = 1$ when the $l$-th student's predicted label matches the teacher's, rising with depth toward $2$ when it does not. Because the weight depends on label mismatch, the metric assigns different uncertainties to two cases with identical KL disagreement: two MIMIC-IV patients with the same disagreement value ($0.1082$) receive normalized uncertainties of $0.0099$ and $0.0211$. That separation is what the authors mean by calling the metric more precise, and it is what allows a single self-distilled network to flag hard or out-of-distribution examples.
Load-bearing premise
The deepest classifier (the teacher) is treated as the correct reference: its predicted label is the outcome in the uncertainty formula and every student is trained to imitate it, so a confidently wrong teacher makes the score low even when the model is actually uncertain.
Editorial extensions
If this is right
- One training pass produces both the classifier and its uncertainty estimate, with training time near that of a single GNN (about 20 s vs 93 s single and 881 s ensemble on MIMIC-IV).
- The weighted metric separates cases the disagreement metric flattens: two MIMIC-IV patients with identical disagreement (0.1082) receive different normalized uncertainties when shallower classifiers contradict the teacher's label.
- On Enzymes, self-distillation shows lower entropy on in-distribution data and higher entropy on out-of-distribution data, indicating usable OOD distinction.
- F1 and ROC AUC are comparable to MC Dropout and deep ensembles (MIMIC-IV F1 0.88 vs 0.85 and 0.84) while the parameter count stays close to a single model.
- The same recipe applies to any GNN classification task, graph or node level, without per-task modification.
Reading between the lines
- Because the teacher's label defines the outcome, the metric measures internal consistency rather than correctness; a natural extension is to add a second reference — e.g., the students' majority vote or an independently trained ensemble — to catch teacher overconfidence.
- The linear and exponential weight functions give different conservativeness; one could test on a larger clinical graph whether the nonlinear variant improves OOD-detection AUROC, since the authors leave the choice as a domain parameter.
- The normalized score is bounded and depth-comparable, so it could directly feed a decision threshold or a conformal-style guarantee for clinical alerting, though the paper does not explore that link.
- Since students are trained to copy the teacher, the metric's sensitivity is capped by how much diversity the soft labels and feature penalties allow; adding a small diversity term to the loss would test whether sensitivity can be increased without losing the efficiency advantage.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a self-distillation framework for uncertainty quantification in GNN graph classification. A multi-classifier GNN is trained so that shallower classifiers mimic the deepest classifier (teacher) via a combined cross-entropy, KL-divergence, and feature-penalty loss. Uncertainty is then measured as a depth-weighted sum of Jensen-Shannon divergences between each student classifier and the teacher, normalized by an upper bound. The method is evaluated on MIMIC-IV and Enzymes datasets, comparing accuracy, training/inference time, calibration, OOD separation, and two illustrative patient examples. The main claims are that the method is more efficient than MC Dropout and ensembles, and that the proposed uncertainty metric is more precise than the standard disagreement metric.
Significance. If fully supported, the paper would offer a practical efficiency gain: multiple GNN classifiers trained in one pass, with uncertainty estimates comparable to more expensive MC Dropout and ensemble baselines. The paper also ships public code and uses standard 5-fold cross-validation, which aids reproducibility. However, the central precision claim is currently supported only by two hand-picked examples, and the OOD claim is tested with entropy rather than the proposed uncertainty metric. The efficiency comparison has an unexplained anomaly on MIMIC-IV. The core training pipeline is plausible and the efficiency story is mostly convincing for Enzymes, but the load-bearing evidence for the new metric's advantages is incomplete.
major comments (4)
- [Section 4.2, Figure 2] The OOD detection evaluation uses predictive entropy, not the proposed uncertainty metric UC from Equations 11 and 14. Since the paper's contribution is the weighted UC metric, the abstract's claim about distinguishing OOD data is not tested for that metric. The authors should report OOD separation using UC (e.g., AUROC for ID vs OOD based on UC scores) on at least one dataset.
- [Section 3.3, Eq. 9 and Finding 3, Table 4] The precision claim is demonstrated on only two manually selected patients, and the result is partly a restatement of the definition: the weight function in Eq. 9 is activated exactly when a student's predicted class differs from the teacher's, so any such patient will have a higher UC by construction. The example shows consistency with Eq. 9, but it does not provide independent evidence that UC tracks predictive uncertainty. A quantitative evaluation over the full test set, such as rank correlation between UC and misclassification or AUROC for detecting incorrect teacher predictions, is needed to support Finding 3.
- [Section 3.3, 'We use the teacher's prediction as the outcome'] The method assumes the deepest classifier is a reliable reference. If the teacher is confidently wrong on a test point, the shallower classifiers are trained to mimic it, so UC will be near zero precisely when uncertainty should be high. The paper does not evaluate cases with teacher errors, nor does it examine the distribution of UC on teacher-misclassified examples. Since the clinical setting is a stated motivation, this is a load-bearing limitation that should be addressed with explicit experiments.
- [Table 2, MIMIC-IV row] The reported self-distillation training time is 20.04 seconds versus 93.10 seconds for a single model, despite the self-distillation network containing additional classifiers. This is implausible under comparable training budgets and makes the efficiency comparison hard to interpret. Please report the number of epochs, stopping criteria, and whether the times are per epoch or total, and clarify how the single-model baseline was trained.
minor comments (5)
- [Table 2, Enzymes row] The MCE value for self-distillation is reported as '0.42±100', which is presumably a typo; it should be a plausible standard deviation such as 0.10.
- [Eq. 13 and Table 4] The normalization in Eq. 13 does not specify whether W(l) is the maximum weight over disagreement patterns or the actual weight. In Table 4, the same UC_max appears to be used for both patients even though Patient 1 has no class mismatches; please clarify the definition of the upper bound.
- [Section 4.2, paragraph after Table 4] The sentence 'Patient 2's misclassification, unlike Patient 1's correct classification by the shallower classifiers' is confusing; it should state explicitly which classifier is misclassifying each patient.
- [Figure 2] The axis label '0.5 -0.1 0.7 1.3 2.0' appears to have formatting issues; please correct the tick labels.
- [Section 4.1, Training and Hardware Specifications] The description 'for the last 20 epochs' depends on the total number of epochs, which is never reported; please include the full training schedule.
Circularity Check
Finding 3's 'higher precision' claim is definitionally encoded in the uncertainty metric's weight function, so the Table 4 demonstration restates Eq. (9) rather than validating it.
-
self definitional
[Section 3.3, Eq. (9); Section 4.2, Finding 3 and Tables 3-4]
"To capture the impact of network depth diversity in such disagreements, we propose a weight function... If a student's prediction matches the outcome, its weight is 1; otherwise, the weight increases with its depth. We use the teacher's prediction as the outcome since all shallower classifiers mimic the teacher model. ... The weight of a classifier at layer l can be computed by W_lin(l) = 1 + L−D(l)/L 1{y_l≠y_teacher}, (9) ..."
The uncertainty metric in Eq. (11) is a weighted sum of JSD(P_l||P_teacher), and the only sample-dependent part of the weight in Eq. (9) is the indicator 1{y_l != y_teacher}. Therefore any sample in which a shallower classifier's predicted class differs from the teacher's automatically receives a higher UC than a sample with identical divergences but unanimous labels. Finding 3 asserts that the metric 'distinguishes scenarios with discrepancies between the teacher classifier's predicted label and shallower classifiers'—that is exactly the condition hard-coded into W. The Table 4 comparison is an illustration of the definition, not an independent empirical test; no fitted parameter or external benchmark is needed to reproduce the larger UC for Patient 2.
full rationale
Finding 3 is not an independent empirical result: Eq. (9) defines the per-layer weight as 1 plus a depth term times the indicator that the student's predicted class differs from the teacher's, so any sample with a class mismatch is guaranteed to receive a larger UC in Eq. (11) than it would under the unweighted disagreement metric. Tables 3-4 select two MIMIC-IV patients with identical unweighted disagreement and opposite mismatch status; the larger UC for Patient 2 follows from the definition of W, not from measured association with predictive error. The rest of the paper—training-time and inference-time efficiency, parameter count, and F1/ROC-AUC parity with MC Dropout and ensembles—is self-contained, externally evaluated on MIMIC-IV and Enzymes, and does not depend on the contested precision claim. There is no load-bearing self-citation: the distillation loss is attributed to the external reference [24], and the authors' own prior work [3,4] is used only for tasks, data construction, and architecture choices. The teacher-as-oracle assumption is a real correctness and calibration risk, but it is an assumption about what the metric measures, not a circular derivation; it is therefore noted but not scored as circularity. Overall, the central precision claim reduces by construction while the efficiency and performance claims remain independent, giving partial circularity.
Assumptions & free parameters
free parameters (3)
- alpha_l (imitation parameter) =
0.6
- lambda_l (feature penalty trade-off) =
0.04
- Weight function form =
W_lin or W_nonlin (Eqs. 9-10)
assumptions (5)
- domain assumption The deepest classifier (teacher) provides the correct reference for measuring disagreement.
- domain assumption Deeper classifiers use richer representations, so disagreement of deeper students with the teacher is more informative.
- domain assumption The self-distillation loss of [24] transfers knowledge effectively for GNNs.
- domain assumption MIMIC-IV patient graphs built as in [4] are a valid modeling of readmission prediction.
- standard math Jensen-Shannon divergence is bounded between 0 and log_b(2) for discrete distributions.
Cite this review
Pith. "Pith review of GNN's Uncertainty Quantification using Self-Distillation." pith.science (2026). https://pith.science/paper/YCLGK6XT
@misc{pith2026250620046,
author = {Pith},
title = {Pith review of: GNN's Uncertainty Quantification using Self-Distillation},
year = {2026},
howpublished = {\url{https://pith.science/paper/YCLGK6XT}},
note = {Machine review of arXiv:2506.20046}
}
read the original abstract
Graph Neural Networks (GNNs) have shown remarkable performance in the healthcare domain. However, what remained challenging is quantifying the predictive uncertainty of GNNs, which is an important aspect of trustworthiness in clinical settings. While Bayesian and ensemble methods can be used to quantify uncertainty, they are computationally expensive. Additionally, the disagreement metric used by ensemble methods to compute uncertainty cannot capture the diversity of models in an ensemble network. In this paper, we propose a novel method, based on knowledge distillation, to quantify GNNs' uncertainty more efficiently and with higher precision. We apply self-distillation, where the same network serves as both the teacher and student models, thereby avoiding the need to train several networks independently. To ensure the impact of self-distillation, we develop an uncertainty metric that captures the diverse nature of the network by assigning different weights to each GNN classifier. We experimentally evaluate the precision, performance, and ability of our approach in distinguishing out-of-distribution data on two graph datasets: MIMIC-IV and Enzymes. The evaluation results demonstrate that the proposed method can effectively capture the predictive uncertainty of the model while having performance similar to that of the MC Dropout and ensemble methods. The code is publicly available at https://github.com/tailabTMU/UQ_GNN.
Figures
Reference graph
Works this paper leans on
-
[24]
IEEE Transactions on Pattern Analysis and Machine Intelligence44(8), 4388–4403 (2022)
Zhang, L., Bao, C., Ma, K.: Self-distillation: Towards efficient and compact neural networks. IEEE Transactions on Pattern Analysis and Machine Intelligence44(8), 4388–4403 (2022). https://doi.org/10.1109/TPAMI.2021.3067100
-
[1]
AIChE Journal68(6), e17696 (2022)
Aouichaoui, A.R.N., Mansouri, S.S., Abildskov, J., Sin, G.: Uncertainty estimation in deep learning-based property models: Graph neural networks applied to the critical properties. AIChE Journal68(6), e17696 (2022). https://doi.org/https: //doi.org/10.1002/aic.17696
-
[2]
Chen, Y., Bian, Y., Xiao, X., Rong, Y., Xu, T., Huang, J.: On self-distilling graph neural network (2021), https://arxiv.org/abs/2011.02255
arXiv 2021
-
[3]
Artificial Intelligence for Medicine pp
Daneshvar, H., Boursalie, O., Samavi, R., Doyle, T.E., Duncan, L., Pires, P., Sassi, R.: Sok: Application of machine learning models in child and youth mental health decision-making. Artificial Intelligence for Medicine pp. 113–132 (2024). https:// doi.org/10.1016/B978-0-443-13671-9.00003-X
-
[4]
Proceedings of the Canadian Conference on Artificial Intelligence (may 27 2022)
Daneshvar, H., Samavi, R.: Heterogeneous Patient Graph Embedding in Readmis- sion Prediction. Proceedings of the Canadian Conference on Artificial Intelligence (may 27 2022). https://doi.org/10.21428/594757db.869abbde
-
[5]
Journal of Machine Learning Research24(43), 1–48 (2023), http://jmlr.org/papers/v24/22-0567.html
Dwivedi, V.P., Joshi, C.K., Luu, A.T., Laurent, T., Bengio, Y., Bresson, X.: Bench- marking graph neural networks. Journal of Machine Learning Research24(43), 1–48 (2023), http://jmlr.org/papers/v24/22-0567.html
work page 2023
-
[6]
Gheshlaghi, S.H., Soltani, N.Y., Ganji, M.: Uncertainty estimation for out-of- distribution detection of whole slide images. In: ICASSP 2025 - 2025 IEEE Interna- tional Conference on Acoustics, Speech and Signal Processing (ICASSP). pp. 1–5 (2025). https://doi.org/10.1109/ICASSP49660.2025.10889349
arXiv 2025
-
[7]
In- ternational Journal of Computer Vision129(6), 1789–1819 (Jun 2021)
Gou, J., Yu, B., Maybank, S.J., Tao, D.: Knowledge distillation: A survey. In- ternational Journal of Computer Vision129(6), 1789–1819 (Jun 2021). https: //doi.org/10.1007/s11263-021-01453-z
Show all 27 references
-
[8]
Zargarbashi, S., Antonelli, S., Bojchevski, A.: Conformal prediction sets for graph neural networks
H. Zargarbashi, S., Antonelli, S., Bojchevski, A.: Conformal prediction sets for graph neural networks. In: Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., Scarlett, J. (eds.) Proceedings of the 40th International Conference on 14 H. Daneshvar & R. Samavi Machi...
2023
-
[9]
In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vish- wanathan, S., Garnett, R
Hamilton, W., Ying, Z., Leskovec, J.: Inductive representation learning on large graphs. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vish- wanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. vol. 30. Curran Associates, Inc...
2017
-
[10]
Hinton, G., Vinyals, O., Dean, J.: Distilling the knowledge in a neural network (2015), https://arxiv.org/abs/1503.02531
2015 arXiv
-
[11]
In: Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., Levine, S
Huang, K., Jin, Y., Candes, E., Leskovec, J.: Uncertainty quantification over graph with conformalized graph neural networks. In: Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., Levine, S. (eds.) Advances in Neu- ral Information Processing Systems. vol. 36, pp. 2669...
2023
-
[12]
Johnson, A., Bulgarelli, L., Pollard, T., Gow, B., Moody, B., Horng, S., Celi, L.A., Mark, R.: MIMIC-IV (2024)
2024
-
[13]
In: Proceedings of the AAAI Symposium Series
Karimi, H., Samavi, R.: Quantifying deep learning model uncertainty in conformal prediction. In: Proceedings of the AAAI Symposium Series. vol. 1, pp. 142–148 (2023)
2023
-
[14]
In: Proceedings of the Thirteenth Symposium on Conformal and Probabilistic Prediction with Applications, PMLR
Karimi, H., Samavi, R.: Evidential uncertainty sets in deep classifiers using confor- mal prediction. In: Proceedings of the Thirteenth Symposium on Conformal and Probabilistic Prediction with Applications, PMLR. vol. 230, pp. 466–489 (2024)
2024
-
[15]
Journal of Cheminformatics14(1), 2 (1 2022)
Kwon, Y., Lee, D., Choi, Y.S., Kang, S.: Uncertainty-aware prediction of chemical reaction yields with graph neural networks. Journal of Cheminformatics14(1), 2 (1 2022). https://doi.org/10.1186/s13321-021-00579-z
2022 doi
-
[16]
In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R
Lakshminarayanan, B., Pritzel, A., Blundell, C.: Simple and scalable predic- tive uncertainty estimation using deep ensembles. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Syst...
2017
-
[17]
In: 2020 IEEE International Conference on Big Data (Big Data)
Liu, Z., Li, X., Peng, H., He, L., Philip, S.Y.: Heterogeneous Similarity Graph Neu- ral Network on Electronic Health Records. In: 2020 IEEE International Conference on Big Data (Big Data). pp. 1196–1205. IEEE (2020)
2020
-
[18]
In: ICML 2020 Workshop on Graph Representation Learning and Beyond (GRL+ 2020) (2020), www.graphlearning.io
Morris, C., Kriege, N.M., Bause, F., Kersting, K., Mutzel, P., Neumann, M.: Tu- dataset: A collection of benchmark datasets for learning with graphs. In: ICML 2020 Workshop on Graph Representation Learning and Beyond (GRL+ 2020) (2020), www.graphlearning.io
2020
-
[19]
Morris, C., Ritzert, M., Fey, M., Hamilton, W.L., Lenssen, J.E., Rattan, G., Grohe, M.: Weisfeiler and Leman Go Neural: Higher-order Graph Neural Networks (2021)
2021
-
[20]
In: Chaudhuri, K., Salakhutdinov, R
Phuong, M., Lampert, C.: Towards understanding knowledge distillation. In: Chaudhuri, K., Salakhutdinov, R. (eds.) Proceedings of the 36th International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 97, pp. 5142–5151. PMLR (09–15 Jun 2019), htt...
2019
-
[21]
Shang, J., Ma, T., Xiao, C., Sun, J.: Pre-training of Graph Augmented Transform- ers for Medication Recommendation (2019)
2019
-
[22]
Vovk, V., Gammerman, A., Shafer, G.: Algorithmic learning in a random world, vol. 29. Springer (2005) GNN’s Uncertainty Quantification using Self-Distillation 15
2005
-
[23]
Wang, F., Liu, Y., Liu, K., Wang, Y., Medya, S., Yu, P.S.: Uncertainty in graph neural networks: A survey (2024), https://arxiv.org/abs/2403.07185
2024 arXiv
-
[25]
In: Proceedings of the 2020 ACM SIGMOD International Conference on Management of Data
Zhang, W., Miao, X., Shao, Y., Jiang, J., Chen, L., Ruas, O., Cui, B.: Reliable data distillation on graph convolutional network. In: Proceedings of the 2020 ACM SIGMOD International Conference on Management of Data. p. 1399–1414. SIG- MOD ’20, Association for Computing Machin...
2020
-
[26]
Zhang, Y., Lee, A.A.: Bayesian semi-supervised learning for uncertainty-calibrated prediction of molecular properties and active learning. Chem. Sci.10, 8154–8163 (2019). https://doi.org/10.1039/C9SC00616H
2019 doi
-
[27]
In: Proceedings of the Conference on Health, Inference, and Learning
Zhu, W., Razavian, N.: Variationally Regularized Graph-Based Representation Learning for Electronic Health Records. In: Proceedings of the Conference on Health, Inference, and Learning. pp. 1–13. CHIL ’21, Association for Computing Machinery,NewYork,NY,USA(2021).https://doi.or...
2021
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.