{"id":"a5d53747-9622-4e3b-8e4c-14972a8ed256","arxiv_id":"2506.20046","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A self-distilled multi-classifier GNN with a depth-weighted JSD disagreement metric quantifies predictive uncertainty efficiently.","lead":"This paper proposes a self-distillation method for quantifying uncertainty in graph neural network predictions, using a weighted disagreement metric among classifiers at different depths. It reports comparable accuracy to ensembles and MC dropout at lower training cost, but its precision and out-of-distribution claims are only partially supported.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The proposed metric treats the deepest classifier as ground truth; if the teacher is wrong, low uncertainty is reported, yet the paper never tests correlation between UC and actual misclassification.","rationale":"The reader's weakest-assumption analysis identified the same core issue: using the teacher's prediction as the outcome in Eq. 8 assumes the deepest classifier is a correct reference. I agree that this is the most load-bearing concern because the paper's novelty and 'higher precision' claim hang entirely on the meaningfulness of the UC metric. The efficiency results, the normalization in Eq. 13, and the training-time inversion are relevant but secondary; they can be corrected or clarified without changing the conceptual foundation. The teacher-as-reference issue is more fundamental: if the teacher is wrong or overconfident, the metric produces low uncertainty for the worst errors, which is exactly the opposite of what clinical UQ should do. The paper provides no quantitative evaluation of UC against teacher misclassification, only two hand-picked cases with correct teacher predictions and an OOD analysis based on entropy. Thus the central precision claim is currently unsupported. The concern is addressable with a simple AUROC analysis, so the conditional verdict is appropriate; no change to the reader's verdict is needed.","tokens_in":10942,"tokens_out":6011,"duration_ms":70450,"concrete_test":"On the held-out test folds of MIMIC-IV and Enzymes, compute the proposed UC (Eq. 11) for every sample, then measure the AUROC of UC for detecting samples where the teacher's predicted label is incorrect versus correct. Compare with the AUROC of MC-dropout entropy and ensemble disagreement on the same task. If UC does not rank teacher errors among the highest-uncertainty samples (e.g., AUROC not significantly above 0.5, or below the baselines), the precision claim is unsupported. Also report the teacher's accuracy relative to each shallower classifier; if the teacher is not consistently the most accurate on ID data, the choice of teacher as reference is unjustified.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The method's uncertainty metric (Eq. 11) is a depth-weighted sum of JSD divergences from the teacher's probability vector, and the weights (Eq. 9) are activated only when a student's predicted class differs from the teacher's. Since all shallower classifiers are trained to minimize KL divergence to the teacher (Eq. 3), their predictions are engineered to mimic the teacher; disagreement with the teacher is thus an artifact of imperfect distillation, not necessarily a signal of predictive uncertainty. If the teacher is confidently wrong on a sample, all students will tend to agree with it and UC will be near zero, exactly when uncertainty should be high. The paper's only demonstration of 'precision' (Table 4) uses two hand-picked examples where the reference classifier is correct; it never evaluates cases with teacher error, and the OOD experiments (Fig. 2) use standard predictive entropy, not the proposed UC. The central claim of 'higher precision' UQ therefore rests on the untested assumption that the deepest classifier is a reliable oracle. This is load-bearing because if it fails, the metric suppresses uncertainty reporting for the most dangerous failures in a clinical setting.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a self-distillation framework for uncertainty quantification in GNN graph classification. A multi-classifier GNN is trained so that shallower classifiers mimic the deepest classifier (teacher) via a combined cross-entropy, KL-divergence, and feature-penalty loss. Uncertainty is then measured as a depth-weighted sum of Jensen-Shannon divergences between each student classifier and the teacher, normalized by an upper bound. The method is evaluated on MIMIC-IV and Enzymes datasets, comparing accuracy, training/inference time, calibration, OOD separation, and two illustrative patient examples. The main claims are that the method is more efficient than MC Dropout and ensembles, and that the proposed uncertainty metric is more precise than the standard disagreement metric.","tokens_in":11148,"tokens_out":6052,"duration_ms":62914,"significance":"If fully supported, the paper would offer a practical efficiency gain: multiple GNN classifiers trained in one pass, with uncertainty estimates comparable to more expensive MC Dropout and ensemble baselines. The paper also ships public code and uses standard 5-fold cross-validation, which aids reproducibility. However, the central precision claim is currently supported only by two hand-picked examples, and the OOD claim is tested with entropy rather than the proposed uncertainty metric. The efficiency comparison has an unexplained anomaly on MIMIC-IV. The core training pipeline is plausible and the efficiency story is mostly convincing for Enzymes, but the load-bearing evidence for the new metric's advantages is incomplete.","major_comments":[{"comment":"The OOD detection evaluation uses predictive entropy, not the proposed uncertainty metric UC from Equations 11 and 14. Since the paper's contribution is the weighted UC metric, the abstract's claim about distinguishing OOD data is not tested for that metric. The authors should report OOD separation using UC (e.g., AUROC for ID vs OOD based on UC scores) on at least one dataset.","section":"Section 4.2, Figure 2"},{"comment":"The precision claim is demonstrated on only two manually selected patients, and the result is partly a restatement of the definition: the weight function in Eq. 9 is activated exactly when a student's predicted class differs from the teacher's, so any such patient will have a higher UC by construction. The example shows consistency with Eq. 9, but it does not provide independent evidence that UC tracks predictive uncertainty. A quantitative evaluation over the full test set, such as rank correlation between UC and misclassification or AUROC for detecting incorrect teacher predictions, is needed to support Finding 3.","section":"Section 3.3, Eq. 9 and Finding 3, Table 4"},{"comment":"The method assumes the deepest classifier is a reliable reference. If the teacher is confidently wrong on a test point, the shallower classifiers are trained to mimic it, so UC will be near zero precisely when uncertainty should be high. The paper does not evaluate cases with teacher errors, nor does it examine the distribution of UC on teacher-misclassified examples. Since the clinical setting is a stated motivation, this is a load-bearing limitation that should be addressed with explicit experiments.","section":"Section 3.3, 'We use the teacher's prediction as the outcome'"},{"comment":"The reported self-distillation training time is 20.04 seconds versus 93.10 seconds for a single model, despite the self-distillation network containing additional classifiers. This is implausible under comparable training budgets and makes the efficiency comparison hard to interpret. Please report the number of epochs, stopping criteria, and whether the times are per epoch or total, and clarify how the single-model baseline was trained.","section":"Table 2, MIMIC-IV row"}],"minor_comments":[{"comment":"The MCE value for self-distillation is reported as '0.42±100', which is presumably a typo; it should be a plausible standard deviation such as 0.10.","section":"Table 2, Enzymes row"},{"comment":"The normalization in Eq. 13 does not specify whether W(l) is the maximum weight over disagreement patterns or the actual weight. In Table 4, the same UC_max appears to be used for both patients even though Patient 1 has no class mismatches; please clarify the definition of the upper bound.","section":"Eq. 13 and Table 4"},{"comment":"The sentence 'Patient 2's misclassification, unlike Patient 1's correct classification by the shallower classifiers' is confusing; it should state explicitly which classifier is misclassifying each patient.","section":"Section 4.2, paragraph after Table 4"},{"comment":"The axis label '0.5 -0.1 0.7 1.3 2.0' appears to have formatting issues; please correct the tick labels.","section":"Figure 2"},{"comment":"The description 'for the last 20 epochs' depends on the total number of epochs, which is never reported; please include the full training schedule.","section":"Section 4.1, Training and Hardware Specifications"}],"recommendation":"major_revision","confidential_remarks":"The manuscript fits the scope of AIiH and the self-distillation pipeline is plausible. My main concern is that the headline claims about precision and OOD detection are under-tested, and one efficiency number is internally inconsistent. These issues are addressable with additional experiments and clarifications, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this applies self-distillation to GNN uncertainty quantification and defines a depth-weighted disagreement metric. The efficiency result is genuine and clearly reported. The precision claim is not yet validated. Send it to referees, but expect a serious revision.\n\nWhat's new: the loss is from Zhang et al. [24], but the weighted JSD uncertainty score (Eq. 11) with class-mismatch depth weights is new, as is the application to graph classification. Table 2 shows real gains: on MIMIC-IV, self-distillation trains in about 20 seconds versus 880 for an ensemble, with similar F1/ROC and comparable calibration error. That is a practical win for clinical or edge settings. Code is public.\n\nWhere it gets soft. The paper claims higher-precision UQ but only shows two hand-picked patients where the teacher is correct. Since students are trained to mimic the teacher, the disagreement signal is partly an artifact of imperfect distillation. The metric treats the teacher as an oracle; if the teacher is confidently wrong, students agree and UC stays near zero. The paper never tests correlation between UC and actual misclassification, which is a load-bearing gap for the precision claim. To their credit, the authors acknowledge the need for more ablation on α, λ, and the weight function.\n\nSecond, the OOD experiments (Figure 2) use predictive entropy, not the proposed UC. The abstract claims OOD detection with the method, but that evidence is not present. Third, the normalization in Eq. 13 is not well-defined because W(l) depends on the indicator function; the upper bound should use maximum weights, not sample-dependent ones. That is a small but fixable math issue. Fourth, the MC Dropout training time on MIMIC-IV is 319 seconds versus 93 for a single model; MC Dropout should not change training cost. Probably a typo, but the table as printed undermines the efficiency comparison.\n\nWho this is for: researchers wanting a cheap UQ option for GNNs, especially in healthcare. The method is simple to reimplement and the code helps. The efficiency story alone justifies peer review. The UQ validation needs to be redone: test UC on misclassified samples, run OOD with UC, and fix the normalization and training-time issue. I would accept for review, with the expectation of substantial revision.","headline":"A plausible efficiency win for GNN UQ via self-distillation, but the precision claim rests on an untested teacher-as-oracle assumption.","tokens_in":11674,"tokens_out":4036,"would_cite":false,"duration_ms":43819,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A single GNN, trained by self-distillation, can output uncertainty estimates that are cheaper than ensemble methods and more precise than the standard disagreement metric.","keywords":["graph neural networks","uncertainty quantification","self-distillation","knowledge distillation","deep ensembles","Jensen-Shannon divergence","out-of-distribution detection","healthcare AI"],"falsifier":"Take a test point whose true label differs from the teacher's most confident prediction, and on which all students (having mimicked the teacher) agree with the teacher; the proposed normalized metric returns a value near zero, while a correctness-aware measure such as true-class probability or disagreement among independently trained models would show high uncertainty.","tokens_in":10733,"feed_emoji":"🩺","tokens_out":9919,"duration_ms":93527,"temperature":0.7,"pith_summary":"The paper tries to establish that predictive uncertainty for graph neural networks (GNNs) can be quantified both accurately and cheaply through self-distillation: a single network with a classifier after every hidden layer, where the deepest classifier plays the teacher and the shallower ones are trained to imitate it. Its central claim is that a depth-weighted disagreement metric — Jensen–Shannon divergence between each student and the teacher, weighted more heavily when a deeper student's predicted label differs from the teacher's — separates cases that the standard disagreement metric treats as identical. If correct, this gives clinicians a trustworthy uncertainty signal (for example, in 30-day readmission prediction) without training several independent networks or running many dropout samples. Experiments on MIMIC-IV and Enzymes show performance comparable to MC Dropout and deep ensembles while keeping training and inference time close to that of a single GNN.","feed_headline":"Self-distillation refines GNN uncertainty in a single pass","feed_subtitle":"Depth-weighted disagreement tells hard cases apart, matching ensemble accuracy at single-model cost.","key_machinery":"The machinery is the self-distillation training objective paired with a depth-weighted JSD disagreement metric. Self-distillation here means a GNN with one classifier after each hidden layer; the deepest classifier is the teacher, and the training loss averages over layers $(1 - \\alpha)$ cross-entropy with the true label plus $\\alpha$ KL divergence between the student's soft outputs and the teacher's soft outputs, together with a weighted $\\ell^2$ penalty on the difference between shallower-layer features and teacher features ($\\alpha$ and $\\lambda$ are set to zero for the teacher). The uncertainty metric compares each student's soft output to the teacher's via JSD, multiplies by a monotone depth weight, and normalizes by the theoretical upper bound, so values are comparable across networks with different numbers of layers. The weight function is the load-bearing object: it converts a depth-only distance into a disagreement signal that fires only when a classifier's predicted class conflicts with the teacher's.","core_discovery":"The central discovery is that the diversity inside a single self-distilled network is usable uncertainty information that the standard disagreement metric throws away. The proposed score is $\\mathrm{UC} = \\sum_{l=1}^{m} W(l) \\cdot \\mathrm{JSD}(P_l \\| P_{\\mathrm{teacher}})$, where $m$ is the number of classifiers, the deepest classifier (the teacher) defines the reference distribution, and $W(l) = 1$ when the $l$-th student's predicted label matches the teacher's, rising with depth toward $2$ when it does not. Because the weight depends on label mismatch, the metric assigns different uncertainties to two cases with identical KL disagreement: two MIMIC-IV patients with the same disagreement value ($0.1082$) receive normalized uncertainties of $0.0099$ and $0.0211$. That separation is what the authors mean by calling the metric more precise, and it is what allows a single self-distilled network to flag hard or out-of-distribution examples.","pith_inferences":["Because the teacher's label defines the outcome, the metric measures internal consistency rather than correctness; a natural extension is to add a second reference — e.g., the students' majority vote or an independently trained ensemble — to catch teacher overconfidence.","The linear and exponential weight functions give different conservativeness; one could test on a larger clinical graph whether the nonlinear variant improves OOD-detection AUROC, since the authors leave the choice as a domain parameter.","The normalized score is bounded and depth-comparable, so it could directly feed a decision threshold or a conformal-style guarantee for clinical alerting, though the paper does not explore that link.","Since students are trained to copy the teacher, the metric's sensitivity is capped by how much diversity the soft labels and feature penalties allow; adding a small diversity term to the loss would test whether sensitivity can be increased without losing the efficiency advantage."],"forward_implications":["One training pass produces both the classifier and its uncertainty estimate, with training time near that of a single GNN (about 20 s vs 93 s single and 881 s ensemble on MIMIC-IV).","The weighted metric separates cases the disagreement metric flattens: two MIMIC-IV patients with identical disagreement (0.1082) receive different normalized uncertainties when shallower classifiers contradict the teacher's label.","On Enzymes, self-distillation shows lower entropy on in-distribution data and higher entropy on out-of-distribution data, indicating usable OOD distinction.","F1 and ROC AUC are comparable to MC Dropout and deep ensembles (MIMIC-IV F1 0.88 vs 0.85 and 0.84) while the parameter count stays close to a single model.","The same recipe applies to any GNN classification task, graph or node level, without per-task modification."],"supporting_citations":[{"why":"Supplies the self-distillation training scheme: layerwise cross-entropy plus KL to the teacher, the feature penalty, and the last-20-epochs alpha/lambda scheduling.","marker":"[24]"},{"why":"Defines the deep-ensemble disagreement metric (KL divergence to the ensemble outcome) that the proposed weighted JSD metric extends.","marker":"[16]"},{"why":"MC Dropout, the Bayesian-style baseline whose accuracy and cost the method is compared against.","marker":"[15]"},{"why":"Provides the MIMIC-IV patient graph construction and pooling strategy used in the experiments.","marker":"[4]"},{"why":"The MIMIC-IV dataset used for readmission prediction and out-of-distribution evaluation.","marker":"[12]"},{"why":"The Enzymes dataset used to test generalizability and out-of-distribution separation.","marker":"[18]"},{"why":"GraphSAGE layers from which the GNN classifiers in the Enzymes experiments are built.","marker":"[9]"},{"why":"The GNN benchmark structure that the Enzyme models follow.","marker":"[5]"}],"fun_headline_variants":["Self-distilled GNNs yield precise uncertainty in one pass","One network, smarter uncertainty: self-distilled GNNs","Weighted disagreement sharpens GNN uncertainty","GNN uncertainty without ensembles via self-distillation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The deepest classifier (the teacher) is treated as the correct reference: its predicted label is the outcome in the uncertainty formula and every student is trained to imitate it, so a confidently wrong teacher makes the score low even when the model is actually uncertain.","fun_headline_variants_meta":{"raw":{"variants":["Self-distilled GNNs yield precise uncertainty in one pass","One network, smarter uncertainty: self-distilled GNNs","Weighted disagreement sharpens GNN uncertainty","GNN uncertainty without ensembles via self-distillation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000199,"raw_usage":{"total_tokens":1388,"prompt_tokens":976,"completion_tokens":412,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":592,"completion_tokens_details":{"reasoning_tokens":348}},"tokens_in":592,"tokens_out":412,"duration_ms":4585,"temperature":1.0,"reasoning_tokens":348,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T22:58:02.181433+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a test point whose true label differs from the teacher's most confident prediction, and on which all students (having mimicked the teacher) agree with the teacher; the proposed normalized metric returns a value near zero, while a correctness-aware measure such as true-class probability or disagreement among independently trained models would show high uncertainty.","supporting_citations":[{"cited_title":"IEEE Transactions on Pattern Analysis and Machine Intelligence44(8), 4388–4403 (2022)","cited_arxiv_id":null,"evidence_quote":"Supplies the self-distillation training scheme: layerwise cross-entropy plus KL to the teacher, the feature penalty, and the last-20-epochs alpha/lambda scheduling."},{"cited_title":"In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R","cited_arxiv_id":null,"evidence_quote":"Defines the deep-ensemble disagreement metric (KL divergence to the ensemble outcome) that the proposed weighted JSD metric extends."},{"cited_title":"Journal of Cheminformatics14(1), 2 (1 2022)","cited_arxiv_id":null,"evidence_quote":"MC Dropout, the Bayesian-style baseline whose accuracy and cost the method is compared against."},{"cited_title":"Proceedings of the Canadian Conference on Artificial Intelligence (may 27 2022)","cited_arxiv_id":null,"evidence_quote":"Provides the MIMIC-IV patient graph construction and pooling strategy used in the experiments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The MIMIC-IV dataset used for readmission prediction and out-of-distribution evaluation."},{"cited_title":"In: ICML 2020 Workshop on Graph Representation Learning and Beyond (GRL+ 2020) (2020), www.graphlearning.io","cited_arxiv_id":null,"evidence_quote":"The Enzymes dataset used to test generalizability and out-of-distribution separation."},{"cited_title":"In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vish- wanathan, S., Garnett, R","cited_arxiv_id":null,"evidence_quote":"GraphSAGE layers from which the GNN classifiers in the Enzymes experiments are built."},{"cited_title":"Journal of Machine Learning Research24(43), 1–48 (2023), http://jmlr.org/papers/v24/22-0567.html","cited_arxiv_id":null,"evidence_quote":"The GNN benchmark structure that the Enzyme models follow."}],"review_version":1}