{"id":"e876e6b5-ab3e-4fdd-8dc7-3bbaef57a3c1","arxiv_id":"2412.14223","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A post-hoc GNN calibration method that groups nodes by confidence and neighborhood prediction similarity and applies group-specific temperatures, reducing expected calibration error on many benchmarks.","lead":"Simi-Mailbox is a new post-hoc calibration method for graph neural networks that groups nodes by both their confidence and the similarity of their neighbors' predictions, then applies a separate temperature to each group to correct miscalibration. It reports lower calibration error than existing GNN calibration baselines across many benchmark graphs and architectures, offering a simple way to make model confidence more trustworthy.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central grouping claim rests on an unvalidated proxy: Msimi (Eq. 7) is never shown to reproduce the s(i)-based miscalibration structure, and no ablation isolates the topology dimension from confidence-only grouping.","rationale":"The reader's weakest assumption is exactly the gap between the s(i)-based analysis and the Msimi-based method, and my stress-test concurs that this is the most load-bearing point. If the proxy fails, the method's conceptual novelty collapses into confidence-based binning with group temperatures, and the empirical ECE tables, while extensive, cannot rescue the topological interpretation because no confidence-only grouped-temperature baseline is reported. The paper has real strengths: broad evaluation across datasets and architectures, including heterophilous graphs and large-scale graphs, plus speed advantages and reproducibility via the anonymous code link. Those strengths support the empirical claim that the method calibrates well in many settings, but they do not establish the paper's explanatory claim about why it works. The requested ablation is feasible with the released code and would settle the concern directly. I therefore keep the reader's CONDITIONAL verdict: acceptance should hinge on demonstrating that the Msimi-based grouping reduces within-group calibration-error variance beyond confidence alone and yields ECE gains beyond confidence-only grouped temperature scaling.","tokens_in":30746,"tokens_out":2419,"duration_ms":24366,"concrete_test":"Run an ablation on Cora, Citeseer, PubMed, and CoraFull with GCN and GAT: fix the number of clusters N and the λ grid, then calibrate using KMeans on (a) normalized confidence only, (b) normalized confidence plus Msimi (Eq. 7), and (c) normalized confidence plus label-based s(i) (Eq. 3), with s(i) computed from validation-set predictions. Report test ECE and the within-group calibration-error variance of Eq. 5 for each grouping. If configuration (b) does not beat (a) by more than one standard deviation and does not reduce V^sim relative to (a), then the topology term is not load-bearing for the method's empirical success.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's motivating analysis and variance evidence (Table 1, Figure 1) are built on label-based neighborhood prediction similarity s(i) = (1/|N_i|) Σ_{j∈N_i} 1[ŷ_i = ŷ_j] (Eq. 3). The proposed method, however, clusters nodes on Msimi(i) = (1/|N_i|) Σ_{j∈N_i} σ(z_i^T z_j) (Eq. 7), a sigmoid of logit dot products. The central premise is that nodes with similar Msimi and similar confidence have similar calibration errors; no experiment in the paper measures whether this premise holds. In particular, no table or figure reports the variance of calibration errors within Msimi-based clusters (the analogue of Eq. 5), and no correlation analysis between Msimi and s(i) is provided. If Msimi does not capture the same miscalibration structure as s(i), the method reduces to confidence-based KMeans binning with per-bin temperatures, and the claimed role of topology in calibration is unsupported. The issue is compounded by the absence of a confidence-only grouped-temperature baseline in Tables 2 and 3, so even the empirical ECE gains do not by themselves demonstrate that the topology term is load-bearing.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper addresses post-hoc calibration of GNN node classifiers. It argues that the common assumption underlying CaGCN and GATS—that nodes with similar neighborhood prediction similarity have similar calibration behavior—does not hold universally, because calibration errors vary with confidence even among nodes of comparable neighborhood similarity. To support this, the paper analyzes calibration errors on CoraFull using label-based neighborhood prediction similarity s(i) (Eq. 3), and proposes SIMI-MAILBOX, which computes logit-based Mailbox similarity Msimi (Eq. 7), clusters nodes via KMeans on normalized confidence and normalized Msimi, and fits one temperature per cluster by minimizing a combination of cross-entropy and a cluster-wise confidence-accuracy loss (Eq. 9). The empirical section compares against TS, VS, ETS, CaGCN, GATS, and recent methods over small, medium, large, and heterophilous graphs, with additional analyses of self-training, OOD detection, runtime, and multiple calibration metrics.","tokens_in":31064,"tokens_out":10372,"duration_ms":97185,"significance":"The paper is practically strong: the method is simple, adds only N temperature parameters, preserves classification accuracy because group-wise temperature scaling is order-preserving, and the empirical evaluation is extensive (10+ datasets, multiple architectures, many metrics, ablations, and a code link). If the grouping premise holds, the method is a useful and efficient post-hoc calibration tool. However, the conceptual contribution stands or falls on whether the logit-based Msimi proxy reproduces the label-based structure used in the motivating analysis; that link is not demonstrated in the current manuscript, and the empirical tables do not include the ablations needed to isolate the topology component.","major_comments":[{"comment":"The motivating analysis (Section 'In-depth Analysis on Neighborhood Similarity', Figure 1, and Table 1) is built on label-based neighborhood prediction similarity s(i) = (1/|N_i|) Σ_{j∈N_i} 1[ŷ_i = ŷ_j] (Eq. 3), but the proposed grouping uses logit-based Mailbox similarity Msimi(i) = (1/|N_i|) Σ_{j∈N_i} σ(z_i^T z_j) (Eq. 7). The paper never validates this proxy: it reports no correlation or rank statistic between Msimi and s(i), no analog of Eq. (5) computed for Msimi-based clusters, and no heatmap showing that accuracy-minus-confidence cells are homogeneous when nodes are grouped by Msimi and confidence. Since confidence is the second clustering feature, the ECE gains in Tables 2 and 3 are consistent with a method that performs per-bin temperature scaling on confidence alone; the claimed role of topology is therefore not yet supported.","section":"Proposed Method, Eq. (7); Preliminaries, Eq. (3)"},{"comment":"The definition of node-wise calibration error D(i) in Eq. (4) is |Acc(B_m) − p̂_i|, where B_m is the confidence interval, not the joint neighborhood-similarity-confidence cell B_m^{(l)}. Consequently, the variance within B_m^{(l)} in Eq. (5) measures only the spread of confidence values inside each cell, not the cell-specific discrepancy between accuracy and confidence. The decrease from Node-wise to Conf. to Neig. Sim. in Table 1 is the expected consequence of partitioning the confidence range into finer subgroups, so it does not establish that nodes with similar s(i) and confidence share similar calibration errors. Please recompute the variance using acc(B_m^{(l)}) or the cell-wise accuracy-confidence gap.","section":"Intuition: Topology Grouping Matters, Eqs. (4)-(6), Table 1"},{"comment":"None of the experimental tables includes a confidence-only grouped-temperature baseline. A direct ablation would apply the same KMeans-plus-group-temperature procedure to (i) normalized confidence alone, (ii) Msimi alone, and (iii) random cluster assignments at the same N, preferably also reporting the within-cluster ECE variance. Without these conditions, the headline ECE reductions over TS, CaGCN, and GATS do not identify which clustering dimension drives the improvement, so the specific benefit of the 'topology-grouping strategy' in the title remains untested.","section":"Experiments, Tables 2-3"}],"minor_comments":[{"comment":"There is a typo in 'ensemble temperature sclaing' that should read 'ensemble temperature scaling'.","section":"Baselines"},{"comment":"The sentence 'The optimal calibration models are chosen based on the lowest validation ECE on training set' should read 'on the validation set', since the training set is not used for validation.","section":"Experimental Setup"},{"comment":"Please clarify whether C_n in L_simi denotes the set of validation nodes in cluster n or all nodes assigned to cluster n; if it denotes all nodes, the average confidence in Eq. (9) includes test nodes while a_val(n) is a validation-only accuracy, which makes the objective ill-defined.","section":"Proposed Method, Eq. (9)"},{"comment":"The caption says 'A lower AUROC indicates better detection performance', but for the Area Under the ROC Curve higher values indicate better detection; this also conflicts with the text describing GNNSafe's high AUROC values as state-of-the-art.","section":"Table 16"}],"recommendation":"major_revision","confidential_remarks":"The missing proxy validation and the confidence-only ablation are potentially easy to add and would determine whether the conceptual contribution is more than confidence binning. I recommend requesting these additions and a corrected variance analysis rather than rejecting the paper, as the method itself is simple, fast, and empirically competitive. Please also ask the authors to address the AUROC caption error, which currently inverts the direction of the metric."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing to know: Simi-Mailbox is a simple, fast, post-hoc calibration method that groups nodes by confidence and a neighborhood-similarity score and learns a temperature per group. The empirical results are broad and mostly convincing. But the paper's explanation for why the method works—that similar neighborhood similarity plus confidence means similar calibration error—rests on a proxy that is never validated, and the method never gets compared to its own confidence-only variant. So the contribution is real but the central narrative is not fully supported.\n\nWhat's good: The paper identifies a genuine failure mode in CaGCN/GATS: they assume a single relationship between neighborhood similarity and calibration, while the heatmaps show both over- and under-confidence at the same similarity level depending on confidence. That is a useful observation and it motivates the grouping idea. The method itself is clean and efficient—group-specific temperatures with only N parameters, optimized with a direct calibration loss. The experiments are extensive: 8 small/medium datasets, 2 large graphs, multiple backbones (GCN, GAT, SAGE, MLP, GIN), heterophilous graphs, sparse labels, OOD, and comparisons with recent baselines (SimCalib, GETS, DCGC, GC with holdout). Code is released. The ECE gains are frequently large and consistent, e.g., Cora GCN from 3.55 to 1.97, Reddit SAGE from 4.66 to 0.73.\n\nThe soft spots are real but addressable. First, the motivating analysis uses label-based neighborhood prediction similarity s(i), while the method clusters on logit-based Mailbox similarity Msimi (Eq. 7). The paper never shows that Msimi reproduces the low within-group calibration-error variance that Table 1 demonstrates for s(i). If Msimi is a bad proxy, the method reduces to confidence-based KMeans binning. Second, no ablation isolates the topology term: no version of the method uses only confidence as the clustering feature. Without that, you cannot tell whether the topology information is load-bearing or incidental. Third, several ECE improvements are within one standard deviation (e.g., Physics GAT is actually slightly worse than GATS). The headline numbers are strong, but not every setting is conclusive. Finally, the paper explicitly defers a theoretical foundation, which is fine for an empirical paper but worth stating.\n\nThe circularity concern in the reader report is a non-issue: temperatures are fit on validation and evaluated on held-out test, so there is no target leakage. The calibration loss directly minimizes within-group discrepancy, which is the intended objective.\n\nBottom line: this is a useful empirical contribution for GNN calibration. A serious referee should engage with it, but the authors should be asked to validate the Msimi proxy and run the confidence-only ablation. If those come out clean, the paper is solid.","headline":"Useful post-hoc calibration recipe; the grouping story is plausible but the topology dimension is not isolated from confidence binning.","tokens_in":31547,"tokens_out":3185,"would_cite":true,"duration_ms":29068,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T07"],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that nodes with the same neighborhood prediction similarity can still be both over- and under-confident depending on their confidence, so calibration should group by both factors and use per-group temperatures.","keywords":["graph neural networks","calibration","temperature scaling","neighborhood similarity","confidence binning","expected calibration error","node classification","post-hoc calibration"],"falsifier":"Compute within-cell variance of calibration error using the Mailbox score $M_{\\text{simi}}(i)$ over confidence bins on CoraFull with GCN and compare it with the paper's Table 1 variance; if the variance is not lower than confidence-only variance, the topology-grouping claim collapses. A cheaper check is to permute $M_{\\text{simi}}(i)$ across nodes and rerun Simi-Mailbox: if ECE stays the same, topology adds nothing beyond confidence binning.","tokens_in":30545,"feed_emoji":"🌡️","tokens_out":6273,"duration_ms":53000,"temperature":0.7,"pith_summary":"Existing GNN calibration methods assume that nodes with similar neighborhood prediction similarity share similar miscalibration, so a single temperature trend suffices. The paper tests this against calibration-error heatmaps and finds that at equal neighborhood similarity, nodes with different confidence can be simultaneously over- and under-confident. It proposes Simi-Mailbox, which clusters nodes by both a logit-based neighborhood similarity score and their own confidence, then fits one temperature per cluster to match average confidence to accuracy. The measured consequence is lower expected calibration error (ECE) than temperature scaling, CaGCN, and GATS across most of sixteen settings.","feed_headline":"Two-factor grouping cuts GNN calibration error below 3 percent","feed_subtitle":"Split nodes by neighborhood similarity and own confidence, then give each group its own temperature.","key_machinery":"The carrying object is the 'Mailbox' neighborhood similarity $M_{\\text{simi}}(i) = \\frac{1}{|N_i|}\\sum_{j\\in N_i}\\sigma(z_i^\\top z_j)$, the average sigmoid of the dot product between a node's logits and its neighbors' logits. Alongside normalized confidence $\\bar{p}_i$, it forms the two-dimensional feature vector used for K-means clustering. The group-specific temperatures $T_n$ are then optimized by a loss that directly minimizes the discrepancy between each group's average scaled confidence and its validation accuracy, with an auxiliary cross-entropy term to preserve predictive behavior. The design is meant to replace a single global or node-wise temperature trend with a small number of cluster-wise temperatures, which is also what makes the calibration step fast.","core_discovery":"The central claim is that the structure of GNN miscalibration is two-dimensional: nodes with comparable neighborhood prediction similarity $s(i) = |N_i|^{-1}\\sum_{j\\in N_i}\\mathbf{1}[\\hat{y}_i = \\hat{y}_j]$ can still have opposite calibration errors depending on their confidence $\\hat{p}_i$. Accordingly, the paper asserts that the single-factor assumption used by prior neighborhood-similarity calibration methods is not universal, and that grouping nodes by both factors yields clusters with homogeneous calibration error. Simi-Mailbox operationalizes this by clustering the two-dimensional feature $[\\text{normalized }\\hat{p}_i,\\ \\text{normalized }M_{\\text{simi}}(i)]$ with K-means and assigning each cluster $C_n$ its own temperature $T_n$, trained with a loss that penalizes the gap between each group's average confidence and its validation accuracy. Since group-wise temperature scaling preserves the ordering of logits, accuracy is preserved. The experimental claim is ECE reductions below 3% on Cora and Citeseer and below 1% on several larger and already well-calibrated graphs.","pith_inferences":["A direct test of the paper's mechanism would be a confidence-only ablation: if clustering on normalized confidence alone matches Simi-Mailbox's ECE, the neighborhood-similarity dimension is not doing the work; the paper does not report that ablation.","The motivating variance analysis uses label agreement $s(i)$, but the method clusters on the sigmoid logit-dot-product proxy $M_{\\text{simi}}(i)$; whether the proxy preserves the same cell structure is untested, so the topology component's contribution remains an open empirical question.","Because group-specific temperatures are cheaper than node-wise temperatures and preserve accuracy, the same grouping could be applied as a lightweight post-hoc layer for uncertainty-driven tasks such as selective prediction or out-of-distribution detection, though the paper's reported out-of-distribution results are mixed."],"forward_implications":["On the eight small- and medium-scale graphs, Simi-Mailbox reports the best or second-best ECE in 15 of 16 GCN/GAT settings, including sub-3% ECE on Cora and Citeseer.","On large-scale graphs, ECE falls below 1% in all reported settings, with Reddit improving from about 8.55% to 0.35% under GCN.","Because a shared group temperature preserves logit order, post-hoc calibration does not change classification accuracy.","Self-training with calibrated pseudo-labels raises accuracy in 17 of 18 settings relative to uncalibrated and CaGCN baselines.","Calibration time is lower than node-wise GNN calibrators because tuning involves only the number of clusters in temperature parameters."],"supporting_citations":[{"why":"Defines GATS, the strongest prior neighborhood-similarity calibration baseline, and supplies the experimental protocol reused here.","marker":"Hsu et al. 2022"},{"why":"Defines CaGCN, the baseline whose uniform confidence-propagation assumption the paper's heatmap analysis challenges.","marker":"Wang et al. 2021"},{"why":"Supplies temperature scaling and vector scaling, the post-hoc baselines that group-specific temperatures extend.","marker":"Guo et al. 2017"},{"why":"Defines the expected calibration error (ECE) metric used for all comparisons.","marker":"Naeini, Cooper, and Hauskrecht 2015"},{"why":"Provides the GCN backbone used for the central heatmap analysis and the main experiments.","marker":"Kipf and Welling 2016"},{"why":"Provides the GAT backbone used in the main experiments.","marker":"Veličković et al. 2017"}],"fun_headline_variants":["Two-factor grouping cuts GNN calibration error","Simi-Mailbox: group by similarity and confidence for better GNN calibration","Group-specific temperatures refine GNN uncertainty estimates","Neighborhood similarity plus confidence: key to GNN calibration"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The logit-based Mailbox score is assumed to capture the same miscalibration structure as the label-based neighborhood similarity used in the motivating analysis, but the paper never shows that the two agree cell by cell.","fun_headline_variants_meta":{"raw":{"variants":["Two-factor grouping cuts GNN calibration error","Simi-Mailbox: group by similarity and confidence for better GNN calibration","Group-specific temperatures refine GNN uncertainty estimates","Neighborhood similarity plus confidence: key to GNN calibration"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000396,"raw_usage":{"total_tokens":2100,"prompt_tokens":997,"completion_tokens":1103,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":613,"completion_tokens_details":{"reasoning_tokens":1038}},"tokens_in":613,"tokens_out":1103,"duration_ms":8648,"temperature":1.0,"reasoning_tokens":1038,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T12:37:05.580886+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute within-cell variance of calibration error using the Mailbox score $M_{\\text{simi}}(i)$ over confidence bins on CoraFull with GCN and compare it with the paper's Table 1 variance; if the variance is not lower than confidence-only variance, the topology-grouping claim collapses. A cheaper check is to permute $M_{\\text{simi}}(i)$ across nodes and rerun Simi-Mailbox: if ECE stays the same, topology adds nothing beyond confidence binning.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines CaGCN, the baseline whose uniform confidence-propagation assumption the paper's heatmap analysis challenges."},{"cited_title":"P.; Cooper, G.; and Hauskrecht, M","cited_arxiv_id":null,"evidence_quote":"Defines the expected calibration error (ECE) metric used for all comparisons."}],"review_version":1}