{"id":"d1efe492-c79f-4733-a552-e308a75dae94","arxiv_id":"2412.03190","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"The paper proposes NCwR-Cost and NCwR-Cov, two ways to let GAT node classifiers abstain, and reports higher accuracy on accepted nodes than softmax-threshold and CF-GNN baselines across four datasets.","lead":"This paper adds an abstention option to graph neural networks, so a node classifier can refuse to predict when uncertain. It reports accuracy gains over confidence-threshold and conformal baselines on citation, legal, and medical graphs.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Main empirical claim compares methods at unmatched coverage levels; e.g., NCwR-Cost has only two coverage points on Pubmed (Table 2), so 'outperforms for all coverage values' is not directly established. A matched-coverage re-analysis is needed.","rationale":"After reading the paper, the central contribution is the empirical demonstration that the proposed reject-option GNNs beat simpler baselines on the accuracy-coverage trade-off. The reader's weakest_assumption points to transductive calibration transfer; I agree that is a risk but I find a more immediate problem: the comparison itself is not coverage-matched. In Table 2, NCwR-Cost coverage is determined by d, producing only two points for Pubmed and five for Cora/Citeseer. Baselines produce their own coverage values. The claim 'always outperforms ... for all coverage values' therefore relies on interpolation or visual overlap of unmatched points, which is not a rigorous comparison. A matched-coverage test would settle whether the advantage is real. The transductive calibration issue is related because it concerns whether the nominal coverage equals actual test coverage; if not, even matched nominal coverage would be misleading. Both concerns are addressable with a reanalysis that measures empirical test coverage and compares at identical coverage levels. Thus the conditional acceptance should explicitly require this reanalysis; the verdict need not change from CONDITIONAL, but the conditions should include matched-coverage comparisons.","tokens_in":12977,"tokens_out":13350,"duration_ms":118614,"concrete_test":"For each dataset and each baseline operating point (SR threshold or CF-GNN alpha), measure its empirical test coverage c. Then for NCwR-Cost, choose a threshold on the reject-class output f_{K+1} (or train with the matching d) so that its test coverage equals c, and record test accuracy. Replot all coverage-accuracy curves at these matched coverage values. If NCwR-Cost is not strictly above the baselines at every c, the claim in Section 5.1 fails. Also report NCwR-Cov's actual test coverage at each nominal target to verify calibration transfer.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim (Section 5.1) is that NCwR-Cost outperforms Softmax-Response for all coverage values and outperforms CF-GNN except at one Cora point. However, Figure 3 plots accuracy versus coverage without ensuring that the compared points share the same coverage. NCwR-Cost's coverage is an emergent property of the rejection cost d (Table 2), yielding a small discrete set: for Pubmed only two coverages (49.3% and 67.8%). Baselines are swept over thresholds/alphas with their own coverage values. Without matching coverage, the assertion 'for all coverage values' rests on interpolation across unmatched points, which is not a valid empirical comparison. A second, related risk is transductive calibration: the threshold for NCwR-Cov is chosen on validation scores (Section 4.3), and the paper assumes the same coverage holds on test, but graph-induced dependence between validation and test nodes may break this. The unmatched-coverage issue alone is sufficient to require re-analysis: the headline comparison should be recomputed at identical empirical test coverage for all methods.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes two GAT-based architectures for transductive node classification with an abstention (reject) option: NCwR-Cov, a coverage-constrained selective classifier modeled on SelectiveNet, and NCwR-Cost, which augments the output with a reject class and uses the consistent cost-based cross-entropy loss of CCF+22. The methods are evaluated on Cora, Citeseer, and PubMed, and on the ILDC legal judgment dataset; SHAP is used to illustrate which text segments drive decisions in two legal cases. The central claim is that NCwR-Cost dominates the Softmax-Response baseline at all coverage values and dominates the CF-GNN conformal baseline except at one Cora point, and that the reject-option models are useful for high-risk legal prediction.","tokens_in":13176,"tokens_out":11459,"duration_ms":101412,"significance":"If the dominance claim could be established at matched coverage, the paper would be a useful practical contribution: it would show that an integrated abstention head in a GAT improves over simple softmax thresholding and conformal set-size rejection for transductive node classification. The paper is built on published consistent losses and clearly credits them; the appendices add results across several base GNNs and two medical datasets, which strengthens the generality claim. The main weakness is that the evaluation protocol does not currently support the headline superiority claim: comparisons are made at unmatched coverage values, error bars are missing for key baselines and for Table 1, and the ILDC experiment lacks all baseline comparisons. The strengths of the paper are the clear problem formulation, the use of standard GAT training, and the concrete application domain; the empirical claims, however, are not yet demonstrated.","major_comments":[{"comment":"The headline claim that 'NCwR-Cost always outperforms Softmax-Response based approach ... in terms of accuracy on unrejected samples for all coverage values' is not established by the reported experiment, because the methods are evaluated at different empirical coverage values. NCwR-Cost's coverage is determined by the rejection cost d and takes a small discrete set of values: for Pubmed only d=0.5 (49.3%) and d=0.6 (67.8%) are reported apart from the d=1 full-coverage point, and for Citeseer the d=0.5 point has 9.7% coverage. The baselines are generated by sweeping over thresholds (Softmax-Response) or over alpha (CF-GNN), so their coverage points do not coincide with these values. Comparing interpolated curves in Figure 3 is not a valid empirical test of 'for all coverage values'. I request a matched-coverage re-analysis: calibrate every method's threshold (or alpha) on the validation set to a common grid of target coverages, report the actual test coverage, and compare accuracy only at identical coverage values.","section":"Section 5.1, Figure 3, Table 2"},{"comment":"The coverage calibration for NCwR-Cov selects tau on the validation set, and the paper assumes that the resulting coverage transfers to the test set; the paper itself notes in Section 4.3 that the training coverage constraint is violated on the test set. The theoretical support for this calibration comes from SelectiveNet [GEY19] and the loss-consistency results in [CCF+22], which are i.i.d. statements, whereas in this transductive setting the GAT embeddings depend on message passing over the whole graph, so validation and test selection scores are dependent. Please report, over repeated random validation/test splits, the achieved test coverage versus target coverage, or provide an exchangeability argument. Without this, the reported coverage rates and the x-axis of Figure 3 are not externally verifiable.","section":"Section 4.3, Table 1, Figure 3"},{"comment":"The ILDC application is presented as a contribution in the abstract and Section 1, but Table 3 reports only NCwR-Cost and NCwR-Cov. There is no Softmax-Response or CF-GNN baseline on this dataset, no comparison with the GAT result from [KYK+23], and no un-rejected-set accuracy from the earlier ILDC systems of [MSN+21]. The sentence 'Our results on LJP task show that these models are very effective in such applications' is therefore unsupported by the data. The paper should add the same baselines used in Section 5.1 on ILDC at matched coverage, or explicitly limit the LJP claim to a demonstration of feasibility.","section":"Section 6, Table 3"},{"comment":"The statistical evidence for the dominance claim is incomplete. The introduction to Section 5 says each experiment is repeated 10 times and average accuracies are reported, but Table 1 gives no standard deviations, Table 4 in Appendix A gives none, and the baseline curves in Figure 3 are shown without any error bars. Table 2 shows nontrivial run-to-run variation (e.g., Citeseer d=0.85 accuracy 75.8 ± 1.61), and the comparisons in Figure 3 are often close, including the single Cora point where CF-GNN is better. The paper should report confidence intervals for every point and a paired comparison over the 10 runs at matched coverage values.","section":"Section 5, Tables 1-3, Figure 3"},{"comment":"The restriction for Pubmed is not justified as written: 'We only experimented with d = [0.5, 0.6] in Pubmed as it has only k = 3 classes and d < k-1/k.' The inequality is not stated correctly or explained, and the omitted values d=0.7, 0.8, and 0.85 are exactly the values that would supply intermediate coverage points for Pubmed. If there is a feasibility condition from [CCF+22], it should be stated with a reference; otherwise the authors should run these values and report the resulting coverage and accuracy points.","section":"Section 4.4, Table 2"}],"minor_comments":[{"comment":"The sentence 'Label smoothing makes this situation worse for NCwR-Cov due to soft labels' refers to a technique that is not introduced anywhere in Section 4; either describe label smoothing in the experimental setup or remove the reference.","section":"Section 5.2"},{"comment":"The columns headed '0-d-1' in Table 2 are never defined in the text; please define the empirical 0-d-1 risk and explain how it is computed.","section":"Table 2"},{"comment":"The text compares coverage '50.4%' for NCwR-Cov with NCwR-Cost, but the experimental tables report 50% target coverage for NCwR-Cov and 42.6% coverage for NCwR-Cost d=0.5 on Cora; the provenance of the 50.4% value in Figure 4 should be clarified.","section":"Section 5.3, Figure 4"},{"comment":"The case presentation says the model's low confidence 'led to rejection,' but the discussion then explains the SHAP output based on a wrong prediction; please clarify whether the example was rejected or accepted with an error.","section":"Section 6.2.2"},{"comment":"The UCI Thyroid and Pima Indians Diabetes results in Tables 5 and 6 are not referenced in the main text and no conclusion is drawn from them; add a pointer in Sections 4 or 5 and state what these results contribute.","section":"Appendix B"},{"comment":"For reproducibility, specify how many runs and which GAT hyperparameters were used for the Softmax-Response and CF-GNN baselines, and how the CF-GNN label-set size was calibrated on the validation set.","section":"Section 4.5"}],"recommendation":"major_revision","confidential_remarks":"The unmatched-coverage concern is genuine and is the main blocker: the headline comparison should be recomputed at identical empirical test coverages for all methods. The requested re-analysis is feasible within the scope of the paper, since threshold calibration on the validation set is already part of the NCwR-Cov procedure and can be applied to the baselines. I also see no circularity problem: the evaluation uses held-out test nodes, and the training losses are properly attributed to prior work. The paper would benefit from a more rigorous statistical treatment and from adding ILDC baselines before it can support its central claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper adapts SelectiveNet and the CCF+22 consistent loss to GNN node classification, which is a genuine new domain. It also brings in a legal judgment dataset as a node classification problem and shows SHAP-based explanations for rejected cases. That part is useful and the writing is clear.\n\nThe central claim is that NCwR-Cost outperforms Softmax-Response and CF-GNN at all coverages. That claim is not actually supported by the experiments. In Figure 3 and Table 2, NCwR-Cost coverage is an emergent property of the rejection cost d, and on Pubmed you only get two coverage points (49.3% and 67.8%). The baselines are swept over thresholds and give their own coverage values, so the comparison is between unmatched points. Saying \"for all coverage values\" requires interpolation across coverage levels where no data exists. This is a real empirical flaw, and the fix is straightforward: compare methods at identical test coverage, either by post-hoc selection or by reporting accuracy at the same achieved coverage. The stress-test note is right about this.\n\nThere is a second issue that bothers me more. The coverage-based method calibrates the selection threshold on validation scores (Section 4.3) and assumes the same coverage transfers to the test set. In transductive node classification, validation and test nodes share the same graph and message passing, so selection scores are not exchangeable. The paper never checks whether the achieved test coverage matches the target. This could distort the accuracy-coverage curves. At minimum, the paper should report achieved coverage on test for each target.\n\nThe baselines also lack error bars. The NCwR methods report standard deviations over 10 runs, but Softmax-Response and CF-GNN points appear as single points, so we cannot tell if differences are meaningful. The ILDC experiments have no baseline comparison at all, which weakens the claim that reject options help in that high-stakes setting.\n\nI want to give credit where it is due. The loss formulations are standard and correctly cited. The t-SNE visualization is a nice check that the reject option is behaving sensibly. And the paper is honest about some limitations, like the label smoothing problem for NCwR-Cov. No code is released, which is a shame but not a fatal flaw.\n\nMy verdict: the underlying idea is plausible and the paper is a reasonable contribution to trustworthy graph learning, but the main empirical claim needs re-analysis at matched coverage, and the calibration transfer needs to be checked. These are fixable. I would send it to review, but I would insist on that re-analysis before acceptance.\n\nFor a colleague: this is worth a quick read if you work on abstention or GNNs, but do not take the performance claims at face value.","headline":"Clean transfer of known reject-option losses to GNN node classification, with a useful legal application, but the headline comparison is not made at matched coverage levels and the transductive calibration issue is unaddressed.","tokens_in":729,"tokens_out":882,"would_cite":false,"duration_ms":25637,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Integrating a reject option into GNN training lets node classifiers abstain on hard nodes and, the authors report, beat both softmax-threshold and conformal set-size baselines in accuracy at a given coverage on citation benchmarks.","keywords":["reject option classification","node classification","graph neural networks","abstention","selective classification","coverage constraint","legal judgment prediction","transductive learning"],"falsifier":"A reader could check whether the reported coverage-accuracy curves survive a non-exchangeable split: for example, hold out an entire connected community (or a time-based split on the ILDC citation graph) so that validation and test selection scores come from different structural neighborhoods. If the validation-calibrated threshold then yields test coverage far below target, and NCwR-Cost's advantage over Softmax-Response disappears, the central claim would be shown to depend on transductive exchangeability.","tokens_in":12788,"feed_emoji":"⚖️","tokens_out":5785,"duration_ms":51916,"temperature":0.7,"pith_summary":"This paper argues that Graph Neural Networks should be trained with an explicit reject option—the ability to abstain when unsure—rather than having abstention bolted on after training. It proposes two integrated training schemes for node classification: NCwR-Cost, which adds a (K+1)th reject class with a cost-weighted cross-entropy loss, and NCwR-Cov, which trains a selection head with a coverage constraint in the style of SelectiveNet. On Cora, Citeseer, and Pubmed, the cost-based variant is reported to outperform both a softmax-confidence threshold and the conformal CF-GNN baseline in accuracy on unrejected samples across almost all coverage levels. The same machinery is applied to Indian legal judgment prediction, where rejected cases are visualized with SHAP to show which textual evidence drives abstention.","feed_headline":"Abstaining GNNs beat confidence-threshold rejection","feed_subtitle":"Cost-based abstention lifts accuracy on accepted nodes at every coverage level tested, and works for legal judgment prediction.","key_machinery":"The load-bearing mechanism is treating rejection as a trainable part of the model rather than a post hoc threshold. For NCwR-Cost the key identity is the rejection-augmented cross-entropy loss $l^d_{ce}(f(h), e_y) = -\\log f_y(h) - (1-d) \\log f_{K+1}(h)$, which interpolates between standard cross-entropy ($d=1$) and a loss that strongly rewards the reject class ($d$ small); because it is calibrated to the 0-d-1 loss, minimizing it is consistent with the intended cost structure. For NCwR-Cov the machinery is the selective-risk objective $r(f,g|S_n) = \\frac{1}{n}\\sum_i l(f(h_i), y_i) g(h_i) / \\phi(g|S_n)$ with a quadratic coverage penalty $\\lambda\\Psi(c - \\phi(g|S_n))$, plus an auxiliary prediction head, and a validation-set threshold calibration to convert the training coverage constraint into a test-time coverage.","core_discovery":"The central claim is that rejection can be integrated into GNN training itself, and that this yields strictly better selective classifiers than post-hoc rejection heuristics. NCwR-Cost treats abstention as an extra class and optimizes the loss $l^d_{ce}(f(h), e_y) = -\\log f_y(h) - (1-d) \\log f_{K+1}(h)$, which is calibrated to the 0-d-1 loss: with a small rejection cost $d$, the model is pushed to assign probability mass to the reject class rather than to misclassify. NCwR-Cov instead follows SelectiveNet, learning a selection score $g(h)$ and minimizing selective risk with a quadratic penalty on coverage, then calibrating the selection threshold $\\tau$ on validation scores to hit the target coverage on test. The paper reports that NCwR-Cost always beats Softmax-Response on accuracy at a given coverage and beats CF-GNN in all but one Cora setting, and that on ILDC the abstaining models reach high accuracy at low coverage, e.g. 97.55% accuracy at 50% coverage for NCwR-Cov.","pith_inferences":["A testable extension would be inductive node classification: if the reject head is trained on one graph and deployed on a graph with unseen topology, the validation-calibrated coverage may fail to transfer; the transductive setup may hide this failure mode.","The cost-based loss is a generic recipe that could be attached to any differentiable classifier, not just GNNs; its advantage over conformal baselines suggests integrated abstention training is a general principle rather than a graph-specific trick.","The SHAP analyses in the legal application connect abstention to specific textual evidence (e.g., suspicious FIR delay), suggesting the reject scores carry interpretable signal that could drive human-in-the-loop review pipelines."],"forward_implications":["On Cora, Citeseer, and Pubmed, NCwR-Cost is reported to beat Softmax-Response at every coverage value tested, and to beat CF-GNN except for one Cora case at 50% coverage.","The cost-based recipe is architecture-agnostic: the appendix shows similar accuracy across GCN, GAT, GraphSAGE, and GATv2 bases, and across 2 to 4 GAT layers.","On ILDC legal judgment, NCwR-Cov achieves 97.55% accuracy at 50% coverage, showing abstention models can be practically useful in high-stakes domains.","The same integrated-reject idea extends to tabular medical data converted into KNN graphs, with results reported on UCI Thyroid and Pima Indians Diabetes."],"supporting_citations":[{"why":"Supplies the coverage-based selective risk objective and architecture that NCwR-Cov adapts.","marker":"[GEY19]"},{"why":"Provides the cost-based rejection loss $l^d_{ce}$ and its calibration to the 0-d-1 loss, which NCwR-Cost uses.","marker":"[CCF+22]"},{"why":"CF-GNN is the conformal set-size baseline that NCwR-Cost and NCwR-Cov are compared against.","marker":"[HJCL23]"},{"why":"GAT is the base GNN architecture used in all experiments.","marker":"[VCC+18]"},{"why":"Provides the Cora, Citeseer, and Pubmed citation datasets used for the main evaluation.","marker":"[SNB+08]"},{"why":"Introduces the graph formulation of ILDC as a node classification task with citations as edges, which the paper extends with reject option heads.","marker":"[KYK+23]"},{"why":"Provides the ILDC dataset and pretrained XLNet embeddings used for legal judgment prediction.","marker":"[MSN+21]"}],"fun_headline_variants":["GNNs that abstain on uncertainty beat fixed rejection thresholds","Integrated abstention improves GNN node classification accuracy","NCwR: GNNs with built-in rejection outperform post-hoc filtering","Reject option in GNNs yields better selective classifiers"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The selective-classification guarantees borrowed from SelectiveNet and the loss-calibration results from the 0-d-1 literature assume i.i.d. examples, but node classification here is transductive—train, validation, and test share one graph, and GAT embeddings are computed by message passing over the whole graph—so the paper assumes that calibrating the selection threshold on validation scores transfers to the test set.","fun_headline_variants_meta":{"raw":{"variants":["GNNs that abstain on uncertainty beat fixed rejection thresholds","Integrated abstention improves GNN node classification accuracy","NCwR: GNNs with built-in rejection outperform post-hoc filtering","Reject option in GNNs yields better selective classifiers"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000485,"raw_usage":{"total_tokens":2387,"prompt_tokens":931,"completion_tokens":1456,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":547,"completion_tokens_details":{"reasoning_tokens":1386}},"tokens_in":547,"tokens_out":1456,"duration_ms":10066,"temperature":1.0,"reasoning_tokens":1386,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T22:40:47.674947+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A reader could check whether the reported coverage-accuracy curves survive a non-exchangeable split: for example, hold out an entire connected community (or a time-based split on the ILDC citation graph) so that validation and test selection scores come from different structural neighborhoods. If the validation-calibrated threshold then yields test coverage far below target, and NCwR-Cost's advantage over Softmax-Response disappears, the central claim would be shown to depend on transductive exchangeability.","supporting_citations":[],"review_version":1}