{"id":"0bf2c30f-5662-434b-b264-eae454dbdfee","arxiv_id":"1908.08588","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A joint 3D UNet and graph neural network for airway segmentation yields a small, significant improvement in centreline false-negative distance over a 3D UNet baseline, with no significant change in Dice or completeness.","lead":"This paper combines a 3D UNet with a graph neural network to segment airways in chest CT scans, using graph connections to pass information between distant image regions. The new models reduce one branch-detection error measure with fewer parameters, but they do not improve Dice score, completeness, or leakage.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported dFN gain may be an artifact of choosing each model's probability threshold from the same test-set ROC used for evaluation, so the central claim is not yet secure.","rationale":"The reader's weakest assumption identifies the same load-bearing issue: thresholds selected from the test ROC are used to compare models on that same test set, making the comparison optimistic and potentially non-replicable. This is the most serious threat to the paper's central quantitative claim because the claimed effect is small (Section 5: 'small and significant improvement') and the statistical evidence rests on paired t-tests over only 12 scans with multiple endpoints. The paper itself reports no external validation, no held-out threshold selection, and no code or data release, so there is no independent check on this bias. My stress-test pass did not find an internal inconsistency in the method or results; the architecture is clearly described and the GNN module is a reasonable, parameter-efficient modification. However, the evaluation methodology is not sufficient to establish the improvement as a property of the model rather than an artifact of test-set-derived operating points. The appropriate verdict remains CONDITIONAL: the finding should be re-evaluated with thresholds fixed by a separate validation procedure or on an external dataset before the performance claim is accepted. I therefore recommend no change to the reader's verdict.","tokens_in":6410,"tokens_out":1919,"duration_ms":21844,"concrete_test":"Re-evaluate the models using thresholds chosen without access to the test labels: for example, select each model's probability threshold on the 4 validation CTs (or via repeated nested cross-validation on the 16 training CTs) to achieve 13% volume leakage, then apply those thresholds to the 12 test CTs and recompute dFN, dFP, Dice, and completeness. If UGnnReg and UGnnDyn no longer show a statistically significant dFN improvement over UNetLev3 (p < 0.05) with a stable mean difference, the test-set threshold selection was responsible for the reported gain. As a secondary check, compare all models at a common fixed threshold of 0.5 using the operating points shown in Figure 2; if the dFN advantage disappears at fixed threshold, the claim is threshold-dependent.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim in Section 5 is that UNet-GNN models show a small and significant improvement in dFN over the baseline UNet at a fixed 13% volume leakage. The load-bearing assumption is that this comparison is a fair estimate of model quality. Section 3.4 states that the per-model thresholds (0.1, 0.04, 0.66, 0.33) are 'estimated' from the test-set ROC curves themselves, and the resulting segmentations are then used to compute dFN, dFP, Dice, and completeness on the same test set. This procedure selects each model's operating point on the evaluation data, which can systematically favor models with more favorable ROC curvature near the chosen leakage level and inflate apparent differences. The effect is amplified by the small test set (12 CTs) and by the large spread in selected thresholds, which indicates very different probability calibrations across models. If the threshold choices are partly overfit to the test set, the reported dFN advantage of UGnnReg and UGnnDyn over UNetLev3 may not replicate on new scans. This concern does not question the architecture's plausibility; it questions whether the specific quantitative claim is established by the current evaluation.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a hybrid 3D UNet-GNN architecture for airway segmentation from chest CTs. The two convolutional layers at the deepest encoding level of a 3D UNet are replaced by a module of four graph-convolutional layers operating on a graph whose nodes are derived from the downsampled feature maps. Two adjacency strategies are tested: a fixed 26-neighbour regular grid and a dynamically computed kNN adjacency in feature space. Using 32 low-dose CTs from the Danish Lung Cancer Screening Trial (16 train, 4 validation, 12 test), the authors compare four models: UNetLev3, UNetLev5, UGnnReg, and UGnnDyn. At a fixed 13% volume-leakage operating point, selected by choosing per-model probability thresholds from the test-set ROC curves, the UNet-GNN models show significantly lower false-negative centreline distance dFN and significantly higher false-positive centreline distance dFP relative to UNetLev3, with no significant differences in Dice or airway completeness. The paper concludes that the GNN-based models can segment slightly more complete airway trees with fewer trainable parameters.","tokens_in":6668,"tokens_out":2691,"duration_ms":29279,"significance":"If the reported dFN improvement is a true effect, the paper provides a useful architectural contribution: replacing the deepest UNet convolutions with graph convolutions can reduce false-negative branch detection at fixed leakage, with a smaller parameter count than the baseline. The authors are transparent about the mixed result pattern (lower dFN but higher dFP, no significant change in Dice or completeness) and about the computational constraints of the GNN module. The work is an empirical evaluation rather than a derivation, and the comparison is grounded in held-out test scans. However, the statistical evidence is weakened by the joint selection of operating thresholds and test-set evaluation, and by multiple uncorrected paired t-tests on only 12 scans. The central claim is therefore plausible but not yet securely established.","major_comments":[{"comment":"The per-model probability thresholds (0.1, 0.04, 0.66, 0.33) are estimated from the test-set ROC curves themselves to achieve a fixed 13% volume leakage, and the resulting segmentations are then used to compute dFN, dFP, Dice, and completeness on the same test set. This procedure selects each model's operating point on the evaluation data, which can systematically favour models with particular ROC curvature and inflate apparent differences between models. The large spread in the selected thresholds also indicates substantially different probability calibrations across models, making it unclear whether the reported dFN gain would generalize to new scans. To support the central claim, the authors should either tune thresholds on a separate validation set (e.g., within cross-validation) or demonstrate that the dFN differences are robust over a range of leakage levels and thresholds rather than only at the single test-selected point.","section":"Section 3.4, Figure 2"},{"comment":"The reported significance values (e.g., p = 0.001 for UGnnDyn vs. UNetLev3 in dFN, p < 0.001 for both GNN models vs. UNetLev3 in dFP) come from multiple two-sided paired t-tests performed across four models and several metrics on a test set of only 12 scans. No correction for multiple comparisons is applied, and with this sample size the p-values are sensitive to a few scans. The authors should report effect sizes, confidence intervals, or corrected p-values, and at minimum state how many comparisons were made. As it stands, the p-values do not provide strong evidence for the specific claim that the GNN variants are superior to UNetLev3 in dFN.","section":"Section 4, Results"},{"comment":"The conclusion that the proposed models \"can segment slightly more complete airway trees\" is not fully supported by the result pattern: dFN improves significantly, but dFP worsens significantly, while Dice and airway completeness are unchanged. A decrease in false-negative centreline distance cannot be equated with a clinically meaningful gain when it is accompanied by a corresponding increase in false positives. The paper should discuss the trade-off more explicitly and, if the authors wish to claim an overall benefit, justify why reducing false negatives is more desirable than increasing false positives for the airway segmentation task.","section":"Section 5, Conclusions"}],"minor_comments":[{"comment":"The sentence \"Initial work of combining CNNs and GNNs was by proposed Shin et al.\" contains a word-order error; it should read \"was proposed by Shin et al.\".","section":"Section 1, Introduction"},{"comment":"The dynamic graph adjacency is computed with k = 26 neighbours, the same as the regular 26-neighbour case, but the search space is constrained to a cube 5 voxels away. It would be helpful to clarify whether the self-connection is included in the adjacency and how the degree matrix in Eq. (1) is defined in that case.","section":"Section 2.2"},{"comment":"The statement that the dynamic graph enables nodes to access \"information well beyond their directly connected neighbours\" is somewhat undercut by the 5-voxel search-space constraint. Please clarify how much of the long-range benefit actually survives the constraint.","section":"Section 2.2"},{"comment":"In the right-hand detailed view, the markers for the operating points at threshold 0.5 should be clearly described in the caption or legend, as the current figure legend is not fully self-contained.","section":"Figure 2"},{"comment":"The loss function in Eq. (3) is written as a single sum over the region of interest, but the dice loss typically involves a sum over classes or a sum over the full image with a masked ground truth. Please clarify the notation so that the masking operation is unambiguous.","section":"Section 3.3"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a well-written empirical study with an honest presentation of results, but the main quantitative claim relies on test-set-dependent threshold selection and multiple uncorrected tests. The authors could reasonably address these issues in a revision with a validation-based threshold selection or a sensitivity analysis; the contribution itself is modest but within the scope of the journal, and the architecture is sufficiently novel to warrant further consideration after strengthened statistical evidence."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe thing to know about arXiv:1908.08588 is that it is a genuine architectural contribution wrapped in an evaluation that needs another pass. The authors replace the two convolutions at the deepest level of a 3D UNet with graph convolutions, trained end-to-end, and compare two adjacency schemes: a fixed 26-neighbour grid and a dynamic k-NN adjacency in feature space. As far as I can tell from the cited work, that end-to-end combination is new; [8] used a GNN only for refinement and [9] trained the CNN and GNN sequentially. The dynamic adjacency in particular is a reasonable idea worth reporting.\n\nThe paper is also honest and clearly written. The baseline is not a strawman; they compare against a larger 5-level UNet as well as the 3-level baseline. They report the parameter counts, and the UNet-GNN models use roughly half the parameters of the 3-level UNet. The main result—a small but statistically significant reduction in false-negative centreline distance at fixed 13% volume leakage—is stated modestly, and the dice and completeness are on par. That pattern is coherent with the claim that the GNN helps detect more peripheral branches.\n\nThe soft spot is exactly where the stress-test note points. The per-model probability thresholds (0.1, 0.04, 0.66, 0.33) are estimated from the test-set ROC curves to hit 13% leakage, and then the same test segmentations are used to compute dFN and dFP. That means the operating point is selected on the evaluation data. With a spread that wide in thresholds, the models are calibrated very differently, and the comparison at a matched leakage level may partly reflect ROC curvature luck rather than a stable model advantage. On 12 test scans, the paired t-tests also should be corrected for multiple comparisons; the raw p-values (0.001 to 0.01) would likely survive a Bonferroni correction for a few tests, but they would be less impressive. There is no external validation and no code, so the reproducibility of the main number is unverified.\n\nNone of this kills the architectural idea. The problem is that the central quantitative claim is not yet secure. If the thresholds had been chosen on the validation set or fixed a priori, I would trust the dFN result. As it stands, I would treat it as a promising but unconfirmed effect.\n\nThis paper deserves a serious referee. The right outcome is probably a revise-and-resubmit asking for a cleaner evaluation protocol—threshold selection on validation data or a fixed threshold, plus a sensitivity analysis around the leakage level. It would be a useful contribution to the airway segmentation and CNN+GNN literature once the numbers are placed on firmer ground.\n\nFor reading group, I would maybe bring it; it is a good example of an honest but under-validated architecture paper.","headline":"A genuinely novel UNet+GNN architecture whose reported dFN gain is not yet secure because the operating thresholds are chosen from the same test ROC.","tokens_in":7180,"tokens_out":3296,"would_cite":false,"duration_ms":32641,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that replacing the deepest convolutions of a 3D UNet with graph convolutions finds more peripheral airway branches in chest CTs, at the same leakage and with fewer parameters.","keywords":["airway segmentation","chest CT","graph neural network","3D UNet","graph convolution","centreline distance","tree-like structure segmentation"],"falsifier":"Run the same four models on an independent held-out set of chest CTs with thresholds fixed using only the training and validation ROC curves, and check whether the UNet-GNN models still show significantly lower $d_{FN}$ at 13% volume leakage; if the gain vanishes, it was an artifact of test-set threshold tuning.","tokens_in":6231,"feed_emoji":"🫁","tokens_out":7097,"duration_ms":63529,"temperature":0.7,"pith_summary":"This paper asks whether a 3D UNet for segmenting airway trees from chest CTs can be improved by replacing the convolutional layers at its deepest level with graph convolutions. The authors build two variants, one with a fixed 26-neighbour graph and one whose adjacency is recomputed during training from nearest neighbours in feature space, and compare them with the baseline UNet. They report a small but significant improvement in false-negative centreline distance, meaning more peripheral branches are detected, with no significant change in Dice overlap or airway completeness at a fixed 13% volume leakage, and with roughly half the trainable parameters of the baseline. If the improvement holds, it suggests that letting segmentation decisions draw on node-connectivity information beyond local image features helps recover thin, branch-like structures.","feed_headline":"Graph neural layers find more airway branches in chest CTs","feed_subtitle":"Graph convolutions at the UNet's deepest level find more airway branches at equal leakage, with fewer parameters.","key_machinery":"The load-bearing component is the GNN module placed at the deepest level of the 3D UNet, replacing the two convolutional layers of the baseline. The dense feature maps at that level are treated as graph nodes (supervoxels), each carrying a feature vector, with connectivity defined by a binary adjacency matrix $A$; the update rule in Eq. (1) combines each node's own transformed features with the degree-normalized features of its neighbours, and stacking four such layers lets nodes access higher-order neighbourhood information. Two adjacency constructions are tested: a predefined regular 26-neighbour grid and a dynamic adjacency recomputed during training from the $k=26$ nearest neighbours in feature space, constrained to a cube of at most 5 voxels away to fit in GPU memory.","core_discovery":"The central claim is that inserting a graph neural network module, consisting of four stacked graph-convolution layers of the form $H^{(l+1)} = \\sigma(H^{(l)}W_0^{(l)} + D^{-1}AH^{(l)}W_1^{(l)})$, at the deepest resolution level of a 3D UNet improves airway-tree segmentation in a specific, measurable way. Compared with the baseline 3-level UNet, the UNet-GNN models achieve a significantly lower false-negative centreline distance $d_{FN}$ ($p \\le 0.01$ for both adjacency types) at an operating point fixed at 13% volume leakage, while Dice overlap, airway completeness, and leakage show no significant difference. The authors interpret the lower $d_{FN}$ as evidence that the models produce slightly more complete airway trees with more or longer peripheral branches, and they note this comes with fewer trainable parameters (about 50k versus about 90k for the baseline; a deeper 5-level UNet with about 1.4M parameters shows no such gain). The two adjacency choices, fixed 26-neighbour connectivity and a dynamically computed $k$-nearest-neighbour graph in feature space with $k=26$, perform similarly.","pith_inferences":["Placing GNN modules at multiple resolution levels, not just the deepest, is a testable extension that could amplify the effect if connectivity information helps at several scales; the paper notes this was not attempted due to GPU memory limits.","The same architecture could transfer to other elongated or tree-like structures in medical imaging, such as vessels, bronchi, or neurons, where peripheral branch detection is the bottleneck and centreline-distance evaluation is the appropriate metric.","Because the dynamic adjacency is limited to a 5-voxel search cube during training, the model only approximates true long-range connectivity; relaxing that constraint with more memory could either strengthen or weaken the reported gain.","If test-set threshold tuning is a major source of the improvement, fixed-threshold deployment on new scans would likely narrow the gap; this is worth checking before clinical use."],"forward_implications":["Replacing the deepest convolutional block with graph convolutions yields a small but significant reduction in false-negative centreline distance, suggesting more complete airway trees at equal leakage.","Dice overlap and airway completeness are unchanged, so the gain is specific to branch detection as measured along the centreline rather than to overall volume overlap.","The UNet-GNN models need about 50k trainable parameters versus about 90k for the baseline and about 1.4M for the 5-level UNet, so graph convolutions can substitute for model capacity in this setting.","A deeper 5-level UNet does not show the same $d_{FN}$ improvement, suggesting extra resolution depth is not the mechanism behind the gain.","Both fixed and dynamically computed graph adjacency produce similar results, so the benefit is not sensitive to whether connectivity is predefined or learned in feature space."],"supporting_citations":[{"why":"Supplies the UNet architecture that the baseline and the UNet-GNN variants are built from.","marker":"[2]"},{"why":"Supplies the graph-convolution update rule with a normalized adjacency matrix used in the GNN module.","marker":"[4]"},{"why":"Documents that airway-extraction methods miss many small peripheral branches, the problem this method targets.","marker":"[5]"},{"why":"Provides the 3D UNet baseline model whose deepest convolutions are replaced by the GNN module.","marker":"[7]"},{"why":"Contributes the earlier graph-neural-network airway extraction approach and the dFN/dFP centreline evaluation measures used for comparison.","marker":"[8]"},{"why":"Prior CNN-plus-GNN vessel segmentation pipeline, the sequential baseline that this work makes joint and end-to-end.","marker":"[9]"},{"why":"Used to generate the reference airway lumen segmentations, with expert correction, that define the ground truth.","marker":"[10]"},{"why":"Supplies the 32 chest CT scans from the screening cohort used in the experiments.","marker":"[13]"}],"fun_headline_variants":["GNN module fills in more airway branches in chest CTs","Graph convolutions cut missed airway branches in CT scans","UNet + graph nets finds extra airway branches with fewer parameters","Graph-enhanced UNet improves airway tree completeness in CT","Lower false-negative airway distance via graph neural layers"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison relies on probability thresholds (0.1, 0.04, 0.66, 0.33) chosen from the test-set ROC curves to fix leakage at 13%, and if those thresholds do not transfer to new scans, the reported improvement in false-negative distance may not reflect real-world performance.","fun_headline_variants_meta":{"raw":{"variants":["GNN module fills in more airway branches in chest CTs","Graph convolutions cut missed airway branches in CT scans","UNet + graph nets finds extra airway branches with fewer parameters","Graph-enhanced UNet improves airway tree completeness in CT","Lower false-negative airway distance via graph neural layers"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000589,"raw_usage":{"total_tokens":2807,"prompt_tokens":1032,"completion_tokens":1775,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":648,"completion_tokens_details":{"reasoning_tokens":1696}},"tokens_in":648,"tokens_out":1775,"duration_ms":12577,"temperature":1.0,"reasoning_tokens":1696,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T11:35:00.222912+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same four models on an independent held-out set of chest CTs with thresholds fixed using only the training and validation ROC curves, and check whether the UNet-GNN models still show significantly lower $d_{FN}$ at 13% volume leakage; if the gain vanishes, it was an artifact of test-set threshold tuning.","supporting_citations":[{"cited_title":"Medical Image Computing and Computer-Assisted Inter- vention MICCAI, pp","cited_arxiv_id":null,"evidence_quote":"Supplies the UNet architecture that the baseline and the UNet-GNN variants are built from."},{"cited_title":"and Welling, M.: Semi-supervised classification with graph convolutional networks","cited_arxiv_id":null,"evidence_quote":"Supplies the graph-convolution update rule with a normalized adjacency matrix used in the GNN module."},{"cited_title":"et al: Extraction of Airways From CT (EXACT09)","cited_arxiv_id":null,"evidence_quote":"Documents that airway-extraction methods miss many small peripheral branches, the problem this method targets."},{"cited_title":"Thoracic Image Analysis Workshop, pp 238-250, 2018","cited_arxiv_id":null,"evidence_quote":"Provides the 3D UNet baseline model whose deepest convolutions are replaced by the GNN module."},{"cited_title":"1st Conference on Medical Imaging with Deep Learning, Amsterdam, 2018","cited_arxiv_id":null,"evidence_quote":"Contributes the earlier graph-neural-network airway extraction approach and the dFN/dFP centreline evaluation measures used for comparison."},{"cited_title":"Deep Vessel Segmentation By Learning Graphical Connectivity","cited_arxiv_id":"1806.02279","evidence_quote":"Prior CNN-plus-GNN vessel segmentation pipeline, the sequential baseline that this work makes joint and end-to-end."},{"cited_title":"Medical image analysis 18, pp","cited_arxiv_id":null,"evidence_quote":"Used to generate the reference airway lumen segmentations, with expert correction, that define the ground truth."},{"cited_title":"Journal of Thoracic Oncology 4.5, pp","cited_arxiv_id":null,"evidence_quote":"Supplies the 32 chest CT scans from the screening cohort used in the experiments."}],"review_version":1}