{"id":"74bc302b-783e-4ec0-be79-feb72a0f9158","arxiv_id":"2506.11677","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A competition solution that segments airways with nn-Unet and uses trachea-focused radiomic features in an SVM to predict 63-week survival in lung fibrosis patients.","lead":"This paper describes a three-stage machine learning pipeline for the AIIB 2023 challenge, using nn-Unet to segment airways in CT scans and an SVM to predict patient survival from radiomic features of the trachea and airway. The authors report a segmentation score of 0.8601 and a classification score of 0.7346 on the challenge validation set.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Combined-feature advantage rests on ~2 patients in 52 validation scans and on validation-set model selection, so the score is not statistically distinguished from the trachea-only baseline.","rationale":"The reader's verdict is already CONDITIONAL, and my concern does not move it; it sharpens the required condition. I considered the trachea-separation heuristic, which the reader named as the weakest assumption, and it is genuinely under-specified: Section 2.2 is ambiguous about whether components are removed above or below z=D/3, and no anatomical validation of the resulting mask is provided. However, the classification claim would remain fragile even with a perfect trachea mask, because the evidence for 'combined features outperform' is a two-patient accuracy gap on 52 validation scans with no error bars, and the same leaderboard was used to select hyperparameters, correlation thresholds, and the feature combination. The paper provides a useful engineering description, and the segmentation score is an independently evaluated leaderboard result, so there is no reason to suspect fabrication; the issue is that the survival-biomarker conclusion is overinterpreted relative to the statistical evidence. A paired McNemar test or an equivalent re-analysis is sufficient to settle whether the gap is real.","tokens_in":4760,"tokens_out":7923,"duration_ms":81446,"concrete_test":"Obtain per-patient validation predictions for all five feature sets in Table 2 (the authors should release them or a runnable Docker image) and apply McNemar's exact test to the paired Combined(Trachea, Airway) vs Only Trachea predictions. With n=52 and an accuracy gap of about 0.0385, the entire advantage is roughly two patients; if the p-value exceeds 0.05, the claim that the combined features outperform the trachea-only set is not supported. Also require the authors to disclose the number of validation submissions used to select hyperparameters, thresholds, and the feature combination, since selection among many leaderboard scores would make the 0.7346 figure optimistically biased.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that Combined(Trachea, Airway) radiomic features outperform other feature sets is supported only by Table 2, computed on 52 validation scans whose labels were obtained through competition submissions. The best row differs from Only Trachea by 0.7308 vs 0.6923 accuracy (a gap of 0.0385, about 2 patients out of 52) and by 0.0577 F1; no confidence intervals or significance tests are reported, and these differences are well within the noise of a 52-sample paired evaluation. The 0.7346 headline score is also a selected maximum: the authors used grid search for SVM hyperparameters (C=8000, gamma=0.01), chose correlation thresholds (0.20 for trachea, 0.41 for airway), and then selected the best of several feature sets from the same validation leaderboard. Selecting among many configurations on the evaluation set produces an optimistically biased estimate, so the reported value is not an unbiased estimate of generalization. The Section 2.2 trachea-separation heuristic is indeed under-specified, but even a perfect trachea mask would not rescue the survival-biomarker conclusion, because the only evidence of its superiority is a small, selected, non-significant validation gap.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper describes a three-stage pipeline for the AIIB 2023 challenge Task 2 (predicting whether a lung fibrosis patient survives beyond 63 weeks). First, nn-Unet segments the airway tree in CT; second, post-processing splits the airway into a trachea mask and an airway bounding box, from which PyRadiomics features are extracted; third, an SVM with RBF kernel classifies patients. The authors report an Overall-Score of 0.8601 for the segmentation task and 0.7346 (Table 2, accuracy 0.7308) for the survival classification on the validation set, and claim that combining trachea and airway features outperforms other feature sets.","tokens_in":5031,"tokens_out":2497,"duration_ms":24839,"significance":"If the reported results were unbiased, the paper would offer a useful engineering contribution and a testable hypothesis about tracheal radiomics as survival biomarkers in fibrotic lung disease. The strengths are the use of a well-established segmentation tool (nn-Unet), a reproducible PyRadiomics feature extraction pipeline, and a clear three-stage description. The main value is the negative/explanatory finding that simple radiomics on a competition validation set may not support strong claims of biomarker superiority. However, the current evaluation is not sufficient to establish the central claims, because the validation set was used for model selection and the reported differences are within sampling noise.","major_comments":[{"comment":"The validation set serves simultaneously as the model-selection set and the evaluation set: SVM hyperparameters (C=8000, gamma=0.01) were chosen by grid search, Pearson correlation thresholds (0.20 and 0.41) were set based on validation performance, and the best of several feature sets was selected from the same validation leaderboard. Consequently, the reported accuracy of 0.7308 (Overall-Score 0.7346) is an optimistically biased estimate of generalization, not an unbiased test. The paper should provide a proper evaluation protocol, e.g., nested cross-validation on the 95 labeled training patients or evaluation on an independent test set, together with confidence intervals.","section":"§2.3 and §3"},{"comment":"The central claim that Combined(Trachea, Airway) outperforms other feature sets rests on a difference of 0.0385 in accuracy (0.7308 vs 0.6923) and 0.0577 in F1-score, i.e., roughly 2 out of 52 validation patients. With no significance test, no confidence intervals, and multiple feature sets compared on the same data, this difference is well within the expected sampling variability. The authors should either demonstrate that the gap is statistically reliable or temper the conclusion to a hypothesis-generating observation.","section":"Table 2"},{"comment":"The trachea-isolation rule is under-specified and load-bearing for all subsequent radiomic features. The description 'retaining solely the biggest connected component within the upper third of the axial plane and omitting them within the lower third' leaves the spatial reference ambiguous (z-coordinate relative to the volume? anatomical landmarks?) and does not state how the 2D-slice operations in Figure 1 correspond to a 3D mask. The authors should specify the exact definition, provide example masks, and assess the sensitivity of the classification results to the choice of the D/3 threshold.","section":"§2.2"},{"comment":"The 'No Trachea' and 'Full Airway' rows show specificity of 0.1923 and 0.1538, respectively, meaning the classifier labels nearly all patients as survivors. This suggests a strong class-imbalance or calibration problem, yet the paper does not discuss how class proportions (36 deaths vs 59 survivors in training) are handled. The authors should report the class distribution in the validation set and clarify whether any class-weight or threshold adjustment was applied.","section":"§3, 'No Trachea' row"}],"minor_comments":[{"comment":"Typo: 'Radiomic Features Extration' should be 'Extraction'.","section":"§2.2 heading"},{"comment":"Typo: 'anOverall-Score' should be 'an Overall-Score'.","section":"Abstract"},{"comment":"The formula for Overall-Score is missing an operator before the leakage term: it reads '(1−leakages)⋆0.3'.","section":"§1.1 Evaluation"},{"comment":"The paper states that 5-fold cross-validation was performed on the training set, but it is not reported which fold/model was used for the validation predictions. Please clarify whether the validation predictions come from a single model retrained on all training data or from an ensemble of folds.","section":"§2.3"},{"comment":"The paper says the results are 'computed on the validation set' but does not give the number of features retained after Pearson correlation for each mask, nor the total number of features fed to the SVM. Reporting these numbers would improve reproducibility.","section":"§3"}],"recommendation":"major_revision","confidential_remarks":"The paper is a competition-solution report and is honest in describing the pipeline, but it overinterprets a validation-set leaderboard result. The load-bearing issue is that the reported scores are not valid generalization estimates, and the main comparative claim is not statistically supported. This is fixable by re-evaluating with nested cross-validation on the 95 labeled patients, but the revised version would need to present the corrected numbers and temper the conclusions accordingly. The trachea-isolation heuristic also needs concrete specification. The fit with the journal's scope is acceptable as a technical application paper, but the novelty is limited."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a clean write-up of an AIIB 2023 competition solution. The segmentation pipeline is solid; nn-Unet with post-processing reaching 0.86 Overall-Score is a credible result. The genuinely new bit is the attempt to isolate the trachea and test whether trachea radiomics carry survival signal. That hypothesis is worth taking seriously, and the engineering heuristic (largest connected component in the upper third of the axial plane) is simple and reproducible.\n\nThe soft spot is not the heuristic; it's the evidence that combined trachea+airway features beat other feature sets. Table 2 is computed on 52 validation scans, and the winning gap over trachea-only is 0.0385 accuracy—roughly two patients. No confidence intervals, no test. The authors also selected SVM hyperparameters (C=8000, gamma=0.01), correlation thresholds (0.20, 0.41), and the winning feature set using the same validation set they then report as the result. That produces an optimistically biased number, and with 52 samples the uncertainty blows any 0.03–0.05 gap wide open. The paper doesn't claim significance, but it does present Combined(Trachea, Airway) as 'the most favorable,' which is only true on this one small, self-selected fold.\n\nThe missing code and data don't help, though the trachea separation rule is described well enough to re-implement. I also want to flag an internal inconsistency: the abstract says 0.7346 for classification, but Table 2 shows 0.7308 accuracy for the combined feature set. That's a minor mismatch but worth fixing.\n\nWho is this for? Anyone looking for a baseline or a sanity check in the AIIB challenge, and radiomics researchers interested in trachea-specific features. It's not yet a clinical biomarker paper. I'd send it to review only with the expectation that the authors disclose the validation-selection procedure, report test set scores if available, and add at least a paired bootstrap or a simple significance test on the gap. Without that, the central claim about combined features is speculation on two patients. If the editors want a competition-report venue, it's fine as-is; if the claim is meant to support a biological finding, it needs independent validation.","headline":"A competent AIIB 2023 competition write-up: solid segmentation, but the central survival-biomarker claim is a validation-selected 2-patient gap with no error bars.","tokens_in":5549,"tokens_out":3327,"would_cite":false,"duration_ms":31042,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a three-stage pipeline using nn-Unet airway segmentation, radiomic features from the trachea and the airway's bounding box, and an SVM classifier predicts 63-week survival in lung-fibrosis patients, achieving an…","keywords":["survival analysis","segmentation","deep learning","radiomics","airway biomarkers","lung fibrosis","trachea","SVM classification"],"falsifier":"A concrete test: on the same CT images, replace the automatic trachea mask with a same-size cylinder of lung tissue in the upper thorax and rerun the SVM; if classification accuracy does not drop, the claimed trachea-specific signal is an artifact. More directly, an independent cohort with manual expert trachea annotations could test whether the automatically isolated trachea reproduces the reported 0.7346 score, and a permutation test on the 120 training cases' labels would settle whether the combined-feature model beats chance despite the small validation set.","tokens_in":4597,"feed_emoji":"🫁","tokens_out":6079,"duration_ms":52042,"temperature":0.7,"pith_summary":"The paper claims that survival after a 63-week horizon in lung-fibrosis patients can be predicted from CT images by first segmenting the airways with nn-Unet, then extracting radiomic features from two regions (the trachea and a minimum bounding box around the entire airway), and finally classifying with a support vector machine. On the AIIB 2023 validation set this three-stage pipeline achieves an overall score of 0.7346 for survival classification, and the combination of trachea and bounding-box-airway features outperforms features from the full airway, trachea alone, or non-trachea regions. The authors interpret this as evidence that the trachea's shape and the overall airway dimensions carry survival-relevant information that standard whole-volume radiomics miss.","feed_headline":"Trachea and airway radiomics predict fibrosis survival at 0.7346","feed_subtitle":"nn-Unet segmentation then SVM on trachea and bounding-box features beats full-airway radiomics.","key_machinery":"The load-bearing mechanism is the trachea-isolation heuristic: in each predicted airway mask, the largest connected component in the upper third of the axial plane is retained and components in the lower third are removed, producing a trachea mask, while the complement yields the non-trachea mask. Radiomic features from the trachea mask and from a minimum bounding box around the full airway mask are then computed with an open-source radiomics package and filtered by Pearson correlation (threshold 0.20 for trachea, 0.41 for airway). These selected features are fed to a support vector machine with an RBF kernel, whose hyperparameters (C=8000, gamma=0.01) are chosen by grid search using 5-fold cross-validation. The mechanism ties together the segmentation quality, the region-choice heuristic, and the feature-selection thresholds.","core_discovery":"The central claim is that a region-wise radiomic analysis of the segmented airway tree, rather than whole-lung or whole-airway radiomics, yields a better survival classifier for fibrotic lung disease. The paper shows that combining radiomic features extracted from the isolated trachea mask with those extracted from a minimum bounding box around the full airway mask produces the best validation-set metrics (accuracy 0.7308, F1-score 0.75, overall score 0.7346), supporting the hypothesis that the trachea's compressed or elongated appearance and the overall airway width are informative biomarkers. The claim is established through a specific pipeline: nn-Unet 3D full-resolution segmentation, a connected-component heuristic to separate the trachea, radiomic feature extraction, Pearson-correlation feature selection at thresholds of 0.20 and 0.41, and an RBF-kernel SVM with C=8000 and gamma=0.01.","pith_inferences":["An editorial extension: the heuristic of 'largest connected component in the upper third' is a proxy for anatomical trachea location; a direct comparison against expert trachea annotations on the same CT scans would test whether the survival signal is truly tracheal or merely a generic upper-airway effect.","Because the validation set has only 52 patients and labels were evaluated by a leaderboard, the reported 0.7346 overall score carries wide uncertainty; a permutation test or leave-one-center-out evaluation on the 120-patient training set would gauge whether the feature combination generalizes beyond the competition split.","The Pearson correlation thresholds (0.20 and 0.41) appear tuned to the validation set; a sensitivity analysis sweeping these thresholds would reveal whether the advantage of the combined feature set is stable or an artifact of the chosen cutoffs.","A testable extension: feed the same bounding-box and trachea radiomic features into a Cox proportional-hazards model on time-to-death data rather than the binarised 63-week outcome, to see whether the same features carry proportional-hazard information."],"forward_implications":["If the central claim holds, airway radiomics centered on the trachea and the airway's minimum bounding box can serve as a reproducible, image-based biomarker for survival in fibrotic lung disease, complementing or replacing lab-dependent FVC measurements.","The pipeline can be applied automatically to a single CT scan without manual annotation, since nn-Unet supplies the masks and the trachea is isolated by a deterministic connected-component rule.","The superior performance of Combined(Trachea, Airway) over full-airway features suggests that fine-grained regional radiomics, rather than whole-structure summaries, should be the default design for similar survival-classification tasks.","The validation on 52 patients in the AIIB 2023 validation set supports pursuing larger prospective studies of tracheal radiomics as a prognostic marker."],"supporting_citations":[{"why":"Supplies the AIIB 2023 challenge dataset and task definitions, the data on which the segmentation and classification are evaluated.","marker":"[3]"},{"why":"Supplies the airway segmentation benchmark and the COVID-19/fibrosis scans used in the challenge, providing the testbed for the pipeline.","marker":"[4]"},{"why":"Provides the nn-U-Net architecture and self-configuration method used in Stage 1 to segment the airways automatically.","marker":"[5]"},{"why":"Provides the radiomics feature extraction algorithm used to compute the trachea and bounding-box features for the SVM classifier.","marker":"[6]"}],"fun_headline_variants":["Trachea plus airway bounding box predicts fibrosis survival","nn-Unet and SVM: trachea and airway box predict survival","Airway radiomics: trachea and box beat whole-airway","0.7346 score: trachea and airway box radiomics"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the largest connected component in the upper third of the axial CT slices reliably isolates the trachea, and that the trachea's appearance plus the overall airway dimensions actually carry survival-relevant information; if this region-selection heuristic fails or the radiomic signal is noise, the classifier's edge over whole-airway features collapses.","fun_headline_variants_meta":{"raw":{"variants":["Trachea plus airway bounding box predicts fibrosis survival","nn-Unet and SVM: trachea and airway box predict survival","Airway radiomics: trachea and box beat whole-airway","0.7346 score: trachea and airway box radiomics"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00124,"raw_usage":{"total_tokens":5055,"prompt_tokens":878,"completion_tokens":4177,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":494,"completion_tokens_details":{"reasoning_tokens":4105}},"tokens_in":494,"tokens_out":4177,"duration_ms":28869,"temperature":1.0,"reasoning_tokens":4105,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T04:02:56.721896+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete test: on the same CT images, replace the automatic trachea mask with a same-size cylinder of lung tissue in the upper thorax and rerun the SVM; if classification accuracy does not drop, the claimed trachea-specific signal is an artifact. More directly, an independent cohort with manual expert trachea annotations could test whether the automatically isolated trachea reproduces the reported 0.7346 score, and a permutation test on the 120 training cases' labels would settle whether the combined-feature model beats chance despite the small validation set.","supporting_citations":[{"cited_title":"Human treelike tubular structure segmentation: A comprehensive review and future perspectives, 12 2022","cited_arxiv_id":null,"evidence_quote":"Supplies the AIIB 2023 challenge dataset and task definitions, the data on which the segmentation and classification are evaluated."},{"cited_title":"Fuzzy attention neural network to tackle discontinuity in airway segmentation","cited_arxiv_id":null,"evidence_quote":"Supplies the airway segmentation benchmark and the COVID-19/fibrosis scans used in the challenge, providing the testbed for the pipeline."},{"cited_title":"nnU-Net : a self-configuring method for deep learning-based biomedical image segmentation","cited_arxiv_id":null,"evidence_quote":"Provides the nn-U-Net architecture and self-configuration method used in Stage 1 to segment the airways automatically."},{"cited_title":"Van Griethuysen, Andriy Fedorov, Chintan Parmar, Ahmed Hosny, Nicole Aucoin, Vivek Narayan, Regina G.H","cited_arxiv_id":null,"evidence_quote":"Provides the radiomics feature extraction algorithm used to compute the trachea and bounding-box features for the SVM classifier."}],"review_version":1}