{"id":"b639bfc2-1c87-48a5-9804-a3192539e5bd","arxiv_id":"2411.16823","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"Nodule-focused AI models outperformed whole-chest-region models for lung cancer classification on internal and external CT datasets, but external validity is threatened by possible pretraining data overlap.","lead":"A study compared two AI systems for lung cancer: one that focuses on individual lung nodules and one that looks at large chest regions. Across three CT datasets, the nodule-focused system scored higher, but the results may be inflated by how the models were pretrained.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"External validation may be contaminated because both models' pretrained weights come from prior work [7] that likely used LUNA16/NLST; the paper never discloses this separation.","rationale":"The reader's weakest assumption identifies the exact load-bearing concern: the independence of external validation from pretraining. This is the most critical point because the paper's headline claim of superior external performance is the only statistically striking result; the internal validation difference is small and not clearly significant. The Methods text explicitly attributes both models' initializations to earlier work [7], and the same earlier study is cited for the external datasets. The paper provides no disclosure of dataset separation, so the possibility of leakage is a concrete, checkable risk rather than a speculative one. I agree with the reader's verdict of rejection because the central claim is not supported without establishing this independence. The recommended verdict remains REJECT (UNCHANGED), as the reader's assessment already captures the necessary outcome. My independent reading also notes dataset-count inconsistencies and the lack of statistical details for subgroup comparisons, but these are secondary to the contamination risk, which alone is sufficient to invalidate the external conclusion.","tokens_in":7528,"tokens_out":3199,"duration_ms":29568,"concrete_test":"Inspect the prior work arXiv:2405.04605 [7] and the released code (https://github.com/fitushar/AI-in-Lung-Health-Benchmarking-Detection-and-Diagnostic-Models-Across-Multiple-CT-Scan-Datasets) to determine whether LUNA16 or NLST were used in any pretraining stage (e.g., SWS++ or false positive reduction). If they were, retrain both models with pretraining restricted to DLND24 only, then re-run external evaluation on LUNA16 and NLST; if the AUC gap narrows substantially (e.g., LUNA16 lesion-level drops from 0.90 toward the internal 0.71), the external claim is invalid. If they were not used, the concern is resolved and the reader's objection is unmet.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—'Lesion-level models demonstrate superior classification performance, especially for external datasets'—rests entirely on external AUCs (LUNA16: 0.90 vs 0.63; NLST: 0.81 vs 0.71). The Methods section 'AI Model Development' states that the lesion-level model uses ResNet50-SWS++ weights pretrained in an earlier study [7], and the chest-region model uses false positive reduction weights from [7]. The external datasets are introduced as 'LUNA16 and NLST used in an earlier study [7]'. The paper never states whether that earlier pretraining or model development used LUNA16 or NLST. If it did, the external test sets are not independent, and the reported external AUCs are inflated by data leakage. The internal validation difference (0.71 vs 0.68) has overlapping confidence intervals and is not statistically established, so the conclusion hinges almost entirely on the external numbers. The paper also contains internal inconsistencies in dataset counts (LUNA16: 888 in abstract, 601 in Table 1, 433 in Figure 1), further undermining confidence in the reported evaluation. Without clarity on the pretraining data, the external validation cannot be considered valid, and the primary conclusion is unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper compares two 3D CNN models for lung cancer classification from chest CT: a lesion-level model that takes 64x64x64 nodule-centric patches and a chest-region model that takes 512x512x8 chest volumes. Both models are trained on an internal dataset (DLND24) and evaluated on the same internal test set plus two external datasets, LUNA16 and NLST. The authors report higher AUC-ROC for the lesion-level model in all settings (internal 0.71 vs 0.68; LUNA16 0.90 vs 0.63; NLST 0.81 vs 0.71), supplemented by subgroup analyses and gradient-based visualizations. The conclusion is that lesion-level models demonstrate superior classification performance, especially for external datasets.","tokens_in":7790,"tokens_out":4530,"duration_ms":41185,"significance":"If the comparison were valid, the finding that nodule-centric input representations outperform whole-chest input representations would provide a practical design guideline for lung cancer screening AI. The attempt to include external validation and subgroup analyses is commendable. However, the central claim rests on external AUC values that may be inflated by data contamination: both models use pretrained weights from a prior study [7] that, according to the manuscript, used LUNA16 and NLST. The internal AUC difference is not statistically significant, so the external results carry the entire argument. The paper does not provide code or sufficient training details, and there are major inconsistencies in the reported dataset sizes. These issues prevent the paper from making a reliable contribution as it stands.","major_comments":[{"comment":"The external validation is potentially contaminated. The lesion-level model uses ResNet50-SWS++ weights pretrained in [7], and the chest-region model uses false-positive-reduction weights from [7]. The manuscript states that LUNA16 and NLST were 'used in an earlier study [7, 11]' but never discloses whether the pretraining in [7] included these datasets. If they did, the external test sets are not independent, and the reported external AUCs (LUNA16 0.90, NLST 0.81) would be inflated by data leakage. This is load-bearing because the paper's primary conclusion—'Lesion-level models demonstrate superior classification performance, especially for external datasets'—depends entirely on those external numbers. The authors must clarify the composition of the pretraining data, and if the external datasets were used, must retrain without them or use genuinely held-out external data.","section":"Materials and Methods, AI Model Development; Patient Data and Imaging Datasets"},{"comment":"The internal AUC difference is not statistically established. The Methods section states that DeLong's test was used, but no p-values are reported anywhere in the Results. The internal lesion-level AUC of 0.71 (95% CI 0.61–0.81) and chest-region AUC of 0.68 (95% CI 0.57–0.77) have heavily overlapping confidence intervals, so the claim of 'consistently outperformed' is not supported for the internal data alone. Without a DeLong p-value or some equivalent statistical test, the internal comparison is inconclusive.","section":"Results, Overall Performance"},{"comment":"The dataset counts are internally inconsistent. The Methods section states LUNA16 includes 888 CT scans; Table 1 reports 601 patients for LUNA16; and Figure 1 reports LUNA16 N=433 with 677 nodules. Similarly, NLST is given as 969 patients in Table 1 and Figure 1, but Figure 1 reports 3128 nodules, while Table 1's nodule-level counts sum to 1192 malignant nodules with no benign count provided. The internal dataset is called DLND24 in the abstract but appears as 'Duke Lung Cancer Screening Dataset' in Table 1 and as DLCSD in the data availability statement. These discrepancies undermine confidence in the reported evaluation and must be reconciled before any further consideration.","section":"Table 1 and Figure 1"},{"comment":"The subgroup analysis claims are not supported by reported statistics. The text states that the lesion-level model showed 'significant advantages' for current smokers, adenocarcinoma, and certain CT manufacturers, but no p-values, confidence intervals, or multiple-comparison corrections are reported for these subgroups. Figure 3 and Figure 4 show error bars, but the corresponding numerical values and test statistics are absent from the text. Without this information, the subgroup conclusions are unverifiable.","section":"Results, Subgroup Analysis"}],"minor_comments":[{"comment":"The phrase 'Area under the curve-curve-receiver operating Characteristic (AUC-ROC)' contains a typo; it should be 'Area under the receiver operating characteristic curve.'","section":"Evaluation Metrics"},{"comment":"References [5] and [19] appear to be the same work (the SPIE Medical Imaging 2022 paper on co-occurring diseases), cited twice with different formatting. This should be consolidated.","section":"References"},{"comment":"Training details are insufficient for reproducibility: no learning rate, batch size, number of epochs, data augmentation, or hardware specifications are provided. The claim that models were 'trained end-to-end' is not enough.","section":"AI Model Development"},{"comment":"The interpretability analysis is used as supporting evidence for the model comparison, but the authors themselves note in the limitations that quantitative metrics for interpretability are needed. The visualizations should be treated as illustrative, not as a standalone confirmation of superiority.","section":"Discussion and Figure 6"}],"recommendation":"reject","confidential_remarks":"The manuscript's own text raises a serious circularity concern: the external datasets are described as 'used in an earlier study' that produced the pretrained weights for both models, yet the paper never establishes separation. Combined with the non-significant internal difference and the internal inconsistencies in dataset counts, the central claim is not supportable. The author is a single author and self-cites several prior works; the dataset availability points to a public Zenodo record, but the evaluation appears sloppy. In my view, the issues are load-bearing and would require new experiments or a major rewrite to address, so I cannot recommend acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nThe short version: this paper asks a good question—whether lesion-level patches beat chest-region patches for lung cancer classification—but the main result isn't defensible as reported because the external validation may be contaminated. If the pretrained weights from the author's earlier work [7] were trained on LUNA16 or NLST, then the external AUCs are inflated and the conclusion falls apart. The paper never says whether that is the case.\n\nWhat's new here: a head-to-head comparison of two patch strategies, with subgroup analyses and external validation, plus interpretability maps. That is a reasonable study design, and it's useful to see the attempt to check generalizability across datasets. The patch specifications and preprocessing are described clearly.\n\nThe soft spots are serious. First, the leakage issue: the Methods say the lesion model uses ResNet50-SWS++ pretrained in [7] and the chest model uses false-positive-reduction weights from [7]. The external datasets are described as 'LUNA16 and NLST used in an earlier study [7]'. Nothing rules out that [7] used those datasets for pretraining. This is not a minor detail; it's the backbone of the external comparison. Second, the internal AUC difference (0.71 vs 0.68) has overlapping confidence intervals, and no DeLong p-values are reported despite the Methods claiming DeLong's test. So the internal result alone is not significant. Third, the dataset numbers don't line up: abstract says LUNA16 has 888 scans, Table 1 says 601 patients, Figure 1 says 433. That makes the evaluation hard to trust. Finally, the subgroup analyses in the text lack CIs and multiple-testing corrections, and the conclusion overclaims clinical utility without any clinical baseline.\n\nOn balance, the study has a sensible design and a real question, but the load-bearing external result rests on an unverified assumption of independence. If the authors can confirm that pretraining never saw LUNA16 or NLST, and if they fix the counting errors, the comparison would be worth taking seriously. As it stands, the central claim is not supported. I'd send it to peer review because the question matters and the flaw is fixable with disclosure, but I would not accept it in this form.\n\nBest,\n[You]","headline":"Good question, weak evidence: the external validation may be contaminated by pretraining on the same datasets, and the internal comparison is not statistically supported, so the superiority claim is not backed by the data as reported.","tokens_in":8270,"tokens_out":2498,"would_cite":false,"duration_ms":22092,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Lesion-level models outperform chest-region models for lung cancer classification on both an internal test set and two external CT cohorts, with the largest difference on LUNA16 (AUC 0.90 versus 0.63).","keywords":["lung cancer","CT screening","lesion-level model","chest-region model","AUC-ROC","external validation","subgroup analysis","Grad-CAM"],"falsifier":"Check the training data behind the pretrained ResNet50-SWS++ and false-positive-reduction weights: if any LUNA16 or NLST scans appear there, rerun both models trained from scratch on DLND24 only and compare the resulting external AUCs; the reported 0.90 versus 0.63 LUNA16 gap should shrink or disappear.","tokens_in":7349,"feed_emoji":"🫁","tokens_out":8556,"duration_ms":72174,"temperature":0.7,"pith_summary":"The paper asks whether an AI lung-cancer classifier should look at the nodule alone or at the broader chest region, and it answers with a head-to-head comparison of two models trained on the same data. The lesion-level model, built on 64×64×64 nodule-centered patches, beats the chest-region model, built on 512×512×8 thoracic volumes, on every dataset tested: AUC 0.71 versus 0.68 internally, 0.90 versus 0.63 on LUNA16, and 0.81 versus 0.71 on NLST. Subgroup analyses show the lesion-level advantage persists across sex, smoking status, key histologies, and CT manufacturers, and gradient heatmaps show it concentrates on the nodule while the chest-region model spreads its attention. A sympathetic reader would take this as evidence that nodule-centric input is the right inductive bias for malignancy classification, with practical payoff in external generalizability and interpretability.","feed_headline":"Nodule-focused AI beats whole-chest model on lung cancer","feed_subtitle":"A tumor-centered 3D patch reaches AUC 0.90 on external LUNA16 scans, versus 0.63 with full-chest input.","key_machinery":"The comparison is carried by two 3D convolutional models that share training data, preprocessing, optimizer, and loss but differ in input focus. The lesion-level model uses ResNet50-SWS++, a ResNet-50 variant pretrained with Strategic WarmStart++, applied to 64×64×64 patches centered on each nodule. The chest-region model takes 512×512×8 volumes spanning the thorax and is initialized from the weights of a false-positive-reduction model from the same prior work. These inputs are the central object of the paper: the identity of everything else is held fixed, so differences in AUC-ROC, subgroup behavior, probability distributions, and Grad-CAM localization are attributed to lesion-centric versus chest-region focus.","core_discovery":"The central claim is that lesion-level models demonstrate superior classification performance, especially for external datasets and challenging subgroups. On DLND24, the internal held-out test set, the lesion-level model reaches AUC 0.71 (95% CI 0.61–0.81) versus 0.68 (0.57–0.77) for the chest-region model. The gap widens externally: on LUNA16, AUC 0.90 (0.87–0.92) versus 0.63 (0.58–0.67), and on NLST, AUC 0.81 (0.79–0.82) versus 0.71 (0.69–0.72). The paper further argues that the lesion-level model's predicted probabilities separate benign from malignant cases cleanly, while the chest-region model's distributions overlap, and that Grad-CAM-family visualizations localize to nodules only for the lesion-level model. The intended conclusion is that nodule-centric models are better suited for precision lung cancer diagnostics.","pith_inferences":["The paper does not disclose what data were used for the prior pretraining of ResNet50-SWS++ and the false-positive-reduction weights; if LUNA16 or NLST contributed to those stages, the reported external AUCs would be inflated and the central conclusion would need to be retested with pretraining restricted to DLND24.","The LUNA16 gap, 0.90 versus 0.63, is large enough to suspect it is not only about input focus: the two datasets define and label nodules differently, and no explicit label harmonization is described, so a matched-label reanalysis is a natural test.","One practical extension the paper leaves implicit is a hybrid pipeline where a nodule detector narrows the chest volume and the lesion-level classifier judges the detected patches; this would make the lesion-level approach applicable to whole-CT screening rather than to pre-annotated nodules."],"forward_implications":["If the lesion-level advantage holds, screening AI can be built around a detector-plus-patch-classifier design, avoiding the need for full-chest context in malignancy classification.","The external results imply that nodule-centric models transfer across scanners and protocols better than chest-region models, since the largest advantage appears on LUNA16 rather than on the institutional test set.","Subgroup results imply that lesion-level models reduce performance gaps in difficult cases such as adenocarcinoma and across different CT manufacturers, supporting their use in more diverse populations.","The interpretability evidence implies that clinicians can check the model's decision by looking at whether its heatmap falls on the nodule, which is closer to a radiologist's own workflow."],"supporting_citations":[{"why":"Supplies the ResNet50-SWS++ architecture and the false-positive-reduction weights that initialize both models, along with the earlier benchmarking setup from which the datasets are drawn.","marker":"[7]"},{"why":"Releases DLND24, the internal dataset used to train and internally validate both models.","marker":"[11]"},{"why":"Provides the LUNA16 external test set and its nodule annotations, the source of the largest reported performance gap.","marker":"[12]"},{"why":"Contributes the NLST-based external cohort used for the second external validation.","marker":"[8]"},{"why":"Supplies the statistical ROC analysis and DeLong's test used for performance comparison and confidence intervals.","marker":"[15]"}],"fun_headline_variants":["Lesion-centric AI beats chest-region model for lung cancer","Tumor-focused 3D patches outperform whole-chest lung cancer AI","External LUNA16: lesion-level AI scores AUC 0.90 vs 0.63","Nodule-level model edges out chest scans in lung cancer test"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central comparison holds only if LUNA16 and NLST were truly external to all stages of model development, including the earlier training of the starting weights, and the paper does not show that this was the case.","fun_headline_variants_meta":{"raw":{"variants":["Lesion-centric AI beats chest-region model for lung cancer","Tumor-focused 3D patches outperform whole-chest lung cancer AI","External LUNA16: lesion-level AI scores AUC 0.90 vs 0.63","Nodule-level model edges out chest scans in lung cancer test"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000265,"raw_usage":{"total_tokens":1699,"prompt_tokens":1127,"completion_tokens":572,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":743,"completion_tokens_details":{"reasoning_tokens":491}},"tokens_in":743,"tokens_out":572,"duration_ms":6171,"temperature":1.0,"reasoning_tokens":491,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T12:52:29.867652+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Check the training data behind the pretrained ResNet50-SWS++ and false-positive-reduction weights: if any LUNA16 or NLST scans appear there, rerun both models trained from scratch on DLND24 only and compare the resulting external AUCs; the reported 0.90 versus 0.63 LUNA16 gap should shrink or disappear.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Releases DLND24, the internal dataset used to train and internally validate both models."},{"cited_title":"Sybil: A Validated Deep Learning Model to Predict Future Lung Cancer Risk From a Single Low-Dose Chest Computed Tomography,","cited_arxiv_id":null,"evidence_quote":"Contributes the NLST-based external cohort used for the second external validation."}],"review_version":1}