REVIEW 4 major objections 6 minor
Comparative Analysis of Machine Learning Models for Lung Cancer Mutation Detection and Staging Using 3D CT Scans
T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read For lung CT radiogenomics, the best pretraining strategy depends on the task: domain-specific features detect KRAS/EGFR mutations, while self-supervised features generalize better to new staging cohorts.
desk verdict Useful new benchmark, but the pretraining attribution is overreach; the design conflates too many variables. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing contrast is between two fixed feature extractors. FMCIB is a convolutional encoder pretrained on 11,467 CT lesions from 2,312 patients; its extracted features feed a fine-tuned XGBoost classifier. Dinov2 is a self-supervised vision model pretrained on general natural images; its per-slice features are pooled by an attention-based multiple-instance learning head that handles 3D nodules without instance-level labels. ABMIL is the mechanism that lets the SSL pipeline work when only nodule-level labels exist, and the comparison is carried by how well each extractor's representations separate mutation status and stage.
What would settle it
Compute patient-level grouped cross-validation, placing all slices and augmentations of one nodule in the same fold, report class prevalence per test fold, and re-run both models; if FMCIB+XGBoost's mutation advantage shrinks to within error bars or Dinov2+ABMIL's staging advantage reverses, the paper's task-dependence claim is not supported.
Extended reading notes
Core claim
This paper argues that no single pretraining strategy dominates in lung CT radiogenomics; the right model depends on the task. On the Radiogenomics cohort, FMCIB+XGBoost—a fixed CT-pretrained feature extractor with a tuned XGBoost classifier—consistently beat Dinov2+ABMIL—a fixed natural-image SSL feature extractor with an attention-based multiple-instance learning head—for KRAS and EGFR mutation detection, with accuracies of 0.846 and 0.883 versus 0.773 and 0.798, and KRAS/EGFR AUCs of 0.689/0.736 versus 0.498/0.498. On the separate Lung-CT-PT-Dx cohort, the pattern reversed for staging: Dinov2+ABMIL reached 0.797 accuracy for T-stage and 0.704 for N-stage, ahead of FMCIB+XGBoost's 0.726 and 0.643. The authors take this as evidence that domain-specific pretraining carries mutation-relevant imaging features, while SSL features transfer better across CT cohorts for staging tasks.
Load-bearing premise
The results depend on the assumption that the repeated 80:20 splits separated patients and augmented copies cleanly and kept mutation-positive cases present in every test fold; if augmented slices from the same nodule landed on both sides of a split, or if positive cases nearly vanished from a test fold, the accuracy and AUC gaps could be artifacts of the split rather than the models.
Editorial extensions
If this is right
- For KRAS and EGFR mutation screening, radiology-pretrained features should be preferred over general-purpose SSL features when the available labeled cohort is similar to the training set.
- For cancer staging, a self-supervised model pretrained on natural images can outperform a domain-specific model on a new CT cohort, so staging benchmarks should not assume domain-specific pretraining is always better.
- Dinov2+ABMIL's zero sensitivity on both mutation tasks means the model is not currently usable for mutation screening; its high specificity comes from predicting the majority class, and any clinical use requires an operating-point or class-balance adjustment.
- Reporting separate mutation and staging results is necessary; a single aggregate accuracy figure would conceal which model to deploy for each clinical question.
Reading between the lines
- The near-random AUC of 0.498 for Dinov2+ABMIL on mutation tasks suggests the ABMIL attention head is not extracting mutation-relevant signal from the SSL features; a threshold sweep or a different pooling strategy might change this, but the paper does not test it.
- A natural extension is an ensemble that uses FMCIB features for mutation calls and Dinov2 features for staging, since the two pipelines show complementary strengths on the same nodule representations.
- Because FMCIB's pretraining data are CT lesions and Dinov2's are natural images, the mutation-detection gap could reflect pretraining dataset similarity rather than a fundamental property of supervised versus self-supervised learning; a controlled comparison would train the same architecture with both pretraining distributions.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper compares two ML pipelines for lung cancer radiogenomics from 3D CT nodule data: FMCIB+XGBoost (a 3D CNN foundation model pretrained on radiology images, used as a frozen feature extractor with an XGBoost classifier) and DINOv2+ABMIL (a 2D Vision Transformer pretrained on ImageNet via self-supervised learning, paired with attention-based multiple-instance learning). Using five repeated 80:20 splits on the Stanford Radiogenomics cohort and the Lung-CT-PT-Dx cohort, the authors report that FMCIB+XGBoost achieves higher accuracy for KRAS and EGFR mutation detection, while DINOv2+ABMIL achieves higher accuracy for T-stage and N-stage prediction on the external Lung-CT-PT-Dx cohort. The Discussion attributes these differences to domain-specific pretraining versus self-supervised generalizability.
Significance. If the findings were cleanly established, the paper would provide a useful comparison of two modeling strategies in a clinically relevant radiogenomics setting, with the interesting observation that the better pipeline differs by task and by cohort. The use of an external cohort and the reporting of accuracy, F1, sensitivity, specificity, and AUC are strengths. However, the significance is currently limited by three structural issues: the pipelines differ in multiple factors simultaneously, the mutation-detection comparison is compromised by one model having zero sensitivity, and the splitting/augmentation protocol is not described tightly enough to rule out leakage. The paper would be strengthened by reanalysis that separates pipeline-level comparison from a claims about pretraining domain, and by reporting class distributions and significance tests.
major comments (4)
- [Methods — Model Architectures; Discussion] The two pipelines differ simultaneously in at least four factors: pretraining corpus (radiology CT vs ImageNet), encoder architecture (3D CNN vs 2D ViT), input representation (3D volumes vs 2D slices), and classifier (XGBoost vs attention-based MIL). Since no factor is varied independently, the observed performance differences (e.g., KRAS accuracy 0.846 vs 0.773, T-stage on Lung-CT-PT-Dx 0.797 vs 0.726) cannot be causally attributed to domain-specific vs self-supervised pretraining, as the Discussion claims. The manuscript should either restrict its conclusions to a comparison of the two complete pipelines or perform an ablation study to isolate the effect of pretraining domain.
- [Results, Table 1; Discussion] DINOv2+ABMIL has sensitivity of 0.0 ± 0.0 for both KRAS and EGFR mutation detection, meaning it never predicts a positive case. Its reported accuracy and specificity therefore reflect the majority-class prevalence rather than any actual discriminative ability. Without per-fold class distributions or prevalence information, the statement that FMCIB+XGBoost 'consistently outperformed' DINOv2+ABMIL is misleading; the accuracy gap may simply reflect different decision thresholds. Please report the number of positive and negative cases per fold and per task, and include balanced metrics or confusion matrices.
- [Methods — Study Cohorts and Experimental Design] The methods describe data augmentation (rotation and shifting for the Stanford cohort) and state only that 'for each fold, data from both cohorts was split into training and testing sets at an 80:20 ratio'. It is not stated whether augmentation was applied before or after the split, nor whether the split was performed at the patient level. If augmented copies of the same nodule appear in both training and test sets, or if multiple nodules from the same patient are distributed across the split, the reported cross-validation results could be inflated. This must be clarified because the generalization claims depend on the absence of leakage.
- [Results; Experimental Design] No statistical significance tests are reported for any comparison. The cross-validation standard deviations are large relative to some reported differences (for example, KRAS AUC 0.689 ± 0.084 vs 0.498 ± 0.005, or Lung-CT-PT-Dx T-stage accuracy 0.797 ± 0.038 vs 0.726 ± 0.036), so the claim of consistent superiority is not statistically supported. Paired tests (e.g., McNemar or a paired bootstrap on per-fold differences) should be used to establish whether the differences are reliable.
minor comments (6)
- [Table 1 and Table 2] The table formatting in the manuscript is broken, with cell values visually merged or misaligned; the tables should be regenerated so that each row and column is legible.
- [Methods — Study Cohorts] The phrase 'the rotation degree was chosen from minus plus 20° and the shifting range is minus plus 15 pixels' should be rephrased for clarity, e.g., 'rotation in uniform range [-20°, +20°] and shifts in [-15, +15] pixels'.
- [Throughout] The model name is spelled inconsistently as 'Dinov2' and 'DINOv2'; please use one standardized spelling throughout.
- [Methods — Model Architectures] Since DINOv2 is a 2D image model and the input data are described as 3D nodule volumes, the manuscript should specify how the 2D slices are extracted from the 3D volume for the DINOv2+ABMIL pipeline.
- [Results and Discussion] The Stanford Radiogenomics cohort sample size and the number of mutation-positive cases are never reported, despite the Discussion acknowledging 'relatively small sample size, particularly for mutation-positive cases'; concrete numbers should be given.
- [Discussion] The statement that a key strength is 'the use of two independent and diverse cohorts' is overstated because the Lung-CT-PT-Dx cohort is used only for staging tasks, not for mutation detection; the manuscript should describe the cohort usage more precisely.
Circularity Check
No significant circularity: empirical benchmark comparing two frozen pretrained feature extractors with trained classifiers on held-out folds; performance numbers are not contained in the inputs by construction.
full rationale
This paper is an empirical benchmark rather than a derivation. FMCIB+XGBoost and Dinov2+ABMIL are complete pipelines applied to fixed CT data, and the reported accuracies, F1-scores, sensitivities, specificities, and AUCs are computed on held-out 20% folds of two external cohorts (Stanford Radiogenomics and Lung-CT-PT-Dx). The outcomes are therefore not derived from the inputs by construction; no equation in the paper equates a fitted parameter with a predicted quantity. The only author-linked component is the FMCIB encoder, which is used as a fixed feature extractor and whose prior validation (anatomical site classification) is unrelated to the mutation and staging labels predicted here. No uniqueness theorem, no self-citation chain, and no ansatz-heavy premise is invoked to force the conclusions. The design does conflate pretraining domain with architecture, input representation, and classifier, so the causal attribution of the performance gap to pretraining strategy is a validity concern rather than a circularity step under the stated criteria. Thus the circularity score is 0.
Assumptions & free parameters
free parameters (3)
- XGBoost hyperparameters (boosting rounds, max depth, learning rate, subsample ratio) =
Grid searched: rounds {50,100,150}, depth {3,5,7}, lr {0.01,0.1,0.2}, subsample {70%,80%,90%}
- CT preprocessing choices (voxel resampling, slice thickness) =
32x32x32 voxels, 1.0 mm slice thickness
- Data augmentation ranges (rotation, shift) =
Rotation +/-20 degrees, shift +/-15 pixels
assumptions (4)
- domain assumption CT image features carry detectable signal for KRAS and EGFR mutation status and T/N stage.
- domain assumption FMCIB is a valid domain-specific feature extractor for lung nodules despite being pretrained on heterogeneous CT lesions.
- domain assumption DINOv2 ImageNet features, after ABMIL aggregation, are comparable inputs for lung CT nodule classification.
- domain assumption Repeated stratified 5-fold CV with 80:20 splits yields unbiased generalization estimates for these cohorts.
Cite this review
Pith. "Pith review of Comparative Analysis of Machine Learning Models for Lung Cancer Mutation Detection and Staging Using 3D CT Scans." pith.science (2026). https://pith.science/paper/SYJ5D3UC
@misc{pith2026250522592,
author = {Pith},
title = {Pith review of: Comparative Analysis of Machine Learning Models for Lung Cancer Mutation Detection and Staging Using 3D CT Scans},
year = {2026},
howpublished = {\url{https://pith.science/paper/SYJ5D3UC}},
note = {Machine review of arXiv:2505.22592}
}
read the original abstract
Lung cancer is the leading cause of cancer mortality worldwide, and non-invasive methods for detecting key mutations and staging are essential for improving patient outcomes. Here, we compare the performance of two machine learning models - FMCIB+XGBoost, a supervised model with domain-specific pretraining, and Dinov2+ABMIL, a self-supervised model with attention-based multiple-instance learning - on 3D lung nodule data from the Stanford Radiogenomics and Lung-CT-PT-Dx cohorts. In the task of KRAS and EGFR mutation detection, FMCIB+XGBoost consistently outperformed Dinov2+ABMIL, achieving accuracies of 0.846 and 0.883 for KRAS and EGFR mutations, respectively. In cancer staging, Dinov2+ABMIL demonstrated competitive generalization, achieving an accuracy of 0.797 for T-stage prediction in the Lung-CT-PT-Dx cohort, suggesting SSL's adaptability across diverse datasets. Our results emphasize the clinical utility of supervised models in mutation detection and highlight the potential of SSL to improve staging generalization, while identifying areas for enhancement in mutation sensitivity.
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.