{"id":"af392714-e6e0-4690-9294-69096dd026f1","arxiv_id":"2505.22592","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"FMCIB+XGBoost beats DINOv2+ABMIL for KRAS/EGFR mutation detection on Stanford CT data, but DINOv2+ABMIL wins T/N staging on an external cohort.","lead":"This paper compared two machine learning models that predict lung cancer mutations and stage from 3D CT scans. The CT-pretrained model was better at mutation detection, while the natural-image-pretrained model generalized better for staging on an external dataset, though limitations in data and code make the results preliminary.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The comparison conflates pretraining domain with architecture, input dimensionality, and classifier; the claimed attribution to domain-specific vs SSL pretraining is not supported by the design.","rationale":"The paper's strongest claim is not merely that two pipelines score differently; it is that the pattern reflects domain-specific pretraining versus SSL. That claim requires the pretraining corpus to be the operative difference. The Methods show the pipelines differ in encoder, input dimensionality, and classifier as well. Since no ablation holds these fixed, the reported numbers are consistent with many causal stories. A leak-free result would still not identify pretraining as the cause. I therefore recommend CONDITIONAL rather than UNCHANGED: the authors should either supply a matched-comparison analysis or revise the conclusion to a pipeline-level comparison. I am not alleging fraud or sloppiness; the concern is about the strength of the inference. The reader's leakage concern is valid but complementary, so agreement is partial.","tokens_in":9287,"tokens_out":10820,"duration_ms":128153,"concrete_test":"Hold the feature extractors fixed and run a matched classifier comparison: take the same 3D nodule volumes, extract FMCIB and DINOv2 (per-slice) features, and train the same classifier (e.g., XGBoost or a linear head) on both; also swap classifiers (FMCIB features + ABMIL, DINOv2 features + XGBoost) where feasible. If the KRAS/EGFR advantage of FMCIB and the T-stage advantage of DINOv2 persist under a common classifier/aggregation, the pretraining-domain attribution gains support; if the rank order flips or narrows to within one standard deviation, the current conclusion is confounded.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim attributes FMCIB+XGBoost's mutation-detection advantage to domain-specific pretraining and DINOv2+ABMIL's staging advantage to self-supervised general pretraining. But the two pipelines differ simultaneously in at least four factors: pretraining corpus (radiology CT vs ImageNet), encoder architecture (3D convolutional FMCIB vs 2D ViT DINOv2), input representation (3D volumes vs 2D slices), and classifier (XGBoost vs attention-based MIL). Because no factor is varied independently, the observed differences (e.g., KRAS accuracy 0.846 vs 0.773; T-stage on Lung-CT-PT-Dx 0.797 vs 0.726) can be explained by architecture or classifier choice as easily as by pretraining domain. The Discussion's causal language ('benefiting from domain-specific training', 'SSL's adaptability') therefore overreaches the experimental design. This is an internal inference problem, not a disagreement with prior consensus: a cleaner comparison or a claim restricted to 'these two complete pipelines perform differently' would be supported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper compares two ML pipelines for lung cancer radiogenomics from 3D CT nodule data: FMCIB+XGBoost (a 3D CNN foundation model pretrained on radiology images, used as a frozen feature extractor with an XGBoost classifier) and DINOv2+ABMIL (a 2D Vision Transformer pretrained on ImageNet via self-supervised learning, paired with attention-based multiple-instance learning). Using five repeated 80:20 splits on the Stanford Radiogenomics cohort and the Lung-CT-PT-Dx cohort, the authors report that FMCIB+XGBoost achieves higher accuracy for KRAS and EGFR mutation detection, while DINOv2+ABMIL achieves higher accuracy for T-stage and N-stage prediction on the external Lung-CT-PT-Dx cohort. The Discussion attributes these differences to domain-specific pretraining versus self-supervised generalizability.","tokens_in":9553,"tokens_out":3381,"duration_ms":40646,"significance":"If the findings were cleanly established, the paper would provide a useful comparison of two modeling strategies in a clinically relevant radiogenomics setting, with the interesting observation that the better pipeline differs by task and by cohort. The use of an external cohort and the reporting of accuracy, F1, sensitivity, specificity, and AUC are strengths. However, the significance is currently limited by three structural issues: the pipelines differ in multiple factors simultaneously, the mutation-detection comparison is compromised by one model having zero sensitivity, and the splitting/augmentation protocol is not described tightly enough to rule out leakage. The paper would be strengthened by reanalysis that separates pipeline-level comparison from a claims about pretraining domain, and by reporting class distributions and significance tests.","major_comments":[{"comment":"The two pipelines differ simultaneously in at least four factors: pretraining corpus (radiology CT vs ImageNet), encoder architecture (3D CNN vs 2D ViT), input representation (3D volumes vs 2D slices), and classifier (XGBoost vs attention-based MIL). Since no factor is varied independently, the observed performance differences (e.g., KRAS accuracy 0.846 vs 0.773, T-stage on Lung-CT-PT-Dx 0.797 vs 0.726) cannot be causally attributed to domain-specific vs self-supervised pretraining, as the Discussion claims. The manuscript should either restrict its conclusions to a comparison of the two complete pipelines or perform an ablation study to isolate the effect of pretraining domain.","section":"Methods — Model Architectures; Discussion"},{"comment":"DINOv2+ABMIL has sensitivity of 0.0 ± 0.0 for both KRAS and EGFR mutation detection, meaning it never predicts a positive case. Its reported accuracy and specificity therefore reflect the majority-class prevalence rather than any actual discriminative ability. Without per-fold class distributions or prevalence information, the statement that FMCIB+XGBoost 'consistently outperformed' DINOv2+ABMIL is misleading; the accuracy gap may simply reflect different decision thresholds. Please report the number of positive and negative cases per fold and per task, and include balanced metrics or confusion matrices.","section":"Results, Table 1; Discussion"},{"comment":"The methods describe data augmentation (rotation and shifting for the Stanford cohort) and state only that 'for each fold, data from both cohorts was split into training and testing sets at an 80:20 ratio'. It is not stated whether augmentation was applied before or after the split, nor whether the split was performed at the patient level. If augmented copies of the same nodule appear in both training and test sets, or if multiple nodules from the same patient are distributed across the split, the reported cross-validation results could be inflated. This must be clarified because the generalization claims depend on the absence of leakage.","section":"Methods — Study Cohorts and Experimental Design"},{"comment":"No statistical significance tests are reported for any comparison. The cross-validation standard deviations are large relative to some reported differences (for example, KRAS AUC 0.689 ± 0.084 vs 0.498 ± 0.005, or Lung-CT-PT-Dx T-stage accuracy 0.797 ± 0.038 vs 0.726 ± 0.036), so the claim of consistent superiority is not statistically supported. Paired tests (e.g., McNemar or a paired bootstrap on per-fold differences) should be used to establish whether the differences are reliable.","section":"Results; Experimental Design"}],"minor_comments":[{"comment":"The table formatting in the manuscript is broken, with cell values visually merged or misaligned; the tables should be regenerated so that each row and column is legible.","section":"Table 1 and Table 2"},{"comment":"The phrase 'the rotation degree was chosen from minus plus 20° and the shifting range is minus plus 15 pixels' should be rephrased for clarity, e.g., 'rotation in uniform range [-20°, +20°] and shifts in [-15, +15] pixels'.","section":"Methods — Study Cohorts"},{"comment":"The model name is spelled inconsistently as 'Dinov2' and 'DINOv2'; please use one standardized spelling throughout.","section":"Throughout"},{"comment":"Since DINOv2 is a 2D image model and the input data are described as 3D nodule volumes, the manuscript should specify how the 2D slices are extracted from the 3D volume for the DINOv2+ABMIL pipeline.","section":"Methods — Model Architectures"},{"comment":"The Stanford Radiogenomics cohort sample size and the number of mutation-positive cases are never reported, despite the Discussion acknowledging 'relatively small sample size, particularly for mutation-positive cases'; concrete numbers should be given.","section":"Results and Discussion"},{"comment":"The statement that a key strength is 'the use of two independent and diverse cohorts' is overstated because the Lung-CT-PT-Dx cohort is used only for staging tasks, not for mutation detection; the manuscript should describe the cohort usage more precisely.","section":"Discussion"}],"recommendation":"major_revision","confidential_remarks":"The authors appear to have a close connection to FMCIB (one of the models under evaluation) but no conflict-of-interest disclosure is included. This is a matter for editorial attention. In addition, the paper would be more suitable for the journal if the claims were narrowed to pipeline-level comparisons and the experimental details on splitting, class imbalance, and leakage were fully specified; in its current form, the central interpretation overreaches the design."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe headline: this is a new empirical benchmark, not a new method. On the whole it's a useful data point, but the central attribution of the results to 'domain-specific pretraining' versus 'SSL' is not supported by the experiment. The two pipelines differ in four ways at once, so the comparison is really between two complete systems, not between pretraining strategies.\n\nWhat's new: this specific comparison of FMCIB+XGBoost and DINOv2+ABMIL on KRAS/EGFR mutation and T/N staging across two cohorts hasn't been published before. The authors report standard metrics, the two-cohort design is a plus, and they are honest about small sample sizes and the lack of interpretability. The most striking result is DINOv2+ABMIL's zero sensitivity on both mutation tasks—that's an important negative result worth knowing.\n\nWhere it's soft: the stress-test note is right. The design changes pretraining corpus, encoder architecture, input representation (3D volumes vs 2D slices), and classifier simultaneously. So any claim like 'domain-specific pretraining helps mutation detection' is speculative. The Discussion uses exactly that language. Also missing: code, class balance per fold, statistical tests, and a clear statement on whether augmentation was applied before or after splitting. Since the Stanford cohort used rotation and shift, if augmented copies of the same nodule crossed into the test set, the accuracy gaps could be inflated. The zero sensitivity also means the AUC differences are driven entirely by specificity; that's worth clarifying.\n\nIs it a serious paper? Yes, it's coherent and the empirical results are likely reproducible in the narrow sense, but the interpretation needs reining in. I'd send it to peer review because the benchmark is new and the question matters, but a referee should require the missing experimental details and a more cautious framing. If those are fixed, it could be a citable comparison for people choosing models for lung CT radiogenomics. For my own work, I wouldn't rely on it for causal claims, but I might cite it as a benchmark.","headline":"Useful new benchmark, but the pretraining attribution is overreach; the design conflates too many variables.","tokens_in":10044,"tokens_out":2511,"would_cite":false,"duration_ms":28666,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"For lung CT radiogenomics, the best pretraining strategy depends on the task: domain-specific features detect KRAS/EGFR mutations, while self-supervised features generalize better to new staging cohorts.","keywords":["lung cancer","radiogenomics","3D CT scans","KRAS mutation","EGFR mutation","TNM staging","self-supervised learning","multiple instance learning"],"falsifier":"Compute patient-level grouped cross-validation, placing all slices and augmentations of one nodule in the same fold, report class prevalence per test fold, and re-run both models; if FMCIB+XGBoost's mutation advantage shrinks to within error bars or Dinov2+ABMIL's staging advantage reverses, the paper's task-dependence claim is not supported.","tokens_in":9136,"feed_emoji":"🫁","tokens_out":7267,"duration_ms":82800,"temperature":0.7,"pith_summary":"This paper tries to settle a practical question: for predicting lung cancer mutations and stage from 3D CT nodule scans, is it better to use features pretrained on radiology images or features pretrained on general images and adapted with self-supervised learning? It compares FMCIB+XGBoost, a radiology-pretrained feature extractor paired with a tuned XGBoost classifier, against Dinov2+ABMIL, a natural-image self-supervised model with an attention-based multiple-instance learning head. The result is task-dependent: the radiology-pretrained pipeline wins for KRAS and EGFR mutation detection, with accuracies of 0.846 and 0.883, while the self-supervised pipeline wins for T-stage and N-stage prediction on a separate cohort, with 0.797 accuracy on T-stage. If the comparison holds, clinicians and model builders should choose pretraining strategy by task rather than assuming one foundation model fits all radiogenomic applications.","feed_headline":"Supervised model beats SSL for KRAS/EGFR; SSL wins staging","feed_subtitle":"Domain-specific CT pretraining detects mutations; self-supervised features transfer better to new staging cohorts.","key_machinery":"The load-bearing contrast is between two fixed feature extractors. FMCIB is a convolutional encoder pretrained on 11,467 CT lesions from 2,312 patients; its extracted features feed a fine-tuned XGBoost classifier. Dinov2 is a self-supervised vision model pretrained on general natural images; its per-slice features are pooled by an attention-based multiple-instance learning head that handles 3D nodules without instance-level labels. ABMIL is the mechanism that lets the SSL pipeline work when only nodule-level labels exist, and the comparison is carried by how well each extractor's representations separate mutation status and stage.","core_discovery":"This paper argues that no single pretraining strategy dominates in lung CT radiogenomics; the right model depends on the task. On the Radiogenomics cohort, FMCIB+XGBoost—a fixed CT-pretrained feature extractor with a tuned XGBoost classifier—consistently beat Dinov2+ABMIL—a fixed natural-image SSL feature extractor with an attention-based multiple-instance learning head—for KRAS and EGFR mutation detection, with accuracies of 0.846 and 0.883 versus 0.773 and 0.798, and KRAS/EGFR AUCs of 0.689/0.736 versus 0.498/0.498. On the separate Lung-CT-PT-Dx cohort, the pattern reversed for staging: Dinov2+ABMIL reached 0.797 accuracy for T-stage and 0.704 for N-stage, ahead of FMCIB+XGBoost's 0.726 and 0.643. The authors take this as evidence that domain-specific pretraining carries mutation-relevant imaging features, while SSL features transfer better across CT cohorts for staging tasks.","pith_inferences":["The near-random AUC of 0.498 for Dinov2+ABMIL on mutation tasks suggests the ABMIL attention head is not extracting mutation-relevant signal from the SSL features; a threshold sweep or a different pooling strategy might change this, but the paper does not test it.","A natural extension is an ensemble that uses FMCIB features for mutation calls and Dinov2 features for staging, since the two pipelines show complementary strengths on the same nodule representations.","Because FMCIB's pretraining data are CT lesions and Dinov2's are natural images, the mutation-detection gap could reflect pretraining dataset similarity rather than a fundamental property of supervised versus self-supervised learning; a controlled comparison would train the same architecture with both pretraining distributions."],"forward_implications":["For KRAS and EGFR mutation screening, radiology-pretrained features should be preferred over general-purpose SSL features when the available labeled cohort is similar to the training set.","For cancer staging, a self-supervised model pretrained on natural images can outperform a domain-specific model on a new CT cohort, so staging benchmarks should not assume domain-specific pretraining is always better.","Dinov2+ABMIL's zero sensitivity on both mutation tasks means the model is not currently usable for mutation screening; its high specificity comes from predicting the majority class, and any clinical use requires an operating-point or class-balance adjustment.","Reporting separate mutation and staging results is necessary; a single aggregate accuracy figure would conceal which model to deploy for each clinical question."],"supporting_citations":[],"fun_headline_variants":["Task decides winner: supervised for mutations, SSL for staging","Supervised model tops KRAS/EGFR; SSL excels at staging","No one-size-fits-all: CT pretraining vs SSL for lung cancer","For lung CT: supervised wins mutations, SSL wins staging","Mutation detection favors supervised; staging favors SSL"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The results depend on the assumption that the repeated 80:20 splits separated patients and augmented copies cleanly and kept mutation-positive cases present in every test fold; if augmented slices from the same nodule landed on both sides of a split, or if positive cases nearly vanished from a test fold, the accuracy and AUC gaps could be artifacts of the split rather than the models.","fun_headline_variants_meta":{"raw":{"variants":["Task decides winner: supervised for mutations, SSL for staging","Supervised model tops KRAS/EGFR; SSL excels at staging","No one-size-fits-all: CT pretraining vs SSL for lung cancer","For lung CT: supervised wins mutations, SSL wins staging","Mutation detection favors supervised; staging favors SSL"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000201,"raw_usage":{"total_tokens":1400,"prompt_tokens":987,"completion_tokens":413,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":603,"completion_tokens_details":{"reasoning_tokens":329}},"tokens_in":603,"tokens_out":413,"duration_ms":4551,"temperature":1.0,"reasoning_tokens":329,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T13:02:51.750015+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute patient-level grouped cross-validation, placing all slices and augmentations of one nodule in the same fold, report class prevalence per test fold, and re-run both models; if FMCIB+XGBoost's mutation advantage shrinks to within error bars or Dinov2+ABMIL's staging advantage reverses, the paper's task-dependence claim is not supported.","supporting_citations":[],"review_version":1}