{"id":"e9270cf7-7752-488b-8e39-6f9748c7e40f","arxiv_id":"2412.21091","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"2D multi-view ResNet ensembles predict IDH mutation from MRI about as well as 3D ResNets (AUROC 0.91 vs 0.90), while MGMT methylation prediction stays near chance.","lead":"This study compared 2D and 3D neural networks that predict two genetic markers in brain tumors from MRI scans. The 2D models matched the 3D models for one marker while using less computing, and the paper shows how hard the other marker is to predict from images.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No confidence intervals or equivalence test support 'no notable difference' between 2D ensembles and 3D; the comparison also confounds dimensionality with model depth and ensembling.","rationale":"Good-faith reading: the paper is a benchmark comparison of 2D vs 3D ResNets on UCSF glioma MRI for IDH and MGMT prediction. The IDH result is plausible and aligns with prior radiogenomics literature. The central claim is an equivalence claim: 2D ensemble models are as effective as 3D models. The most load-bearing condition for that claim is not merely whether largest slices capture IDH features, but whether the observed AUROC differences are distinguishable from noise and whether the comparison is structurally fair. Neither condition is met: the analysis uses single-split point estimates without uncertainty, and it compares a 2D ensemble with deeper architectures to 3D single models with only shallow depths. The MGMT results, including AUROCs below 0.5 and an internal T1/T1c inconsistency, further suggest unresolved label or evaluation issues, though the central IDH claim is independent. I do not see fraud or a fatal flaw; this is a useful preliminary benchmark. However, the stated conclusion goes beyond the evidence. The reader's conditional verdict remains appropriate; the weakest assumption the reader named, largest-slice selection, is a real but secondary vulnerability. My concern does not change the verdict, so verdict_should_be is UNCHANGED.","tokens_in":9125,"tokens_out":4434,"duration_ms":43714,"concrete_test":"On the test predictions, compute patient-level bootstrap 95% confidence intervals for the AUROC difference between the best 2D ensemble and best 3D model, and for the mean 2D-ensemble vs mean 3D difference; also run a paired equivalence test with a prespecified margin such as ±0.05 AUROC. If the CI spans the equivalence margin or the equivalence test is inconclusive, the claim of 'no notable difference' is unsupported. To address the confound, retrain 3D ResNet50/101/152 and/or a 3D ensemble from the same view volumes; if deeper 3D models outperform the 2D ensemble, the comparison was not fair. Also report compute/memory metrics to substantiate 'reduced resource utilization.'","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that 2D ensemble models predict IDH status as effectively as 3D models, with reduced resource use. This requires showing that the 2D and 3D test AUROCs are statistically indistinguishable, or at least that any difference is small relative to uncertainty. The paper reports only point estimates from a single 60/20/20 split: best 2D ensemble AUROC 0.9096 vs best 3D AUROC 0.8999; mean 0.8782 vs 0.8586. No confidence intervals, DeLong tests, bootstrap resampling, or repeated splits are provided. With a test set of roughly 99 patients and about 21% IDH-mutant prevalence, these raw differences are well within plausible sampling noise, so 'no notable differences' is not established by the presented analysis. A second confound compounds this: the 2D arm uses an ensemble of three view-specific models combined by logistic regression and includes ResNet50/101/152, while the 3D arm uses only single ResNet10/18/34 models. Thus apparent 2D/3D parity may reflect ensemble size or model depth rather than dimensionality. The paper itself notes 3D AUROC improves with depth, so the omission of deeper 3D models matters. 'Reduced resource utilization' is asserted without reporting FLOPs, memory, or training time. The largest-slice assumption is a further vulnerability, but it is secondary: even if the 2D inputs preserve all relevant features, the equivalence claim still lacks statistical support. Minor internal inconsistency: the best MGMT model is described as T1 in Results but T1c in Discussion.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript compares 2D and 3D ResNet models for predicting IDH and MGMT mutation status from preoperative multiparametric MRI (T1, T1c, FLAIR) in the UCSF glioma dataset. The 2D models use the single largest tumor slice in each of the axial, coronal, and sagittal views, with the three view-specific models combined by logistic regression; the 3D models use the full tumor volume as input. The authors report a best IDH test AUROC of 0.9096 for a 2D ResNet50 (T1) and 0.8999 for a 3D ResNet34 (T1c), while MGMT prediction was near chance, with a best 2D AUROC of 0.6168. The central claim, stated in the Discussion, is that 2D ensemble models can predict IDH status as effectively as 3D models with reduced resource utilization.","tokens_in":9618,"tokens_out":3950,"duration_ms":39728,"significance":"If the IDH result is robust, the manuscript would provide a useful empirical data point: 2D ensembles operating on largest tumor slices may match the accuracy of 3D volume processing, with potential savings in compute and memory. The study uses a reasonably sized public dataset, performs a patient-level split, trains multiple ResNet depths in both 2D and 3D, uses weighted loss for class imbalance, and honestly discusses the MGMT negative result in the context of conflicting literature. However, the significance is conditional: the central equivalence claim rests on point estimates from a single split, with no confidence intervals, repeated validation, or resource measurements. As presented, the evidence supports a descriptive comparison of specific models on one internal test set, not a general claim of 2D/3D equivalence.","major_comments":[{"comment":"The central claim that 2D ensemble models perform as effectively as 3D models is not statistically supported. The paper reports only point estimates from one 60/20/20 split: best 2D AUROC 0.9096 versus best 3D AUROC 0.8999, and mean 0.8782 versus 0.8586. No confidence intervals, bootstrap resampling, repeated splits, DeLong tests, or equivalence tests are provided. With a test set of roughly 99 patients and about 21% IDH-mutant prevalence, these differences are well within plausible sampling noise. The phrase 'no notable differences' therefore cannot be concluded from the presented analysis.","section":"Discussion, 'There are no notable differences' paragraph"},{"comment":"The comparison confounds input dimensionality with model depth and ensembling. The 2D arm ensembles three view-specific models and includes ResNet50, ResNet101, and ResNet152, while the 3D arm uses single models only up to ResNet34. The paper itself reports a positive depth-performance trend for 3D models (Figures 1-2), so the apparent 2D/3D parity could be explained by higher model capacity or by ensembling rather than by dimensionality. To support the central claim, the design should include depth-matched single-model comparisons or ensemble 3D models as well.","section":"Methods, '2D approach' and Results, Tables 2-3"},{"comment":"The MGMT result is internally inconsistent. The Results state that the best MGMT model is the 2D ResNet152 using T1 (AUROC 0.6168), while the Discussion states that this model used T1c. In addition, no MGMT performance tables are provided, so the reader cannot verify the claim that '14 out of 16 models' fell below 0.5, nor can per-view, per-sequence, or per-depth results be inspected. The sequence should be corrected and the MGMT tables added.","section":"Results, MGMT prediction; Abstract"},{"comment":"The 2D equivalence claim depends on the assumption that the single largest masked slice in each orthogonal view preserves the imaging features needed for IDH classification. No slice-level ablation or comparison with multi-slice or full-volume 2D inputs is provided to show that features outside the largest slice, or at the tumor margin, are not needed. This is a load-bearing assumption for the fairness of the 2D/3D comparison and should be explicitly tested or clearly stated as a limitation.","section":"Methods, '2D approach' slice selection"},{"comment":"The claim of 'reduced resource utilization' for the 2D approach is asserted without any supporting measurements. No FLOPs, parameter counts, memory footprints, training time, or inference time are reported for either the 2D or 3D models. Please provide quantitative resource comparisons or weaken the claim to reflect that resource use was not directly measured.","section":"Results, resource utilization claim"}],"minor_comments":[{"comment":"'USCF glioma dataset' should be 'UCSF glioma dataset' to match the cited dataset name.","section":"Abstract and Methods"},{"comment":"The layout of Table 2 interleaves model names, views, modalities, validation AUROC, and test AUROC in a way that makes row-by-row mapping very difficult; consider a conventional wide table with one row per architecture-sequence combination.","section":"Table 2"},{"comment":"The statement '14 out of 16 models' is ambiguous; clarify whether this counts the 2D arm only and specify which 16 configurations are included.","section":"Results, MGMT prediction"},{"comment":"No seed information or number of training runs per configuration is reported; adding this would help the reader assess the variability of the point estimates.","section":"Methods, training details"},{"comment":"The discussion of the MGMT index cutoff is useful, but the statement that 'an MGMT index of 0 was defined as negative' should be reconciled with the wide range of MGMT index values in Table 1 and the possibility of a continuous-score analysis.","section":"Discussion, MGMT cutoff"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is within the scope of an imaging-focused journal, and the empirical comparison of many ResNet variants on a public dataset has some value. The main issue is evidential: the central equivalence claim is not supported by the single-split, no-interval analysis, and the 2D/3D comparison is confounded by depth and ensembling. These issues can in principle be addressed with additional experiments or by substantially restating the claim, so I do not recommend rejection. The self-citations are methodologically relevant and not disproportionate. If the authors cannot provide repeated split or bootstrap analyses, the Discussion should be revised to present the findings as descriptive rather than as evidence of equivalence."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know this paper is a straightforward benchmark: seven 2D ResNet variants (single largest slice per view, ensembled across three views) against three 3D ResNets on UCSF MRI for IDH and MGMT prediction. The headline IDH numbers are plausible (best 2D AUROC 0.9096, best 3D 0.8999) and consistent with prior work. The MGMT results are poor, which the authors openly discuss and attribute to label noise and an arbitrary positivity cutoff. That honesty is a real strength.\n\nWhat is new is the systematic comparison itself: 2D multi-view ensembles vs 3D volumes on the same public dataset, with a clear finding that 2D reaches roughly the same IDH AUROC. That is practically useful for groups deciding where to spend compute. The methods are described well enough to reproduce the general pipeline, and the literature review is fair and current.\n\nThe soft spots are statistical, not conceptual. The central claim — 'no notable differences' between 2D and 3D — is supported only by point estimates from a single 60/20/20 split. With about 99 test patients and ~21% IDH-mutant prevalence, the 0.01 AUROC gap between the best models is well inside sampling noise. No confidence intervals, no DeLong test, no bootstrap, no repeated splits. The paper also confounds dimensionality with model depth and ensembling: the 2D arm uses ensembles of ResNet50/101/152, while the 3D arm uses single ResNet10/18/34. Since the paper itself shows 3D improves with depth, deeper 3D models might close or reverse the gap. And 'reduced resource utilization' is asserted without reporting FLOPs, memory, or training time. The largest-slice assumption is a reasonable approximation, but it remains untested; I would not call it a fatal flaw.\n\nOne minor internal inconsistency: the best MGMT model is T1 in the Results but T1c in the Discussion. Also, no code or hyperparameters are provided, which hurts reproducibility but does not invalidate the findings.\n\nOverall, this is an honest, clearly-written empirical study that confirms IDH is predictable from MRI and suggests 2D ensembles are competitive with 3D at lower cost. The main claim is plausible but not statistically established. A serious referee could fix this by asking for error bars or a non-inferiority test, and for compute measurements. I would send it to peer review rather than desk-reject; the comparison is useful and the weaknesses are addressable.","headline":"Useful 2D-vs-3D ResNet benchmark for IDH prediction, but the 'no notable difference' claim needs error bars and the MGMT results are honestly reported but not informative.","tokens_in":10046,"tokens_out":1164,"would_cite":false,"duration_ms":13331,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A three-view 2D ResNet ensemble can classify IDH mutation status from glioma MRI as accurately as 3D volume models, with the best 2D model at AUROC 0.9096.","keywords":["radiogenomics","glioma","IDH mutation","MGMT promoter methylation","ResNet","2D versus 3D CNN","MRI","ensemble model"],"falsifier":"Train the same 2D ResNet ensemble on all tumor slices within each view, or on randomly chosen slices, and compare test AUROC with the 3D baseline on the same patient split; if the 2D advantage disappears or reverses, the paper's equivalence claim depends on the largest-slice selection rather than on 2D processing itself.","tokens_in":8972,"feed_emoji":"🧠","tokens_out":7076,"duration_ms":64736,"temperature":0.7,"pith_summary":"The paper compares two ways of reading MRI to predict molecular markers in glioma: 2D ResNet models that look at the largest tumor slice from each of three anatomical views, ensembled by logistic regression, and 3D ResNet models that consume the entire segmented tumor volume. For IDH mutation status, the paper's central claim is that the 2D ensemble matches 3D performance: the best 2D model reached a test AUROC of 0.9096, the best 3D model 0.8999, and the mean 2D AUROC (0.8782) was slightly higher than the 3D mean (0.8586). The paper reads this as evidence that 3D volume processing is not necessary for accurate IDH classification from MRI and costs more resources. For MGMT promoter methylation, the same comparison failed: the best 2D model scored 0.6168 and most 3D models fell below chance, leading the paper to question whether MGMT status is reliably encoded in MRI at all. The contribution is the systematic head-to-head comparison of 2D and 3D ResNet variants on the same dataset and evaluation protocol.","feed_headline":"2D MRI views match 3D scans for IDH mutation detection","feed_subtitle":"Three-view 2D ResNet ensemble hits AUROC 0.91 while 3D volume models do no better.","key_machinery":"Residual networks (ResNets), convolutional networks with skip connections that allow deeper training. The 2D arm uses one ResNet per anatomical view, with each network fed the single axial, coronal, or sagittal slice that contains the largest tumor area, and logistic regression fuses the three view-level predictions. The 3D arm replaces the single slice with the full segmented tumor volume as input. The load-bearing design choice is the largest-tumor-area slice per view: it converts a volume problem into three 2D problems while retaining some three-dimensional context, and it is what makes the claimed resource saving possible.","core_discovery":"The central claim, stated in the discussion, is that '2D ensemble models, employing three different views, can predict IDH mutation status as effectively as 3D models, without notable differences in performance but with reduced resource utilization.' Concretely, a 2D ResNet50 ensemble on T1-weighted MRI reached a test AUROC of 0.9096, while the best 3D model, ResNet34 on contrast-enhanced T1, reached 0.8999. The paper also reports a secondary pattern: within both 2D and 3D families, deeper ResNet variants tended to produce higher IDH AUROCs, most clearly among 3D models. For MGMT, the paper's finding is negative: the best model barely exceeded chance and most 3D models scored below 0.5 AUROC, which it attributes to a possible absence of imaging signal or to a suboptimal positive/negative labeling cutoff.","pith_inferences":["An untested implication is that the largest-slice protocol may be leaving useful volumetric context unused: feeding several adjacent slices per view, a 2.5D input, could test whether the 2D arm can be pushed even higher.","The MGMT labeling cutoff, with index 0 as negative and 1-17 as positive, is a plausible confound; modeling the methylation index as a continuous target or using a standardized assay would be a direct test before concluding MRI carries no MGMT signal.","If the 2D/3D equivalence survives external multi-institutional validation, IDH classification could run on standard 2D computational infrastructure, lowering the barrier for clinical deployment.","The observed complexity trend in 3D models raises the untested possibility that deeper 3D architectures than ResNet34, not included here, could eventually separate from the 2D ensemble."],"forward_implications":["IDH mutation status can be predicted non-invasively from preoperative MRI with a test AUROC near 0.91 using 2D ResNet ensembles.","3D volume processing does not add a meaningful accuracy advantage over 2D multi-view ensembles for IDH classification in this dataset.","T1-weighted and contrast-enhanced T1 sequences carry more IDH-discriminative signal than FLAIR in both 2D and 3D models.","Deeper ResNet variants tend to improve IDH prediction, most consistently in the 3D models.","MGMT promoter methylation status is not reliably predicted from MRI here; most models performed at or below chance, so the paper treats MGMT prediction as an unresolved problem."],"supporting_citations":[{"why":"supplies the preoperative glioma MRI dataset with IDH and MGMT labels for all training, tuning, and testing.","marker":"[16]"},{"why":"motivates the patient-level split into training, tuning, and test sets that keeps the evaluation independent.","marker":"[17]"},{"why":"provides the medical imaging data-handling and augmentation pipeline used to prepare 2D and 3D inputs.","marker":"[18]"},{"why":"supports the weighted binary cross-entropy loss used to handle class imbalance during training.","marker":"[19]"},{"why":"defines AUROC as the performance metric on which model selection and comparison rest.","marker":"[20]"},{"why":"documents prior MGMT prediction results on multi-institutional challenge data that contextualize the paper's negative MGMT finding.","marker":"[28]"},{"why":"supplies the critique of data leakage, small samples, and label heterogeneity that the paper uses to explain inflated MGMT claims in other work.","marker":"[30]"}],"fun_headline_variants":["2D MRI ensemble matches 3D for IDH mutation detection","2D views rival 3D for IDH status, cutting compute load","2D ResNet equals 3D for IDH prediction, uses fewer resources","IDH status: 2D MRI matches 3D, with lower computational cost","MGMT prediction remains elusive in both 2D and 3D MRI models"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison is fair only if the single largest tumor slice in each of the three views carries the imaging features that separate IDH-mutant from IDH-wildtype tumors; if that signal is spread across other slices or lives at the tumor margin, the 2D models are handicapped before the comparison starts.","fun_headline_variants_meta":{"raw":{"variants":["2D MRI ensemble matches 3D for IDH mutation detection","2D views rival 3D for IDH status, cutting compute load","2D ResNet equals 3D for IDH prediction, uses fewer resources","IDH status: 2D MRI matches 3D, with lower computational cost","MGMT prediction remains elusive in both 2D and 3D MRI models"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000272,"raw_usage":{"total_tokens":1734,"prompt_tokens":1148,"completion_tokens":586,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":764,"completion_tokens_details":{"reasoning_tokens":483}},"tokens_in":764,"tokens_out":586,"duration_ms":6159,"temperature":1.0,"reasoning_tokens":483,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T23:02:21.100024+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same 2D ResNet ensemble on all tumor slices within each view, or on randomly chosen slices, and compare test AUROC with the 3D baseline on the same patient split; if the 2D advantage disappears or reverses, the paper's equivalence claim depends on the largest-slice selection rather than on 2D processing itself.","supporting_citations":[{"cited_title":"The University of California San Francisco preoperative diffuse glioma MRI dataset","cited_arxiv_id":null,"evidence_quote":"supplies the preoperative glioma MRI dataset with IDH and MGMT labels for all training, tuning, and testing."},{"cited_title":"Mitigating bias in radiology machine learning: 1","cited_arxiv_id":null,"evidence_quote":"motivates the patient-level split into training, tuning, and test sets that keeps the evaluation independent."},{"cited_title":"Mitigating bias in radiology machine learning: 2","cited_arxiv_id":null,"evidence_quote":"supports the weighted binary cross-entropy loss used to handle class imbalance during training."},{"cited_title":"Mitigating bias in radiology machine learning: 3","cited_arxiv_id":null,"evidence_quote":"defines AUROC as the performance metric on which model selection and comparison rest."},{"cited_title":"A comparison of three different deep learning-based models to predict the MGMT promoter methylation status in glioblastoma using brain MRI","cited_arxiv_id":null,"evidence_quote":"documents prior MGMT prediction results on multi-institutional challenge data that contextualize the paper's negative MGMT finding."},{"cited_title":"MRI-based deep learning tools for mgmt promoter methylation detection: a thorough evaluation","cited_arxiv_id":null,"evidence_quote":"supplies the critique of data leakage, small samples, and label heterogeneity that the paper uses to explain inflated MGMT claims in other work."}],"review_version":1}