{"id":"c78b7b1d-224b-448b-bf30-1f3f112a53be","arxiv_id":"1908.01506","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A fully automatic 3D CNN pipeline (segmentation plus classification) grades brain tumors as glioblastoma versus lower-grade glioma with about 91-92% accuracy on public and external hospital MRI data.","lead":"The authors built a two-step deep learning system that automatically finds brain tumors on MRI and then decides whether each tumor is an aggressive glioblastoma or a lower-grade glioma. The system reached about 91-92% accuracy on both a public test set and an independent hospital dataset, suggesting it may work outside the lab where it was trained.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Untested assumption that automatic-segmentation ROI errors do not affect grading leaves the reported accuracy and cross-center robustness unsecured.","rationale":"The reader's weakest assumption identifies exactly the untested premise that segmentation-ROI variations do not affect grading. My stress-test confirms this is the most load-bearing point in the argument: the entire end-to-end system's accuracy is mediated by the quality of the automatically extracted ROI, but the paper only evaluates the two stages in isolation. The classification network's behavior on distorted or biased ROIs is unknown, and the external Ghent result cannot be interpreted as a robustness guarantee without knowing whether the segmentation step performs comparably there. This concern is concrete and addressable by a straightforward experiment comparing manual versus automatic ROI extraction on the same test patients and reporting Ghent segmentation Dice. The reader's conditional verdict is appropriate: the paper should be accepted only if such sensitivity analysis is provided or the claim is softened. I see no additional internal inconsistency or leakage that would change the verdict, and the reported accuracies are plausible conditional on the untested assumption holding.","tokens_in":3837,"tokens_out":7095,"duration_ms":76539,"concrete_test":"Run an end-to-end sensitivity analysis on the TCIA test set and the Ghent set: extract ROIs both from the automatic U-Net segmentations and from manual/expert tumor masks, feed both through the trained classifier, and compare per-dataset accuracy, AUC, and per-class errors. Also compute U-Net Dice on the Ghent set. If manual-ROI accuracy exceeds automatic-ROI accuracy by more than a few points, or if Ghent Dice is substantially below the reported 88.53% BraTS Dice, the assumption in Section 3 is false and the claimed robustness is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central accuracy claim depends on the assertion in Section 3 that 'small variations between manual and predicted segmentations won't have a strong influence on the tumour ROI.' This is load-bearing because the 112x112x112 ROI is the only input to the classifier, and its geometry is determined entirely by the U-Net output. The paper reports segmentation Dice and classification accuracy as separate numbers, but never performs the joint comparison: classification from manually defined ROIs versus automatically defined ROIs on the same patients. It also does not report U-Net Dice on the Ghent University Hospital data, even though the Ghent accuracy is the sole basis for the cross-center robustness conclusion. If the U-Net makes systematic errors on external data—under-segmenting non-enhancing tumor, over-including edema, or missing the full tumor extent—the classifier's input distribution shifts. Since the classification network may have been trained on ROIs produced by the same automatic segmentation, it could have absorbed the public-data segmentation bias, making the reported 91.96% on Ghent harder to interpret. Without a direct test of how segmentation errors propagate to grading, the robustness claim is not established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a fully automatic two-stage pipeline for binary glioma grading (glioblastoma versus lower-grade glioma, WHO grades II/III) from pre-therapy structural MRI. The first stage segments the whole tumor with a 3D U-Net; the second extracts a 112x112x112 ROI from the T1ce image inside the predicted mask and classifies it with a 3D residual CNN. The pipeline is trained on 549 patients from public datasets (BraTS 2018 training data, TCGA-GBM, TCGA-LGG, LGG-1p19qDeletion) and evaluated on 80 held-out TCIA patients and on 112 patients from Ghent University Hospital. Reported performance includes a whole-tumor Dice of 88.53% on 76 BraTS 2018 test cases, and classification accuracy of 91.25% on the TCIA test set and 91.96% on the Ghent set. The authors conclude that state-of-the-art grading performance is achieved and that the system is robust to data from different centres.","tokens_in":4014,"tokens_out":2229,"duration_ms":23384,"significance":"If the reported results hold, the paper makes a useful empirical contribution: it demonstrates a fully automatic, two-stage deep-learning system for glioma grading on a relatively large multi-source dataset, with an independent external evaluation. The use of public BraTS/TCIA data and an unseen single-center dataset is a strength, as are the explicit train/validation/test splits and the statement that test patients were excluded from segmentation training. The paper is an extended abstract and does not claim a mathematical derivation, so the main scientific value is the measured accuracy on held-out data. However, the strength of the cross-center robustness claim depends on an untested assumption about the interaction between segmentation errors and classification accuracy, and the reported performance metrics lack uncertainty quantification. These issues are addressable and do not require a change in the core methodology.","major_comments":[{"comment":"The assertion that 'small variations between manual and predicted segmentations won't have a strong influence on the tumour ROI' is load-bearing for the fully automatic claim, but it is never tested. The paper reports segmentation Dice and classification accuracy as separate numbers, but does not compare classification accuracy using manual ROIs versus automatic ROIs on the same patients. Because the 112x112x112 ROI is the only input to the classifier, systematic under-segmentation or over-inclusion of tissue could shift the classifier's input distribution. Please provide such a comparison, or clearly restrict the robustness claim to the specific automatic pipeline as evaluated.","section":"Section 3, first paragraph"},{"comment":"The paper does not specify whether the ROIs used to train the classification network were generated by the automatic U-Net, by manual segmentations, or by a mixture. This is critical for interpreting the reported accuracies: if the classifier was trained on manual ROIs and tested on automatic ROIs, a train/test mismatch exists; if it was trained on automatic ROIs, the network may have absorbed the segmentation bias of the public data. Please state explicitly how the training ROIs were obtained.","section":"Section 2.3"},{"comment":"The U-Net Dice score is reported only for the 76 BraTS 2018 test patients, not for the Ghent University Hospital data. The Ghent accuracy of 91.96% is the sole basis for the conclusion of robustness to different centres and imaging protocols, but without any measure of segmentation quality or input-distribution shift on the external data, this conclusion is not fully supported. Please report segmentation performance on the Ghent data or otherwise analyze the effect of cross-center variation on the classifier inputs.","section":"Sections 2.2 and 3"},{"comment":"No confidence intervals or statistical significance tests are provided for accuracy, AUC, MCC, sensitivity, or specificity. With 80 and 112 test patients, the difference between 91.25% and 91.96% accuracy is within sampling noise, and sensitivity/specificity estimates have wide intervals. Please add bootstrap confidence intervals or another form of uncertainty quantification for the headline metrics.","section":"Table 1"}],"minor_comments":[{"comment":"The author names appear as 'Roel V an Holen' with spacing irregularities; please correct the typesetting.","section":"Author affiliations"},{"comment":"The table header text 'Ghent Univeristy Hospital' contains a typo; it should be 'Ghent University Hospital'.","section":"Table 1 caption"},{"comment":"The term 'dice score' should be capitalized as 'Dice score' for consistency with standard usage.","section":"Section 2.2"},{"comment":"The hyperparameters 'lrinit = 1·10−4' and 'lrinit = 1 · 10−3' would be clearer as 'initial learning rate' spelled out, and the multiplication dot should be typeset consistently.","section":"Section 2.2"},{"comment":"The figure caption contains a garbled substring ('7x7x7 conv, 64 /23x3x3 conv...') that appears to be a layout artifact; please regenerate the figure and caption so that the architecture is readable.","section":"Figure 1"},{"comment":"The claim that performance 'matches state-of-the-art algorithms in the most recent BraTS 2018 challenge' is not accompanied by a direct comparison table or citation to specific challenge results; please clarify the basis of this comparison.","section":"Section 3"}],"recommendation":"major_revision","confidential_remarks":"The paper is an extended abstract, and the main scientific contribution—an empirical accuracy measurement on a fully automatic pipeline—is plausible. The most important fix is to either add the manual-versus-automatic ROI comparison or soften the robustness conclusion; this is feasible within the scope of the data already used. The absence of confidence intervals is a recurring issue in this area, but for a journal version it should be addressed. I do not see any circularity or data leakage in the described splits."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a solid extended abstract that does one thing better than most papers in this niche — it validates the full pipeline on a genuinely held-out external hospital dataset. The architecture is standard (3D U-Net plus 3D residual classifier), but the scale is not: 549 patients for training, 112 patients from Ghent University Hospital for final testing, with explicit separation between segmentation training and test data. The reported accuracies (91.25% on TCIA, 91.96% on Ghent) are plausible and the AUC/MCC numbers help. The authors also avoid self-citation and are honest about the data sources. That is real credit where it's due.\n\nThe soft spots are real but not disqualifying. The stress-test note is on target: the paper asserts in Section 3 that small variations between manual and predicted segmentations won't strongly influence the ROI, but never tests this directly. The classifier sees only the automatically segmented, resized 112x112x112 ROI. If the U-Net systematically under-segments non-enhancing tumor or over-includes edema on the Ghent data, the input distribution shifts, and the 91.96% accuracy becomes harder to interpret. We don't even get U-Net Dice on the Ghent set, which is the only basis for the cross-center robustness claim. Also missing: confidence intervals and comparisons to other methods on the same test sets, so \"state-of-the-art\" is not established. The single external center is a limitation but a minor one for a four-page abstract.\n\nThe paper does not do anything circular or incoherent. The problem is an untested assumption, not a misrepresented result. For an MIDL extended abstract, this is a legitimate feasibility study. I would not call the robustness claim proven, but I would call it a reasonable hypothesis that deserves a proper test.\n\nWho is this for? People building CAD pipelines for brain tumors and referees who want a clean example of external validation design. I'd send this to peer review rather than desk reject — it's a legitimate empirical study with a clear gap that a full version could address (ROI sensitivity analysis, segmentation Dice on the external set, confidence intervals). The stress-test note identifies the paper's actual weak spot, not a manufactured one. My bottom line: conditionally accept, with that missing analysis as the main condition.","headline":"A clinically plausible but incremental two-stage 3D CNN for glioma grading whose external validation is undermined by an untested assumption that segmentation errors don't affect classification.","tokens_in":704,"tokens_out":732,"would_cite":true,"duration_ms":35289,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fully automatic MRI pipeline grades gliomas with 91% accuracy","keywords":["glioma grading","glioblastoma","lower-grade glioma","3D convolutional neural network","MRI segmentation","U-Net","computer-aided diagnosis","pre-therapy MRI"],"falsifier":"On the same public test patients, feed the classifier ROIs defined by expert manual segmentation and compare accuracy to the 91.25% obtained with automatic masks; a large drop would directly falsify the paper's assertion that segmentation errors do not influence the tumour ROI.","tokens_in":3605,"feed_emoji":"🧠","tokens_out":10615,"duration_ms":96141,"temperature":0.7,"pith_summary":"This paper tries to establish that glioma grading—the decision between high-grade glioblastoma and lower-grade glioma—can be made non-invasively and without any manual steps from standard pre-therapy MRI. The proposed system is a two-stage neural network: a 3D U-Net first segments the whole tumour, and a 3D residual classifier then labels the extracted tumour region. On a public multi-centre test cohort the system reaches 91.25% accuracy, and on independent data from a second hospital it reaches 91.96%. A sympathetic reader would care because manual MRI grading is unreliable, while biopsy needed for tissue diagnosis can shorten survival; a dependable automatic grader could guide initial treatment decisions.","feed_headline":"A fully automatic MRI pipeline grades brain tumors with 91% accuracy","feed_subtitle":"Segmentation plus a 3D classifier reaches high accuracy on public data and a second hospital's scans.","key_machinery":"The load-bearing mechanism is the cascade of two networks, with the tumour ROI extracted from the automatic segmentation as the central object. The first stage is a 3D U-Net—a convolutional network with a contracting path and an expanding path that outputs a per-voxel whole-tumour mask—built with 25 features at the highest resolution, instance normalisation and leaky ReLUs, and trained with a soft dice loss. The T1ce volume inside that predicted mask is resized to 112x112x112 and fed to the classifier, a 3D residual network with one 7x7x7 convolution, four residual blocks, instance normalisation and ReLU after every convolution, then average pooling and a two-class output. Randomly zeroing the T1, T2 or FLAIR channels during segmentation training makes the system robust to missing modality sequences. The resizing step is what turns tumours of different shapes and sizes into a fixed input so the classifier only sees the tumour region.","core_discovery":"The central claim is that this fully automatic two-stage pipeline—segmentation with a 3D U-Net, classification of the tumour region of interest with a 3D residual CNN—achieves state-of-the-art binary glioma grading on a large, heterogeneous, multi-centre dataset. On 80 held-out public test patients the classifier reaches 91.25% accuracy with an AUC of 96.29%, and on 112 independent patients from another hospital it reaches 91.96% accuracy with an AUC of 93.39%. The segmenter itself scores a whole-tumour dice coefficient of 88.53% on a public benchmark test set, which the authors say matches top-performing entries in the most recent challenge. They further assert that small differences between manual and predicted segmentations have little effect on the tumour ROI, so the segmentation quality is sufficient for grading. Their conclusion is that state-of-the-art performance is achievable with a system trained on a large heterogeneous dataset and that the method generalises to different imaging protocols and centres.","pith_inferences":["The paper leaves implicit that the fixed-cube resize makes tumour size a potentially usable signal, since larger lesions are more often glioblastoma; a volume-matched control experiment would reveal whether the classifier learns size or tissue-level patterns.","An extension the authors do not attempt is applying the same cascade to finer distinctions—WHO grade II versus III, or IDH-mutant versus IDH-wild-type disease—so it is unknown how far the learned features generalise beyond the coarse binary split.","A single end-to-end network trained jointly on segmentation and classification could plausibly outperform the two-stage design, and the reported accuracy gives a concrete baseline for such a comparison."],"forward_implications":["If the accuracy holds in clinical use, initial management decisions—watch-and-wait versus aggressive resection with chemoradiotherapy—could be made from MRI alone, without biopsy or manual tumour grading.","Because the segmentation network was trained to tolerate missing input channels, the full pipeline could be used at centres that do not acquire all four standard MRI sequences.","The strong result on a second hospital's data implies the method is not tied to one scanner or protocol, provided the preprocessing steps are reproduced.","A fully automatic grader could be run at the time of the first scan, giving an immediate malignancy estimate while histopathology is pending or unavailable."],"supporting_citations":[{"why":"Provides the public multimodal benchmark whose training data train the segmentation U-Net and whose test data measure its dice score.","marker":"Menze et al., 2015"},{"why":"Supplies expert segmentation labels and radiomic features for the TCGA-GBM and TCGA-LGG MRI collections included in the study.","marker":"Bakas et al., 2017a"},{"why":"Defines the 3D U-Net architecture the segmentation stage closely follows.","marker":"Isensee et al., 2018"},{"why":"Represents the top BraTS 2018 segmentation performance against which the 88.53% dice score is matched.","marker":"Myronenko, 2018"},{"why":"Describes the public imaging archive that hosts the TCGA-GBM, TCGA-LGG and LGG-1p19q collections used as data.","marker":"Clark et al., 2013"},{"why":"Provides the TCGA-GBM radiology data that contribute training and test patients.","marker":"Scarpace et al., 2016"},{"why":"Provides the TCGA-LGG radiology data that contribute training and test patients.","marker":"Pedano et al., 2016"},{"why":"Provides the LGG-1p19qDeletion collection, an additional source of lower-grade glioma cases.","marker":"Erickson et al., 2017"}],"fun_headline_variants":["Automatic MRI brain tumor grading hits 91-92% accuracy","Deep learning on MRI auto-grades glioma with 91% accuracy","MRI-only AI grades brain tumor grade at 91% accuracy","3D CNN on MRI achieves 91% accuracy for brain tumor grading","Fully automatic brain tumor grading from MRI reaches 91% accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The pipeline assumes that the tumour region automatically cut out by the 3D U-Net and squeezed into a fixed cube still contains all the image clues needed to distinguish glioblastoma from lower-grade glioma; if common segmentation mistakes leave out or add in the wrong tissue, the reported accuracy would not transfer.","fun_headline_variants_meta":{"raw":{"variants":["Automatic MRI brain tumor grading hits 91-92% accuracy","Deep learning on MRI auto-grades glioma with 91% accuracy","MRI-only AI grades brain tumor grade at 91% accuracy","3D CNN on MRI achieves 91% accuracy for brain tumor grading","Fully automatic brain tumor grading from MRI reaches 91% accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001502,"raw_usage":{"total_tokens":6015,"prompt_tokens":922,"completion_tokens":5093,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":538,"completion_tokens_details":{"reasoning_tokens":5002}},"tokens_in":538,"tokens_out":5093,"duration_ms":34011,"temperature":1.0,"reasoning_tokens":5002,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T15:10:20.049710+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On the same public test patients, feed the classifier ROIs defined by expert manual segmentation and compare accuracy to the 91.25% obtained with automatic masks; a large drop would directly falsify the paper's assertion that segmentation errors do not influence the tumour ROI.","supporting_citations":[{"cited_title":"Data From LGG-1p19qDeletion","cited_arxiv_id":null,"evidence_quote":"Provides the LGG-1p19qDeletion collection, an additional source of lower-grade glioma cases."}],"review_version":1}