{"id":"deeb5e27-2935-4e6f-87fd-fda6150d18ad","arxiv_id":"2506.20786","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Fine-tuned SAM and SAM 2 with high-quality bounding-box prompts achieve higher Dice scores than zero-shot nnU-Net on pediatric brain tumor segmentation, but nnU-Net remains more practical.","lead":"This study compares five AI models for brain tumor segmentation on MRI scans, testing zero-shot and fine-tuned performance under different prompt qualities. It finds that SAM-based models can beat the standard nnU-Net when given very precise bounding boxes, but such precise prompts are unrealistic in daily clinical practice.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline comparison is unequal: fine-tuned SAM/SAM2 with ground-truth bounding boxes are compared with zero-shot nnU-Net, and nnU-Net was never fine-tuned on the target dataset, so the claim that they 'exceed nnU-Net' is not backed by a comparable baseline.","rationale":"I read the paper's goal as benchmarking promptable SAM-based models against nnU-Net on the same BraTS datasets, with special care about data leakage. The strongest claim is not just that SAM/SAM2 can achieve high Dice, but that they exceed nnU-Net; the authors then temper this by asserting nnU-Net remains dominant in practice. For that claim to hold, the comparison must be between models that received comparable training opportunity. It does not: nnU-Net is used zero-shot, while SAM/SAM2 are fine-tuned on the pediatric dataset and are given oracle bounding boxes. The paper openly discloses this in Sec. IV-C, and the reader's rationale also mentions the fine-tuned-versus-zero-shot asymmetry, so this is not a hidden flaw. However, the abstract phrases the result as 'exceeds nnU-Net's segmentation performance,' which overstates what was measured. The single-sequence FLAIR input for SAM is a real limitation, but it biases against SAM and thus is not the most decisive threat to the claim; the training asymmetry is. A concrete resolution is to fine-tune nnU-Net on the same splits and compare. If nnU-Net remains below the SAM scores, the paper's conclusion is strengthened; if not, the headline requires revision. I therefore keep the reader's conditional verdict unchanged. I did not find evidence of methodological fraud or hidden data leakage; the code is provided and the data-leakage discussion is a strength. The remaining issues, such as missing confidence intervals, non-nested hyperparameter selection, and oracle prompts, are secondary to the training-asymmetry concern and would also need attention in a revision.","tokens_in":11229,"tokens_out":6443,"duration_ms":69878,"concrete_test":"Fine-tune nnU-Net on the same BraTS 2023 Pediatrics folds used for SAM/SAM2, with the same 50/25/25 split and early stopping, then evaluate on the held-out test folds. Compute per-fold Dice and report means with confidence intervals. If the fine-tuned nnU-Net reaches or exceeds 0.894, the headline 'exceeds nnU-Net' is not supported; if it remains below 0.860, the concern is resolved and the original comparison stands.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Sec. IV-C states that all promptable models were fine-tuned on the BraTS 2023 Pediatrics dataset 'except for nnU-Net due to computational resource limitations.' The abstract and Discussion nevertheless use the fine-tuned SAM/SAM2 high-quality box scores (0.894/0.893, Table III) as the headline result and compare them with a zero-shot nnU-Net score (0.860, Table II). This is not a like-for-like comparison: nnU-Net was trained only on BraTS 2021 adult data and applied zero-shot to the pediatric dataset, whereas SAM/SAM2 received in-domain pediatric fine-tuning. The paper itself cites BraTS solutions in which nnU-Net is part of winning ensembles, but those systems were trained on the target data, not run zero-shot. Consequently, the strongest quantitative claim, that promptable models exceed nnU-Net, could be an artifact of unequal training effort. The complementary conclusion that nnU-Net remains dominant because high-quality prompts are impractical is a clinical-practicality argument, not a benchmark result, and it is not tested against an in-domain fine-tuned nnU-Net. The oracle high-quality box prompt (tightest possible box around the ground-truth tumour, Sec. III-C) reinforces that the favorable comparison operates under information unavailable at inference time.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper benchmarks five medical image segmentation approaches (SAM, SAM 2, MedSAM, SAM-Med-3D, and nnU-Net) on the BraTS 2023 adult glioma and pediatrics datasets, evaluating promptable models under high-, medium-, and low-quality point and bounding-box prompts, in both zero-shot and fine-tuned settings. The authors report Dice scores showing that fine-tuned SAM and SAM 2 with high-quality bounding-box prompts reach 0.894 and 0.893 on the pediatrics dataset, exceeding the 0.860 Dice of zero-shot nnU-Net, but they conclude that nnU-Net remains the dominant clinical choice because highly accurate prompts are impractical. The paper includes inference and training time measurements and makes code available.","tokens_in":11476,"tokens_out":5621,"duration_ms":67148,"significance":"If the comparisons were like-for-like, the paper would provide a useful benchmark for model selection in brain tumour segmentation, and the prompt-quality sensitivity analysis would be a valuable reference for researchers deploying promptable models. The authors deserve credit for using the BraTS 2023 pediatric dataset to mitigate data leakage, for evaluating a consistent set of prompt types across models, and for releasing their code. However, the central quantitative claim is weakened by an unequal comparison: fine-tuned promptable models are compared against a zero-shot nnU-Net, and the input-channel configuration differs between the SAM-based models and nnU-Net. The lack of uncertainty quantification further limits the evidentiary weight of the headline differences.","major_comments":[{"comment":"The headline comparison is not like-for-like. Section IV-C states that all promptable models were fine-tuned on the BraTS 2023 pediatrics dataset 'except for nnU-Net due to computational resource limitations,' and Table III reports fine-tuned SAM/SAM 2 high-quality box Dice scores of 0.894/0.893, which the abstract and Section V compare with the zero-shot nnU-Net score of 0.860 from Table II. Because nnU-Net was trained on adult BraTS 2021 data and applied zero-shot, the conclusion that SAM/SAM 2 'exceeds nnU-Net's segmentation performance' is not supported as a benchmark result; it may reflect unequal training effort. The authors should either fine-tune nnU-Net on the pediatrics dataset or explicitly restrict the claim to 'fine-tuned promptable models versus zero-shot nnU-Net' and clearly state the asymmetry in every summary of the result.","section":"Section IV-C, Table III; Abstract"},{"comment":"The comparison is also confounded by input dimensionality. Section III-B states that only the FLAIR sequence, replicated to three channels, is used for all SAM-based models 'for fairness of comparison against SAM-Med-3D,' while nnU-Net receives all four MRI sequences. This makes the comparison against nnU-Net unequal in the opposite direction. If multi-sequence input improved the SAM-based models' performance, the reported gaps could change materially. The authors should either train or evaluate nnU-Net on FLAIR-only input, or evaluate the SAM-based models with four sequences where architecturally possible, or explicitly frame the result as 'single-sequence promptable models versus four-sequence non-promptable model.'","section":"Section III-B"},{"comment":"No uncertainty quantification is provided for any of the reported Dice scores. With 99 pediatric patients, the difference between SAM's fine-tuned high-quality box score (0.894) and nnU-Net's zero-shot score (0.860) could fall within sampling variability, and the same applies to several other comparisons in Tables I-III. The authors should report per-fold or per-patient variance, confidence intervals, and paired statistical tests (e.g., Wilcoxon signed-rank) for the central pairwise comparisons, especially those used to support the 'exceeds nnU-Net' claim.","section":"Tables I-III; Section IV-B and IV-C"},{"comment":"The prompt-quality ordering is partly constructed rather than discovered. In Section III-C, high-quality box prompts are defined as the tightest bounding box around the ground-truth tumour, and point-prompt quality is defined by the number of points sampled from the ground-truth mask. It is therefore unsurprising that high-quality prompts outperform low-quality prompts, and the Discussion's conclusion that high-quality prompts are 'impractical in a clinical setting' is a practicality argument rather than a benchmark result. The paper should separate the oracle-prompt upper-bound benchmarking from a clinically meaningful evaluation that uses human-like prompts, and should not treat the prompt-quality ordering as evidence about clinical usability.","section":"Section III-C; Section V"},{"comment":"The fine-tuning protocol may optimistically bias the reported fine-tuned scores. Section III-E describes a 'modified 4-fold cross validation' in which hyperparameters are chosen by average validation Dice across folds, early stopping and checkpoint selection are also performed on the same validation folds, and the final scores are then computed on the corresponding test folds. This is not a nested cross-validation, and the repeated use of the validation folds for model selection means the test-fold scores may be optimistically biased. The authors should report the variance across folds and, ideally, add a fully held-out evaluation or nested procedure to support the fine-tuned results.","section":"Section III-E"}],"minor_comments":[{"comment":"Section III-B lists U-Net as one of the five benchmarked models, but no U-Net results appear in any of the results tables; the paper should either report U-Net results or remove U-Net from the model list.","section":"Section III-B; Tables I-III"},{"comment":"The in-text citations appear to be misaligned with the bibliography in several places; for example, SAM-Med3D is cited as [10] in the introduction, but reference [10] is the BraTS 2021 paper. The authors should reconcile all citations with the reference list.","section":"References"},{"comment":"The text says early stopping is applied 'when the validation Dice score stopped decreasing'; this should presumably read 'stopped increasing.'","section":"Section III-E"},{"comment":"The evaluation metric section states that Dice scores are computed on the 3D scan, but the BraTS benchmark conventionally reports Dice separately for whole tumour, tumour core, and enhancing tumour; the paper should state which tumour subregions are included in the reported Dice scores.","section":"Section III-D"},{"comment":"The phrase 'prompts overlayed with the ground truth mask' contains a typo; it should be 'overlaid.'","section":"Fig. 1 caption"}],"recommendation":"major_revision","confidential_remarks":"The manuscript contains a useful benchmark resource, but the central comparative claim is not supported by the experimental design as presented. The most important fix is to make the nnU-Net baseline comparable, either by fine-tuning it on the pediatric dataset or by clearly limiting the conclusion to zero-shot nnU-Net. The lack of error bars and the input-channel mismatch are additional load-bearing issues. If these are addressed, the paper could be acceptable for a venue that values reproducible benchmarking; in its current form, the abstract's 'exceeds nnU-Net' claim is likely to mislead readers."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is a genuinely useful benchmarking effort: it puts five segmentation models on the same two BraTS 2023 datasets, varies prompt quality in a sensible way, fine-tunes the promptable models on the pediatrics set, and releases the code. The attention to data leakage—using the pediatrics dataset because it wasn't in MedSAM's training—is a good call, and the inference/training time tables are a nice addition that most similar papers skip.\n\nThe main problem is the headline result. The abstract says SAM and SAM 2 reach Dice scores of 0.894 and 0.893, which \"exceeds nnU-Net's segmentation performance.\" But that comparison is between fine-tuned SAM models given high-quality, ground-truth-derived bounding boxes and a zero-shot nnU-Net that was never fine-tuned on the target dataset. The authors admit in Section IV-C that nnU-Net wasn't fine-tuned due to computational limits. That's an honest admission, but it doesn't make the comparison fair. The conclusion that nnU-Net remains dominant because accurate prompts are impractical is a clinical-practicality argument, not a measured benchmark result. It may well be true, but this paper doesn't test it against an in-domain fine-tuned nnU-Net.\n\nOther soft spots: the high-quality box prompt is the tightest possible box around the ground-truth tumour, which is an oracle condition; the SAM-based models get only the FLAIR sequence (replicated to three channels) while nnU-Net gets all four sequences; and there are no confidence intervals or error bars anywhere, so we can't tell if the small gaps between SAM 2 and nnU-Net are meaningful. The hyperparameter selection is a non-nested 50/25/25 split, which is acceptable for a benchmark but should be stated as a limitation.\n\nThat said, the zero-shot results on the pediatrics dataset are still a useful reference table, and the fine-tuning curves show a real, if unsurprising, pattern: point prompts improve more than box prompts because they start worse. The authorship is careful about stating data leakage and input constraints in the methods.\n\nWho should read this: anyone comparing promptable segmentation models on medical MRI and wanting a quick, reproducible baseline on BraTS. It deserves peer review, but the abstract and discussion need to be reworded so the fine-tuned-versus-zero-shot asymmetry is explicit, and ideally the authors should add a fine-tuned nnU-Net baseline or drop the \"exceeds\" claim.\n\nVerdict: engage with it, but read the tables critically and don't take the headline comparison at face value.","headline":"A useful benchmark, but the headline claim that fine-tuned SAM models beat nnU-Net is an apples-to-oranges comparison that needs rephrasing or a proper nnU-Net baseline.","tokens_in":11965,"tokens_out":1708,"would_cite":true,"duration_ms":21156,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuned SAM and SAM 2 outscore nnU-Net on pediatric brain-tumour MRI when handed near-perfect bounding-box prompts, yet nnU-Net remains the clinical default because such prompts are impractical to obtain.","keywords":["Image Segmentation","Model Prompting","Model Benchmarking","SAM","nnU-Net","Deep Learning","BraTS","MRI brain tumour segmentation"],"falsifier":"Repeat the pediatric benchmark feeding the SAM-family models all four MRI sequences (multi-sequence input) and, separately, drawing bounding-box prompts automatically from nnU-Net's own predicted masks. If fine-tuned SAM or SAM 2 then beats nnU-Net's 0.860 with point prompts or with cheap automatic boxes, the paper's conclusion that accurate prompts are impractical — and hence that nnU-Net remains clinically dominant — is falsified.","tokens_in":11018,"feed_emoji":"🧠","tokens_out":27806,"duration_ms":261301,"temperature":0.7,"pith_summary":"This paper asks whether general-purpose, promptable segmentation models — SAM, SAM 2, MedSAM, and SAM-Med-3D — can compete with nnU-Net, the self-configuring U-Net that has dominated medical segmentation challenges, on brain-tumour MRI. The authors benchmark all five models on the BraTS 2023 adult and pediatric datasets at three prompt-quality levels for both points and bounding boxes, then fine-tune the promptable models on the pediatric data. With a near-perfect bounding box, fine-tuned SAM and SAM 2 reach Dice scores of 0.894 and 0.893 on the pediatric set, above nnU-Net's 0.860, yet the authors conclude that nnU-Net stays the clinical default because prompts at that accuracy would not be available in practice. Prompt quality is the decisive variable: loosening the box costs about 0.1 of Dice in zero-shot inference, and point prompts trail box prompts by a wide margin even after fine-tuning. The result matters because it separates what these models can do from what they can be asked to do, pointing at prompt acquisition rather than segmentation ability as the real obstacle to clinical use.","feed_headline":"Fine-tuned SAM models beat nnU-Net on pediatric brain tumours","feed_subtitle":"Prompt quality decides the winner — and near-perfect prompts are exactly what clinics can't supply.","key_machinery":"The argument is carried by a controlled benchmarking protocol rather than by a single new algorithm. The load-bearing pieces are the BraTS 2023 pediatrics dataset, which none of the models saw during training and which therefore isolates genuine zero-shot and fine-tuned performance; a three-tier prompt-quality ladder — bounding boxes that are tight, inflated and shifted by 10%, or inflated and shifted by 30%, alongside point prompts of 10, 5, and 1 points — which turns prompt imprecision into a measured variable; and a modified 4-fold fine-tuning scheme (50% train, 25% validation, 25% test per fold, early stopping, best-checkpoint selection) applied to every SAM-family model. Dice scores are computed on the compiled 3D mask so that slice-wise 2D models, the volumetric SAM-Med-3D, and nnU-Net are compared on one metric, and inference times are measured per patient. One design choice shapes the entire comparison: the SAM-family models receive only the FLAIR sequence, replicated to three channels, 'for fairness of comparison against SAM-Med-3D,' while nnU-Net receives all four sequences, making the fairness of that single-sequence restriction the pivot on which the conclusion turns.","core_discovery":"The paper's central claim is that promptable foundation models are capable of out-segmenting the strongest specialised medical segmentation network, provided the prompt is accurate enough — and that this requirement is precisely what keeps them out of clinical use. On the BraTS 2023 pediatrics dataset, chosen because no evaluated model was trained on it (in contrast to the adult set, where the paper notes nnU-Net's 0.958 and several SAM-family scores are inflated by data leakage), zero-shot SAM 2 with the tightest bounding box scores 0.886 Dice, already above nnU-Net's 0.860. After fine-tuning on the 99-patient pediatric set, SAM reaches 0.894 and SAM 2 reaches 0.893 with high-quality boxes, and SAM 2 also exceeds nnU-Net with medium-quality boxes (0.881). The authors nevertheless conclude that nnU-Net 'remains the dominant medical image segmentation network,' because a high-quality prompt encodes the tumour's true extent — knowledge a clinician does not have. Point prompts still trail even after fine-tuning: the best single-point results are 0.737 for SAM-Med-3D and 0.670–0.686 for SAM and SAM 2, and the best ten-point result (0.814 for SAM-Med-3D) stays below nnU-Net. In zero-shot inference, high-quality boxes beat low-quality boxes by roughly 0.1 Dice on average, showing how steeply performance depends on prompt fidelity.","pith_inferences":["A natural extension the paper leaves untested: because the single-sequence input rule was adopted for fairness with SAM-Med-3D, feeding SAM, SAM 2, and MedSAM three distinct MRI sequences (for instance T1Gd, T2, and FLAIR in their three channels) could plausibly push their Dice higher still and either shrink or erase the gap to nnU-Net.","The most direct clinical test of the dominance claim is to automate the prompt: take nnU-Net's own prediction as the bounding box for fine-tuned SAM or SAM 2, and compare that pipeline against nnU-Net alone on the pediatric set — if it wins, the 'prompts are impractical' objection collapses.","With only 99 pediatric patients, the large fine-tuning gains (roughly doubling single-point Dice for the 2D models) suggest promptable generalist models are data-efficient on niche medical domains; rerunning the protocol on other tumour types or on CT would show whether the model ranking is stable.","MedSAM's collapse into blob-like masks that ignore the bounding box looks like domain shift rather than lack of capacity; a longer fine-tuning schedule or box-only training would test whether it can be rescued on brain MRI, something the paper's fixed protocol could not reveal."],"forward_implications":["With tight bounding-box prompts, fine-tuned SAM and SAM 2 beat zero-shot nnU-Net on leakage-free pediatric data (0.894 and 0.893 versus 0.860), so the practical ceiling of promptable models is set by prompt quality, not segmentation capacity.","Point prompting is not yet clinically competitive: even after fine-tuning, the best single-point and ten-point Dice scores for the 2D models (0.686 and 0.775 for SAM 2, with the volumetric SAM-Med-3D reaching 0.737 and 0.814) remain below nnU-Net's 0.860.","Prompt quality dominates zero-shot outcomes — high-quality boxes beat low-quality boxes by roughly 0.1 Dice — and lower-quality prompts gain the most from fine-tuning, which the authors read as evidence that fine-tuning mainly teaches models to resolve prompt ambiguity.","BraTS-style challenges withhold test masks, so promptable models cannot be entered in them at all; this benchmark had to synthesize prompts from public training masks, which the paper gives as the reason competition rankings have not captured the SAM family's performance.","Measured inference times (0.8 seconds per patient for SAM-Med-3D, 7.2 for SAM 2, 22 for nnU-Net, 72 for SAM, 120 for MedSAM) show that runtime practicality, not just accuracy, underlies the paper's conclusion that nnU-Net remains dominant."],"supporting_citations":[{"why":"Defines the self-configuring nnU-Net architecture that serves as the benchmark baseline and the reference point for the dominance conclusion.","marker":"[4]"},{"why":"Supplies the specific pre-trained nnU-Net (trained on BraTS 2021) that this study loads to perform zero-shot inference on both datasets.","marker":"[16]"},{"why":"Defines the original SAM, the promptable ViT segmenter whose zero-shot and fine-tuned box and point prompts are evaluated.","marker":"[6]"},{"why":"Defines SAM 2, including the video segmentation mode used here to propagate prompts every tenth slice.","marker":"[8]"},{"why":"Defines MedSAM, the medical SAM variant that the paper finds to underperform, producing blob-like masks that stray from the prompt.","marker":"[7]"},{"why":"Defines SAM-Med3D, the volumetric point-promptable model that scores highest among point-prompt methods and supplies the 10-point prompt convention.","marker":"[9]"},{"why":"The BraTS 2021 benchmark paper; grounds the dataset lineage and the challenge format that withholds test masks, which the paper cites to explain why promptable models cannot compete in BraTS-style challenges.","marker":"[10]"},{"why":"Defines the BraTS 2023 pediatrics dataset, the no-leakage test bed on which the zero-shot and fine-tuning comparisons rest.","marker":"[23]"},{"why":"Documents how data leakage inflates reported performance; the paper's reason for treating the pediatric dataset as the trustworthy comparison set.","marker":"[22]"},{"why":"Shows FLAIR is the best single MRI sequence for brain-tumour segmentation; grounds the decision to give the SAM-family models only the FLAIR sequence, the paper's pivotal fairness assumption.","marker":"[25]"}],"fun_headline_variants":["SAM beats nnU-Net only with perfect prompts","Prompt quality decides brain tumour segmentation winner","Perfect boxes let SAM top nnU-Net, but clinics can't","nnU-Net still dominant as SAM demands impossible prompts","Prompt-bound: SAM tops nnU-Net only with perfect boxes"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison assumes it is fair to feed the SAM-family models only the FLAIR MRI sequence (replicated to fill their three input channels) while nnU-Net receives all four sequences; if the extra sequences materially improve the promptable models, the reported gap to nnU-Net could shrink or reverse.","fun_headline_variants_meta":{"raw":{"variants":["SAM beats nnU-Net only with perfect prompts","Prompt quality decides brain tumour segmentation winner","Perfect boxes let SAM top nnU-Net, but clinics can't","nnU-Net still dominant as SAM demands impossible prompts","Prompt-bound: SAM tops nnU-Net only with perfect boxes"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001597,"raw_usage":{"total_tokens":6441,"prompt_tokens":1097,"completion_tokens":5344,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":713,"completion_tokens_details":{"reasoning_tokens":5263}},"tokens_in":713,"tokens_out":5344,"duration_ms":35162,"temperature":1.0,"reasoning_tokens":5263,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T22:41:40.867537+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Repeat the pediatric benchmark feeding the SAM-family models all four MRI sequences (multi-sequence input) and, separately, drawing bounding-box prompts automatically from nnU-Net's own predicted masks. If fine-tuned SAM or SAM 2 then beats nnU-Net's 0.860 with point prompts or with cheap automatic boxes, the paper's conclusion that accurate prompts are impractical — and hence that nnU-Net remains clinically dominant — is falsified.","supporting_citations":[],"review_version":1}