REVIEW 5 major objections 5 minor 26 references
AI-Driven MRI-based Brain Tumour Segmentation Benchmarking
T0 review · 5 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read Fine-tuned SAM and SAM 2 outscore nnU-Net on pediatric brain-tumour MRI when handed near-perfect bounding-box prompts, yet nnU-Net remains the clinical default because such prompts are impractical to obtain.
desk verdict A useful benchmark, but the headline claim that fine-tuned SAM models beat nnU-Net is an apples-to-oranges comparison that needs rephrasing or a proper nnU-Net baseline. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The argument is carried by a controlled benchmarking protocol rather than by a single new algorithm. The load-bearing pieces are the BraTS 2023 pediatrics dataset, which none of the models saw during training and which therefore isolates genuine zero-shot and fine-tuned performance; a three-tier prompt-quality ladder — bounding boxes that are tight, inflated and shifted by 10%, or inflated and shifted by 30%, alongside point prompts of 10, 5, and 1 points — which turns prompt imprecision into a measured variable; and a modified 4-fold fine-tuning scheme (50% train, 25% validation, 25% test per fold, early stopping, best-checkpoint selection) applied to every SAM-family model. Dice scores are computed on the compiled 3D mask so that slice-wise 2D models, the volumetric SAM-Med-3D, and nnU-Net are compared on one metric, and inference times are measured per patient. One design choice shapes the entire comparison: the SAM-family models receive only the FLAIR sequence, replicated to three channels, 'for fairness of comparison against SAM-Med-3D,' while nnU-Net receives all four sequences, making the fairness of that single-sequence restriction the pivot on which the conclusion turns.
What would settle it
Repeat the pediatric benchmark feeding the SAM-family models all four MRI sequences (multi-sequence input) and, separately, drawing bounding-box prompts automatically from nnU-Net's own predicted masks. If fine-tuned SAM or SAM 2 then beats nnU-Net's 0.860 with point prompts or with cheap automatic boxes, the paper's conclusion that accurate prompts are impractical — and hence that nnU-Net remains clinically dominant — is falsified.
Extended reading notes
Core claim
The paper's central claim is that promptable foundation models are capable of out-segmenting the strongest specialised medical segmentation network, provided the prompt is accurate enough — and that this requirement is precisely what keeps them out of clinical use. On the BraTS 2023 pediatrics dataset, chosen because no evaluated model was trained on it (in contrast to the adult set, where the paper notes nnU-Net's 0.958 and several SAM-family scores are inflated by data leakage), zero-shot SAM 2 with the tightest bounding box scores 0.886 Dice, already above nnU-Net's 0.860. After fine-tuning on the 99-patient pediatric set, SAM reaches 0.894 and SAM 2 reaches 0.893 with high-quality boxes, and SAM 2 also exceeds nnU-Net with medium-quality boxes (0.881). The authors nevertheless conclude that nnU-Net 'remains the dominant medical image segmentation network,' because a high-quality prompt encodes the tumour's true extent — knowledge a clinician does not have. Point prompts still trail even after fine-tuning: the best single-point results are 0.737 for SAM-Med-3D and 0.670–0.686 for SAM and SAM 2, and the best ten-point result (0.814 for SAM-Med-3D) stays below nnU-Net. In zero-shot inference, high-quality boxes beat low-quality boxes by roughly 0.1 Dice on average, showing how steeply performance depends on prompt fidelity.
Load-bearing premise
The comparison assumes it is fair to feed the SAM-family models only the FLAIR MRI sequence (replicated to fill their three input channels) while nnU-Net receives all four sequences; if the extra sequences materially improve the promptable models, the reported gap to nnU-Net could shrink or reverse.
Editorial extensions
If this is right
- With tight bounding-box prompts, fine-tuned SAM and SAM 2 beat zero-shot nnU-Net on leakage-free pediatric data (0.894 and 0.893 versus 0.860), so the practical ceiling of promptable models is set by prompt quality, not segmentation capacity.
- Point prompting is not yet clinically competitive: even after fine-tuning, the best single-point and ten-point Dice scores for the 2D models (0.686 and 0.775 for SAM 2, with the volumetric SAM-Med-3D reaching 0.737 and 0.814) remain below nnU-Net's 0.860.
- Prompt quality dominates zero-shot outcomes — high-quality boxes beat low-quality boxes by roughly 0.1 Dice — and lower-quality prompts gain the most from fine-tuning, which the authors read as evidence that fine-tuning mainly teaches models to resolve prompt ambiguity.
- BraTS-style challenges withhold test masks, so promptable models cannot be entered in them at all; this benchmark had to synthesize prompts from public training masks, which the paper gives as the reason competition rankings have not captured the SAM family's performance.
- Measured inference times (0.8 seconds per patient for SAM-Med-3D, 7.2 for SAM 2, 22 for nnU-Net, 72 for SAM, 120 for MedSAM) show that runtime practicality, not just accuracy, underlies the paper's conclusion that nnU-Net remains dominant.
Reading between the lines
- A natural extension the paper leaves untested: because the single-sequence input rule was adopted for fairness with SAM-Med-3D, feeding SAM, SAM 2, and MedSAM three distinct MRI sequences (for instance T1Gd, T2, and FLAIR in their three channels) could plausibly push their Dice higher still and either shrink or erase the gap to nnU-Net.
- The most direct clinical test of the dominance claim is to automate the prompt: take nnU-Net's own prediction as the bounding box for fine-tuned SAM or SAM 2, and compare that pipeline against nnU-Net alone on the pediatric set — if it wins, the 'prompts are impractical' objection collapses.
- With only 99 pediatric patients, the large fine-tuning gains (roughly doubling single-point Dice for the 2D models) suggest promptable generalist models are data-efficient on niche medical domains; rerunning the protocol on other tumour types or on CT would show whether the model ranking is stable.
- MedSAM's collapse into blob-like masks that ignore the bounding box looks like domain shift rather than lack of capacity; a longer fine-tuning schedule or box-only training would test whether it can be rescued on brain MRI, something the paper's fixed protocol could not reveal.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper benchmarks five medical image segmentation approaches (SAM, SAM 2, MedSAM, SAM-Med-3D, and nnU-Net) on the BraTS 2023 adult glioma and pediatrics datasets, evaluating promptable models under high-, medium-, and low-quality point and bounding-box prompts, in both zero-shot and fine-tuned settings. The authors report Dice scores showing that fine-tuned SAM and SAM 2 with high-quality bounding-box prompts reach 0.894 and 0.893 on the pediatrics dataset, exceeding the 0.860 Dice of zero-shot nnU-Net, but they conclude that nnU-Net remains the dominant clinical choice because highly accurate prompts are impractical. The paper includes inference and training time measurements and makes code available.
Significance. If the comparisons were like-for-like, the paper would provide a useful benchmark for model selection in brain tumour segmentation, and the prompt-quality sensitivity analysis would be a valuable reference for researchers deploying promptable models. The authors deserve credit for using the BraTS 2023 pediatric dataset to mitigate data leakage, for evaluating a consistent set of prompt types across models, and for releasing their code. However, the central quantitative claim is weakened by an unequal comparison: fine-tuned promptable models are compared against a zero-shot nnU-Net, and the input-channel configuration differs between the SAM-based models and nnU-Net. The lack of uncertainty quantification further limits the evidentiary weight of the headline differences.
major comments (5)
- [Section IV-C, Table III; Abstract] The headline comparison is not like-for-like. Section IV-C states that all promptable models were fine-tuned on the BraTS 2023 pediatrics dataset 'except for nnU-Net due to computational resource limitations,' and Table III reports fine-tuned SAM/SAM 2 high-quality box Dice scores of 0.894/0.893, which the abstract and Section V compare with the zero-shot nnU-Net score of 0.860 from Table II. Because nnU-Net was trained on adult BraTS 2021 data and applied zero-shot, the conclusion that SAM/SAM 2 'exceeds nnU-Net's segmentation performance' is not supported as a benchmark result; it may reflect unequal training effort. The authors should either fine-tune nnU-Net on the pediatrics dataset or explicitly restrict the claim to 'fine-tuned promptable models versus zero-shot nnU-Net' and clearly state the asymmetry in every summary of the result.
- [Section III-B] The comparison is also confounded by input dimensionality. Section III-B states that only the FLAIR sequence, replicated to three channels, is used for all SAM-based models 'for fairness of comparison against SAM-Med-3D,' while nnU-Net receives all four MRI sequences. This makes the comparison against nnU-Net unequal in the opposite direction. If multi-sequence input improved the SAM-based models' performance, the reported gaps could change materially. The authors should either train or evaluate nnU-Net on FLAIR-only input, or evaluate the SAM-based models with four sequences where architecturally possible, or explicitly frame the result as 'single-sequence promptable models versus four-sequence non-promptable model.'
- [Tables I-III; Section IV-B and IV-C] No uncertainty quantification is provided for any of the reported Dice scores. With 99 pediatric patients, the difference between SAM's fine-tuned high-quality box score (0.894) and nnU-Net's zero-shot score (0.860) could fall within sampling variability, and the same applies to several other comparisons in Tables I-III. The authors should report per-fold or per-patient variance, confidence intervals, and paired statistical tests (e.g., Wilcoxon signed-rank) for the central pairwise comparisons, especially those used to support the 'exceeds nnU-Net' claim.
- [Section III-C; Section V] The prompt-quality ordering is partly constructed rather than discovered. In Section III-C, high-quality box prompts are defined as the tightest bounding box around the ground-truth tumour, and point-prompt quality is defined by the number of points sampled from the ground-truth mask. It is therefore unsurprising that high-quality prompts outperform low-quality prompts, and the Discussion's conclusion that high-quality prompts are 'impractical in a clinical setting' is a practicality argument rather than a benchmark result. The paper should separate the oracle-prompt upper-bound benchmarking from a clinically meaningful evaluation that uses human-like prompts, and should not treat the prompt-quality ordering as evidence about clinical usability.
- [Section III-E] The fine-tuning protocol may optimistically bias the reported fine-tuned scores. Section III-E describes a 'modified 4-fold cross validation' in which hyperparameters are chosen by average validation Dice across folds, early stopping and checkpoint selection are also performed on the same validation folds, and the final scores are then computed on the corresponding test folds. This is not a nested cross-validation, and the repeated use of the validation folds for model selection means the test-fold scores may be optimistically biased. The authors should report the variance across folds and, ideally, add a fully held-out evaluation or nested procedure to support the fine-tuned results.
minor comments (5)
- [Section III-B; Tables I-III] Section III-B lists U-Net as one of the five benchmarked models, but no U-Net results appear in any of the results tables; the paper should either report U-Net results or remove U-Net from the model list.
- [References] The in-text citations appear to be misaligned with the bibliography in several places; for example, SAM-Med3D is cited as [10] in the introduction, but reference [10] is the BraTS 2021 paper. The authors should reconcile all citations with the reference list.
- [Section III-E] The text says early stopping is applied 'when the validation Dice score stopped decreasing'; this should presumably read 'stopped increasing.'
- [Section III-D] The evaluation metric section states that Dice scores are computed on the 3D scan, but the BraTS benchmark conventionally reports Dice separately for whole tumour, tumour core, and enhancing tumour; the paper should state which tumour subregions are included in the reported Dice scores.
- [Fig. 1 caption] The phrase 'prompts overlayed with the ground truth mask' contains a typo; it should be 'overlaid.'
Circularity Check
No significant circularity: the main benchmarking claims are externally grounded in BraTS evaluations, and no fitted parameter or self-citation is renamed as a prediction.
full rationale
This paper is an empirical benchmark rather than a derivation chain, so the main circularity patterns do not apply. The central comparisons (SAM, SAM 2, MedSAM, SAM-Med-3D, and nnU-Net on BraTS 2023 adult and pediatric datasets) are supported by Dice scores computed on public data, with fine-tuned models evaluated on separate test folds (Section III-E, Table III). No parameter is fitted to a subset and then reported as a prediction of that same subset; the zero-shot results are external, and the fine-tuned results are held-out test-fold evaluations. There are no load-bearing self-citations: the reference list contains no works by the present authors, and no uniqueness theorem or ansatz is imported from the authors' prior work. The prompt-quality analysis labels prompts by their proximity to ground-truth masks (Section III-C), so both the prompt labels and the Dice metric reference the ground truth; however, the paper does not derive the performance ordering from that definition, and the measured ordering is not strictly forced by construction (e.g., MedSAM's medium-quality box score of 0.633 exceeds its high-quality box score of 0.572 in Table II). The unequal comparison of fine-tuned SAM models against zero-shot nnU-Net, explicitly acknowledged in Section IV-C ('except for nnU-Net due to computational resource limitations'), is a limitation on the strength of the headline claim but is not circular reasoning. Likewise, the conclusion that nnU-Net remains dominant because high-quality prompts are impractical in clinical settings is a practicality argument rather than a circular derivation. Overall, the paper's claims are self-contained with respect to circularity, though they carry separate fairness and generalizability risks.
Assumptions & free parameters
free parameters (4)
- Box prompt scaling (medium) =
10% enlargement, 10% shift
- Box prompt scaling (low) =
30% enlargement, 20% shift
- Point prompt counts =
1, 5, 10
- Validation prompt mix =
50% single point, 50% perfect box
assumptions (4)
- domain assumption FLAIR sequence is the best single MRI sequence for brain tumor segmentation
- domain assumption The ground truth segmentation masks are accurate and reliable
- domain assumption The 4-fold cross-validation with hyperparameter selection on validation folds yields unbiased test performance
- domain assumption The BraTS 2023 adult and pediatric datasets share identical preprocessing and resolution
Cite this review
Pith. "Pith review of AI-Driven MRI-based Brain Tumour Segmentation Benchmarking." pith.science (2026). https://pith.science/paper/IDT7ZJVW
@misc{pith2026250620786,
author = {Pith},
title = {Pith review of: AI-Driven MRI-based Brain Tumour Segmentation Benchmarking},
year = {2026},
howpublished = {\url{https://pith.science/paper/IDT7ZJVW}},
note = {Machine review of arXiv:2506.20786}
}
read the original abstract
Medical image segmentation has greatly aided medical diagnosis, with U-Net based architectures and nnU-Net providing state-of-the-art performance. There have been numerous general promptable models and medical variations introduced in recent years, but there is currently a lack of evaluation and comparison of these models across a variety of prompt qualities on a common medical dataset. This research uses Segment Anything Model (SAM), Segment Anything Model 2 (SAM 2), MedSAM, SAM-Med-3D, and nnU-Net to obtain zero-shot inference on the BraTS 2023 adult glioma and pediatrics dataset across multiple prompt qualities for both points and bounding boxes. Several of these models exhibit promising Dice scores, particularly SAM and SAM 2 achieving scores of up to 0.894 and 0.893, respectively when given extremely accurate bounding box prompts which exceeds nnU-Net's segmentation performance. However, nnU-Net remains the dominant medical image segmentation network due to the impracticality of providing highly accurate prompts to the models. The model and prompt evaluation, as well as the comparison, are extended through fine-tuning SAM, SAM 2, MedSAM, and SAM-Med-3D on the pediatrics dataset. The improvements in point prompt performance after fine-tuning are substantial and show promise for future investigation, but are unable to achieve better segmentation than bounding boxes or nnU-Net.
Figures
Reference graph
Works this paper leans on
-
[1]
Brain tumor segmentation based on region of interest-aided localization and segmentation U- Net,
S. Li, J. Liu, and Z. Song, “Brain tumor segmentation based on region of interest-aided localization and segmentation U- Net,” International Journal of Machine Learning and Cybernetics, vol. 13, no. 9, 2022, doi: 10.1007/s13042-022-01536-4
-
[2]
The Medical Segmentation Decathlon,
M. Antonelli et al. , “The Medical Segmentation Decathlon,” Nat Commun, vol. 13, no. 1, 2022, doi: 10.1038/s41467-022-30695-9
-
[3]
U-net: Convolutional networks for biomedical image segmentation,
O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 2015. doi: 10.1007/978- 3-319-24574- 4_28
doi:10.1007/978- 2015
-
[4]
nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation,
F. Isensee, P. F. Jaeger, S. A. A. Kohl, J. Petersen, and K. H. Maier-Hein, “nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation,” Nat Methods , vol. 18, no. 2, 2021, doi: 10.1038/s41592-020-01008-z
-
[5]
Medical Image Segmentation Review: The Success of U- Net,
R. Azad et al., “Medical Image Segmentation Review: The Success of U- Net,” IEEE Trans Pattern Anal Mach Intell , 2024, doi: 10.1109/TPAMI.2024.3435571
arXiv 2024
-
[6]
A. Kirillov et al. , “Segment Anything,” in Proceedings of the IEEE International Conference on Computer Vision , 2023. doi: 10.1109/ICCV51070.2023.00371
arXiv 2023
-
[7]
Segment anything in medical images,
J. Ma, Y. He, F. Li, L. Han, C. You, and B. Wang, “Segment anything in medical images,” Nat Commun, vol. 15, no. 1, 2024, doi: 10.1038/s41467- 024-44824-z
doi:10.1038/s41467- 2024
-
[8]
SAM 2: Segment Anything in Images and Videos,
N. Ravi et al., “SAM 2: Segment Anything in Images and Videos,” Aug. 2024, [Online]. Available: http://arxiv.org/abs/2408.00714
arXiv 2024
Show all 26 references
-
[9]
SAM-Med3D: Towards General-purpose Segmentation Models for Volumetric Medical Images,
H. Wang et al., “SAM-Med3D: Towards General-purpose Segmentation Models for Volumetric Medical Images,” Oct. 2023, [Online]. Available: http://arxiv.org/abs/2310.15161
2023 arXiv
-
[10]
The RSNA-ASNR-MICCAI BraTS 2021 Benchmark on Brain Tumor Segmentation and Radiogenomic Classification,
U. Baid et al., “The RSNA-ASNR-MICCAI BraTS 2021 Benchmark on Brain Tumor Segmentation and Radiogenomic Classification,” Jul. 2021, [Online]. Available: http://arxiv.org/abs/2107.02314
2021 arXiv
-
[11]
Image Segmentation Using Deep Learning: A Survey,
S. Minaee, Y. Boykov, F. Porikli, A. Plaza, N. Kehtarnavaz, and D. Terzopoulos, “Image Segmentation Using Deep Learning: A Survey,” IEEE Trans Pattern Anal Mach Intell, vol. 44, no. 7, 2022, doi: 10.1109/TPAMI.2021.3059968
2022
-
[12]
Why rankings of biomedical image analysis competitions should be interpreted with care,
L. Maier- Hein et al. , “Why rankings of biomedical image analysis competitions should be interpreted with care,” Nat Commun , vol. 9, no. 1, 2018, doi: 10.1038/s41467-018-07619-7
2018 doi
-
[13]
DeepIGeoS: A Deep Interactive Geodesic Framework for Medical Image Segmentation,
G. Wang et al. , “DeepIGeoS: A Deep Interactive Geodesic Framework for Medical Image Segmentation,” IEEE Trans Pattern Anal Mach Intell, vol. 41, no. 7, 2019, doi: 10.1109/TPAMI.2018.2840695
2019
-
[14]
3D U- net: Learning dense volumetric segmentation from sparse annotation,
Ö. Çiçek, A. Abdulkadir, S. S. Lienkamp, T. Brox, and O. Ronneberger, “3D U- net: Learning dense volumetric segmentation from sparse annotation,” in Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformat...
2016 doi
-
[15]
nnU-Net for Brain Tumor Segmentation,
F. Isensee, P. F. Jäger, P. M. Full, P. Vollmuth, and K. H. Maier -Hein, “nnU-Net for Brain Tumor Segmentation,” in Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 2021. d oi: 10.1007/978-3-0...
2021 doi
-
[16]
Extending nn- UNet for Brain Tumor Segmentation,
H. M. Luu and S. H. Park, “Extending nn- UNet for Brain Tumor Segmentation,” in Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 2022. doi: 10.1007/978-3-031-09002-8_16
2022 doi
-
[17]
Multimodal CNN Networks for Brain Tumor Segmentation in MRI: A BraTS 2022 Challenge Solution,
R. A. Zeineldin, M. E. Karar, O. Burgert, and F. Mathis -Ullrich, “Multimodal CNN Networks for Brain Tumor Segmentation in MRI: A BraTS 2022 Challenge Solution,” in Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Note...
2022 doi
-
[18]
How we won BraTS 2023 Adult Glioma challenge? Just faking it! Enhanced Synthetic Data Augmentation and Model Ensemble for brain tumour segmentation,
A. Ferreira et al. , “How we won BraTS 2023 Adult Glioma challenge? Just faking it! Enhanced Synthetic Data Augmentation and Model Ensemble for brain tumour segmentation,” Feb. 2024, [Online]. Available: http://arxiv.org/abs/2402.17317
2023 arXiv
-
[19]
Segment anything model for medical image analysis: An experimental study,
M. A. Mazurowski, H. Dong, H. Gu, J. Yang, N. Konz, and Y. Zhang, “Segment anything model for medical image analysis: An experimental study,” Med Image Anal , vol. 89, p. 102918, Oct. 2023, doi: 10.1016/J.MEDIA.2023.102918
2023
-
[20]
Biomedical SAM 2: Segment Anything in Biomedical Images and Videos,
Z. Yan et al. , “Biomedical SAM 2: Segment Anything in Biomedical Images and Videos,” Aug. 2024, [Online]. Available: http://arxiv.org/abs/2408.03286
2024 arXiv
-
[21]
The SRI24 multichannel atlas of normal adult human brain structure,
T. Rohlfing, N. M. Zahr, E. V. Sullivan, and A. Pfefferbaum, “The SRI24 multichannel atlas of normal adult human brain structure,” Hum Brain Mapp, vol. 31, no. 5, 2010, doi: 10.1002/hbm.20906
2010 doi
-
[22]
Data leakage inflates prediction performance in connectome- based machine learning models,
M. Rosenblatt, L. Tejavibulya, R. Jiang, S. Noble, and D. Scheinost, “Data leakage inflates prediction performance in connectome- based machine learning models,” Nat Commun , vol. 15, no. 1, 2024, doi: 10.1038/s41467-024-46150-w
2024 doi
-
[23]
The Brain Tumor Segmentation (BraTS) Challenge 2023: Focus on Pediatrics (CBTN- CONNECT-DIPGR- ASNR-MICCAI BraTS -PEDs),
A. F. Kazerooni et al. , “The Brain Tumor Segmentation (BraTS) Challenge 2023: Focus on Pediatrics (CBTN- CONNECT-DIPGR- ASNR-MICCAI BraTS -PEDs),” May 2023, [Online]. Available: http://arxiv.org/abs/2305.17033
2023 arXiv
-
[24]
Exploring a Datasets Statistical Effect Size Impact on Model Performance, and Data Sample-Size Sufficiency,
A. Hatamian, L. Levine, H. E. Oskouie, and M. Sarrafzadeh, “Exploring a Datasets Statistical Effect Size Impact on Model Performance, and Data Sample-Size Sufficiency,” Jan. 2025, [Online]. Available: http://arxiv.org/abs/2501.02673 8
2025 arXiv
-
[25]
Brain tumour segmentation with incomplete imaging data,
J. K. Ruffle, S. Mohinta, R. Gray, H. Hyare, and P. Nachev, “Brain tumour segmentation with incomplete imaging data,” Brain Commun, vol. 5, no. 2, 2023, doi: 10.1093/braincomms/fcad118
2023 doi
-
[26]
Segment Anything in Medical Images and Videos: Benchmark and Deployment,
J. Ma et al. , “Segment Anything in Medical Images and Videos: Benchmark and Deployment,” Aug. 2024, [Online]. Available: http://arxiv.org/abs/2408.033
2024
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.