REVIEW 4 major objections 5 minor 17 references
Assessing the generalization performance of SAM for ureteroscopy scene understanding
T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read Fine-tuned SAM keeps high IoU on unseen ureteroscopy datasets, beating U-Nets by up to 23 points.
desk verdict Useful OOD benchmark with a real but confounded headline; needs controlled baselines before the SAM generalization claim can stand. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the Segment Anything Model, a promptable transformer-based segmentation architecture pretrained on a very large corpus of images and fine-tuned here with a mask-decoder head on 117 to 156 expert-labeled ureteroscopy images. The argument is that its pretrained visual features encode generic object-boundary knowledge that transfers across endoscope and CCD domains, whereas the U-Net encoder-decoder learned only dataset-specific features from the small training set. The comparison is conducted under identical training data, 512-by-512 inputs, and Dice plus cross-entropy loss, with quantitative claims resting on Accuracy, Dice, and IoU measured on held-out test partitions.
What would settle it
Train each U-Net variant on the same Dataset A images but initialized with weights from a large generic image dataset, or trained for 200 epochs instead of 80, and evaluate on Datasets B, C, and D; if any variant reaches SAM's IoU within a few points, the result would show the gap comes from pretraining or compute rather than from the SAM architecture's intrinsic generalization.
Extended reading notes
Core claim
The central claim is that the Segment Anything Model, fine-tuned on a small annotated set of ureteroscopy frames, generalizes across kidney-stone image distributions far better than U-Net-style architectures trained from scratch on the same data. On the in-distribution test, SAM and U-Net are statistically indistinguishable; on each cross-dataset transfer, SAM's IoU remains high while every U-Net falls by roughly 20 to 40 points. For the three-class task, a SAM trained on Dataset C maintains a mean IoU of 86.12 on Dataset A out-of-distribution, and the same model can be applied to two-class segmentation without retraining. The paper interprets this as evidence that SAM's pretrained, promptable representations are a reliable basis for clinical scene understanding under real-world variability.
Load-bearing premise
The comparison treats a U-Net trained from scratch for 80 epochs on roughly 117 to 156 annotated images as a fair baseline for SAM, whose weights start from pretraining on about 11 million images; if a U-Net with pretrained weights or more training would close the out-of-distribution gap, the claim that SAM generalizes better collapses.
Editorial extensions
If this is right
- Clinical annotation effort could drop: a single fine-tuned SAM can segment kidney stones across multiple ureteroscopy camera systems without per-dataset retraining.
- Automated stone-classification pipelines could use SAM as the segmentation front end, receiving reliable masks from previously unseen procedures instead of heavily degraded U-Net masks.
- The same model can serve both two-class (stone/tissue) and three-class (stone/laser/tissue) tasks, reducing the number of specialized models a clinical team must maintain.
- Out-of-distribution error rates stay near 6 to 13 percent for SAM, versus 24 to 48 percent for the U-Net variants, so downstream analyses would receive far fewer mis-segmented frames in new clinical settings.
Reading between the lines
- The paper's design conflates architecture with pretraining: SAM begins from a corpus of roughly 11 million images while U-Nets start from random weights, so the result may primarily demonstrate the value of large-scale pretraining for medical domain shift rather than a unique property of SAM's transformer design.
- Because the three-class model trained on Dataset C scored higher on out-of-distribution Dataset A than on Dataset C's second test partition, dataset difficulty itself may drive transfer more than domain distance; readers should not infer a monotonic generalization ranking from these four distributions.
- A testable extension is benchmarking zero-shot prompted SAM, with no fine-tuning, against the fine-tuned version; if prompted SAM matches the fine-tuned results, fine-tuning may be unnecessary for deployment.
- Another extension is examining failure modes on mis-segmented out-of-distribution frames, since average IoU near 87 to 94 still leaves nontrivial per-pixel errors on edge cases like fragmented stones and laser fibers.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper compares the Segment Anything Model (SAM) with U-Net, Residual U-Net, and Attention U-Net for kidney-stone segmentation in ureteroscopy images. The two-class experiment trains all models on Dataset A and evaluates them on the in-distribution test set of A and on out-of-distribution test sets B, C, and D; the three-class experiment trains SAM on Dataset C and evaluates it on C and A. The headline finding is that SAM maintains IoU values in the 86.6–93.8 range on out-of-distribution data, whereas the U-Net variants fall to 51.8–76.2, yielding margins the abstract reports as up to 23 percent.
Significance. If the reported generalization gap is real, the result would be practically valuable for automating ureteroscopy scene analysis across camera systems and imaging conditions. The evaluation is a genuine forward test: performance is measured against independent ground-truth masks and held-out datasets, with no quantities fitted to the test results. The significance of the contribution is currently limited, however, by the uncontrolled comparison (unequal pretraining and epoch budgets), unresolved dataset-size inconsistencies, and the absence of statistical testing, so the evidence does not yet support the abstract's architectural claim. The manuscript also includes no code or data availability statement, which limits reproducibility.
major comments (4)
- [Section 2.4, 'Experimental setup'; Table 1] The comparison that drives the abstract's claim is confounded: the U-Net variants are trained only on Dataset A (no pretrained weights are stated) for 80 epochs, while SAM is initialized from the SA-1B pretrained weights and trained for 200 epochs. There is no ImageNet-pretrained CNN baseline, no compute-matched training schedule, and no ablation without SAM's pretrained initialization. Because of these differences, the 20-point OOD IoU gaps (e.g., 93.74 vs. 60.75 on Dataset B) cannot be attributed to SAM's architecture, and the phrase 'significantly enhanced generalization capabilities' overstates what the experiment shows.
- [Table 1] The row label 'DA -> DB' appears twice; the second occurrence corresponds to Dataset D in the text of Section 3.1. Since the 'up to 23 percent' margin is computed from that row, the label error must be corrected before the quantitative claims can be trusted.
- [Section 2.1, Section 2.4, Table 2] The reported sizes of the datasets are inconsistent: Dataset C is 138 images in Section 2.1 and 153 images in Section 2.4 (and 117 + 21 + 15 = 153 in Table 2), while Section 3.2 refers to a '138-image collection' for Dataset A, which Section 2.1 states contains 156 images. These inconsistencies affect the experimental setup and support counts and must be resolved.
- [Abstract, Section 3.1] No significance tests or confidence intervals are reported for any comparison, despite the abstract's use of 'significantly.' Given the large standard deviations (e.g., SAM IoU 93.74 ± 9.55 vs. U-Net 60.75 ± 22.20 on Dataset B), paired statistical tests or bootstrap intervals are needed to support the 'significantly enhanced' wording.
minor comments (5)
- [Figure 1 caption] The phrase 'Di and Di represent different data distributions' contains a typo; it should read 'D_i and D_j' (the same subscript error appears in Section 2.2).
- [Section 3.1] The sentence 'SAM trained on Distribution A and tested on in-distribution data maintains an error rate of 4.03%, outperforming all U-Net-based models' is not supported by Table 1, where Residual U-Net also has a 4.03% error rate.
- [Section 3.2] The paragraph opens with a reference to 'all datasets described in Section 3.2' that should be Section 2.1, and the second mention of 'Section 3.2' in that paragraph appears to intend Section 3.1.
- [Section 2.4] The 'Error Rate' metric is never defined; in Table 1 its values equal 100 − IoU, so the definition should be stated explicitly.
- [General] The manuscript contains no code or data availability statement, which limits the reproducibility of Tables 1 and 2.
Circularity Check
No circularity: the generalization claims are forward evaluations against independent held-out ground-truth masks, not derivations from fitted inputs.
full rationale
This paper is an empirical benchmark, not a derivation chain. The central claim—that SAM generalizes better than U-Net variants on out-of-distribution ureteroscopy data—is supported by direct evaluation of models trained on Dataset A and tested on the ground-truth test partitions of Datasets B, C, and D. No parameter is fitted to the reported test results, no prediction is constructed from the measured quantities, and no load-bearing result is imported from a self-citation. The paper's own equations (none are present) are not used to define the outcome. The main methodological weakness is that SAM starts from SA-1B pretrained weights and trains for 200 epochs while the U-Net baselines are trained from scratch for only 80 epochs, which is a fairness concern about the comparison, not circularity, because the reported OOD numbers are genuine forward inferences on independent data. Internal inconsistencies such as the duplicate 'DA -> DB' row in Table 1 and the 138 vs. 153 image count for Dataset C are quality issues but do not make the evaluation circular. The manuscript is therefore self-contained as an empirical study, and the circularity score is 0.
Assumptions & free parameters
assumptions (3)
- domain assumption The four datasets represent distinct clinical distributions, and the random 80/20 splits for A, B, and D do not leak same-video frames into test partitions.
- domain assumption U-Net variants trained from scratch for 80 epochs on 156 images are a fair baseline for SAM initialized with SA-1B pretrained weights.
- domain assumption Ground-truth masks annotated by clinical specialists are accurate and consistent enough to support the reported metrics.
Cite this review
Pith. "Pith review of Assessing the generalization performance of SAM for ureteroscopy scene understanding." pith.science (2026). https://pith.science/paper/HXU3B4DN
@misc{pith2026250517210,
author = {Pith},
title = {Pith review of: Assessing the generalization performance of SAM for ureteroscopy scene understanding},
year = {2026},
howpublished = {\url{https://pith.science/paper/HXU3B4DN}},
note = {Machine review of arXiv:2505.17210}
}
read the original abstract
The segmentation of kidney stones is regarded as a critical preliminary step to enable the identification of urinary stone types through machine- or deep-learning-based approaches. In urology, manual segmentation is considered tedious and impractical due to the typically large scale of image databases and the continuous generation of new data. In this study, the potential of the Segment Anything Model (SAM) -- a state-of-the-art deep learning framework -- is investigated for the automation of kidney stone segmentation. The performance of SAM is evaluated in comparison to traditional models, including U-Net, Residual U-Net, and Attention U-Net, which, despite their efficiency, frequently exhibit limitations in generalizing to unseen datasets. The findings highlight SAM's superior adaptability and efficiency. While SAM achieves comparable performance to U-Net on in-distribution data (Accuracy: 97.68 + 3.04; Dice: 97.78 + 2.47; IoU: 95.76 + 4.18), it demonstrates significantly enhanced generalization capabilities on out-of-distribution data, surpassing all U-Net variants by margins of up to 23 percent.
Figures
Reference graph
Works this paper leans on
-
[1]
Leila Moftakhar, Fatemeh Jafari, Masoumeh Ghoddusi Johari, Ramin Rezaeian- zadeh, Seyed Vahid Hosseini, and Abbas Rezaianzadeh. Prevalence and risk factors of kidney stone disease in population aged 40–70 years old in kharameh cohort study: a cross-sectional population-based study in southern iran.BMC urology, 22(1):205, 2022. 14 M. Villagrana et al
work page 2022
-
[2]
Ben H Chew, Larry E Miller, Brian Eisner, Samir Bhattacharyya, and Naeem Bhojani. Prevalence, incidence, and determinants of kidney stones in a nationally representative sample of us adults.JU Open Plus, 2(1):e00006, 2024
work page 2024
-
[3]
Re- currence rates of urinary calculi according to stone composition and morphology
Michel Daudon, Paul Jungers, Dominique Bazin, and James C Williams. Re- currence rates of urinary calculi according to stone composition and morphology. Urolithiasis, 46:459–470, 2018
work page 2018
-
[4]
Michel Daudon and Paul Jungers. Clinical value of crystalluria and quantitative morphoconstitutional analysis of urinary calculi.Nephron Physiology, 98(2):p31– p36, 2004
work page 2004
-
[5]
On the in vivo recognition of kidney stones using machine learning.IEEE Access, 12:10736–10759, 2024
Francisco Lopez-Tiro, Vincent Estrade, Jacques Hubert, Daniel Flores-Araiza, Miguel Gonzalez-Mendoza, Gilberto Ochoa, and Christian Daul. On the in vivo recognition of kidney stones using machine learning.IEEE Access, 12:10736–10759, 2024
work page 2024
-
[6]
Deep morphological recognition of kidney stones using intra-operative endoscopic digital videos
Vincent Estrade, Michel Daudon, Emmanuel Richard, Jean-Christophe Bernhard, Franck Bladou, Gregoire Robert, Laurent Facq, and Baudouin Denis de Senneville. Deep morphological recognition of kidney stones using intra-operative endoscopic digital videos. Physics in Medicine & Biology, 67(16):165006, 2022
work page 2022
-
[7]
Mauricio Mendez-Ruiz, Francisco Lopez-Tiro, Daniel Flores-Araiza, Jonathan El- Beze, Gilberto Ochoa-Ruiz, Miguel Gonzalez-Mendoza, Jacques Hubert, Andres Mendez-Vazquez, and Christian Daul. On the generalization capabilities of fsl methods through domain adaptation: a case study in endoscopic kidney stone image classification. InMexican International Conf...
work page 2022
-
[8]
Vincent Estrade, Michel Daudon, Emmanuel Richard, Jean-Christophe Bernhard, Franck Bladou, Grégoire Robert, and Baudouin Denis de Senneville. Towards automatic recognition of pure and mixed stones using intra-operative endoscopic digital images. BJU international, 129(2):234–242, 2022
work page 2022
Show all 17 references
-
[9]
Effective deep learning for semantic segmentation based bleeding zone detection in capsule endoscopy images
Tonmoy Ghosh, Linfeng Li, and Jacob Chakareski. Effective deep learning for semantic segmentation based bleeding zone detection in capsule endoscopy images. In 2018 25th IEEE International Conference on Image Processing (ICIP), pages 3034–3038. IEEE, 2018
2018
-
[10]
Assessing deep learning methods for the identification of kidney stones in endoscopic images
Francisco Lopez, Andres Varelo, Oscar Hinojosa, Mauricio Mendez, Dinh-Hoan Trinh, Yonathan ElBeze, Jacques Hubert, Vincent Estrade, Miguel Gonzalez, Gilberto Ochoa, et al. Assessing deep learning methods for the identification of kidney stones in endoscopic images. In2021 43rd...
2021
-
[11]
Multi-class motion-based semantic segmentation for ureteroscopy and laser lithotripsy
Soumya Gupta, Sharib Ali, Louise Goldsmith, Ben Turney, and Jens Rittscher. Multi-class motion-based semantic segmentation for ureteroscopy and laser lithotripsy. Computerized Medical Imaging and Graphics, 101:102112, 2022
2022
-
[12]
U-net: Convolutional networks for biomedical image segmentation
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical image computing and computer-assisted intervention–MICCAI 2015: 18th international conference, Mu- nich, Germany, October 5-9, 2015, proceedings, par...
2015
-
[13]
Recurrent residual u-net for medical image segmentation.Journal of medical imaging, 6(1):014006–014006, 2019
Md Zahangir Alom, Chris Yakopcic, Mahmudul Hasan, Tarek M Taha, and Vi- jayan K Asari. Recurrent residual u-net for medical image segmentation.Journal of medical imaging, 6(1):014006–014006, 2019
2019
-
[14]
Residual-attention unet++: a nested residual-attention u-net for medical image segmentation.Applied Sciences, 12(14):7149, 2022
Zan Li, Hong Zhang, Zhengzhen Li, and Zuyue Ren. Residual-attention unet++: a nested residual-attention u-net for medical image segmentation.Applied Sciences, 12(14):7149, 2022. Assessing the generalization performance of SAM for ureteroscopy scene 15
2022
-
[15]
Segment anything
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 4015–4026, 2023
2023
-
[16]
Evaluation and understanding of automated urinary stone recognition methods.BJU Int., 130(6):786–798, 2022
Jonathan El Beze, Charles Mazeaud, Christian Daul, Gilberto Ochoa-Ruiz, Michel Daudon, Pascal Eschwège, and Jacques Hubert. Evaluation and understanding of automated urinary stone recognition methods.BJU Int., 130(6):786–798, 2022
2022
-
[17]
Classification of stones according to michel daudon: a narrative review
Mariela Corrales, Steeve Doizi, Yazeed Barghouthy, Olivier Traxer, and Michel Daudon. Classification of stones according to michel daudon: a narrative review. European Urology Focus, 7(1):13–21, 2021
2021
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.