Pith. sign in

REVIEW 4 major objections 5 minor 17 references

Assessing the generalization performance of SAM for ureteroscopy scene understanding

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Pith's one-line read Fine-tuned SAM keeps high IoU on unseen ureteroscopy datasets, beating U-Nets by up to 23 points.

desk verdict Useful OOD benchmark with a real but confounded headline; needs controlled baselines before the SAM generalization claim can stand. read the letter →

arxiv 2505.17210 v1 pith:HXU3B4DN submitted 2025-05-22 eess.IV cs.AIcs.CVcs.LG

classification eess.IVcs.AIcs.CVcs.LG
keywords SegmentAnythingModelkidneystonesegmentationureteroscopyout-of-distributiongeneralizationsemanticU-Netcomparisonlaserfiber
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether a fine-tuned Segment Anything Model can automate kidney-stone segmentation in ureteroscopy videos well enough to survive changes in camera, tissue appearance, and imaging conditions that break conventional segmentation networks. It trains SAM and three U-Net variants on the same in-vivo endoscopic images, then tests all models on three unseen datasets. On the training distribution SAM matches the U-Nets, with IoU around 95 to 96, but on out-of-distribution data SAM keeps IoU between 86.6 and 93.8 while the best U-Net variant drops to between 51.8 and 76.2. The paper also shows a three-class SAM trained on one in-vivo dataset retains accuracy when moved to another and can segment two classes without retraining.

What carries the argument

The load-bearing object is the Segment Anything Model, a promptable transformer-based segmentation architecture pretrained on a very large corpus of images and fine-tuned here with a mask-decoder head on 117 to 156 expert-labeled ureteroscopy images. The argument is that its pretrained visual features encode generic object-boundary knowledge that transfers across endoscope and CCD domains, whereas the U-Net encoder-decoder learned only dataset-specific features from the small training set. The comparison is conducted under identical training data, 512-by-512 inputs, and Dice plus cross-entropy loss, with quantitative claims resting on Accuracy, Dice, and IoU measured on held-out test partitions.

What would settle it

Train each U-Net variant on the same Dataset A images but initialized with weights from a large generic image dataset, or trained for 200 epochs instead of 80, and evaluate on Datasets B, C, and D; if any variant reaches SAM's IoU within a few points, the result would show the gap comes from pretraining or compute rather than from the SAM architecture's intrinsic generalization.

Watch

Extended reading notes

Core claim

The central claim is that the Segment Anything Model, fine-tuned on a small annotated set of ureteroscopy frames, generalizes across kidney-stone image distributions far better than U-Net-style architectures trained from scratch on the same data. On the in-distribution test, SAM and U-Net are statistically indistinguishable; on each cross-dataset transfer, SAM's IoU remains high while every U-Net falls by roughly 20 to 40 points. For the three-class task, a SAM trained on Dataset C maintains a mean IoU of 86.12 on Dataset A out-of-distribution, and the same model can be applied to two-class segmentation without retraining. The paper interprets this as evidence that SAM's pretrained, promptable representations are a reliable basis for clinical scene understanding under real-world variability.

Load-bearing premise

The comparison treats a U-Net trained from scratch for 80 epochs on roughly 117 to 156 annotated images as a fair baseline for SAM, whose weights start from pretraining on about 11 million images; if a U-Net with pretrained weights or more training would close the out-of-distribution gap, the claim that SAM generalizes better collapses.

Editorial extensions

If this is right

  • Clinical annotation effort could drop: a single fine-tuned SAM can segment kidney stones across multiple ureteroscopy camera systems without per-dataset retraining.
  • Automated stone-classification pipelines could use SAM as the segmentation front end, receiving reliable masks from previously unseen procedures instead of heavily degraded U-Net masks.
  • The same model can serve both two-class (stone/tissue) and three-class (stone/laser/tissue) tasks, reducing the number of specialized models a clinical team must maintain.
  • Out-of-distribution error rates stay near 6 to 13 percent for SAM, versus 24 to 48 percent for the U-Net variants, so downstream analyses would receive far fewer mis-segmented frames in new clinical settings.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's design conflates architecture with pretraining: SAM begins from a corpus of roughly 11 million images while U-Nets start from random weights, so the result may primarily demonstrate the value of large-scale pretraining for medical domain shift rather than a unique property of SAM's transformer design.
  • Because the three-class model trained on Dataset C scored higher on out-of-distribution Dataset A than on Dataset C's second test partition, dataset difficulty itself may drive transfer more than domain distance; readers should not infer a monotonic generalization ranking from these four distributions.
  • A testable extension is benchmarking zero-shot prompted SAM, with no fine-tuning, against the fine-tuned version; if prompted SAM matches the fine-tuned results, fine-tuning may be unnecessary for deployment.
  • Another extension is examining failure modes on mis-segmented out-of-distribution frames, since average IoU near 87 to 94 still leaves nontrivial per-pixel errors on edge cases like fragmented stones and laser fibers.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper compares the Segment Anything Model (SAM) with U-Net, Residual U-Net, and Attention U-Net for kidney-stone segmentation in ureteroscopy images. The two-class experiment trains all models on Dataset A and evaluates them on the in-distribution test set of A and on out-of-distribution test sets B, C, and D; the three-class experiment trains SAM on Dataset C and evaluates it on C and A. The headline finding is that SAM maintains IoU values in the 86.6–93.8 range on out-of-distribution data, whereas the U-Net variants fall to 51.8–76.2, yielding margins the abstract reports as up to 23 percent.

Significance. If the reported generalization gap is real, the result would be practically valuable for automating ureteroscopy scene analysis across camera systems and imaging conditions. The evaluation is a genuine forward test: performance is measured against independent ground-truth masks and held-out datasets, with no quantities fitted to the test results. The significance of the contribution is currently limited, however, by the uncontrolled comparison (unequal pretraining and epoch budgets), unresolved dataset-size inconsistencies, and the absence of statistical testing, so the evidence does not yet support the abstract's architectural claim. The manuscript also includes no code or data availability statement, which limits reproducibility.

major comments (4)
  1. [Section 2.4, 'Experimental setup'; Table 1] The comparison that drives the abstract's claim is confounded: the U-Net variants are trained only on Dataset A (no pretrained weights are stated) for 80 epochs, while SAM is initialized from the SA-1B pretrained weights and trained for 200 epochs. There is no ImageNet-pretrained CNN baseline, no compute-matched training schedule, and no ablation without SAM's pretrained initialization. Because of these differences, the 20-point OOD IoU gaps (e.g., 93.74 vs. 60.75 on Dataset B) cannot be attributed to SAM's architecture, and the phrase 'significantly enhanced generalization capabilities' overstates what the experiment shows.
  2. [Table 1] The row label 'DA -> DB' appears twice; the second occurrence corresponds to Dataset D in the text of Section 3.1. Since the 'up to 23 percent' margin is computed from that row, the label error must be corrected before the quantitative claims can be trusted.
  3. [Section 2.1, Section 2.4, Table 2] The reported sizes of the datasets are inconsistent: Dataset C is 138 images in Section 2.1 and 153 images in Section 2.4 (and 117 + 21 + 15 = 153 in Table 2), while Section 3.2 refers to a '138-image collection' for Dataset A, which Section 2.1 states contains 156 images. These inconsistencies affect the experimental setup and support counts and must be resolved.
  4. [Abstract, Section 3.1] No significance tests or confidence intervals are reported for any comparison, despite the abstract's use of 'significantly.' Given the large standard deviations (e.g., SAM IoU 93.74 ± 9.55 vs. U-Net 60.75 ± 22.20 on Dataset B), paired statistical tests or bootstrap intervals are needed to support the 'significantly enhanced' wording.
minor comments (5)
  1. [Figure 1 caption] The phrase 'Di and Di represent different data distributions' contains a typo; it should read 'D_i and D_j' (the same subscript error appears in Section 2.2).
  2. [Section 3.1] The sentence 'SAM trained on Distribution A and tested on in-distribution data maintains an error rate of 4.03%, outperforming all U-Net-based models' is not supported by Table 1, where Residual U-Net also has a 4.03% error rate.
  3. [Section 3.2] The paragraph opens with a reference to 'all datasets described in Section 3.2' that should be Section 2.1, and the second mention of 'Section 3.2' in that paragraph appears to intend Section 3.1.
  4. [Section 2.4] The 'Error Rate' metric is never defined; in Table 1 its values equal 100 − IoU, so the definition should be stated explicitly.
  5. [General] The manuscript contains no code or data availability statement, which limits the reproducibility of Tables 1 and 2.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the generalization claims are forward evaluations against independent held-out ground-truth masks, not derivations from fitted inputs.

full rationale

This paper is an empirical benchmark, not a derivation chain. The central claim—that SAM generalizes better than U-Net variants on out-of-distribution ureteroscopy data—is supported by direct evaluation of models trained on Dataset A and tested on the ground-truth test partitions of Datasets B, C, and D. No parameter is fitted to the reported test results, no prediction is constructed from the measured quantities, and no load-bearing result is imported from a self-citation. The paper's own equations (none are present) are not used to define the outcome. The main methodological weakness is that SAM starts from SA-1B pretrained weights and trains for 200 epochs while the U-Net baselines are trained from scratch for only 80 epochs, which is a fairness concern about the comparison, not circularity, because the reported OOD numbers are genuine forward inferences on independent data. Internal inconsistencies such as the duplicate 'DA -> DB' row in Table 1 and the 138 vs. 153 image count for Dataset C are quality issues but do not make the evaluation circular. The manuscript is therefore self-contained as an empirical study, and the circularity score is 0.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The central claim rests on the datasets being distinct, the annotations being trustworthy, and the baseline comparison being fair. The paper states split ratios and annotation protocols but does not control for pretraining or provide code. No free parameters or invented entities are introduced; the 'efficiency' claim is an over-interpretation of results rather than a new entity.

assumptions (3)
  • domain assumption The four datasets represent distinct clinical distributions, and the random 80/20 splits for A, B, and D do not leak same-video frames into test partitions.
    The OOD results assume each dataset is a cleanly separated distribution. Section 2.1 states random splits for A, B, and D, so same-video leakage could inflate in-distribution scores and make OOD gains look larger.
  • domain assumption U-Net variants trained from scratch for 80 epochs on 156 images are a fair baseline for SAM initialized with SA-1B pretrained weights.
    Section 2.4 compares models without controlling for pretraining or compute budget. If a pretrained or longer-trained U-Net would close the gap, the central claim is weakened.
  • domain assumption Ground-truth masks annotated by clinical specialists are accurate and consistent enough to support the reported metrics.
    The paper says masks were 'meticulously annotated' (Section 2.1) but provides no inter-observer variability, and the three-class subset of Dataset A has only 44 images.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Assessing the generalization performance of SAM for ureteroscopy scene understanding." pith.science (2026). https://pith.science/paper/HXU3B4DN

@misc{pith2026250517210,
  author       = {Pith},
  title        = {Pith review of: Assessing the generalization performance of SAM for ureteroscopy scene understanding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HXU3B4DN}},
  note         = {Machine review of arXiv:2505.17210}
}
read the original abstract

The segmentation of kidney stones is regarded as a critical preliminary step to enable the identification of urinary stone types through machine- or deep-learning-based approaches. In urology, manual segmentation is considered tedious and impractical due to the typically large scale of image databases and the continuous generation of new data. In this study, the potential of the Segment Anything Model (SAM) -- a state-of-the-art deep learning framework -- is investigated for the automation of kidney stone segmentation. The performance of SAM is evaluated in comparison to traditional models, including U-Net, Residual U-Net, and Attention U-Net, which, despite their efficiency, frequently exhibit limitations in generalizing to unseen datasets. The findings highlight SAM's superior adaptability and efficiency. While SAM achieves comparable performance to U-Net on in-distribution data (Accuracy: 97.68 + 3.04; Dice: 97.78 + 2.47; IoU: 95.76 + 4.18), it demonstrates significantly enhanced generalization capabilities on out-of-distribution data, surpassing all U-Net variants by margins of up to 23 percent.

Figures

Figures reproduced from arXiv: 2505.17210 by the authors.

Figure 1
Figure 1. Comparison framework for kidney stone segmentation methods. In tradi [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. The four datasets (distributions) of kidney stone images are displayed [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. A qualitative comparison is presented across rows (Datasets A-D) and [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Qualitative comparison of segmentation results for three classes (kidney [PITH_FULL_IMAGE:figures/full_fig_p012_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

17 extracted references · 14 canonical work pages

  1. [1]

    Leila Moftakhar, Fatemeh Jafari, Masoumeh Ghoddusi Johari, Ramin Rezaeian- zadeh, Seyed Vahid Hosseini, and Abbas Rezaianzadeh. Prevalence and risk factors of kidney stone disease in population aged 40–70 years old in kharameh cohort study: a cross-sectional population-based study in southern iran.BMC urology, 22(1):205, 2022. 14 M. Villagrana et al

  2. [2]

    Prevalence, incidence, and determinants of kidney stones in a nationally representative sample of us adults.JU Open Plus, 2(1):e00006, 2024

    Ben H Chew, Larry E Miller, Brian Eisner, Samir Bhattacharyya, and Naeem Bhojani. Prevalence, incidence, and determinants of kidney stones in a nationally representative sample of us adults.JU Open Plus, 2(1):e00006, 2024

  3. [3]

    Re- currence rates of urinary calculi according to stone composition and morphology

    Michel Daudon, Paul Jungers, Dominique Bazin, and James C Williams. Re- currence rates of urinary calculi according to stone composition and morphology. Urolithiasis, 46:459–470, 2018

  4. [4]

    Clinical value of crystalluria and quantitative morphoconstitutional analysis of urinary calculi.Nephron Physiology, 98(2):p31– p36, 2004

    Michel Daudon and Paul Jungers. Clinical value of crystalluria and quantitative morphoconstitutional analysis of urinary calculi.Nephron Physiology, 98(2):p31– p36, 2004

  5. [5]

    On the in vivo recognition of kidney stones using machine learning.IEEE Access, 12:10736–10759, 2024

    Francisco Lopez-Tiro, Vincent Estrade, Jacques Hubert, Daniel Flores-Araiza, Miguel Gonzalez-Mendoza, Gilberto Ochoa, and Christian Daul. On the in vivo recognition of kidney stones using machine learning.IEEE Access, 12:10736–10759, 2024

  6. [6]

    Deep morphological recognition of kidney stones using intra-operative endoscopic digital videos

    Vincent Estrade, Michel Daudon, Emmanuel Richard, Jean-Christophe Bernhard, Franck Bladou, Gregoire Robert, Laurent Facq, and Baudouin Denis de Senneville. Deep morphological recognition of kidney stones using intra-operative endoscopic digital videos. Physics in Medicine & Biology, 67(16):165006, 2022

  7. [7]

    On the generalization capabilities of fsl methods through domain adaptation: a case study in endoscopic kidney stone image classification

    Mauricio Mendez-Ruiz, Francisco Lopez-Tiro, Daniel Flores-Araiza, Jonathan El- Beze, Gilberto Ochoa-Ruiz, Miguel Gonzalez-Mendoza, Jacques Hubert, Andres Mendez-Vazquez, and Christian Daul. On the generalization capabilities of fsl methods through domain adaptation: a case study in endoscopic kidney stone image classification. InMexican International Conf...

  8. [8]

    Towards automatic recognition of pure and mixed stones using intra-operative endoscopic digital images

    Vincent Estrade, Michel Daudon, Emmanuel Richard, Jean-Christophe Bernhard, Franck Bladou, Grégoire Robert, and Baudouin Denis de Senneville. Towards automatic recognition of pure and mixed stones using intra-operative endoscopic digital images. BJU international, 129(2):234–242, 2022

Show all 17 references
  1. [9]

    Effective deep learning for semantic segmentation based bleeding zone detection in capsule endoscopy images

    Tonmoy Ghosh, Linfeng Li, and Jacob Chakareski. Effective deep learning for semantic segmentation based bleeding zone detection in capsule endoscopy images. In 2018 25th IEEE International Conference on Image Processing (ICIP), pages 3034–3038. IEEE, 2018

  2. [10]

    Assessing deep learning methods for the identification of kidney stones in endoscopic images

    Francisco Lopez, Andres Varelo, Oscar Hinojosa, Mauricio Mendez, Dinh-Hoan Trinh, Yonathan ElBeze, Jacques Hubert, Vincent Estrade, Miguel Gonzalez, Gilberto Ochoa, et al. Assessing deep learning methods for the identification of kidney stones in endoscopic images. In2021 43rd...

  3. [11]

    Multi-class motion-based semantic segmentation for ureteroscopy and laser lithotripsy

    Soumya Gupta, Sharib Ali, Louise Goldsmith, Ben Turney, and Jens Rittscher. Multi-class motion-based semantic segmentation for ureteroscopy and laser lithotripsy. Computerized Medical Imaging and Graphics, 101:102112, 2022

  4. [12]

    U-net: Convolutional networks for biomedical image segmentation

    Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical image computing and computer-assisted intervention–MICCAI 2015: 18th international conference, Mu- nich, Germany, October 5-9, 2015, proceedings, par...

  5. [13]

    Recurrent residual u-net for medical image segmentation.Journal of medical imaging, 6(1):014006–014006, 2019

    Md Zahangir Alom, Chris Yakopcic, Mahmudul Hasan, Tarek M Taha, and Vi- jayan K Asari. Recurrent residual u-net for medical image segmentation.Journal of medical imaging, 6(1):014006–014006, 2019

  6. [14]

    Residual-attention unet++: a nested residual-attention u-net for medical image segmentation.Applied Sciences, 12(14):7149, 2022

    Zan Li, Hong Zhang, Zhengzhen Li, and Zuyue Ren. Residual-attention unet++: a nested residual-attention u-net for medical image segmentation.Applied Sciences, 12(14):7149, 2022. Assessing the generalization performance of SAM for ureteroscopy scene 15

  7. [15]

    Segment anything

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 4015–4026, 2023

  8. [16]

    Evaluation and understanding of automated urinary stone recognition methods.BJU Int., 130(6):786–798, 2022

    Jonathan El Beze, Charles Mazeaud, Christian Daul, Gilberto Ochoa-Ruiz, Michel Daudon, Pascal Eschwège, and Jacques Hubert. Evaluation and understanding of automated urinary stone recognition methods.BJU Int., 130(6):786–798, 2022

  9. [17]

    Classification of stones according to michel daudon: a narrative review

    Mariela Corrales, Steeve Doizi, Yazeed Barghouthy, Olivier Traxer, and Michel Daudon. Classification of stones according to michel daudon: a narrative review. European Urology Focus, 7(1):13–21, 2021

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.