{"id":"41965935-6cb6-44f1-bda3-19e5142cf07b","arxiv_id":"2505.17210","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Fine-tuned SAM generalizes much better than U-Net variants to unseen kidney stone image distributions, with out-of-distribution IoU margins of up to 23 percentage points.","lead":"SAM, a foundation model for image segmentation, beats standard U-Net models at finding kidney stones in endoscopy images from unseen hospitals and cameras by up to 23 percentage points. The result suggests foundation models may be more practical than task-specific CNNs for automating stone segmentation when training data is scarce and clinical sites vary.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"SAM's claimed OOD advantage is confounded by pretraining and unequal training budget; the U-Net baselines are trained from scratch for 80 epochs while SAM starts from SA-1B.","rationale":"The reader's weakest assumption correctly identifies the pretraining/training-budget confound as the most load-bearing issue. The paper's experimental design cannot distinguish whether SAM's high OOD performance comes from its architecture or from starting with SA-1B weights and training for 2.5x more epochs. This is a genuine threat to the central claim: if ImageNet-initialized U-Nets close most of the gap, the conclusion 'SAM surpasses U-Nets due to superior generalization' collapses. No countervailing evidence in the manuscript addresses this confound; there are no pretrained CNN baselines, no compute-matched experiments, and no code or data to reproduce the results. The mechanical errors (duplicated table row, inconsistent dataset count) are secondary but reinforce that the reported comparisons should not be taken at face value. A CONDITIONAL verdict is therefore appropriate: the claim is plausible but not yet adequately supported. The stress-test pass does not identify a different, more severe objection; the pretraining confound is the central issue, and the concrete test would settle whether it actually explains the observed gap.","tokens_in":9688,"tokens_out":2395,"duration_ms":20024,"concrete_test":"Retrain U-Net, Residual U-Net, and Attention U-Net with ImageNet-pretrained encoders under the same 200-epoch budget, optimizer, and loss as SAM, using the same Dataset A training split; evaluate on the identical B, C, and D test partitions and compare IoU. If the OOD IoU gap relative to SAM drops below roughly 10 points (e.g., from 23 on Dataset D to under 10), the claimed architectural superiority is not established. Additionally, correct the duplicated 'DA -> DB' row in Table 1 to 'DA -> DD' and re-check whether the reported margins remain consistent.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that SAM generalizes better than U-Net variants on out-of-distribution ureteroscopy data—depends on a fair comparison, but Section 2.4 ('Experimental setup') describes an unequal protocol. All models are 'trained exclusively on Dataset A,' yet U-Net, Residual U-Net, and Attention U-Net are trained from scratch for 80 epochs, while SAM is trained for 200 epochs and, as the Segment Anything Model, is initialized from weights pretrained on SA-1B (roughly 11 million images). The paper reports no ImageNet-pretrained or otherwise pretrained CNN baseline, no compute-matched training, and no ablation of SAM without its pretrained initialization. Therefore, the observed 20+ point IoU gaps on Datasets B, C, and D could be attributed to SAM's massive pretraining and longer training schedule rather than to any architectural property that 'demonstrates significantly enhanced generalization capabilities' (abstract). The manuscript also contains mechanical inconsistencies that undermine confidence in the reported numbers: Table 1 lists the row 'DA -> DB' twice (the final row should correspond to Dataset D), and Database C is described as 138 images in Section 2.1 but 153 images in Section 2.4. These issues do not by themselves invalidate the qualitative pattern, but they strengthen the need for a controlled comparison before the headline claim can be accepted.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper compares the Segment Anything Model (SAM) with U-Net, Residual U-Net, and Attention U-Net for kidney-stone segmentation in ureteroscopy images. The two-class experiment trains all models on Dataset A and evaluates them on the in-distribution test set of A and on out-of-distribution test sets B, C, and D; the three-class experiment trains SAM on Dataset C and evaluates it on C and A. The headline finding is that SAM maintains IoU values in the 86.6–93.8 range on out-of-distribution data, whereas the U-Net variants fall to 51.8–76.2, yielding margins the abstract reports as up to 23 percent.","tokens_in":9896,"tokens_out":7978,"duration_ms":87972,"significance":"If the reported generalization gap is real, the result would be practically valuable for automating ureteroscopy scene analysis across camera systems and imaging conditions. The evaluation is a genuine forward test: performance is measured against independent ground-truth masks and held-out datasets, with no quantities fitted to the test results. The significance of the contribution is currently limited, however, by the uncontrolled comparison (unequal pretraining and epoch budgets), unresolved dataset-size inconsistencies, and the absence of statistical testing, so the evidence does not yet support the abstract's architectural claim. The manuscript also includes no code or data availability statement, which limits reproducibility.","major_comments":[{"comment":"The comparison that drives the abstract's claim is confounded: the U-Net variants are trained only on Dataset A (no pretrained weights are stated) for 80 epochs, while SAM is initialized from the SA-1B pretrained weights and trained for 200 epochs. There is no ImageNet-pretrained CNN baseline, no compute-matched training schedule, and no ablation without SAM's pretrained initialization. Because of these differences, the 20-point OOD IoU gaps (e.g., 93.74 vs. 60.75 on Dataset B) cannot be attributed to SAM's architecture, and the phrase 'significantly enhanced generalization capabilities' overstates what the experiment shows.","section":"Section 2.4, 'Experimental setup'; Table 1"},{"comment":"The row label 'DA -> DB' appears twice; the second occurrence corresponds to Dataset D in the text of Section 3.1. Since the 'up to 23 percent' margin is computed from that row, the label error must be corrected before the quantitative claims can be trusted.","section":"Table 1"},{"comment":"The reported sizes of the datasets are inconsistent: Dataset C is 138 images in Section 2.1 and 153 images in Section 2.4 (and 117 + 21 + 15 = 153 in Table 2), while Section 3.2 refers to a '138-image collection' for Dataset A, which Section 2.1 states contains 156 images. These inconsistencies affect the experimental setup and support counts and must be resolved.","section":"Section 2.1, Section 2.4, Table 2"},{"comment":"No significance tests or confidence intervals are reported for any comparison, despite the abstract's use of 'significantly.' Given the large standard deviations (e.g., SAM IoU 93.74 ± 9.55 vs. U-Net 60.75 ± 22.20 on Dataset B), paired statistical tests or bootstrap intervals are needed to support the 'significantly enhanced' wording.","section":"Abstract, Section 3.1"}],"minor_comments":[{"comment":"The phrase 'Di and Di represent different data distributions' contains a typo; it should read 'D_i and D_j' (the same subscript error appears in Section 2.2).","section":"Figure 1 caption"},{"comment":"The sentence 'SAM trained on Distribution A and tested on in-distribution data maintains an error rate of 4.03%, outperforming all U-Net-based models' is not supported by Table 1, where Residual U-Net also has a 4.03% error rate.","section":"Section 3.1"},{"comment":"The paragraph opens with a reference to 'all datasets described in Section 3.2' that should be Section 2.1, and the second mention of 'Section 3.2' in that paragraph appears to intend Section 3.1.","section":"Section 3.2"},{"comment":"The 'Error Rate' metric is never defined; in Table 1 its values equal 100 − IoU, so the definition should be stated explicitly.","section":"Section 2.4"},{"comment":"The manuscript contains no code or data availability statement, which limits the reproducibility of Tables 1 and 2.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The paper is a modest empirical contribution whose central claim is probably directionally correct but not established at the current level of experimental control. The authors should add a pretrained CNN baseline, equalize training budgets, run significance tests, and fix the table and data-size errors before resubmission. I would not recommend acceptance in the present form, but the topic is within the journal's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: worth reading, worth revising, not worth citing as-is. What is actually new is a concrete out-of-distribution evaluation of fine-tuned SAM against three U-Net variants on four ureteroscopy datasets, plus a three-class setup with laser fiber. The held-out design is genuine: numbers come from forward evaluations against expert masks, and nothing is fitted to the reported test results. Credit where due—the datasets are distinct, the per-class IoU in the three-class task is useful, and the clinical motivation is concrete.\n\nThe soft spots are real but mostly fixable. The central comparison is unequal: SAM starts from SA-1B pretrained weights and trains 200 epochs, while the U-Nets train from scratch for 80. That means the 20+ point OOD gap cannot be attributed to SAM's architecture. The paper needs an ImageNet-pretrained U-Net baseline or a compute-matched training budget, or at minimum the claim should be about pretraining plus fine-tuning rather than about SAM as such. The duplicated table row (DA->DB should be DA->DD) is a mechanical error, but combined with the dataset C inconsistency (138 vs 153 images) it makes the numbers harder to trust. No significance tests are reported, and with the high variances on Dataset D the margins could overlap. The 'multi-class without retraining' claim is oversold: it is two separately trained binary models fused, not one true multi-class SAM. No code or data is shipped, which limits reproducibility but does not make the evaluation circular.\n\nOverall, the empirical pattern is plausible and the qualitative direction matches what one expects from large-scale pretraining. The paper would be publishable after a serious revision that adds controlled baselines, fixes the table and dataset counts, runs significance tests, and softens the efficiency and 'without retraining' claims. The central argument, once rephrased, mostly holds up: fine-tuned SAM generalizes better than from-scratch U-Nets on this medical imaging benchmark. What is not established is how much of that comes from SAM's architecture versus its pretraining and longer schedule.\n\nYes, send it to peer review. The benchmark is useful, the application matters, and the flaws are correctable. I would not cite the headline as it stands, but I would cite the cleaned-up benchmark.","headline":"Useful OOD benchmark with a real but confounded headline; needs controlled baselines before the SAM generalization claim can stand.","tokens_in":10469,"tokens_out":1625,"would_cite":false,"duration_ms":13723,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuned SAM keeps high IoU on unseen ureteroscopy datasets, beating U-Nets by up to 23 points.","keywords":["Segment Anything Model","kidney stone segmentation","ureteroscopy","out-of-distribution generalization","semantic segmentation","U-Net comparison","laser fiber segmentation"],"falsifier":"Train each U-Net variant on the same Dataset A images but initialized with weights from a large generic image dataset, or trained for 200 epochs instead of 80, and evaluate on Datasets B, C, and D; if any variant reaches SAM's IoU within a few points, the result would show the gap comes from pretraining or compute rather than from the SAM architecture's intrinsic generalization.","tokens_in":9466,"feed_emoji":"🩺","tokens_out":5122,"duration_ms":36409,"temperature":0.7,"pith_summary":"This paper asks whether a fine-tuned Segment Anything Model can automate kidney-stone segmentation in ureteroscopy videos well enough to survive changes in camera, tissue appearance, and imaging conditions that break conventional segmentation networks. It trains SAM and three U-Net variants on the same in-vivo endoscopic images, then tests all models on three unseen datasets. On the training distribution SAM matches the U-Nets, with IoU around 95 to 96, but on out-of-distribution data SAM keeps IoU between 86.6 and 93.8 while the best U-Net variant drops to between 51.8 and 76.2. The paper also shows a three-class SAM trained on one in-vivo dataset retains accuracy when moved to another and can segment two classes without retraining.","feed_headline":"Fine-tuned SAM beats U-Nets on unseen kidney-stone images","feed_subtitle":"SAM loses almost nothing in-distribution but keeps IoU high where U-Nets drop as much as 23 points.","key_machinery":"The load-bearing object is the Segment Anything Model, a promptable transformer-based segmentation architecture pretrained on a very large corpus of images and fine-tuned here with a mask-decoder head on 117 to 156 expert-labeled ureteroscopy images. The argument is that its pretrained visual features encode generic object-boundary knowledge that transfers across endoscope and CCD domains, whereas the U-Net encoder-decoder learned only dataset-specific features from the small training set. The comparison is conducted under identical training data, 512-by-512 inputs, and Dice plus cross-entropy loss, with quantitative claims resting on Accuracy, Dice, and IoU measured on held-out test partitions.","core_discovery":"The central claim is that the Segment Anything Model, fine-tuned on a small annotated set of ureteroscopy frames, generalizes across kidney-stone image distributions far better than U-Net-style architectures trained from scratch on the same data. On the in-distribution test, SAM and U-Net are statistically indistinguishable; on each cross-dataset transfer, SAM's IoU remains high while every U-Net falls by roughly 20 to 40 points. For the three-class task, a SAM trained on Dataset C maintains a mean IoU of 86.12 on Dataset A out-of-distribution, and the same model can be applied to two-class segmentation without retraining. The paper interprets this as evidence that SAM's pretrained, promptable representations are a reliable basis for clinical scene understanding under real-world variability.","pith_inferences":["The paper's design conflates architecture with pretraining: SAM begins from a corpus of roughly 11 million images while U-Nets start from random weights, so the result may primarily demonstrate the value of large-scale pretraining for medical domain shift rather than a unique property of SAM's transformer design.","Because the three-class model trained on Dataset C scored higher on out-of-distribution Dataset A than on Dataset C's second test partition, dataset difficulty itself may drive transfer more than domain distance; readers should not infer a monotonic generalization ranking from these four distributions.","A testable extension is benchmarking zero-shot prompted SAM, with no fine-tuning, against the fine-tuned version; if prompted SAM matches the fine-tuned results, fine-tuning may be unnecessary for deployment.","Another extension is examining failure modes on mis-segmented out-of-distribution frames, since average IoU near 87 to 94 still leaves nontrivial per-pixel errors on edge cases like fragmented stones and laser fibers."],"forward_implications":["Clinical annotation effort could drop: a single fine-tuned SAM can segment kidney stones across multiple ureteroscopy camera systems without per-dataset retraining.","Automated stone-classification pipelines could use SAM as the segmentation front end, receiving reliable masks from previously unseen procedures instead of heavily degraded U-Net masks.","The same model can serve both two-class (stone/tissue) and three-class (stone/laser/tissue) tasks, reducing the number of specialized models a clinical team must maintain.","Out-of-distribution error rates stay near 6 to 13 percent for SAM, versus 24 to 48 percent for the U-Net variants, so downstream analyses would receive far fewer mis-segmented frames in new clinical settings."],"supporting_citations":[{"why":"Supplies the SAM architecture, pretraining strategy, and promptable design that the paper fine-tunes and evaluates.","marker":"[15]"},{"why":"Defines the baseline U-Net architecture whose in-distribution score SAM matches and whose out-of-distribution degradation is the central comparison.","marker":"[12]"},{"why":"Provides the residual U-Net variant used as a second baseline in the same generalization comparison.","marker":"[13]"},{"why":"Provides the attention U-Net variant used as a third baseline in the same generalization comparison.","marker":"[14]"},{"why":"Source of the in-vivo endoscopic Dataset A images and annotations used for training and in-distribution testing.","marker":"[6]"},{"why":"Source of the ex-vivo endoscopic Dataset B images used as one out-of-distribution test set.","marker":"[16]"},{"why":"Source of the ex-vivo CCD-camera Dataset D images used as another out-of-distribution test set.","marker":"[17]"}],"fun_headline_variants":["SAM keeps kidney-stone IoU high where U-Nets drop 23 points","SAM beats U-Nets on unseen kidney-stone images","U-Nets lose 23 points on new data; SAM doesn't","SAM beats U-Nets by up to 23 points on unseen data","SAM generalizes, U-Nets don't: up to 23-point IoU gap"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison treats a U-Net trained from scratch for 80 epochs on roughly 117 to 156 annotated images as a fair baseline for SAM, whose weights start from pretraining on about 11 million images; if a U-Net with pretrained weights or more training would close the out-of-distribution gap, the claim that SAM generalizes better collapses.","fun_headline_variants_meta":{"raw":{"variants":["SAM keeps kidney-stone IoU high where U-Nets drop 23 points","SAM beats U-Nets on unseen kidney-stone images","U-Nets lose 23 points on new data; SAM doesn't","SAM beats U-Nets by up to 23 points on unseen data","SAM generalizes, U-Nets don't: up to 23-point IoU gap"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000934,"raw_usage":{"total_tokens":3978,"prompt_tokens":905,"completion_tokens":3073,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":521,"completion_tokens_details":{"reasoning_tokens":2974}},"tokens_in":521,"tokens_out":3073,"duration_ms":17357,"temperature":1.0,"reasoning_tokens":2974,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:50:04.432517+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train each U-Net variant on the same Dataset A images but initialized with weights from a large generic image dataset, or trained for 200 epochs instead of 80, and evaluate on Datasets B, C, and D; if any variant reaches SAM's IoU within a few points, the result would show the gap comes from pretraining or compute rather than from the SAM architecture's intrinsic generalization.","supporting_citations":[{"cited_title":"Recurrent residual u-net for medical image segmentation.Journal of medical imaging, 6(1):014006–014006, 2019","cited_arxiv_id":null,"evidence_quote":"Provides the residual U-Net variant used as a second baseline in the same generalization comparison."},{"cited_title":"Residual-attention unet++: a nested residual-attention u-net for medical image segmentation.Applied Sciences, 12(14):7149, 2022","cited_arxiv_id":null,"evidence_quote":"Provides the attention U-Net variant used as a third baseline in the same generalization comparison."},{"cited_title":"Deep morphological recognition of kidney stones using intra-operative endoscopic digital videos","cited_arxiv_id":null,"evidence_quote":"Source of the in-vivo endoscopic Dataset A images and annotations used for training and in-distribution testing."},{"cited_title":"Evaluation and understanding of automated urinary stone recognition methods.BJU Int., 130(6):786–798, 2022","cited_arxiv_id":null,"evidence_quote":"Source of the ex-vivo endoscopic Dataset B images used as one out-of-distribution test set."}],"review_version":1}