REVIEW 4 major objections 5 minor 30 references
SAM Carries the Burden: A Semi-Supervised Approach Refining Pseudo Labels for Medical Segmentation
T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read Training a U-Net on pseudo labels refined by SAM raises Dice scores from 74.29% to 84.17% on wrist bones and from 66.63% to 74.87% on teeth.
desk verdict The wrist headline number is likely inflated by patient overlap between the unlabeled set and the test set; the dental experiment and the prompt-extraction scheme are still worth a careful revision. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is prompt extraction from imperfect predictions. For each predicted class mask, the pipeline keeps only the connected component with the highest area-normalised likelihood, applies morphological dilation to make the prompt robust, then feeds SAM an axis-aligned bounding box, the component's centre of mass as a positive seed point, and the centres of all other classes as negative seed points. SAM's mask output is then fed back as a dense prompt in a self-refinement loop. This converts a crude, over- or under-segmented U-Net prediction into a clean, object-aware pseudo label without any human interaction.
What would settle it
Run the pipeline on a test set of radiographs where the initial U-Net fails to detect some objects at all (for example, low-contrast casts or atypical anatomy) and compare the Dice of the SAM-refined pseudo labels on those images with the unrefined predictions; if refinement only fixes partially covered objects and never recovers wholly missed ones, the score on such cases should stay near the baseline.
Extended reading notes
Core claim
The central claim is that SAM's abstract object understanding can carry the burden of annotation quality in semi-supervised medical segmentation when it is prompted, not with hand-placed clicks, but with prompts extracted automatically from a model trained on a few dozen labelled images. The paper shows that a U-Net trained on these SAM-refined pseudo labels reaches 84.17% Dice on 17 paediatric wrist bones and 74.87% on teeth, outperforming the same U-Net trained only on labelled data, a supervised nnU-Net, and a Mean Teacher semi-supervised baseline. On the dental dataset the SAM-refined pipeline matches a fully supervised U-Net trained on ten times more annotations. The paper also reports that refinement with SAM outperforms intensity-based Random Walk post-processing and MedSAM, and that the self-refinement step (passing SAM its own previous mask as an additional prompt) is a measurable contributor.
Load-bearing premise
The method assumes the initial segmentation produced by the weakly trained U-Net partially covers every object of interest; if an object is missed entirely, the extracted box and seed points cannot guide SAM to it and the pseudo label for that object will be wrong.
Editorial extensions
If this is right
- On paediatric wrist radiographs, training a U-Net on SAM-refined pseudo labels raises Dice from 74.29% to 84.17%, exceeding both nnU-Net (77.6%) and Mean Teacher (79.3%).
- On the dental radiograph dataset, the same recipe reaches 74.87% Dice using 45 labelled images plus 405 unlabelled images, matching the 74.19% of a U-Net trained on all 450 labelled images.
- Using SAM for refinement beats intensity-based Random Walk post-processing and the medical-domain MedSAM variant on both datasets, and the self-refinement step contributes 2.21% Dice on the wrist dataset.
- Across an ablation study with 1 to 43 labelled training images, the SAM-refined pseudo-label training outperforms all comparison methods at every training-set size.
Reading between the lines
- The same prompting recipe should transfer to other single-instance-per-class segmentation problems, but applying it to objects that appear many times per image, such as cells or multiple lesions, would require a detection or instance-separation step before the mask-cleaning stage.
- Because SAM is used off-the-shelf and never fine-tuned, the pipeline's cost is essentially one extra inference pass per unlabelled image; a natural next comparison is against human-in-the-loop annotation time, which would show whether the refinement step actually saves expert hours.
- The paper's failure analysis suggests a testable extension: use the overlap between the initial mask and SAM's refined mask as a confidence filter, discarding pseudo labels where SAM disagrees strongly with the prompt, which could improve robustness on the worst test cases.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a semi-supervised segmentation pipeline that uses a U-Net trained on a small labelled set to produce initial masks for unlabelled images, cleans these masks, extracts bounding-box and seed-point prompts, and feeds them to SAM to obtain refined pseudo labels. A new model is then trained on the unlabelled images with these pseudo labels. The method is evaluated on paediatric wrist bone segmentation (GRaZPEDWRI-DX, 43 labelled training images and 500 unlabelled images) and teeth segmentation on dental radiographs (45 labelled and 405 unlabelled images). The authors report Dice-score improvements from 74.29% to 84.17% for the wrist and from 66.63% to 74.87% for teeth, outperforming nnU-Net, Mean Teacher, Random Walk, and MedSAM baselines, and they include an ablation study with successively fewer labelled training samples.
Significance. If the reported gains hold, the paper would provide a practical and reusable recipe for leveraging SAM to turn cheap, noisy predictions into high-quality pseudo labels for medical segmentation with very limited annotations. The strengths are the release of code and manual wrist annotations, the inclusion of several relevant baselines (nnU-Net, Mean Teacher, MedSAM, Random Walk), and a clearly described pipeline with hyperparameter optimization on a separate validation split. The main empirical claim, however, rests on a nine-image wrist test set and on a single training run per condition, and there is a potentially severe patient-leakage issue in the construction of the wrist unlabelled set. These issues need to be resolved before the headline results can be considered reliable.
major comments (4)
- [§3.1, Datasets] The unlabelled wrist subset Y is formed by sampling 500 images randomly from Dw \ X, which excludes only the 62 manually annotated images, not all radiographs belonging to the patients in those 62 images. Since GRaZPEDWRI-DX contains roughly 10k images from about 5.9k patients, many patients have multiple radiographs, so it is highly likely that Y contains additional images of the nine test patients. Because the final model fφ is trained on (Y, R), any such overlap gives the model direct experience with test patients' radiographs, and the reported 84.17% versus 74.29% wrist comparison in Table 1 may reflect patient identity rather than the benefit of SAM-refined pseudo labels. Please make Y patient-disjoint from Xtest (ideally from all of X) or report a patient-level analysis showing that the results are unchanged when overlapping images are removed.
- [Table 1 and §3.1] The wrist test set consists of only nine images, and all results are reported for a single training run without significance tests. The reported standard deviations are large (for example, 84.2±8.0 for the proposed method versus 79.3±4.7 for Mean Teacher in the wrist setting), so the headline differences are not statistically established. Please report per-image Dice scores or provide paired significance tests across test cases, and run multiple seeds or cross-validation folds so that the variability of the training procedure is quantified.
- [§3.1, Comparison Methods] The nnU-Net baseline is modified relative to its standard configuration: the authors extend it with a binary cross-entropy loss and train for 100 epochs without cross-validation. The abstract and Section 4 claim that the method outperforms state-of-the-art supervised learning, namely nnU-Net. As implemented, this modified baseline may underestimate nnU-Net's performance. Please justify the modification or include the default nnU-Net configuration as an additional baseline; if the default configuration is infeasible, state the computational cost that motivates the modification.
- [Fig. 2 and §3.1, Ablation Study] The ablation study in Fig. 2 is performed with a single randomly selected subset for each value of n and a single training run, so the claim that the method reaches top performance with as few as 20 labelled images is not supported with uncertainty estimates. Please report multiple subset draws and training runs, or add error bands, so that the crossover point at approximately n=20 is credible.
minor comments (5)
- [§2, Problem statement] In the notation, |Y| ≪ |X| should read |Y| ≫ |X|, since the text states that the amount of unlabelled data is much greater than the labelled data.
- [§3.1, Datasets] The word 'disjunct' should be 'disjoint', and the definition of X should make explicit that it contains the 62 wrist images or the 450 dental images used for the respective experiments, to avoid ambiguity in the notation Y = D \ X.
- [§3.2, Table 1] The dental result for fully supervised training with all 450 annotations is mentioned in the text but not shown in Table 1; adding it to the table would make the comparison with the semi-supervised result easier to verify.
- [§3.2, Fig. 3] For Figures 3, 4, and 5, it is unclear whether the reported Dice scores are averaged over images, over classes, or over both; please state the aggregation procedure in the caption or in Section 3.1.
- [§4, Limitations] The stated limitations in Section 4—one instance per class per image due to mask cleaning, and the requirement that the initial segmentation partially cover each object—are relevant and should be evaluated explicitly on the dental data, where multiple instances of the same class can appear in an image and where initial masks may miss teeth entirely.
Circularity Check
No significant circularity; the only author-overlapping citation is non-load-bearing.
full rationale
The paper makes an empirical claim: training a U-Net on SAM-refined pseudo labels improves Dice. This claim is supported by held-out test evaluation against external baselines, not by a derivation that reduces to its own inputs. The pseudo-label chain is: train f_theta on Xtrain, predict P on unlabelled Y, clean P via connected-component and morphological processing, extract bounding-box and seed-point prompts, apply the off-the-shelf pre-trained SAM to obtain refined masks R, and train f_phi on (Y,R). Each stage is operationally defined, and prompt hyperparameters are optimized on a separate validation split rather than on the test set. The paper's stated limitation that 'the initial segmentation for prompt extraction must partially cover the object of interest' is an input-distribution assumption, not a definitional equivalence between the claimed result and an input. The only self-citation is reference [3] (Bigalke and Heinrich, MICCAI 2023), cited in the conclusion only to motivate possible future application to domain adaptation; it does not carry the central claim. The possible patient overlap between the randomly sampled wrist unlabelled subset Y and the nine test patients is a data-leakage and experimental-validity concern, not a circular-reasoning step, and should be assessed by a separate validity review. Therefore the claimed derivation is not circular; the score of 2 reflects only the minor, non-load-bearing self-citation rather than any circular step.
Assumptions & free parameters
free parameters (2)
- Morphological structure element and radius for mask cleaning =
isotropic squared structure element, radius 8 pixels
- SAM prompt configuration =
bounding box + positive and negative seed points + self-refinement mask
assumptions (4)
- domain assumption The initial U-Net segmentation mask must partially cover the object of interest for prompt extraction to work.
- domain assumption Each semantic class appears at most once per image, so the connected-component selection can choose a single component.
- domain assumption Off-the-shelf SAM, trained on natural images, provides useful zero-shot segmentations on medical X-ray images.
- domain assumption The unlabelled data distribution is similar to the labelled data, so pseudo-label training improves test performance.
Cite this review
Pith. "Pith review of SAM Carries the Burden: A Semi-Supervised Approach Refining Pseudo Labels for Medical Segmentation." pith.science (2026). https://pith.science/paper/YYRKV3F6
@misc{pith2026241112602,
author = {Pith},
title = {Pith review of: SAM Carries the Burden: A Semi-Supervised Approach Refining Pseudo Labels for Medical Segmentation},
year = {2026},
howpublished = {\url{https://pith.science/paper/YYRKV3F6}},
note = {Machine review of arXiv:2411.12602}
}
abstract
Semantic segmentation is a crucial task in medical imaging. Although supervised learning techniques have proven to be effective in performing this task, they heavily depend on large amounts of annotated training data. The recently introduced Segment Anything Model (SAM) enables prompt-based segmentation and offers zero-shot generalization to unfamiliar objects. In our work, we leverage SAM's abstract object understanding for medical image segmentation to provide pseudo labels for semi-supervised learning, thereby mitigating the need for extensive annotated training data. Our approach refines initial segmentations that are derived from a limited amount of annotated data (comprising up to 43 cases) by extracting bounding boxes and seed points as prompts forwarded to SAM. Thus, it enables the generation of dense segmentation masks as pseudo labels for unlabelled data. The results show that training with our pseudo labels yields an improvement in Dice score from $74.29\,\%$ to $84.17\,\%$ and from $66.63\,\%$ to $74.87\,\%$ for the segmentation of bones of the paediatric wrist and teeth in dental radiographs, respectively. As a result, our method outperforms intensity-based post-processing methods, state-of-the-art supervised learning for segmentation (nnU-Net), and the semi-supervised mean teacher approach. Our Code is available on GitHub.
Figures
Figures from the paper (2 more)
Reference graph
Works this paper leans on
-
[1]
In: 2014 11th International Conference on Electronics, Computer and Computation (ICECCO)
Adeshina, S.A., Lindner, C., Cootes, T.F.: Automatic segmentation of carpal area bones with random forest regression voting for estimat- ing skeletal maturity in infants. In: 2014 11th International Conference on Electronics, Computer and Computation (ICECCO). pp. 1–4 (2014). https://doi.org/10.1109/ICECCO.2014.6997559
-
[2]
In: Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining
Akiba, T., Sano, S., Yanase, T., Ohta, T., Koyama, M.: Optuna: A next- generation hyperparameter optimization framework. In: Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. p. 2623–2631. KDD ’19, Association for Computing Machinery, New York, NY, USA (2019). https://doi.org/10.1145/3292500.3330701, https:/...
arXiv 2019
-
[3]
Bigalke, A., Heinrich, M.P.: A denoised mean teacher for domain adaptive point cloud registration. In: Greenspan, H., Madabhushi, A., Mousavi, P., Salcudean, S., Duncan, J., Syeda-Mahmood, T., Taylor, R. (eds.) Medical Image Computing and Computer Assisted Intervention – MICCAI 2023. pp. 666–676. Springer Nature Switzerland, Cham (2023)
work page 2023
-
[4]
Chen, T., Mai, Z., Li, R., Chao, W.L.: Segment anything model (sam) en- hanced pseudo labels for weakly supervised semantic segmentation. ArXiv abs/2305.05803 (2023)
arXiv 2023
-
[5]
Cheng, J., Ye, J., Deng, Z., Chen, J., Li, T.X., Wang, H., Su, Y., Huang, Z., Chen, J., Jiang, L., Sun, H., He, J., Zhang, S., Zhu, M., Qiao, Y.: Sam-med2d (2023)
work page 2023
-
[6]
MONAI Label: A framework for AI-assisted Interactive Labeling of 3D Medical Images
Diaz-Pinto, A., Alle, S., Nath, V., Tang, Y., Ihsani, A., Asad, M., P´ erez-Garc ´ ıa, F., Mehta, P., Li, W., Flores, M., et al.: Monai label: A framework for ai-assisted interactive labeling of 3d medical images. arXiv preprint arXiv:2203.12362 (2022)
work page Pith review arXiv 2022
-
[7]
Multimedia Tools and Applications pp
Faisal, A., Khalil, A., Chai, H.Y., Lai, K.W.: X-ray carpal bone segmentation and area measurement. Multimedia Tools and Applications pp. 1–12 (2021). https://doi.org/10.1007/s11042-021-11281-5
-
[8]
IEEE Transactions on Pattern Analysis and Machine Intelligence 28, 1768–1783 (2006)
Grady, L.J.: Random walks for image segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence 28, 1768–1783 (2006)
work page 2006
Show all 30 references
-
[9]
In: Proc IEEE Comput Soc Conf Comput Vis Pattern Recognit (June 2016) 10 Keuth et al
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proc IEEE Comput Soc Conf Comput Vis Pattern Recognit (June 2016) 10 Keuth et al
2016
-
[10]
In: Proc IEEE Comput Soc Conf Comput Vis Pattern Recognit (October 2019)
Howard, A., Sandler, M., Chu, G., Chen, L.C., Chen, B., Tan, M., Wang, W., Zhu, Y., Pang, R., Vasudevan, V., Le, Q.V., Adam, H.: Searching for mobilenetv3. In: Proc IEEE Comput Soc Conf Comput Vis Pattern Recognit (October 2019)
2019
-
[11]
https://doi.org/https://doi.org/10.1016/j.media.2023.103061, https://www
Huang, Y., Yang, X., Liu, L., Zhou, H., Chang, A., Zhou, X., Chen, R., Yu, J., Chen, J., Chen, C., Liu, S., Chi, H., Hu, X., Yue, K., Li, L., Grau, V., Fan, D.P., Dong, F., Ni, D.: Segment anything model for medical images? Medical Image Analysis 92, 103061 (2024). https://doi...
2024
-
[12]
Huang, Z., Liu, H., Zhang, H., Li, X., Liu, H., Xing, F., Laine, A.F., Angelini, E.D., Hendon, C.P., Gan, Y.: Push the boundary of sam: A pseudo-label correction framework for medical segmentation (2023)
2023
-
[13]
https://doi.org/10.1038/s41592-020-01008-z
Isensee, F., Jaeger, P.F., Kohl, S.A.A., Petersen, J., Maier-Hein, K.H.: nnU-net: a self-configuring method for deep learning-based biomedical image segmentation 18(2), 203–211. https://doi.org/10.1038/s41592-020-01008-z
-
[14]
Computers in biology and medicine 169, 107840 (2022)
Jiao, R., Zhang, Y., Ding, L., Cai, R., Zhang, J.: Learning with limited annota- tions: A survey on deep semi-supervised learning for medical image segmentation. Computers in biology and medicine 169, 107840 (2022)
2022
-
[15]
2023 IEEE/CVF International Conference on Computer Vision (ICCV) pp
Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A.C., Lo, W.Y., Doll´ ar, P., Girshick, R.B.: Segment anything. 2023 IEEE/CVF International Conference on Computer Vision (ICCV) pp. 3992– 4003 (2023)
2023
-
[16]
In: Inter- national Conference on Learning Representations (2017), https://openreview.net/ forum?id=BJ6oOfqge
Laine, S., Aila, T.: Temporal ensembling for semi-supervised learning. In: Inter- national Conference on Learning Representations (2017), https://openreview.net/ forum?id=BJ6oOfqge
2017
-
[17]
ICML 2013 Workshop : Challenges in Representation Learning (WREPL) (07 2013)
Lee, D.H.: Pseudo-label : The simple and efficient semi-supervised learning method for deep neural networks. ICML 2013 Workshop : Challenges in Representation Learning (WREPL) (07 2013)
2013
-
[18]
In: International Conference on Neural Information Processing (2023)
Li, N., Xiong, L., Qiu, W., Pan, Y., Luo, Y., Zhang, Y.: Segment anything model for semi-supervised medical image segmentation via selecting reliable pseudo-labels. In: International Conference on Neural Information Processing (2023)
2023
-
[19]
https://doi.org/10.34740/KAGGLE/DSV/5884500, https://www.kaggle.com/ dsv/5884500
Loop, H.I.T.: Teeth segmentation on dental x-ray images (2023). https://doi.org/10.34740/KAGGLE/DSV/5884500, https://www.kaggle.com/ dsv/5884500
2023
-
[20]
Nature Communications 15(1), 654 (2024)
Ma, J., He, Y., Li, F., Han, L., You, C., Wang, B.: Segment anything in medical images. Nature Communications 15(1), 654 (2024)
2024
-
[21]
https://doi.org/10.1038/s41597-022-01328-z, https://www.nature.com/ articles/s41597-022-01328-z
Nagy, E., Janisch, M., Hrˇ zi´ c, F., Sorantin, E., Tschauner, S.: A pedi- atric wrist trauma x-ray dataset (GRAZPEDWRI-DX) for machine learning 9(1), 222. https://doi.org/10.1038/s41597-022-01328-z, https://www.nature.com/ articles/s41597-022-01328-z
-
[22]
In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F
Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomed- ical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) Medical Image Computing and Computer-Assisted Intervention – MICCAI
-
[23]
ACM SIGGRAPH 2004 Papers (2004)
Rother, C., Kolmogorov, V., Blake, A.: ”grabcut”: interactive foreground extrac- tion using iterated graph cuts. ACM SIGGRAPH 2004 Papers (2004)
2004
-
[24]
Advances in neural information processing systems 30 (2017) SAM Carries the Burden 11
Tarvainen, A., Valpola, H.: Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. Advances in neural information processing systems 30 (2017) SAM Carries the Burden 11
2017
-
[25]
IEEE transactions on medical imaging 37(7), 1562–1573 (2018)
Wang, G., Li, W., Zuluaga, M.A., Pratt, R., Patel, P.A., Aertsen, M., Doel, T., David, A.L., Deprest, J., Ourselin, S., et al.: Interactive medical image segmen- tation using deep learning with image-specific fine tuning. IEEE transactions on medical imaging 37(7), 1562–1573 (2018)
2018
-
[26]
ArXiv abs/2312.06316 (2023)
Zhang, Y., Cheng, Y., Qi, Y.: Semisam: Exploring sam for enhancing semi- supervised medical image segmentation with extremely limited annotations. ArXiv abs/2312.06316 (2023)
2023 arXiv
-
[27]
Computers in biology and medicine 171, 108238 (2024)
Zhang, Y., Shen, Z., Jiao, R.: Segment anything model for medical image seg- mentation: Current applications and future directions. Computers in biology and medicine 171, 108238 (2024)
2024
-
[28]
ArXiv abs/2308.13759 (2023), https: //api.semanticscholar.org/CorpusID:261245481
Zhang, Y., Zhou, T., Wang, S., Wu, Y., Gu, P., Chen, D.: Samdsk: Combin- ing segment anything model with domain-specific knowledge for semi-supervised learning in medical image segmentation. ArXiv abs/2308.13759 (2023), https: //api.semanticscholar.org/CorpusID:261245481
2023 arXiv
-
[29]
In: Shen, D., Liu, T., Peters, T.M., Staib, L.H., Essert, C., Zhou, S., Yap, P.T., Khan, A
Zheng, H., Lin, L., Hu, H., Zhang, Q., Chen, Q., Iwamoto, Y., Han, X., Chen, Y.W., Tong, R., Wu, J.: Semi-supervised segmentation of liver using adversarial learning with deep atlas prior. In: Shen, D., Liu, T., Peters, T.M., Staib, L.H., Essert, C., Zhou, S., Yap, P.T., Khan,...
2019
-
[2015]
pp. 234–241. Springer International Publishing, Cham (2015)
2015
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.