{"id":"6aa8019b-17d3-439a-a6f0-fd0dbc20d48a","arxiv_id":"2411.12602","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"SAM-refined pseudo labels from a few labeled examples let a U-Net reach or exceed fully supervised performance on pediatric wrist and dental X-ray segmentation.","lead":"This paper trains a U-Net on a small labeled set and uses SAM to refine its predictions on unlabeled images, turning them into pseudo labels for further training. On pediatric wrist and dental X-ray segmentation, this lifts Dice scores by about ten points, matching fully supervised performance with far fewer manual annotations.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Wrist unlabelled set Y may share patients with the 9-image test set; if so, the 84.17% result reflects patient leakage rather than SAM refinement.","rationale":"The reader correctly notes the small 9-image wrist test set, single runs, and absent significance tests, but their stated weakest assumption (initial mask must partially cover the object) is an acknowledged failure mode, not an experimental flaw. The patient-leakage risk I identify is more load-bearing: if Y contains images of the test patients, the central wrist comparison is invalid regardless of how well the method works. The teeth experiment uses an image-level custom split and is not subject to this particular leakage, so the paper should not be rejected outright; rather, it should be conditional on verifying the wrist split. The ablation against the non-refined pseudo-label variant (P) is a real strength, as it partially controls for the effect of simply adding data, and the public code makes the proposed check feasible. If the check comes back clean, the reader's original conditional verdict stands on statistical grounds.","tokens_in":8325,"tokens_out":10343,"duration_ms":111041,"concrete_test":"Using the public GRaZPEDWRI-DX metadata (or the authors' GitHub code), recover the patient ID of each of the 62 annotated images and of the 500 sampled Y images. Compute the intersection of patient IDs between Y and the 9 test patients (and also the validation patients). If the intersection is non-empty, re-run the wrist experiment with Y restricted to patients not present in X and report whether the 84.17% result survives; if the intersection is empty, the concern is resolved and the remaining limitation is statistical size.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"Sec. 3.1 states the 62 manually annotated wrist images are split 'over the number of patients' into 43/10/9, but the unlabelled subset Y is 'sample[d] 500 images randomly from Dw \\ X', i.e., only the 62 annotated images are excluded, not all images belonging to those patients. GRaZPEDWRI-DX contains ~10k images from ~5.9k patients, so many patients have multiple radiographs. With 500 images sampled from the remaining ~9.9k, it is highly likely that Y includes other images of the 9 test patients. Since the final model fφ is trained on (Y,R), any such overlap gives the model direct experience with the test patients' radiographs, and the reported 84.17% vs 74.29% wrist gain could be inflated by patient identity rather than by SAM-refined pseudo labels. The paper neither states that Y is patient-disjoint from X nor reports a patient-level check. This is the most load-bearing issue because the wrist result is the paper's strongest headline number; if the overlap exists, that comparison is no longer a valid evaluation of the method.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a semi-supervised segmentation pipeline that uses a U-Net trained on a small labelled set to produce initial masks for unlabelled images, cleans these masks, extracts bounding-box and seed-point prompts, and feeds them to SAM to obtain refined pseudo labels. A new model is then trained on the unlabelled images with these pseudo labels. The method is evaluated on paediatric wrist bone segmentation (GRaZPEDWRI-DX, 43 labelled training images and 500 unlabelled images) and teeth segmentation on dental radiographs (45 labelled and 405 unlabelled images). The authors report Dice-score improvements from 74.29% to 84.17% for the wrist and from 66.63% to 74.87% for teeth, outperforming nnU-Net, Mean Teacher, Random Walk, and MedSAM baselines, and they include an ablation study with successively fewer labelled training samples.","tokens_in":8550,"tokens_out":3374,"duration_ms":34061,"significance":"If the reported gains hold, the paper would provide a practical and reusable recipe for leveraging SAM to turn cheap, noisy predictions into high-quality pseudo labels for medical segmentation with very limited annotations. The strengths are the release of code and manual wrist annotations, the inclusion of several relevant baselines (nnU-Net, Mean Teacher, MedSAM, Random Walk), and a clearly described pipeline with hyperparameter optimization on a separate validation split. The main empirical claim, however, rests on a nine-image wrist test set and on a single training run per condition, and there is a potentially severe patient-leakage issue in the construction of the wrist unlabelled set. These issues need to be resolved before the headline results can be considered reliable.","major_comments":[{"comment":"The unlabelled wrist subset Y is formed by sampling 500 images randomly from Dw \\ X, which excludes only the 62 manually annotated images, not all radiographs belonging to the patients in those 62 images. Since GRaZPEDWRI-DX contains roughly 10k images from about 5.9k patients, many patients have multiple radiographs, so it is highly likely that Y contains additional images of the nine test patients. Because the final model fφ is trained on (Y, R), any such overlap gives the model direct experience with test patients' radiographs, and the reported 84.17% versus 74.29% wrist comparison in Table 1 may reflect patient identity rather than the benefit of SAM-refined pseudo labels. Please make Y patient-disjoint from Xtest (ideally from all of X) or report a patient-level analysis showing that the results are unchanged when overlapping images are removed.","section":"§3.1, Datasets"},{"comment":"The wrist test set consists of only nine images, and all results are reported for a single training run without significance tests. The reported standard deviations are large (for example, 84.2±8.0 for the proposed method versus 79.3±4.7 for Mean Teacher in the wrist setting), so the headline differences are not statistically established. Please report per-image Dice scores or provide paired significance tests across test cases, and run multiple seeds or cross-validation folds so that the variability of the training procedure is quantified.","section":"Table 1 and §3.1"},{"comment":"The nnU-Net baseline is modified relative to its standard configuration: the authors extend it with a binary cross-entropy loss and train for 100 epochs without cross-validation. The abstract and Section 4 claim that the method outperforms state-of-the-art supervised learning, namely nnU-Net. As implemented, this modified baseline may underestimate nnU-Net's performance. Please justify the modification or include the default nnU-Net configuration as an additional baseline; if the default configuration is infeasible, state the computational cost that motivates the modification.","section":"§3.1, Comparison Methods"},{"comment":"The ablation study in Fig. 2 is performed with a single randomly selected subset for each value of n and a single training run, so the claim that the method reaches top performance with as few as 20 labelled images is not supported with uncertainty estimates. Please report multiple subset draws and training runs, or add error bands, so that the crossover point at approximately n=20 is credible.","section":"Fig. 2 and §3.1, Ablation Study"}],"minor_comments":[{"comment":"In the notation, |Y| ≪ |X| should read |Y| ≫ |X|, since the text states that the amount of unlabelled data is much greater than the labelled data.","section":"§2, Problem statement"},{"comment":"The word 'disjunct' should be 'disjoint', and the definition of X should make explicit that it contains the 62 wrist images or the 450 dental images used for the respective experiments, to avoid ambiguity in the notation Y = D \\ X.","section":"§3.1, Datasets"},{"comment":"The dental result for fully supervised training with all 450 annotations is mentioned in the text but not shown in Table 1; adding it to the table would make the comparison with the semi-supervised result easier to verify.","section":"§3.2, Table 1"},{"comment":"For Figures 3, 4, and 5, it is unclear whether the reported Dice scores are averaged over images, over classes, or over both; please state the aggregation procedure in the caption or in Section 3.1.","section":"§3.2, Fig. 3"},{"comment":"The stated limitations in Section 4—one instance per class per image due to mask cleaning, and the requirement that the initial segmentation partially cover each object—are relevant and should be evaluated explicitly on the dental data, where multiple instances of the same class can appear in an image and where initial masks may miss teeth entirely.","section":"§4, Limitations"}],"recommendation":"major_revision","confidential_remarks":"The paper is a straightforward empirical application of SAM to pseudo-label refinement, with modest novelty but potentially useful practical value. The most serious concern is the likely patient overlap between the wrist unlabelled set and the test set, which would invalidate the headline wrist result as reported. The lack of significance testing and the modified nnU-Net baseline are secondary but also need attention. If the authors can re-run the wrist experiments with a patient-disjoint unlabelled set and provide significance estimates, the paper could become acceptable for publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the wrist headline number—84.17% vs 74.29%—is likely not a clean comparison. The authors split the 62 labeled images by patient into 43/10/9, but then sample the 500 unlabeled images as a random draw from every remaining image in Dw \\ X. Since the dataset has multiple radiographs per patient, that draw almost certainly includes other images of the nine test patients. Training the final model on those images gives it direct experience with the test patients, so the reported gain could reflect patient identity, not SAM-refined pseudo labels. The paper never checks patient-level overlap. That is a load-bearing flaw.\n\nWhat is actually new is the automatic prompt-extraction scheme: largest-component cleaning, center-of-mass positive point, other classes' centers as negative points, and SAM's mask self-refinement. It is a sensible way to convert weak predictions into useful SAM prompts. The paper is clearly written, the ablation in Fig. 2 is consistent across label counts, and the dental experiment (45 labeled, 405 unlabeled) is free of this leakage because the unlabeled set is drawn from the training split, not from the whole dataset. The authors are honest about limitations, including that the initial mask must partly cover the object.\n\nSoft spots in proportion. The leakage is the big one; it needs a patient-disjoint unlabeled split or at least a patient-overlap analysis. After that, the wrist test set is only nine images, results are from a single run, and there are no significance tests. The nnU-Net baseline is modified in a nonstandard way (BCE, no cross-validation); that is defensible but should be described as such. And the GitHub link is mentioned in the abstract but no URL or commit hash appears in the manuscript.\n\nIf the leakage is real, the central wrist claim is not established. The method is still plausible, and the dental result is a partial counterweight, so I would not desk-reject. I would send it to peer review with a strong request to fix the evaluation. This is a fixable flaw, not a fundamental one.","headline":"The wrist headline number is likely inflated by patient overlap between the unlabeled set and the test set; the dental experiment and the prompt-extraction scheme are still worth a careful revision.","tokens_in":9068,"tokens_out":4657,"would_cite":false,"duration_ms":43112,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Training a U-Net on pseudo labels refined by SAM raises Dice scores from 74.29% to 84.17% on wrist bones and from 66.63% to 74.87% on teeth.","keywords":["semantic segmentation","Segment Anything Model","semi-supervised learning","pseudo labels","medical image segmentation","U-Net","Dice score","paediatric wrist bones"],"falsifier":"Run the pipeline on a test set of radiographs where the initial U-Net fails to detect some objects at all (for example, low-contrast casts or atypical anatomy) and compare the Dice of the SAM-refined pseudo labels on those images with the unrefined predictions; if refinement only fixes partially covered objects and never recovers wholly missed ones, the score on such cases should stay near the baseline.","tokens_in":8154,"feed_emoji":"🩻","tokens_out":5601,"duration_ms":49176,"temperature":0.7,"pith_summary":"This paper proposes a recipe for learning medical image segmenters from very few annotated images: let a weakly supervised U-Net propose imperfect masks, convert those masks into bounding-box and seed-point prompts, and hand the prompts to the off-the-shelf Segment Anything Model (SAM) to produce refined pseudo labels. The refined labels are then used to train a second U-Net on unlabelled data. On paediatric wrist radiographs this raises Dice from 74.29% to 84.17%, and on dental radiographs from 66.63% to 74.87%, beating the nnU-Net and Mean Teacher baselines. The paper argues that the combination matters: unrefined pseudo labels give no benefit, while SAM-refined labels match fully supervised performance on teeth (74.87% with 45 labelled images versus 74.19% with all 450).","feed_headline":"SAM-refined labels lift wrist Dice to 84%, teeth to 75%","feed_subtitle":"Training a U-Net on SAM-refined pseudo labels beats supervised and semi-supervised baselines on two medical tasks.","key_machinery":"The load-bearing mechanism is prompt extraction from imperfect predictions. For each predicted class mask, the pipeline keeps only the connected component with the highest area-normalised likelihood, applies morphological dilation to make the prompt robust, then feeds SAM an axis-aligned bounding box, the component's centre of mass as a positive seed point, and the centres of all other classes as negative seed points. SAM's mask output is then fed back as a dense prompt in a self-refinement loop. This converts a crude, over- or under-segmented U-Net prediction into a clean, object-aware pseudo label without any human interaction.","core_discovery":"The central claim is that SAM's abstract object understanding can carry the burden of annotation quality in semi-supervised medical segmentation when it is prompted, not with hand-placed clicks, but with prompts extracted automatically from a model trained on a few dozen labelled images. The paper shows that a U-Net trained on these SAM-refined pseudo labels reaches 84.17% Dice on 17 paediatric wrist bones and 74.87% on teeth, outperforming the same U-Net trained only on labelled data, a supervised nnU-Net, and a Mean Teacher semi-supervised baseline. On the dental dataset the SAM-refined pipeline matches a fully supervised U-Net trained on ten times more annotations. The paper also reports that refinement with SAM outperforms intensity-based Random Walk post-processing and MedSAM, and that the self-refinement step (passing SAM its own previous mask as an additional prompt) is a measurable contributor.","pith_inferences":["The same prompting recipe should transfer to other single-instance-per-class segmentation problems, but applying it to objects that appear many times per image, such as cells or multiple lesions, would require a detection or instance-separation step before the mask-cleaning stage.","Because SAM is used off-the-shelf and never fine-tuned, the pipeline's cost is essentially one extra inference pass per unlabelled image; a natural next comparison is against human-in-the-loop annotation time, which would show whether the refinement step actually saves expert hours.","The paper's failure analysis suggests a testable extension: use the overlap between the initial mask and SAM's refined mask as a confidence filter, discarding pseudo labels where SAM disagrees strongly with the prompt, which could improve robustness on the worst test cases."],"forward_implications":["On paediatric wrist radiographs, training a U-Net on SAM-refined pseudo labels raises Dice from 74.29% to 84.17%, exceeding both nnU-Net (77.6%) and Mean Teacher (79.3%).","On the dental radiograph dataset, the same recipe reaches 74.87% Dice using 45 labelled images plus 405 unlabelled images, matching the 74.19% of a U-Net trained on all 450 labelled images.","Using SAM for refinement beats intensity-based Random Walk post-processing and the medical-domain MedSAM variant on both datasets, and the self-refinement step contributes 2.21% Dice on the wrist dataset.","Across an ablation study with 1 to 43 labelled training images, the SAM-refined pseudo-label training outperforms all comparison methods at every training-set size."],"supporting_citations":[{"why":"supplies the weakly trained U-Net whose imperfect masks are the input to the refinement pipeline","marker":"[22]"},{"why":"provides the promptable Segment Anything Model that turns extracted boxes and seed points into refined masks","marker":"[15]"},{"why":"provides the paediatric wrist x-ray images and manual annotations used for training, validation, and testing","marker":"[21]"},{"why":"provides the dental radiographs and expert annotations for the teeth segmentation task","marker":"[19]"},{"why":"serves as the state-of-the-art supervised segmentation baseline the method must outperform","marker":"[13]"},{"why":"serves as the semi-supervised Mean Teacher baseline the method is compared against","marker":"[24]"},{"why":"serves as the medical-domain SAM variant used to test whether a medical-specific model improves refinement","marker":"[20]"},{"why":"serves as the intensity-based post-processing baseline for refinement comparison","marker":"[8]"}],"fun_headline_variants":["SAM pseudo labels lift wrist Dice to 84%, teeth to 75%","SAM boosts semi-supervised segmentation: 84% wrist, 75% teeth","SAM-refined labels beat nnU-Net and Mean Teacher in segmentation","SAM: from few annotated cases to 84% Dice on wrist bones","SAM refines pseudo labels, outdoes supervised baselines on medical scans"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes the initial segmentation produced by the weakly trained U-Net partially covers every object of interest; if an object is missed entirely, the extracted box and seed points cannot guide SAM to it and the pseudo label for that object will be wrong.","fun_headline_variants_meta":{"raw":{"variants":["SAM pseudo labels lift wrist Dice to 84%, teeth to 75%","SAM boosts semi-supervised segmentation: 84% wrist, 75% teeth","SAM-refined labels beat nnU-Net and Mean Teacher in segmentation","SAM: from few annotated cases to 84% Dice on wrist bones","SAM refines pseudo labels, outdoes supervised baselines on medical scans"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000837,"raw_usage":{"total_tokens":3663,"prompt_tokens":969,"completion_tokens":2694,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":585,"completion_tokens_details":{"reasoning_tokens":2595}},"tokens_in":585,"tokens_out":2694,"duration_ms":20956,"temperature":1.0,"reasoning_tokens":2595,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T17:20:50.196081+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the pipeline on a test set of radiographs where the initial U-Net fails to detect some objects at all (for example, low-contrast casts or atypical anatomy) and compare the Dice of the SAM-refined pseudo labels on those images with the unrefined predictions; if refinement only fixes partially covered objects and never recovers wholly missed ones, the score on such cases should stay near the baseline.","supporting_citations":[{"cited_title":"2023 IEEE/CVF International Conference on Computer Vision (ICCV) pp","cited_arxiv_id":null,"evidence_quote":"provides the promptable Segment Anything Model that turns extracted boxes and seed points into refined masks"},{"cited_title":"Advances in neural information processing systems 30 (2017) SAM Carries the Burden 11","cited_arxiv_id":null,"evidence_quote":"serves as the semi-supervised Mean Teacher baseline the method is compared against"},{"cited_title":"IEEE Transactions on Pattern Analysis and Machine Intelligence 28, 1768–1783 (2006)","cited_arxiv_id":null,"evidence_quote":"serves as the intensity-based post-processing baseline for refinement comparison"}],"review_version":1}