{"id":"1401b2cb-a81c-455c-ad44-7232726548d8","arxiv_id":"2501.16740","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A lightweight SAM student trained with MSE plus perceptual loss reaches SAM-like Dice scores on some medical datasets but falls behind on breast ultrasound, with no error bars or prompt details.","lead":"This paper distills the large Segment Anything Model (SAM) into a smaller ResNet-based model for medical image segmentation, using a mix of pixel and perceptual losses. Across four medical datasets the small model roughly matches or beats the big model on two, but clearly lags on breast ultrasound.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'comparable or superior' claim is contradicted by the paper's own Table 1 on Breast Ultrasound and Kvasir, and no prompting protocol is reported for the SAM baselines, so Table 1 cannot be reconstructed or trusted.","rationale":"The reader's weakest assumption—missing prompting protocol—is real and load-bearing, but I would put the primary weight on the paper's own Table 1, which directly contradicts the 'comparable or superior' claim on two of the four datasets. The prompt ambiguity matters because it is the only conceivable way the baseline numbers could be dismissed, but even granting the authors every benefit of the doubt on protocol, the reported KD SAM result on Breast Ultrasound is 0.8216 versus 0.8985 for MobileSAM and 0.9051 for SAM, a gap too large to call 'comparable.' The paper also provides no error bars, no statistical tests, no code, and no split details, so the central empirical claim cannot be independently checked. The method itself is a minor variation on MobileSAM's decoupled distillation with an added perceptual loss, and Equation 3 is internally inconsistent about whether the perceptual features come from VGG or from the ViT/ResNet encoders, though that inconsistency is secondary to the empirical failure. I see no machine-checked proofs or reproducible artifacts that would offset these problems. Therefore the reader's REJECT verdict is appropriate: the central claim is not supported by the evidence presented, and the most decisive next step is a controlled, prompt-matched reproduction.","tokens_in":5452,"tokens_out":3783,"duration_ms":39350,"concrete_test":"Obtain the authors' code and rerun all four datasets with a single identical prompt protocol for every model—for example, ground-truth bounding box, plus a center-point variant—on identical fixed splits with at least 5 seeds, reporting mean and standard deviation of Dice. If Breast Ultrasound remains near 0.82 and Kvasir near 0.86 while SAM/MobileSAM stay near 0.90/0.87, the abstract claim is false. If KD SAM rises to within about one point of the baselines, the original Table 1 was a prompt-protocol artifact. The run must also report prompt type, prompt count, seeds, and per-seed Dice values.","verdict_should_be":"REJECT","load_bearing_attack":"The central claim in the abstract and Section 5 is that KD SAM achieves comparable or superior performance to SAM and MobileSAM with fewer parameters. The load-bearing condition is that the Table 1 numbers are a fair, apples-to-apples comparison. This condition fails in two ways. First, Table 1 itself contradicts the claim: on Breast Ultrasound KD SAM scores 0.8216 versus 0.9051 for SAM and 0.8985 for MobileSAM, an 8-point gap; on Kvasir-SEG it scores 0.8586 versus 0.8715 and 0.8719. So on two of four datasets the method is not comparable. Second, the paper never states what prompts were used for SAM and MobileSAM (point, box, or mask; ground-truth box or user clicks; how many prompts), and SAM's accuracy is highly prompt-sensitive. Section 4 gives dataset names and optimizer hyperparameters but no evaluation protocol, prompt settings, seeds, or train/test splits. Without this, the baseline rows in Table 1 are not well-defined: the Breast Ultrasound gap could be a genuine method failure or an artifact of an unfair baseline evaluation. The parameter claim is also internally hedged: MobileSAM has 5M parameters while KD SAM has 26.4M, so 'fewer parameters than baseline models' is only true compared with SAM, not with MobileSAM.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes KD SAM, a two-phase knowledge-distillation approach for compressing the Segment Anything Model (SAM) for medical image segmentation. In the first phase, a ResNet-50 encoder is trained to mimic SAM's ViT-H encoder using a combination of mean squared error and perceptual loss. In the second phase, the SAM decoder is fine-tuned on medical segmentation datasets with Dice loss while the distilled encoder is frozen. The authors evaluate KD SAM on Kvasir-SEG, ISIC 2017, Fetal Head Ultrasound, and Breast Ultrasound, reporting Dice coefficients in Table 1 and claiming comparable or superior performance to SAM and MobileSAM with significantly fewer parameters.","tokens_in":5680,"tokens_out":3038,"duration_ms":28579,"significance":"If the claimed performance held, a 26.4M-parameter medical image segmenter approaching SAM's accuracy would be practically valuable for resource-constrained settings. The paper also addresses a real problem, since SAM and MobileSAM have limitations in medical segmentation. However, the manuscript's central claim is not supported by its own reported numbers, and the experimental protocol is too incomplete to establish the validity of the comparison. The method itself is plausible and builds on existing distillation ideas, but the evidence as presented is not sufficient to justify the conclusions.","major_comments":[{"comment":"The central claim that KD SAM achieves 'comparable or superior performance to the baseline models' is contradicted by the numbers in Table 1 on two of four datasets. On Breast Ultrasound, KD SAM obtains 0.8216 versus 0.9051 for SAM and 0.8985 for MobileSAM, an approximately 8-point gap. On Kvasir-SEG, KD SAM scores 0.8586 versus 0.8715 and 0.8719. The abstract and Section 5 therefore overstate the results; the claim is only partially supported (Fetal Head, ISIC) and even there the differences are within a range that requires statistical validation.","section":"Abstract and Table 1"},{"comment":"The evaluation protocol for the SAM and MobileSAM baselines is not specified. SAM is a prompt-based interactive model whose accuracy depends strongly on the prompt type (point, box, mask) and prompt quality. The paper never states what prompts were used for the baseline models, how many prompts, or whether ground-truth boxes were used. Without this information, the baseline Dice scores in Table 1 are not well-defined and the comparison cannot be reproduced or trusted. This is a load-bearing omission because the entire conclusion hinges on a fair baseline comparison.","section":"Section 4 and Table 1"},{"comment":"The statement that KD SAM has 'significantly fewer parameters' than the baseline models is misleading. The paper reports 26.4M parameters for KD SAM versus 632M for SAM, which is indeed far fewer, but MobileSAM has only 5M parameters. Thus KD SAM has substantially more parameters than MobileSAM. The sentence in Section 5 acknowledges this but the abstract does not. The efficiency claim should be qualified to compare against SAM only, or should explain why 26.4M is still advantageous relative to MobileSAM in the medical imaging context.","section":"Section 5, parameter comparison"},{"comment":"The definition of perceptual loss in Eq. (3) is internally inconsistent. The text states that perceptual loss uses a pre-trained VGG network, but Eq. (3) defines φ^T_l and φ^S_l as feature maps from 'the ViT and ResNet encoders, respectively.' The notation also applies C_l, H_l, W_l, which are spatial/depth dimensions suited to convolutional feature maps, but ViT outputs are sequences of token embeddings. As written, the loss cannot be implemented without additional clarification of how VGG is used and how the feature maps are aligned.","section":"Section 3.1, Eq. (3)"},{"comment":"The inference protocol for KD SAM itself is not described. The paper says the decoder is fine-tuned with Dice loss, but it does not state whether prompts are used at test time and, if so, what kind and how many. For an interactive segmentation model, the test-time prompting protocol is essential for interpreting the reported Dice scores. Without this, the reader cannot determine what the numbers in Table 1 mean, even for the proposed method.","section":"Sections 3.2 and 4"}],"minor_comments":[{"comment":"The paper does not specify train/validation/test splits, dataset preprocessing details, or the random seeds used for training and evaluation. These details are necessary for reproducibility.","section":"Section 4"},{"comment":"The modification of ResNet-50 (reducing channels from 2048 to 256 and adding upsampling layers) is described only briefly; the exact architecture and how the spatial dimensions are matched to the ViT feature maps should be given explicitly.","section":"Section 4"},{"comment":"The sentence 'ResNet-50 as several considerations drove the student model' in Section 3.1 is ungrammatical; it should be rephrased, e.g., 'Several considerations drove the selection of ResNet-50 as the student model.'","section":"Section 5"},{"comment":"Figures 2 and 3 are referenced but not explained in the text; the reader is not told what the box plot and bar chart show or how the qualitative examples in Figure 3 were selected.","section":"Section 5"},{"comment":"The related work discussion would benefit from a direct comparison with MobileSAM's decoupled distillation approach, since the proposed method appears similar in spirit; the paper should clarify the novel contribution relative to MobileSAM.","section":"Section 2"}],"recommendation":"reject","confidential_remarks":"The gap between the abstract's claim and the data in Table 1 is severe and not a simple wording issue; the experimental protocol is also too incomplete to salvage the comparison without new experiments. The authors would need to rerun evaluations with a clearly specified prompting protocol and, ideally, report error bars and statistical tests. Until then, the manuscript does not meet the bar for publication. This is not a case of disagreement with consensus; it is a case of internal inconsistency between the stated claims and the reported results."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a modest, clearly written engineering paper that applies MobileSAM's decoupled distillation to a ResNet-50 student and adds a perceptual loss. The headline claim of 'comparable or superior' performance does not survive contact with Table 1: on Breast Ultrasound KD SAM scores 0.8216 Dice versus 0.9051 for SAM and 0.8985 for MobileSAM, an 8-point gap; on Kvasir it is 0.8586 versus 0.8715 and 0.8719. On half the datasets the student is clearly worse, and on the other two the gains are marginal. The abstract and Section 5 overstate what the table shows.\n\nWhat is new is limited but real: the specific combination of a modified ResNet-50 student (channels reduced to 256) with unweighted MSE plus perceptual loss for encoder distillation, evaluated on four medical datasets, is not in the cited papers. The decoupled two-phase training is practical and clearly described, and the related work is honestly cited, including the paper's own admission that the recipe is 'as done by MobileSAM'. The qualitative figures provide some visual support.\n\nThe soft spots are mostly in the evaluation. No prompting protocol is given for SAM or MobileSAM, and SAM's accuracy is highly prompt-dependent. Without knowing whether the baselines used a center point, a bounding box, or multiple clicks, the numbers in Table 1 are not well-defined. There are no error bars, seeds, or statistical tests, and the train/test split is not described. The decoder is fine-tuned per dataset and evaluated on held-out data, which is fine in principle, but the details are missing. Equation (3) is internally inconsistent: the text says perceptual loss uses VGG features, but the equation defines phi_l^T and phi_l^S as feature maps from the ViT and ResNet encoders. Also, the 'significantly fewer parameters' claim is only true against SAM (632M); MobileSAM has 5M, so KD SAM at 26.4M is not the smallest lightweight option.\n\nWho is this for? A reader tracking lightweight SAM variants for medical segmentation might want a peek, but the results as presented do not establish an advantage over MobileSAM. The paper deserves a serious referee, though major revisions are needed: report the prompt protocol, add error bars and reproducibility details, fix Eq. (3), and rewrite the claim in the abstract. I would not cite this as a reliable comparison until those issues are addressed.","headline":"A reasonable engineering variant of MobileSAM whose own Table 1 undercuts the 'comparable or superior' claim.","tokens_in":6298,"tokens_out":3367,"would_cite":false,"duration_ms":28558,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that KD SAM, a 26.4M-parameter distillation of SAM trained with MSE plus perceptual loss and decoder fine-tuning, reaches comparable or better Dice than SAM and MobileSAM on most of the four medical datasets.","keywords":["knowledge distillation","Segment Anything Model","medical image segmentation","perceptual loss","encoder distillation","decoder fine-tuning","Dice loss","ResNet-50"],"falsifier":"Run SAM, MobileSAM, and KD SAM on the same held-out test folds with identical, fixed prompts (e.g., ground-truth bounding boxes for every image) and compare Dice; the central claim is falsified if KD SAM's mean Dice is not within the paper's reported margins on Fetal Head and ISIC 2017, or if its deficits on Kvasir-SEG and Breast Ultrasound are larger than reported.","tokens_in":5175,"feed_emoji":"🩺","tokens_out":13429,"duration_ms":112355,"temperature":0.7,"pith_summary":"KD SAM is the paper's proposal for shrinking the Segment Anything Model for medical image segmentation: it replaces SAM's 632M-parameter ViT-H encoder with a modified ResNet-50 (26.4M total parameters) and then fine-tunes the SAM decoder. The training is decoupled -- the encoder is distilled first using the sum of mean squared error and a perceptual loss from a pre-trained feature extractor, and the decoder is fine-tuned afterwards with Dice loss while the encoder stays frozen. The paper reports Dice scores of 0.9774 on Fetal Head Ultrasound, 0.9114 on ISIC 2017, 0.8586 on Kvasir-SEG, and 0.8216 on Breast Ultrasound, and it summarizes these as comparable or superior to SAM and MobileSAM on most of these tasks. If the claim holds, interactive prompt-driven segmentation at SAM-like accuracy could run on hardware with roughly one twenty-fourth of SAM's parameter budget.","feed_headline":"26.4M-parameter SAM student matches the original on most medical sets","feed_subtitle":"The 26.4M student keeps Dice close to the 632M original, a step toward on-device use.","key_machinery":"The machinery that carries the argument is the decoupled distillation pipeline behind KD SAM: a combined loss $L_{\\mathrm{Combined}} = L_{\\mathrm{MSE}} + L_{\\mathcal{P}}$ on encoder feature maps, where $L_{\\mathrm{MSE}}$ is pixel-wise mean squared error between teacher and student feature maps and $L_{\\mathcal{P}}$ is a perceptual loss computed from selected layers of a pre-trained convolutional feature extractor; the student encoder is a ResNet-50 with channels trimmed to 256 and upsampling layers to match the teacher's spatial resolution. After encoder distillation, the SAM decoder is fine-tuned with Dice loss while the encoder is frozen, so the decoder adapts to the student's feature distribution.","core_discovery":"The central claim is that a decoupled distillation can preserve SAM's medical segmentation quality at a fraction of its size. The paper trains a modified ResNet-50 encoder to imitate SAM's ViT-H feature maps under the combined loss $L_{\\mathrm{Combined}} = L_{\\mathrm{MSE}} + L_{\\mathcal{P}}$, where $L_{\\mathrm{MSE}}$ is the mean squared error between teacher and student feature maps and $L_{\\mathcal{P}}$ is a perceptual loss from selected pre-trained feature-extractor layers, then fine-tunes the SAM mask decoder with Dice loss while the encoder is frozen. The resulting 26.4M-parameter KD SAM is reported to reach Dice scores of 0.9774 on Fetal Head Ultrasound and 0.9114 on ISIC 2017, above both baselines, while scoring 0.8586 on Kvasir-SEG and 0.8216 on Breast Ultrasound, below one or both baselines. The paper's summary claim is that KD SAM is comparable or superior to SAM and MobileSAM across most of the four datasets.","pith_inferences":["The paper never specifies the prompts used to obtain the SAM and MobileSAM baselines, so the cross-model ranking in Table 1 is provisional until the evaluation protocol is pinned down; this is an editorial caution, not a result in the paper.","The same two-stage loss could be ablated by training the student with MSE only, perceptual only, and the sum, to see which term drives the gains on fetal-head and melanoma boundaries; the paper presents only the combined loss.","Because the drop is largest on Breast Ultrasound, a modality with heterogeneous echo patterns, the distillation may transfer unevenly across imaging domains; per-modality calibration of the perceptual loss layers is a natural next experiment.","With MobileSAM at 5M parameters and KD SAM at 26.4M, there is a wide middle range of the accuracy-efficiency curve; one could sweep student backbones to locate the Pareto frontier, which the paper does not explore."],"forward_implications":["At 26.4M parameters, KD SAM is roughly 24x smaller than SAM, so a faithful reproduction of the reported results would put prompt-driven medical segmentation in reach of mobile and edge hardware.","The reported Fetal Head (0.9774) and ISIC 2017 (0.9114) Dice scores exceed both SAM and MobileSAM, indicating that on these two tasks the distilled student does not pay an accuracy penalty.","The decoupled recipe -- encoder distillation with MSE plus perceptual loss, followed by frozen-encoder decoder fine-tuning with Dice loss -- is the reusable component that other distillation efforts could adopt.","The paper's own numbers show the tradeoff is not uniform: Kvasir-SEG is 0.8586 versus 0.8715 for SAM, and Breast Ultrasound is 0.8216 versus 0.9051, so the claim of comparable or superior performance rests on how the per-dataset results are aggregated."],"supporting_citations":[{"why":"Defines the Segment Anything Model, which supplies the ViT-H teacher encoder and the decoder that KD SAM fine-tunes, and serves as the primary accuracy baseline.","marker":"[1]"},{"why":"Introduces MobileSAM, the lightweight baseline that motivates the work and that KD SAM is compared against in Table 1.","marker":"[3]"},{"why":"Provides the perceptual loss term used alongside MSE in the encoder distillation objective.","marker":"[4]"},{"why":"Supplies ResNet-50, the student backbone chosen for the distilled encoder.","marker":"[12]"},{"why":"Supplies the pre-trained feature extractor used to define the perceptual loss.","marker":"[13]"},{"why":"Provides the Kvasir-SEG polyp segmentation dataset used for evaluation.","marker":"[14]"},{"why":"Provides the ISIC 2017 skin-lesion dataset used for evaluation.","marker":"[15]"},{"why":"Provides the Fetal Head Ultrasound dataset used for evaluation.","marker":"[16]"},{"why":"Provides the Breast Ultrasound dataset used for evaluation.","marker":"[17]"}],"fun_headline_variants":["SAM distilled to 26.4M params, holds its own on medical scans","Tiny SAM: 96% smaller, still sharp on medical images","KD SAM: 26.4M params, Dice on par with 632M SAM","From 632M to 26.4M: SAM stays sharp for medical tasks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison assumes that SAM and MobileSAM were evaluated under the same prompt protocol -- same type and quality of point or box prompts -- as KD SAM, but the paper never reports what prompts the baselines received.","fun_headline_variants_meta":{"raw":{"variants":["SAM distilled to 26.4M params, holds its own on medical scans","Tiny SAM: 96% smaller, still sharp on medical images","KD SAM: 26.4M params, Dice on par with 632M SAM","From 632M to 26.4M: SAM stays sharp for medical tasks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00104,"raw_usage":{"total_tokens":4365,"prompt_tokens":923,"completion_tokens":3442,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":539,"completion_tokens_details":{"reasoning_tokens":3354}},"tokens_in":539,"tokens_out":3442,"duration_ms":23131,"temperature":1.0,"reasoning_tokens":3354,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T11:02:31.092371+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run SAM, MobileSAM, and KD SAM on the same held-out test folds with identical, fixed prompts (e.g., ground-truth bounding boxes for every image) and compare Dice; the central claim is falsified if KD SAM's mean Dice is not within the paper's reported margins on Fetal Head and ISIC 2017, or if its deficits on Kvasir-SEG and Breast Ultrasound are larger than reported.","supporting_citations":[{"cited_title":"The Segment Any- thing Model (SAM) [1] has established itself as a powerful tool in this domain, leveraging a Vision Transformer (ViT)","cited_arxiv_id":null,"evidence_quote":"Defines the Segment Anything Model, which supplies the ViT-H teacher encoder and the decoder that KD SAM fine-tunes, and serves as the primary accuracy baseline."},{"cited_title":"Initially de- veloped for classification tasks [5], it has been adapted for dense prediction tasks such as semantic segmentation [6] and object detection [7]","cited_arxiv_id":null,"evidence_quote":"Introduces MobileSAM, the lightweight baseline that motivates the work and that KD SAM is compared against in Table 1."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the perceptual loss term used alongside MSE in the encoder distillation objective."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the pre-trained feature extractor used to define the perceptual loss."},{"cited_title":"Differ- entiable feature aggregation search for knowledge dis- tillation,","cited_arxiv_id":null,"evidence_quote":"Provides the ISIC 2017 skin-lesion dataset used for evaluation."},{"cited_title":"Yolact: Real-time instance segmentation,","cited_arxiv_id":null,"evidence_quote":"Provides the Breast Ultrasound dataset used for evaluation."}],"review_version":1}