{"id":"46b7326a-b924-41d1-b3bc-c9a46f84cffe","arxiv_id":"2501.06836","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"A zero-initialized attention adapter placed in SAM's mask decoder matches full fine-tuning on medical segmentation and improves cross-domain generalization with under 1% trainable parameters.","lead":"This paper adds a small learnable adapter to the mask decoder of the Segment Anything Model (SAM), adapting it to medical images while training under 1% of SAM's parameters. The method aims to match full fine-tuning accuracy and improve generalization to unseen scanners and sites.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Decoder-placement claim is confounded by MedSAM initialization: all medical experiments start from a medical-adapted encoder, and the only SAM-initialized experiment (HQSeg-44K) contradicts the decoder-only advantage.","rationale":"The most load-bearing concern is that the paper's central novelty, the decoder-placement insight, is supported only by experiments in which the frozen encoder is MedSAM, i.e., an encoder already fine-tuned on a massive medical corpus with plausible overlap or similarity to the test domains. The reader's weakest assumption correctly identified the decoder-only premise as chosen rather than derived, but framed it as whether SAM's encoder features are domain-agnostic. In fact, the medical experiments never test that premise because they do not use the original SAM encoder: they use MedSAM, whose features are medical-adapted rather than domain-agnostic. The only experiment with official SAM weights, HQSeg-44K, shows encoder adaptation beating decoder adaptation and encoder-based methods beating SAM-DA, which is the opposite of the abstract's general claim. This is not an accusation of dishonesty; the use of MedSAM is stated in Section 4.2, but the paper's scope goes beyond what that setup supports. The proposed control experiment, re-running Tables 2 and 3 with official SAM weights, would directly settle whether decoder placement is a general property of SAM or an artifact of MedSAM initialization. The statistical caveat in Appendix C further weakens the 'statistically significant superiority' phrasing in Section 5.2, but it is secondary to the confound. Because the method may still be valuable as a parameter-efficient way to fine-tune MedSAM, rejection is too strong; a conditional acceptance requiring the SAM-initialized control and an overlap audit is the appropriate outcome. This reinforces rather than changes the reader's conditional verdict, but for a more specific reason.","tokens_in":13023,"tokens_out":7854,"duration_ms":82774,"concrete_test":"Run the fully supervised and domain-generalization protocols of Tables 2 and 3 on Retouch, MRI, and WMH using the official SAM ViT-B/16 checkpoint instead of MedSAM, keeping all hyperparameters and the same adapter code, and compare SAM-DA against full fine-tuning, LoRA, and Med-SA on both in-domain test sets and the Cirrus, UCL, and Singapore generalization targets. If SAM-DA no longer matches full fine-tuning or no longer beats encoder adapters in generalization, the decoder-placement claim is specific to MedSAM-initialized encoders. As a secondary check, audit MedSAM's SA-Med2D-20M training corpus for overlap with the three medical datasets or near-duplicate devices or sites, and report results on a guaranteed-disjoint target domain if overlap cannot be excluded.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"Section 5.3 attributes SAM-DA's generalization advantage to adapter placement in the decoder, and the abstract presents this as a property of SAM. However, all medical experiments initialize SAM ViT-B with MedSAM weights (Section 4.2), so the frozen encoder is not the 'SAM' that the paper says struggles on medical data; it is already a medical-domain-adapted encoder trained on a large corpus of medical image-mask pairs whose overlap with the Retouch, MRI, and WMH targets is not assessed. The only experiment with official SAM weights, HQSeg-44K (Tables 4 and 9), shows the opposite ranking in-domain: encoder adapter 80.8 vs. decoder adapter 79.6, and encoder-based LoRA and Med-SA beat SAM-DA by roughly 4 IoU points. This is exactly the pattern expected if decoder-only adaptation helps only because the encoder has already absorbed the medical domain. The domain-generalization results in Table 3 therefore do not establish that decoder placement is broadly superior for SAM; they establish at most that a decoder adapter is a cheap way to fine-tune a MedSAM-pretrained model on a specific source domain. The paper should either scope its claims to MedSAM-initialized models or demonstrate the decoder advantage from official SAM weights. Additionally, the paper's own statistical appendix flags that the single paired t-test used to support significance is 'not completely justified,' yet Section 5.2 asserts statistically significant superiority without reporting tests for the generalization comparisons.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SAM-DA, a parameter-efficient fine-tuning method for the Segment Anything Model (SAM) that inserts a zero-initialized, prompt-based attention adapter into the mask decoder. The adapter uses a small set of learnable prompt tokens as keys and values, with decoder embeddings as queries, and a gating factor that starts at zero. The method is evaluated on four datasets (Retouch, MRI, WMH, HQSeg-44K) under fully supervised training, domain generalization, and test-time domain adaptation, and is compared against full fine-tuning, decoder-only fine-tuning, LoRA, Med-SA, and HQ-SAM. The core claims are that the decoder placement yields strong generalization and that the method matches or exceeds state-of-the-art while training less than 1% of SAM's parameters.","tokens_in":13353,"tokens_out":4859,"duration_ms":46319,"significance":"If the central claims were fully supported, SAM-DA would be a practically valuable contribution: it is simple, parameter-efficient, and the experimental setup is more careful than is typical (four seeds, validation-set hyperparameter tuning, per-dataset splits, and an honest statistical appendix). The method is clearly described and the ablations are informative. However, the paper's headline claims are not fully backed by its own tables, and the decoder-placement advantage is confounded by the use of MedSAM initialization in all medical experiments. The core architectural insight is plausible and worth publishing after appropriate scoping and additional analysis.","major_comments":[{"comment":"The abstract and introduction claim that SAM-DA 'outperform[s] existing methods' and achieves 'comparable performances to full fine-tuning,' but Table 2 shows otherwise on two of the four datasets: on WMH, Med-SA achieves 44.7 IoU while SAM-DA achieves 44.2, and on HQSeg-44K, LoRA (83.1) and Med-SA (83.8) both exceed SAM-DA (79.6). The claims should be scoped to reflect the actual rankings, or the presentation should emphasize the settings where the method is genuinely superior.","section":"Abstract and Table 2"},{"comment":"The claim that decoder placement is broadly superior for SAM is confounded by initialization. All medical experiments initialize SAM ViT-B with MedSAM weights, and the only experiment with official SAM weights, HQSeg-44K, shows the opposite ranking: the encoder adapter reaches 80.8 IoU versus 79.6 for the decoder adapter, and both LoRA and Med-SA outperform SAM-DA by roughly 4 points. The domain-generalization results in Table 3 therefore do not establish a general decoder advantage for SAM; they may hold only for MedSAM-initialized models. The paper should either scope its claims to MedSAM-initialized models or demonstrate the decoder advantage from official SAM weights on a medical task.","section":"Section 4.2 and Section 5.3 (Tables 4 and 9)"},{"comment":"Section 5.2 asserts 'statistically significant superiority' of SAM-DA on zero-shot generalization compared to Med-SA and LoRA, but no significance tests are reported for the results in Table 3. The only significance test in the appendix is a paired t-test for the fully supervised Retouch case, and the authors themselves state that this test is 'not completely justified' because image-wise mIoU scores are not directly comparable across images. The paper should either report an appropriate statistical analysis for the generalization claims or soften the language accordingly.","section":"Section 5.2 and Appendix C"},{"comment":"The test-time domain adaptation results in Table 8 are reported without any error bars or indication of the number of runs. The paper states that experiments are repeated four times for the fully supervised setting, but it is unclear whether this applies to TTDA. Given that the differences between methods are small (e.g., 67.5 for SAM-DA vs. 67.0 for LoRA on Retouch), the lack of variance information makes it difficult to assess whether the reported improvements are meaningful.","section":"Section 4.4 and Table 8"},{"comment":"The claim that initializing the gating factor to zero 'ensure[s] no disruption during the early stages of adaptation' is only valid if the linear projection Lineart in Eq. (5) is initialized as an identity map. As written, T'_ℓ = Lineart(Tℓ + gℓ·S'_ℓ) with gℓ=0 still transforms Tℓ through Lineart, which would alter the embedding unless that layer is specially initialized. The paper should specify the initialization of Lineart and, if needed, use a residual connection so that the zero-init guarantee actually holds.","section":"Equation (5) and Section 3.2"}],"minor_comments":[{"comment":"The note that HQSeg-44K 'uses the same set for validation and testing' is confusing and potentially problematic if hyperparameters are tuned on the test set. Please clarify whether any hyperparameter selection was performed on that split and how the reported IoU values were obtained.","section":"Table 1"},{"comment":"The phrase 'this dataset can be considered quite distinct' is vague. It would be clearer to say that HQSeg-44K is much larger than the others and contains natural images, which explains the different behavior of PEFT methods.","section":"Section 5.1"},{"comment":"The quantitative values in the Figure 4 caption are not fully explained; please add a sentence describing how the IoU scores are computed and which methods the numbers refer to.","section":"Figure 4"},{"comment":"The related work section mentions only a few recent SAM-based PEFT works; citing additional recent methods such as SAM-Adapter and other prompt-based adaptations would help position the contribution.","section":"Section 2.1"}],"recommendation":"major_revision","confidential_remarks":"The paper is technically sound in its experimental protocol (four seeds, validation tuning, honest appendix) but the strongest claims are not supported by the reported tables. The MedSAM-initialization confound is the most serious issue: the paper's own HQSeg-44K experiment with official SAM weights contradicts the decoder-placement advantage. The authors should either substantially narrow the claims or add experiments that separate the effects of initialization from adapter placement. The statistical appendix is commendably honest, but the paper should not rely on a self-admittedly unjustified test to support a significance claim. If the authors address these points, the paper could become a useful contribution to the PEFT-for-SAM literature."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear [name],\n\nThe short version: this is a competent, practical PEFT paper with one genuinely useful result—adapting only SAM's mask decoder with a zero-initialized attention adapter keeps up with full fine-tuning on medical data while training well under 1% of parameters—and one interpretive claim that does not yet hold up.\n\nWhat's new: the adapter itself is a direct transplant of LLaMA-Adapter's zero-init attention into SAM's decoder, so the novelty is configuration, not mechanism. That is fine. The empirical finding that decoder-only adaptation generalizes better across medical scanners than encoder-focused adapters (LoRA, Med-SA) is interesting, and the fully supervised experiments are done with care: four seeds, validation-based hyperparameter selection, and an unusually honest statistical appendix. The TTDA experiments, though simpler, are a reasonable extra.\n\nNow the soft spots. The abstract says the adapter 'outperform[s] existing methods', but Table 2 shows it is best on MRI and Retouch, second on WMH, and well behind LoRA and Med-SA on HQSeg-44K. Fine to claim competitive; overclaiming invites referees to distrust the rest.\n\nMore important: the decoder-placement conclusion is confounded by MedSAM initialization. All medical experiments start from MedSAM weights, so the frozen encoder is already medical-adapted. The one experiment with official SAM weights—HQSeg-44K—shows the encoder adapter does better in-domain (80.8 vs 79.6), which is exactly the opposite of the paper's story. Section 5.3 treats decoder placement as the cause of the generalization gains, but in the medical experiments the encoder has already absorbed much of the medical domain. The paper needs to either scope its claims to MedSAM-initialized models or demonstrate the decoder advantage starting from official SAM.\n\nMinor issues: Table 8 (TTDA) has no error bars; Section 5.2 asserts statistical significance for generalization without reporting a test, while the appendix concedes the one test they do run is 'not completely justified.'\n\nNet: this is a serious empirical paper worth a real referee. Revise the abstract, confront the MedSAM confound, add uncertainty to the TTDA numbers, and it would be a solid contribution. As it stands, I'd send it out rather than desk reject.\n\nBest,\n[Your name]","headline":"Useful practical adapter with solid experiments, but the decoder-placement story is partly a MedSAM initialization effect and the abstract overclaims.","tokens_in":13874,"tokens_out":2979,"would_cite":true,"duration_ms":27068,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A zero-initialized attention adapter placed in SAM's mask decoder achieves comparable performance to full fine-tuning while training less than 1% of SAM's parameters, and improves generalization to unseen medical domains.","keywords":["SAM","parameter-efficient fine-tuning","domain adaptation","medical image segmentation","test-time domain adaptation","decoder adapter","zero-init attention","semantic segmentation"],"falsifier":"Take a medical dataset where the domain shift is primarily in the image encoder, such as training on retinal OCT and testing on a different modality like ultrasound or CT, and compare SAM-DA against an encoder adapter with matched parameters. If the encoder adapter equals or beats the decoder adapter on the unseen domain, the paper's central claim that decoder-only adaptation is preferable fails.","tokens_in":12859,"feed_emoji":"🩺","tokens_out":3702,"duration_ms":33804,"temperature":0.7,"pith_summary":"This paper argues that a medical imaging model built on SAM can be adapted with fewer than 1% of the model's parameters by inserting a zero-initialized attention adapter only in the mask decoder, leaving the image encoder frozen. The authors claim this decoder-only adapter matches or beats full fine-tuning on fully supervised segmentation and test-time domain adaptation, and that it generalizes to unseen domains better than encoder-focused adapters such as LoRA or Med-SA. They validate this on three medical datasets (retinal OCT, prostate MRI, brain MRI) plus the natural-image HQSeg-44K benchmark. The reason a reader should care: if true, adapting large segmentation foundation models to new medical settings becomes cheap enough to do per clinic, per device, or even per test image.","feed_headline":"Decoder-only SAM adapter matches full fine-tuning with <1% of parameters","feed_subtitle":"Zero-init attention in the mask decoder keeps the frozen encoder intact and boosts generalization to unseen medical domains.","key_machinery":"The central object is the SAM Decoder Adapter (SAM-DA): at each of the two layers of SAM's mask-decoder transformer, a learnable prompt $A_\\ell$ is fed as keys and values into a zero-initialized cross-attention block whose queries are the decoder's own dense embeddings $T_\\ell$. A learnable gating scalar $g_\\ell$, initialized to zero, scales the attention output before it is added to the embeddings, so training starts from the unmodified pretrained model. This is the LLaMA-Adapter mechanism transplanted from NLP into SAM. Its job is to let the decoder recombine the frozen encoder's features for a new domain while leaving the encoder untouched; the gating acts as a warm start that grows the correction gradually.","core_discovery":"The central claim is that the mask decoder of SAM is the right place to adapt the model for medical segmentation, and that a parameter-efficient adapter there can perform comparably to full fine-tuning while training 0.66M parameters out of roughly 90.6M (less than 1%). The paper shows that on domain generalization (training on one device or site, testing on another), the decoder adapter outperforms encoder-adaptation methods, with gains up to 8.5 and 13.8 IoU points on Retouch and WMH, respectively. The paper also reports a limitation: on the large HQSeg-44K dataset, encoder adapters exceed it (LoRA 83.1, Med-SA 83.8 vs. 79.6), indicating that decoder-only adaptation loses when abundant training data is available.","pith_inferences":["A natural extension the paper does not run is combining the decoder adapter with a light encoder adapter; the paper's own ablations show the two placements capture complementary strengths, so a hybrid might recover the HQSeg-44K gap while preserving generalization.","The zero-init gating suggests that the adapter learns a residual correction on the decoder's feature re-weighting; measuring the rank of the learned correction across datasets would test whether it behaves as a low-rank perturbation of the attention maps.","Because the paper uses only SAM's ViT-B/16 backbone with MedSAM weights, the claim should be re-checked on larger SAM variants (ViT-L/H) and on 3D medical data, which the authors note require architectural changes."],"forward_implications":["If the claim holds, any SAM-based segmentation model can be adapted to a new medical imaging protocol using only a few hundred thousand trainable parameters, making deployment on a single GPU or even at the edge feasible.","Test-time domain adaptation becomes practical: the adapter can be fit per image in a handful of iterations, and the paper reports the best IoU on all three medical datasets under this setting.","The generalization results imply that encoder freezing is not just a cost-saving measure but also a form of regularization, since updating the encoder hurts zero-shot transfer to unseen devices and sites.","On large natural-image datasets the advantage disappears, so the method's benefits are specific to the small-data medical regime."],"supporting_citations":[{"why":"Defines SAM and its encoder-prompter-decoder architecture, the base model that SAM-DA adapts.","marker":"[15]"},{"why":"LoRA, the primary parameter-efficient fine-tuning baseline whose encoder-focused approach SAM-DA is compared against.","marker":"[12]"},{"why":"LLaMA-Adapter, the source of the zero-initialized attention adapter mechanism transplanted into SAM's decoder.","marker":"[49]"},{"why":"HQ-SAM, a decoder-focused baseline and the source of the HQSeg-44K dataset used for large-scale evaluation.","marker":"[14]"},{"why":"Med-SA, the medical-specific encoder adapter baseline that SAM-DA must outperform.","marker":"[39]"},{"why":"MedSAM, whose pretrained weights initialize SAM for all medical experiments.","marker":"[28]"}],"fun_headline_variants":["Decoder SAM adapter: <1% params, matches fine-tuning","SAM decoder adapter beats full tuning with <1% trainable","Minimal SAM decoder tweak adapts to medical images","SAM's mask decoder is the key to medical adaptation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The frozen image encoder already extracts features that are good enough for medical images, so only the decoder needs adapting; the paper's own ablations show this holds for small medical datasets but not for the large HQSeg-44K set, where encoder adaptation wins.","fun_headline_variants_meta":{"raw":{"variants":["Decoder SAM adapter: <1% params, matches fine-tuning","SAM decoder adapter beats full tuning with <1% trainable","Minimal SAM decoder tweak adapts to medical images","SAM's mask decoder is the key to medical adaptation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000122,"raw_usage":{"total_tokens":1037,"prompt_tokens":826,"completion_tokens":211,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":442,"completion_tokens_details":{"reasoning_tokens":143}},"tokens_in":442,"tokens_out":211,"duration_ms":3109,"temperature":1.0,"reasoning_tokens":143,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:49:42.016938+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a medical dataset where the domain shift is primarily in the image encoder, such as training on retinal OCT and testing on a different modality like ultrasound or CT, and compare SAM-DA against an encoder adapter with matched parameters. If the encoder adapter equals or beats the decoder adapter on the unseen domain, the paper's central claim that decoder-only adaptation is preferable fails.","supporting_citations":[{"cited_title":"Berg, Wan-Yen Lo, Piotr Doll ´ar, and Ross Girshick","cited_arxiv_id":null,"evidence_quote":"Defines SAM and its encoder-prompter-decoder architecture, the base model that SAM-DA adapts."},{"cited_title":"LoRA: Low- rank adaptation of large language models","cited_arxiv_id":null,"evidence_quote":"LoRA, the primary parameter-efficient fine-tuning baseline whose encoder-focused approach SAM-DA is compared against."},{"cited_title":"Segment anything in high quality","cited_arxiv_id":null,"evidence_quote":"HQ-SAM, a decoder-focused baseline and the source of the HQSeg-44K dataset used for large-scale evaluation."},{"cited_title":"Segment anything in medical images","cited_arxiv_id":null,"evidence_quote":"MedSAM, whose pretrained weights initialize SAM for all medical experiments."}],"review_version":1}