{"id":"d0280037-5618-47af-86d2-ac42de89dfad","arxiv_id":"2508.11032","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Layer-wise search-based merging of SAM, MedSAM, and MedicoSAM improves average Dice on 25 tasks by 6.67 points when tuned per task and 4.37 points when tuned across tasks.","lead":"MedSAMix merges the general-purpose Segment Anything Model with medical fine-tuned variants by searching layer-wise merge recipes on a few labeled examples, with no retraining. It reports higher Dice scores across 25 medical segmentation tasks, but the gains are measured against zero-shot baselines that did not get the same calibration data.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Calibration-label asymmetry, not basin compatibility, is the key threat: MedSAMix-S uses 20% labeled calibration during search while SAM-based baselines are zero-shot, so the 6.67% specialized gain may reflect label access rather than the merging mechanism.","rationale":"The paper's internal logic is coherent: if the three SAM-initialized models lie in compatible loss basins, layer-wise merge recipes should exist, and SMAC can search for them on a calibration set. The empirical consistency across 25 tasks and the promised code/checkpoint release are positive evidence. However, the decisive threat to the central claim is not primarily basin compatibility; it is that the headline comparison gives MedSAMix labeled calibration data while giving SAM-based baselines none. The reader's rationale already identifies this asymmetry, but the formally stated weakest assumption is the basin-compatibility premise. I therefore mark partial agreement. A few-shot fine-tuned MedSAM with the same calibration data is the natural control and is absent; without it, the 6.67% and 4.37% numbers are not interpretable as effects of merging. The multi-task generalization claim is also weakened because the eight search tasks are contained in the 25-task evaluation set, so no held-out task generalization is demonstrated. These concerns do not warrant rejection, but they strengthen the need for the CONDITIONAL verdict already given, with the additional explicit condition of matched calibration-exposed baselines.","tokens_in":15945,"tokens_out":5968,"duration_ms":68747,"concrete_test":"Run a matched calibration-exposed baseline on the same 25 tasks: take MedSAM (and separately MedicoSAM), apply a lightweight fine-tune or few-shot adaptation using exactly the same 20% calibration split and the same box prompts used for MedSAMix-S, and evaluate on the same 80% test split. If the average Dice of this calibration-exposed single model reaches or exceeds the 79.64 reported for MedSAMix-S, then the 6.67% improvement is attributable to label access and search rather than to model merging.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that MedSAMix consistently improves domain-specific and multi-task segmentation performance (Abstract; Table 1). The load-bearing condition is that the comparison isolates the effect of model merging. It does not. In Section 4, 80% of each dataset is reserved for testing and the remaining 20% is used as a calibration set for MedSAMix during the merging search; SAM, MedSAM, and MedicoSAM are evaluated zero-shot with no access to that data. Fixed merging baselines in Table 2 (TIES, TA, Linear, SLERP) also receive no calibration or search. Consequently, the reported 6.67% average gain over MedicoSAM conflates three factors: (a) access to labeled calibration samples, (b) automatic per-task configuration search, and (c) the merging operation itself. A conventional few-shot fine-tune or calibration-exposed adaptation of MedSAM could plausibly close or exceed the gap; the paper provides no such control. This is an evaluation-protocol weakness, not an internal inconsistency, but it directly threatens the quantitative headline. The multi-task claim has a related gap: the eight tasks used for Pareto search are included among the 25 evaluation tasks, so generalization to genuinely unseen tasks is not measured.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript introduces MedSAMix, a training-free layer-wise model merging framework for SAM-based medical image segmentation. Given a base SAM model and two fine-tuned variants (MedSAM and MedicoSAM) that share the same architecture, MedSAMix defines a search space of per-layer-group merging operations (TIES, task arithmetic, linear combination, SLERP) and uses SMAC Bayesian optimization with either a single-task or a multi-task (ParEGO) objective to discover merging configurations. The objective is evaluated on a 20% calibration split of each dataset, and the resulting merged model is evaluated on a held-out 80% test split across 25 segmentation tasks. The authors report improvements of 6.67 percentage points over MedicoSAM for single-task merging and 4.37 percentage points for multi-task merging, and claim consistent improvements over all baselines across the 25 tasks.","tokens_in":16218,"tokens_out":4471,"duration_ms":50615,"significance":"If the evaluation protocol is sound, the contribution is significant: the paper demonstrates a practical, gradient-free way to combine existing public SAM checkpoints that can improve both specialized and general medical segmentation performance, and the 25-task benchmark with a genuinely held-out 80% test split is a strength. The search-space formulation is clear, the use of SMAC is appropriate, and the authors provide code and weights for reproducibility. However, the primary quantitative claim currently rests on an asymmetric evaluation in which MedSAMix uses labeled calibration data for configuration search while all baselines, including the fixed merging methods, are evaluated zero-shot. As a result, the magnitude of the improvement attributable specifically to model merging is not yet established, and the multi-task generalization claim is weakened because the tasks used for Pareto search are included among the evaluation tasks.","major_comments":[{"comment":"The comparison is asymmetric. The 20% calibration split is explicitly used to search for MedSAMix's merging configuration, whereas SAM, MedSAM, MedicoSAM, and the fixed merging baselines (TIES, TA, Linear, SLERP) are evaluated zero-shot without access to that data. Consequently, the reported gains—6.67 percentage points over MedicoSAM and the superiority over the fixed merging baselines—conflate the effect of the merging operation with the effect of access to labeled calibration samples and per-task automatic search. To isolate the benefit of the layer-wise search, please add calibration-exposed controls, for example a few-shot fine-tune or prompt-only adaptation of MedSAM on the same 20% split, or a grid-selected fixed merging recipe using the same calibration set. Without such controls, the headline improvement cannot be attributed to model merging itself.","section":"Section 4, 'Evaluations' paragraph, and Table 2"},{"comment":"The multi-task generalization claim is not fully supported because the eight tasks used for the Pareto-front search (optic disk, tumor, vascular, lateral ventricle, mice-lung, cardiac, nasal pharynx, prostate) are a subset of the 25 evaluation tasks. The MedSAMix-M average in Table 1 is therefore computed partly on tasks that were in-sample for configuration selection, so the reported 4.37% gain over the second-best model does not demonstrate generalization to genuinely unseen tasks. Please report results separately on the 17 tasks not used in the search, or use a leave-some-out protocol over search-task subsets, so that the multi-task regime can be evaluated on truly held-out tasks.","section":"Section 4, 'Objective and Optimizer', and Section 5.2"}],"minor_comments":[{"comment":"The numbers '6.67%' and '4.37%' are described as 'improvements' but correspond to absolute percentage-point differences in Dice (79.64 vs. 72.97 and 77.34 vs. 72.97). Please state 'percentage points' or report relative improvements to avoid ambiguity.","section":"Abstract and Table 2"},{"comment":"Table 4 reports three runs of MedSAMix-M with some variation (e.g., 77.14, 77.40, 77.33 average), but Table 1 reports a single run. Please clarify whether Table 1 is one selected run or an average, and consider reporting variance for the main results.","section":"Section 4, 'Evaluations', and Table 1"},{"comment":"The merged architecture is reported only for MedSAMix-M. Since MedSAMix-S is a per-task configuration for each of the 25 tasks, the authors should either show an example or explicitly state that the per-task configurations are provided in the released code.","section":"Table 5"},{"comment":"The sensitivity analysis in Figure 4 does not show error bars or repeated-run variability, even though Section B.1 demonstrates that optimization runs have some variance. Adding error bars would make the claims about insensitivity to granularity and the optimal number of tasks more robust.","section":"Section 5.4, Figure 4"},{"comment":"The statement that MedSAMix works 'without the need for additional data or retraining' should be qualified: the search does use the 20% labeled calibration split. This is distinct from retraining, but it is still additional data access relative to the zero-shot baselines.","section":"Section 6, Discussion"}],"recommendation":"major_revision","confidential_remarks":"The calibration-asymmetry issue is the main barrier to accepting the quantitative claims. I would encourage the authors to treat it as the primary revision burden and to add calibration-exposed baselines; the benchmark itself and the merging formulation are likely to be of interest to the community. No novelty-disclosure concerns beyond what is stated in the paper."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version: the paper does something real and useful—it shows that a layer-wise, automatically searched merge of SAM, MedSAM, and MedicoSAM reliably beats each model alone on 25 medical segmentation tasks, with a clean 80/20 test/calibration split. That is not nothing. The search space is sensible, SMAC with ParEGO is a reasonable choice, and the multi-task run stability in the supplement is a genuine plus. Building on the authors' own automated merging framework is legitimate here; the cited prior work is the direct foundation.\n\nWhere it gets soft is the evaluation protocol. MedSAMix-S searches on 20% labeled calibration data per task; the SAM-based baselines are zero-shot with no access to that data. The fixed merging baselines also get no calibration or search. So the reported 6.67% gain over MedicoSAM is not the marginal value of merging; it conflates label access, per-task configuration search, and the merging operation itself. A few-shot fine-tuned or calibration-exposed MedSAM control would tell you how much of the gain is simply having labels. Without it, the headline margin is not credible as a merging-effect claim. The multi-task claim is also weaker than it looks: the eight tasks used for Pareto search are among the 25 evaluated, so 'generalization' here is interpolation, not extrapolation to unseen tasks. The Limitation section acknowledges only task diversity, not this protocol issue.\n\nMinor: no error bars on single-task MedSAMix-S, and no comparison with other automated merging methods (e.g., evolutionary recipes), so the 'automated beats manual' point is only made against fixed recipes, not against other searchers.\n\nThe basin-compatibility assumption is fine: all models share SAM's architecture and initialization, exactly the regime where weight arithmetic is expected to work, and the paper is honest about excluding adapter-based variants.\n\nBottom line: a solid engineering contribution that deserves referee time, but the quantitative claims need to be re-cast after matched baselines. I'd send it to review with a request for the calibration-exposed controls. If those close the gap, the method is still a cheap, private-friendly way to combine specialists; if they don't, the paper still has value but the abstract needs rewriting.","headline":"A useful, well-defined model-merging recipe for SAM-based medical segmentation, but the headline gains overstate the method because MedSAMix sees labeled calibration data while baselines are zero-shot.","tokens_in":16759,"tokens_out":2505,"would_cite":false,"duration_ms":24417,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"MedSAMix claims that a search over layer-wise merge recipes, using only calibration images, turns generalist SAM plus its fine-tuned medical variants into a segmenter that beats every single model on 25 medical tasks.","keywords":["medical image segmentation","model merging","Segment Anything Model","training-free","Bayesian optimization","layer-wise merging","multi-objective optimization","zero-shot segmentation"],"falsifier":"On a representative set of medical scans, compute the loss barrier along linear interpolation between SAM and MedSAM, and between SAM and MedicoSAM; if the barrier is steep, with loss well above both endpoint models, the shared-basin assumption is false and the reported merged-model Dice gains should not reproduce.","tokens_in":15751,"feed_emoji":"🩺","tokens_out":9166,"duration_ms":90654,"temperature":0.7,"pith_summary":"The paper sets out to show that the best medical image segmenter may be a blend of existing models rather than any single one of them. MedSAMix is a training-free procedure that searches over layer-wise merge recipes, choosing per layer group among task arithmetic, TIES, linear combination, and spherical interpolation, using only a calibration split of the target data as reward. The paper reports that the resulting merged model beats the generalist SAM where fine-tuned medical specialists underperform, and beats the specialists where they are strong, with average Dice improvements of 6.67% on specialized tasks and 4.37% across a 25-task multi-task evaluation. A reader should care because the gains come without retraining, gradient updates, or data sharing, so separately trained clinical models could be combined into one stronger universal segmenter.","feed_headline":"Searching merge recipes beats every single model on 25 medical tasks","feed_subtitle":"Training-free layer-wise merging of SAM with its medical variants gains 6.67% on specialized tasks.","key_machinery":"The carrying mechanism is a layer-wise, searched merge recipe over the SAM architecture. The image encoder, prompt encoder, and mask decoder are partitioned into layer groups (with granularity itself searchable), and for each group the search chooses one of four merging operations—task arithmetic, TIES (trim, elect sign, and merge), linear combination, or spherical linear interpolation (SLERP)—plus method-specific hyperparameters. A Bayesian optimizer with a random-forest surrogate proposes configurations, and the reward is segmentation Dice on a calibration set, either for one task or scalarized across tasks to find a Pareto front. This converts model merging from a manually fixed hyperparameter choice into an automatic, per-task configuration search.","core_discovery":"MedSAMix claims that SAM, MedSAM, and MedicoSAM—all initialized from the same SAM weights—converge into a shared loss basin, so their layer weights can be mixed by arithmetic operations, and that the best mixture is not uniform but varies layer by layer and task by task. The method therefore treats the merge recipe itself as an optimization variable: each layer group can use a different merging operation and different hyperparameters, and a Bayesian search over these recipes is driven by Dice on a calibration set. For a single task, the search finds a recipe that outperforms both the best specialist model and the generalist base. For universal use, a multi-objective version of the same search produces a Pareto-optimal recipe that the paper reports as beating all baselines on 18 of 25 tasks and the best competing universal model by 4.37% average Dice. The authors interpret these results as evidence that fine-tuning bias can be corrected at the model level, without additional data or training.","pith_inferences":["The same searched layer-wise mixing could rescue any SAM-family specialist that underperforms its base on a niche, provided the variant keeps the base architecture and shared initialization.","A testable extension is to run the search on a held-out dataset absent from the 25 tasks, or with corrupted calibration labels; if gains vanish, the method is mostly fitting the calibration distribution rather than discovering transferable skill combinations.","The Pareto front of merge recipes could be used as a per-prediction model-selection mechanism, choosing among several merged checkpoints for different inputs at inference time."],"forward_implications":["A clinic can fine-tune a local specialist, merge it with a generalist foundation model using a small calibration set, and obtain a better segmenter without sharing data or retraining.","For a specific clinical task, the single-task search yields a merged model that outperforms both the best specialist and the generalist base on that task.","For universal segmentation, the multi-task Pareto search yields one merged model that beats the strongest baseline on 18 of 25 tasks and the best universal competitor by 4.37% average Dice.","Fixed merging strategies applied uniformly are inconsistent across tasks, so the automatic layer-wise search is what carries the reported gains.","The procedure is training-free and inference-only, with 120 search trials completed in roughly 70 minutes on two 8GB GPUs in the reported single-task setting."],"supporting_citations":[{"why":"Supplies the base SAM architecture and weights from which all candidate models are fine-tuned.","marker":"Kirillov et al. (2023)"},{"why":"Provides MedSAM, the specialist medical model that MedSAMix merges with SAM.","marker":"Ma et al. (2024a)"},{"why":"Provides MedicoSAM, the second specialist medical model used as a merge candidate.","marker":"Archit et al. (2025)"},{"why":"Supplies the loss-basin transfer result that justifies merging fine-tuned models initialized from the same weights.","marker":"Neyshabur et al. (2020)"},{"why":"Supplies the Bayesian optimization algorithm used to search merging configurations.","marker":"Lindauer et al. (2022)"},{"why":"Supplies the Pareto scalarization method used for the multi-task objective.","marker":"Knowles (2006)"},{"why":"Supplies the random forest surrogate model used inside the optimizer.","marker":"Breiman (2001)"},{"why":"Supplies task arithmetic, one of the merge operations included in the search space.","marker":"Ilharco et al. (2022)"},{"why":"Supplies TIES-Merging, one of the merge operations included in the search space.","marker":"Yadav et al. (2024)"},{"why":"Supplies SLERP, one of the merge operations included in the search space.","marker":"White (2016)"}],"fun_headline_variants":["Training-free merge of SAM and MedSAM gains 6.67%","Layer-wise merge recipes beat specialists on medical tasks","Zero-order search finds best model merge for segmentation","Auto-optimized merging boosts medical segmentation accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that after fine-tuning from the same initial SAM weights, SAM, MedSAM, and MedicoSAM end up in compatible regions of parameter space, so mixing their layer weights produces a working model rather than a broken one; if that premise fails, the search has nothing useful to combine.","fun_headline_variants_meta":{"raw":{"variants":["Training-free merge of SAM and MedSAM gains 6.67%","Layer-wise merge recipes beat specialists on medical tasks","Zero-order search finds best model merge for segmentation","Auto-optimized merging boosts medical segmentation accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001232,"raw_usage":{"total_tokens":5089,"prompt_tokens":1003,"completion_tokens":4086,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":619,"completion_tokens_details":{"reasoning_tokens":4022}},"tokens_in":619,"tokens_out":4086,"duration_ms":30759,"temperature":1.0,"reasoning_tokens":4022,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T17:29:20.634440+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a representative set of medical scans, compute the loss barrier along linear interpolation between SAM and MedSAM, and between SAM and MedicoSAM; if the barrier is steep, with loss well above both endpoint models, the shared-basin assumption is false and the reported merged-model Dice gains should not reproduce.","supporting_citations":[{"cited_title":"What is being transferred in transfer learning? Advances in neural information processing systems, 33: 0 512--523, 2020","cited_arxiv_id":null,"evidence_quote":"Supplies the loss-basin transfer result that justifies merging fine-tuned models initialized from the same weights."},{"cited_title":"Smac3: A versatile bayesian optimization package for hyperparameter optimization","cited_arxiv_id":null,"evidence_quote":"Supplies the Bayesian optimization algorithm used to search merging configurations."}],"review_version":2}