REVIEW 4 major objections 5 minor 25 references
A dual-adaptive SAM3 design matches or beats full fine-tuning and standard MoE for medical segmentation while cutting MoE parameter overhead by over 80 percent.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
Hierarchical multimodal MoE with low-rank expert deltas adapts frozen SAM3 for medical segmentation, reporting ~5-point Dice gains over SAM3 and lower MoE overhead.
T0 review reviewed 2026-07-12 challenge →
load-bearing objection Solid PEFT+MoE engineering on SAM3 for medical CT/MRI with real multi-dataset gains; hierarchy story is narrative and one table is mislabeled, but the dual design is still worth a look. the 4 major comments →
Dual-Adaptive SAM3: Hierarchical Routing over Low-Rank Expert Layers for Parameter-Efficient Medical Image Segmentation
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
Core claim
Dual-Adaptive SAM3 shows that task-aware sparse routing over low-rank expert deltas, applied only to selected feed-forward layers of a frozen SAM3 fusion decoder, is sufficient to specialize the model to diverse medical concepts, matching or exceeding the accuracy of full fine-tuning and dense MoE while reducing MoE parameter overhead by more than 80 percent and lifting Dice scores about five percent above prior medical SAM adaptations.
What carries the argument
The Dual-Adaptive MoE layer: a Dynamic Expert Router that scores experts from the joint visual-token, global-context and concept embedding, plus Decomposed Parameterized Experts that express each expert as the shared frozen SAM3 FFN base plus a low-rank trainable delta (A B^T).
Load-bearing premise
That freezing the image and text encoders plus attention layers, and inserting the adaptive experts only at three fixed decoder depths, is enough for the model to specialize to medical domains without those frozen parts becoming the real performance bottleneck.
What would settle it
Re-run the same four-dataset benchmark with Dual-Adaptive MoE layers placed at other decoder depths or with the image encoder unfrozen; if the Dice and Hausdorff gains disappear or the parameter advantage collapses, the hierarchical dual-adaptation claim fails.
If this is right
- Open-vocabulary medical segmentation can be specialized without full-model fine-tuning or large MoE memory cost.
- Sparse multimodal routing conditioned on both image and text concept becomes a practical drop-in replacement for static adapters in clinical vision-language models.
- Low-rank expert deltas on frozen base weights can keep total trainable parameters small enough for resource-constrained hospital deployment.
- Hierarchical expert placement can be used as a design pattern for other coarse-to-fine medical reasoning tasks.
Where Pith is reading between the lines
- The same DER+DPE pattern could transfer to other multimodal foundation models that already possess frozen encoders and a fusion decoder.
- If the two-stage warm-up then routing-calibration schedule is the true stabilizer, simpler one-stage MoE medical adapters may systematically under-perform.
- Interpretable routing maps produced by the DER could serve as an audit trail for which visual-text cues drove each organ mask.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes Dual-Adaptive SAM3 (DA-SAM3), which adapts the vision-language SAM3 model to medical image segmentation by replacing selected FFN blocks in the fusion decoder with Dual-Adaptive MoE layers. Each layer combines a Dynamic Expert Router (DER) that sparsely gates experts from joint visual-token, domain-context, and concept embeddings (Eqs. 2–3, top-k=2) with Decomposed Parameterized Experts (DPE) that keep a frozen pretrained FFN base W0 and train only low-rank deltas ΔWi=AiBi⊤ (Eqs. 4–6). MoE layers are inserted only at hierarchical depths {L/6, L/4, L/2}, with image/text encoders, MHSA, and MSCA frozen. A two-stage training schedule (expert warm-up then router fine-tuning) uses Dice+Focal plus load-balance and sparsity losses. On Synapse, MMWHS, BTCV, and ACDC, Table 1 reports DSC/HD that match or exceed fully fine-tuned SAM3, standard MoE, and several SAM adaptations, with an abstract claim of ~5% SOTA gain and >80% MoE parameter-overhead reduction. Table 2 ablations attribute gains to DER and DPE under the fixed hierarchy.
Significance. If the dual-adaptive design is the true source of the gains and the efficiency numbers hold, the work offers a practical PEFT path for open-vocabulary medical segmentation that is more parameter-efficient than dense MoE and more task-conditioned than static LoRA. The combination of multimodal sparse routing with frozen-base low-rank experts is a clear, reusable recipe, and the public-split results plus promised code are useful for the community. The hierarchical clinical-reasoning narrative is an attractive framing even if currently under-validated. Strengths that should be credited: competitive multi-dataset numbers against strong baselines (Table 1), explicit DER/DPE ablations (Table 2), and an open code link.
major comments (4)
- [Abstract; Table 1; §3.3] Abstract and §3.3 claim a “notable 5% gain over current state-of-the-art methods.” Table 1 shows DA-SAM3 Synapse DSC 85.12 vs MoE-SAM 84.71 (+0.41) and vs Swin-UMamba 83.48 (+1.64); the ~4.4-point jump is mainly vs the SAM3 baseline (80.75). The 5% SOTA claim is therefore overstated relative to the strongest published comparators and should be restated as absolute/relative gains against named baselines, not a blanket SOTA percentage.
- [Table 2; §3.4] Table 2 is labeled “Synapse CT, MMWHS, ACDC,” yet the third-column numbers for SAM3 (DSC 72.24 / HD 8.080) and full DA-SAM3 (77.35 / 5.587) exactly match Table 1’s BTCV column, not ACDC (91.93 / 1.064 and 85.03 / 5.280). This is a load-bearing reporting error: either the ablation was run on BTCV and mislabeled, or ACDC numbers were overwritten. The table must be corrected and the corresponding text in §3.4 revised so that ablation conclusions rest on the intended datasets.
- [§2.1–2.2; Table 2] The central efficiency–accuracy story rests on placing Dual-Adaptive MoE only at depths {L/6, L/4, L/2} and freezing the image encoder, text encoder, MHSA, and MSCA (§2.1–2.2), with the stages narrated as “coarse global alignment / semantic identification / boundary refinement.” No ablation varies the depth set, compares full-decoder MoE, or unfreezes attention; no routing-activation or feature-map analysis shows stage-specific specialization. Table 2’s DER/DPE ablations keep the same fixed hierarchy, so they do not establish that the hierarchical placement (vs any PEFT on the fusion module) drives the gains. A minimal depth-set or full-vs-partial MoE ablation is needed to support attribution and the clinical-reasoning claim.
- [Abstract; §1; §2.3] The claim of “reducing MoE parameter overhead by over 80%” (Abstract; §1) is never quantified with a parameter table (trainable params, total params, or FLOPs vs full fine-tune, standard MoE, and LoRA). Without those numbers, the dual-adaptive efficiency contribution cannot be verified. Please add an explicit parameter/compute comparison for the configurations used in Tables 1–2.
minor comments (5)
- [§3.2; §2.3] §3.2 states “the top-k value to half of the total feature count,” while §2.3 and the MoE setup use k=2 with 4 experts. Clarify the intended top-k rule.
- [§2.3 Eq. (3)] Eq. (3) concatenates [hj_l; h_ctx; StopGrad(ec)] but does not specify dimensions or how Wr is shaped when token, context, and concept dims differ; a short dimension note would help reproducibility.
- [Fig. 2; §3.3] Fig. 2 is described as showing precise organ alignment vs MoE-SAM/SAM3 misclassifications, but the caption and body give no dataset/case IDs or failure-mode analysis; adding identifiers would strengthen the qualitative claim.
- [Throughout] Minor typos and spacing: “AsshowninFig.1”, “WeintroducetheDynamicExpertRouter”, “over80%”, “notable5%gain”, “Lseg =L Dice +L Focal Loss” — consistent spacing and math formatting would improve readability.
- [§1; §2.3] Related work cites DeRS [9] as inspiration for DPE; a one-sentence contrast (what is new beyond upcycling pretrained FFN bases into low-rank experts) would clarify novelty relative to that concurrent line.
Circularity Check
No circularity: empirical architecture paper whose Dice/HD claims are external public-split measurements, not algebraic restatements of its design choices or losses.
full rationale
DA-SAM3 is a standard PEFT/MoE engineering paper. The Dual-Adaptive MoE construction (DER routing scores in Eq. 3, DPE low-rank deltas in Eqs. 4–5, hierarchical placement at {L/6, L/4, L/2}, two-stage training with Dice+Focal + load-balance + sparsity losses) defines a trainable model; the reported DSC/HD numbers in Tables 1–2 are ordinary held-out evaluations on public Synapse/MMWHS/BTCV/ACDC splits. Nothing in the derivation chain equates a fitted hyperparameter or routing weight to the evaluation metric by construction. The “coarse-to-fine clinical reasoning” narrative and the fixed depth set are unvalidated design choices (as the skeptic notes), but they are not circular: they do not redefine the target quantities. Self-citations are ordinary (DeRS inspiration, SAM3 base) and not load-bearing uniqueness theorems. Score 0 is therefore the correct, proportionate finding.
Axiom & Free-Parameter Ledger
free parameters (6)
- number of experts N_e
- top-k experts per token
- low-rank dimension r of ΔW_i = A_i B_i^T
- load-balance weight λ1 and sparsity weight λ2
- MoE insertion depths {L/6, L/4, L/2}
- learning rate and related training knobs
axioms (5)
- domain assumption Freezing SAM3 image encoder, text encoder, MHSA, and MSCA while only training Dual-Adaptive MoE layers and LN preserves useful pretrained knowledge and is sufficient for medical domain adaptation.
- domain assumption Joint routing on token features, pooled visual context, and concept embedding (with StopGrad on e_c) selects clinically relevant experts better than random or non-multimodal routing.
- domain assumption Representing each expert as frozen W0 plus low-rank delta (DeRS-style) retains enough expressivity for medical morphological variation.
- domain assumption Official public splits of Synapse, MMWHS, BTCV, and ACDC are adequate and comparable benchmarks for the stated SOTA comparisons.
- standard math Standard linear algebra and Transformer residual computation (GELU FFN form, top-k gating) behave as usual.
invented entities (3)
-
Dynamic Expert Router (DER)
no independent evidence
-
Decomposed Parameterized Experts (DPE)
no independent evidence
-
Dual-Adaptive MoE Layer / DA-SAM3 hierarchical specialization stages
no independent evidence
Cite this review
Pith. "Pith review of Dual-Adaptive SAM3: Hierarchical Routing over Low-Rank Expert Layers for Parameter-Efficient Medical Image Segmentation." pith.science (2026). https://pith.science/paper/JTEWTUAE
@misc{pith2026260702571,
author = {Pith},
title = {Pith review of: Dual-Adaptive SAM3: Hierarchical Routing over Low-Rank Expert Layers for Parameter-Efficient Medical Image Segmentation},
year = {2026},
howpublished = {\url{https://pith.science/paper/JTEWTUAE}},
note = {Machine review of arXiv:2607.02571}
}
read the original abstract
The Segment Anything Model with Concepts (SAM3) heralds a new paradigm for open-vocabulary segmentation through natural language interaction, offering significant potential for medical image analysis. However, effectively adapting such a powerful vision-language model to the diverse and nuanced domain of medical imaging remains a key challenge. Naive fine-tuning is parameter-inefficient, while standard Mixture-of-Experts (MoE) methods introduce prohibitive computational overhead, limiting their clinical applicability. To address this, we propose Dual-Adaptive SAM3 (DA-SAM3), a novel framework that achieves both high segmentation accuracy and extreme parameter efficiency via a dual-adaptive specialization mechanism. Our first adaptation is task-aware: a Dynamic Expert Router (DER) that sparsely activates the most relevant experts by jointly reasoning about the visual input and the textual concept prompt, mimicking a clinical consultation process. Our second adaptation is parameter-aware: a Decomposed Parameterized Experts (DPE) design that represents each expert as a shared frozen base (inherited from the pretrained SAM3) and a lightweight trainable low-rank delta, reducing MoE parameter overhead by over 80\%. Extensive experiments on multiple public medical segmentation benchmarks demonstrate that Dual-Adaptive SAM3 not only matches or exceeds the accuracy of fully fine-tuned SAM3 and standard MoE baselines, but also achieves a notable 5\% gain over current state-of-the-art methods, with interpretable results validating its effectiveness. The code is available at: https://github.com/Reconsider80/DA-SAM3.
Figures
Reference graph
Works this paper leans on
-
[1]
Deep learning techniques for automatic mri cardiac multi- structures segmentation and diagnosis: is the problem solved?IEEE transactions on medical imaging, 37(11):2514–2525, 2018
Olivier Bernard, Alain Lalande, Clement Zotti, Frederick Cervenansky, Xin Yang, Pheng-Ann Heng, Irem Cetin, Karim Lekadir, Oscar Camara, Miguel Angel Gon- zalez Ballester, et al. Deep learning techniques for automatic mri cardiac multi- structures segmentation and diagnosis: is the problem solved?IEEE transactions on medical imaging, 37(11):2514–2525, 2018
2018
-
[2]
Sam 3: Segment anything with concepts.arXiv preprint arXiv:2511.16719, 2025
Nicolas Carion, Laura Gustafson, Yuan-Ting Hu, Shoubhik Debnath, Ronghang Hu, Didac Suris, Chaitanya Ryali, Kalyan Vasudev Alwala, Haitham Khedr, An- drew Huang, et al. Sam 3: Segment anything with concepts.arXiv preprint arXiv:2511.16719, 2025
Pith/arXiv arXiv 2025
-
[3]
JienengChen,YongyiLu,QihangYu,XiangdeLuo,EhsanAdeli,YanWang,LeLu, Alan L Yuille, and Yuyin Zhou. Transunet: Transformers make strong encoders for medical image segmentation.arXiv preprint arXiv:2102.04306, 2021
Pith/arXiv arXiv 2021
-
[4]
Unleashing the potential of sam for medical adaptation via hierarchical decoding
Zhiheng Cheng, Qingyue Wei, Hongru Zhu, Yan Wang, Liangqiong Qu, Wei Shao, and Yuyin Zhou. Unleashing the potential of sam for medical adaptation via hierarchical decoding. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3511–3522, 2024
2024
-
[5]
Multi-organ segmentation over partially labeled datasets with multi-scale feature abstraction.IEEE Transactions on Medical Imag- ing, 39(11):3619–3629, 2020
Xi Fang and Pingkun Yan. Multi-organ segmentation over partially labeled datasets with multi-scale feature abstraction.IEEE Transactions on Medical Imag- ing, 39(11):3619–3629, 2020
2020
-
[6]
Desam: Decoupled segment anything model for generalizable medical image segmentation
Yifan Gao, Wei Xia, Dingdu Hu, Wenkui Wang, and Xin Gao. Desam: Decoupled segment anything model for generalizable medical image segmentation. InInter- national Conference on Medical Image Computing and Computer-Assisted Inter- vention, pages 509–519. Springer, 2024
2024
-
[7]
Swin unetr: Swin transformers for semantic segmentation of brain tumors in mri images
Ali Hatamizadeh, Vishwesh Nath, Yucheng Tang, Dong Yang, Holger R Roth, and Daguang Xu. Swin unetr: Swin transformers for semantic segmentation of brain tumors in mri images. InInternational MICCAI brainlesion workshop, pages 272–
-
[8]
Lora: Low-rank adaptation of large language models.ICLR, 1(2):3, 2022
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models.ICLR, 1(2):3, 2022
2022
-
[9]
Ders: Towards extremely efficient upcycled mixture-of-experts models
Yongqi Huang, Peng Ye, Chenyu Huang, Jianjian Cao, Lin Zhang, Baopu Li, Gang Yu, and Tao Chen. Ders: Towards extremely efficient upcycled mixture-of-experts models. InProceedings of the Computer Vision and Pattern Recognition Confer- ence, pages 10056–10066, 2025
2025
-
[10]
A-eval: A benchmark for cross-dataset and cross-modality evaluation of abdominal multi-organ segmenta- tion.Medical Image Analysis, 101:103499, 2025
Ziyan Huang, Zhongying Deng, Jin Ye, Haoyu Wang, Yanzhou Su, Tianbin Li, Hui Sun, Junlong Cheng, Jianpin Chen, Junjun He, et al. A-eval: A benchmark for cross-dataset and cross-modality evaluation of abdominal multi-organ segmenta- tion.Medical Image Analysis, 101:103499, 2025
2025
-
[11]
nnu-net: a self-configuring method for deep learning-based biomedical image segmentation.Nature methods, 18(2):203–211, 2021
Fabian Isensee, Paul F Jaeger, Simon AA Kohl, Jens Petersen, and Klaus H Maier- Hein. nnu-net: a self-configuring method for deep learning-based biomedical image segmentation.Nature methods, 18(2):203–211, 2021
2021
-
[12]
Segment anything
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. InProceedings of the IEEE/CVF international conference on computer vision, pages 4015–4026, 2023
2023
-
[13]
Miccaimulti-atlaslabelingbeyondthecranialvault–workshopand challenge
Bennett Landman, Zhoubing Xu, Juan Igelsias, Martin Styner, Thomas Langerak, andArnoKlein. Miccaimulti-atlaslabelingbeyondthecranialvault–workshopand challenge. InProc. MICCAI multi-atlas labeling beyond cranial vault—workshop challenge, volume 5, page 12. Munich, Germany, 2015. 10 Y. Chen et al
2015
-
[14]
Moe-sam: Enhancing sam for medical image segmentation with mixture-of-experts
Ruocheng Li, Lei Wu, Jingjun Gu, Qi Xu, Wanyi Chen, Xiaoxu Cai, and Jiajun Bu. Moe-sam: Enhancing sam for medical image segmentation with mixture-of-experts. InInternational Conference on Medical Image Computing and Computer-Assisted Intervention, pages 367–377. Springer, 2025
2025
-
[15]
Beyond adapting sam: Towards end-to-end ultrasound image segmentation via auto prompting
Xian Lin, Yangyang Xiang, Li Yu, and Zengqiang Yan. Beyond adapting sam: Towards end-to-end ultrasound image segmentation via auto prompting. InInter- national Conference on Medical Image Computing and Computer-Assisted Inter- vention, pages 24–34. Springer, 2024
2024
-
[16]
Swin-umamba: Mamba-based unet with imagenet-based pretraining
Jiarun Liu, Hao Yang, Hong-Yu Zhou, Yan Xi, Lequan Yu, Cheng Li, Yong Liang, Guangming Shi, Yizhou Yu, Shaoting Zhang, et al. Swin-umamba: Mamba-based unet with imagenet-based pretraining. InInternational conference on medical im- age computing and computer-assisted intervention, pages 615–625. Springer, 2024
2024
-
[17]
Segment anything in medical images.Nature Communications, 15(1):654, 2024
Jun Ma, Yuting He, Feifei Li, Lin Han, Chenyu You, and Bo Wang. Segment anything in medical images.Nature Communications, 15(1):654, 2024
2024
-
[18]
Mednext: transformer-driven scaling of convnets for medical image segmentation
Saikat Roy, Gregor Koehler, Constantin Ulrich, Michael Baumgartner, Jens Pe- tersen, Fabian Isensee, Paul F Jaeger, and Klaus H Maier-Hein. Mednext: transformer-driven scaling of convnets for medical image segmentation. InIn- ternational Conference on Medical Image Computing and Computer-Assisted In- tervention, pages 405–415. Springer, 2023
2023
-
[19]
Tal Shaharabany, Aviad Dahan, Raja Giryes, and Lior Wolf. Autosam: Adapt- ing sam to medical images by overloading the prompt encoder.arXiv preprint arXiv:2306.06370, 2023
Pith/arXiv arXiv 2023
-
[20]
Sictta: Single image continual test time adaptation for medical image segmentation.Medical Image Analysis, 108:103859, 2026
Jianghao Wu, Xinya Liu, Guotai Wang, and Shaoting Zhang. Sictta: Single image continual test time adaptation for medical image segmentation.Medical Image Analysis, 108:103859, 2026
2026
-
[21]
Medical sam adapter: Adapting segment anything model for medical image segmentation.Medical image analysis, 102:103547, 2025
Junde Wu, Ziyue Wang, Mingxuan Hong, Wei Ji, Huazhu Fu, Yanwu Xu, Min Xu, and Yueming Jin. Medical sam adapter: Adapting segment anything model for medical image segmentation.Medical image analysis, 102:103547, 2025
2025
-
[22]
Accel- erating vision foundation model for efficient medical image segmentation.Medical Physics, 53(1):e70193, 2026
Xian-Tao Wu, Xiao-Diao Chen, Wen Wu, Weiyin Ma, and Haichuan Song. Accel- erating vision foundation model for efficient medical image segmentation.Medical Physics, 53(1):e70193, 2026
2026
-
[23]
Jin Ye, Junlong Cheng, Jianpin Chen, Zhongying Deng, Tianbin Li, Haoyu Wang, Yanzhou Su, Ziyan Huang, Jilong Chen, Lei Jiang, et al. Sa-med2d-20m dataset: Segment anything in 2d medical imaging with 20 million masks.arXiv preprint arXiv:2311.11969, 2023
Pith/arXiv arXiv 2023
-
[24]
Kaidong Zhang and Dong Liu. Customized segment anything model for medical image segmentation.arXiv preprint arXiv:2304.13785, 2023
Pith/arXiv arXiv 2023
-
[25]
Multi-scale patch and multi-modality atlases for whole heart segmentation of mri.Medical image analysis, 31:77–87, 2016
Xiahai Zhuang and Juan Shen. Multi-scale patch and multi-modality atlases for whole heart segmentation of mri.Medical image analysis, 31:77–87, 2016
2016
This paper was first reviewed by grok-4.5 on July 12, 2026.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.