Pith. sign in

REVIEW 3 major objections 4 minor 1 cited by

SEG-SAM: Semantic-Guided SAM for Unified Medical Image Segmentation

T0 review · 3 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read SEG-SAM claims that adding a semantic-aware decoder, LLM text knowledge, and cross-mask alignment lets one SAM-based model outperform prior SAM-based methods in binary medical segmentation and task-specific methods in semantic medical…

desk verdict Binary results are credible; the semantic headline isn't supported yet due to an unfair baseline protocol and a missing MaskSAM comparison—but the architecture deserves a serious referee. read the letter →

arxiv 2412.12660 v1 pith:R2DJGGLI submitted 2024-12-17 cs.CV

Shuangping Huang , Hao Liang , Qingfeng Wang , Chulong Zhong , Zijian Zhou , Miaojing Shi This is my paper · ORCID
classification cs.CV
keywords medicalimagesegmentationSegmentAnythingModelsemantic-awaredecodertext-to-visionsemanticenhancementcross-maskspatialalignmentunifiedMed2D-16Mpromptable
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

SEG-SAM is a unified medical image segmentation model built on SAM that tries to show one promptable model can produce both accurate binary masks and correct semantic labels without task-specific retraining. It adds a separate semantic-aware decoder to SAM's original mask decoder, feeds it category knowledge extracted from a large language model, and enforces spatial agreement between the two decoders' outputs. On Med2D-16M across CT, MR, PET, and other modalities, the paper reports that this design outperforms prior SAM-based methods on binary segmentation (average DICE 73.15 with points, 81.46 with boxes) and also outperforms task-specific methods on semantic segmentation (average mDSC 75.28 and 80.90). The value of the claim, if right, is that a single model with one prompt can serve diverse clinical segmentation tasks while giving clinicians both the object boundary and its category.

What carries the argument

The load-bearing mechanism is the semantic-aware decoder (SAWD), a parallel attention-based decoder that consumes a segmentation-oriented token plus prompt tokens and outputs a semantic mask, plus classification-oriented tokens that predict categories for unprompted objects. The text-to-vision semantic enhancement (T2VSE) module encodes LLM-generated category descriptions and folds them into the prompt tokens via a text summary token and cross-attention. A cross-mask spatial alignment loss, written as $1 - \text{IoU}(\hat{M}_s, \hat{M}_b)$, supervises consistency between the semantic and binary masks. The SAWD is what lets the model keep SAM's original binary decoder while adding semantic learning without conflicting gradient signals.

What would settle it

Evaluate the method on a dataset constructed to contain spatially overlapping masks of different categories (for example, a tumor region labeled inside a liver region) and compare the semantic decoder's per-category accuracy and the cross-mask alignment loss against a dataset of disjoint categories; if accuracy collapses on overlaps, the unique-category assumption behind $M_s=[C_s, M_b]$ is the breaking point.

Watch

Extended reading notes

Core claim

The central claim is that semantic supervision, added through a decoder that is architecturally separate from SAM's semantic-agnostic decoder, resolves the binary-versus-semantic conflict and improves both tasks. For a prompted object, a segmentation-oriented token decodes a semantic mask, while classification-oriented tokens predict categories of unprompted objects to supply context. LLM-generated text descriptions of each category are summarized into a token and appended to the visual prompt tokens, and a cross-mask spatial alignment loss pulls the semantic and binary masks toward each other. The paper reports that this combination beats both SAM-based unified models and conventional task-specific segmenters on the Med2D-16M benchmark and generalizes to unseen KiTS23, BTCV, and AMOS datasets.

Load-bearing premise

The whole semantic-supervision pipeline rests on the assumption that every binary mask in every training dataset carries exactly one unambiguous category label, so that semantic ground truth can be assembled as the category combined with the binary mask.

Editorial extensions

If this is right

  • With one prompt, SEG-SAM can return both a binary mask and a semantic label, so a single deployed model could replace several task-specific segmenters.
  • The reported gains on binary segmentation imply that semantic context, including unprompted-object categories, is a usable signal for sharpening prompted-object boundaries, not just for labeling.
  • Because the semantic-aware decoder is a plug-in, the paper's ablation suggests other SAM-based medical segmenters can be upgraded to semantic prediction without full retraining from scratch.
  • The cross-dataset results imply that semantic knowledge learned from Med2D-16M transfers to new organs and modalities that were not in the training set.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable consequence the paper does not run: if a dataset contains overlapping masks of different categories (for example, a tumor region labeled inside a liver region), the assumed one-category-per-mask construction would give the semantic decoder contradictory labels; measuring per-category mDSC on such overlaps would show where the method's ceiling is.
  • The LLM descriptions are generated once from a fixed template. A cheap sensitivity experiment would replace them with category names only, or with descriptions from a different model, to see how much of the gain is genuinely linguistic versus just an extra token.
  • The cross-mask alignment loss forces spatial equality between semantic and binary masks; for inherently ambiguous boundaries this could bias one decoder. A soft alignment or uncertainty-weighted variant is a natural extension to test.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes SEG-SAM, a unified medical image segmentation model built on SAM. It adds a semantic-aware decoder (with a segmentation-oriented token for the prompted object and classification-oriented tokens for unprompted objects), a text-to-vision semantic enhancement module that injects LLM-generated medical category descriptions into the prompt tokens, and a cross-mask spatial alignment loss that encourages agreement between the semantic and binary decoders. Experiments on Med2D-16M report improved binary segmentation over SAM-based methods and improved semantic segmentation over four conventional architectures, with additional cross-dataset tests on KiTS23, BTCV, and AMOS.

Significance. If the semantic segmentation claim is substantiated, SEG-SAM would be a useful step toward a single promptable model that produces both binary and semantically labeled masks across modalities. The architectural idea is plausible and the ablations (Table 4 and Table 8) indicate that each proposed component contributes. The binary comparison in Table 1 is reasonably controlled, using the same prompt protocol for all SAM-based methods, and the plug-in experiments in Table 5 are informative. However, the headline semantic superiority over task-specific methods currently rests on an unsuitable evaluation protocol and on the omission of the closest prior semantic SAM baseline, so the paper's central claim is not yet supported as written.

major comments (3)
  1. [Sec. 4.2, Table 2] The semantic comparison trains nnUNet, UNETR, nnFormer, and U-Mamba as 2D-slice models on a merged 200-class dataset, a setting that strips these 3D architectures of their native inductive biases (e.g., nnUNet's self-configuring pipeline and volumetric context). The resulting margin (75.28 vs 50.57 average mDSC against U-Mamba) is therefore not credible evidence of superiority over task-specific methods. Please evaluate these baselines in their native 3D per-task setting (or at least per-dataset 2D models such as U-Net trained separately on each dataset) and report those results. The abstract's phrase "task-specific methods" also does not match Table 2, where all methods are trained on the same multi-dataset merge.
  2. [Sec. 2 and Sec. 4.2] MaskSAM (Ref. [35]), the only cited prior SAM-based semantic segmentation method, is discussed in Related Work but is absent from all experiments. Since the paper claims state-of-the-art semantic medical segmentation, the most relevant baseline must be included with the same training and prompting protocol, or the exclusion must be explicitly justified. The SAM-based semantic baselines that are included (Med2D*, MedSAM*, Med-SA*) are all created by inserting the proposed semantic-aware decoder into other methods, so they do not measure SEG-SAM against existing semantic SAM approaches.
  3. [Sec. 3.2.1, Eq. (10)] The semantic ground truth is defined as Ms = [Cs, Mb] under the assumption that every binary mask in every merged dataset carries exactly one unambiguous category. The paper does not analyze how many of the ~200 categories and ~16M masks violate this, for example masks that span multiple organs or cases where organ and lesion masks overlap spatially. Under such conditions, the semantic decoder receives contradictory labels and the cross-mask spatial alignment loss becomes ill-posed. Please report statistics on label ambiguity in Med2D-16M and, if ambiguous cases exist, describe how the supervision and the consistency loss handle them.
minor comments (4)
  1. [Eq. (9)] The equation concatenates prompt tokens tp into the text cross-attention query and then appends MLP(t'_text) to tp to form t'_p. This appears to duplicate prompt information; please clarify the exact role of tp inside the cross-attention and whether the final t'_p contains the original tp unchanged.
  2. [Eq. (10)] The IoU loss is written for masks Ms and Mb, but the paper does not specify how these continuous probability maps are binarized before computing the intersection and union. Please state the thresholding or soft-IoU variant used.
  3. [Sec. 3.3] The training procedure says 'randomly select 5 foreground objects in each image'; if an image contains fewer than five foreground masks, the paper does not state how sampling is handled. Please clarify.
  4. [Introduction and Sec. 6.2] The intro claims 'we are the first to introduce semantic learning into SAM,' which is contradicted by the cited MaskSAM work and by other promptable semantic segmentation models; please soften this claim. Also, the supplementary template feeds a highlighted example image to the LLM, so the generated text may describe that example rather than class-general characteristics; please state whether the same description is used for the entire class and discuss the effect on cross-dataset generalization.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the semantic decoder, LLM text module, and cross-mask alignment are all supervised or conditioned on external labels and fixed text inputs, with no equation reducing a prediction to a fitted parameter or to the paper's own output.

full rationale

The derivation chain is self-contained and externally anchored. The semantic ground truth Ms = [Cs, Mb] (Sec. 3.2.1) is constructed from dataset annotations, where each binary mask carries a unique category label; the semantic decoder is trained on this label-derived target, not on its own output. The text-to-vision module (Sec. 3.2.2, Eq. 9) consumes ChatGPT-generated descriptions as fixed embeddings; the only provenance concern is that the supplementary template (Sec. 6.2) provides a highlighted example image as visual reference, which makes the text dataset-derived rather than purely external, but this does not make the predicted mask equivalent to the text or to any fitted value. The cross-mask alignment loss (Eq. 10) encourages IoU agreement between two decoder outputs, both of which are supervised by the same ground-truth masks; this is a consistency regularizer, not a pseudo-label loop. The paper's self-citation [44] is a related-work pointer on text-promptable segmentation and is not load-bearing for any claimed result. Concerns about baseline fairness (e.g., retraining 3D-native nnUNet/U-Mamba as unified 2D models and omitting MaskSAM) are experimental-protocol issues, not circularity. No equation in the paper reduces a prediction to its input by construction, so the circularity score is 0.

Assumptions & free parameters 3 free parameters · 4 assumptions · 3 invented entities

SEG-SAM's central result rests on three unpaid premises: SAM's frozen visual backbone transfers to medical images, merged medical masks have unique category labels, and LLM-generated text descriptions are accurate enough to improve segmentation. The paper provides ablations supporting each component internally, but none of these premises is validated by external evidence beyond the reported benchmarks.

free parameters (3)
  • Loss weight lambda for L_bce = 10
    Set empirically in Sec. 4.1 ('We empirically set λ = 10'); controls the contribution of unprompted-object classification and affects both DSC and mDSC.
  • Number of classification-oriented tokens N = not stated (likely class count)
    N CO-tokens are initialized for unprompted-object classification in Sec. 3.2.1; the exact value is unspecified, making reproduction dependent on guessing the class inventory.
  • Training subset fraction = 1/4
    Sec. 4.1: only a quarter of the Med2D-16M training set is used for all methods due to compute limits; relative rankings could change on the full training set.
assumptions (4)
  • domain assumption SAM's pre-trained image encoder and prompt encoder transfer to medical images when fine-tuned on a subset of Med2D-16M.
    All experiments build on SAM components as described in Sec. 3.1-3.2; if the natural-image features do not transfer, the improvements vanish.
  • domain assumption Every binary mask in the merged dataset is associated with exactly one semantic category, so Ms=[Cs,Mb] is well-defined.
    Invoked in Sec. 3.2.1 to construct semantic ground truth; the paper does not analyze label overlap or ambiguity across roughly 200 categories.
  • domain assumption LLM-generated text descriptions contain accurate, transferable shape/texture/position knowledge for all medical categories.
    Sec. 3.2.2 uses GPT-4 descriptions as the sole source of text knowledge; only three examples are shown in the supplementary.
  • domain assumption The two decoders should produce identical masks for the prompted object, so enforcing IoU=1 in the alignment loss is correct.
    Eq. 10 assumes Decsaw and Decsag segment the same spatial region despite different semantic and background focuses; boundary disagreements are treated as errors.
invented entities (3)
  • Semantic-oriented token (SO-Token)
    purpose: Learned token in the semantic-aware decoder that decodes the semantic mask of the prompted object.
    No external validation; its contribution is shown only in the paper's ablation tables.
  • Classification-oriented tokens (CO-Tokens)
    purpose: Learned tokens that classify unprompted objects and feed category priors into the SO-Token.
    No external falsifiable handle; effectiveness is measured only in SEG-SAM's own experiments.
  • Text summary token
    purpose: Aggregates LLM text embeddings and injects them into the prompt tokens via T2VSE for the binary decoder.
    No external evidence that this token's summaries transfer beyond the reported benchmarks.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SEG-SAM: Semantic-Guided SAM for Unified Medical Image Segmentation." pith.science (2026). https://pith.science/paper/R2DJGGLI

@misc{pith2026241212660,
  author       = {Pith},
  title        = {Pith review of: SEG-SAM: Semantic-Guided SAM for Unified Medical Image Segmentation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/R2DJGGLI}},
  note         = {Machine review of arXiv:2412.12660}
}
read the original abstract

Recently, developing unified medical image segmentation models gains increasing attention, especially with the advent of the Segment Anything Model (SAM). SAM has shown promising binary segmentation performance in natural domains, however, transferring it to the medical domain remains challenging, as medical images often possess substantial inter-category overlaps. To address this, we propose the SEmantic-Guided SAM (SEG-SAM), a unified medical segmentation model that incorporates semantic medical knowledge to enhance medical segmentation performance. First, to avoid the potential conflict between binary and semantic predictions, we introduce a semantic-aware decoder independent of SAM's original decoder, specialized for both semantic segmentation on the prompted object and classification on unprompted objects in images. To further enhance the model's semantic understanding, we solicit key characteristics of medical categories from large language models and incorporate them into SEG-SAM through a text-to-vision semantic module, adaptively transferring the language information into the visual segmentation task. In the end, we introduce the cross-mask spatial alignment strategy to encourage greater overlap between the predicted masks from SEG-SAM's two decoders, thereby benefiting both predictions. Extensive experiments demonstrate that SEG-SAM outperforms state-of-the-art SAM-based methods in unified binary medical segmentation and task-specific methods in semantic medical segmentation, showcasing promising results and potential for broader medical applications.

Figures

Figures reproduced from arXiv: 2412.12660 by the authors.

Figure 1
Figure 1. The comparison of our SEG-SAM with task-specific [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of the SEG-SAM framework. Give an image [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Qualitative comparisons with other methods on 8 modalities. We compared with U-Mambda [ [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. MedSeg-R: Medical Image Segmentation with Clinical Reasoning

    cs.CV 2025-06 conditional novelty 6.0 of 10

    MedSeg-R injects structured location, texture, and shape priors into a frozen SAM backbone, improving Dice scores on small and overlapping medical structures across multiple modalities.

Reference graph

Works this paper leans on

48 extracted references · 27 canonical work pages · cited by 1 Pith paper

  1. [35]

    Masksam: Towards auto-prompt sam with mask classifi- cation for medical image segmentation

    Bin Xie, Hao Tang, Bin Duan, Dawen Cai, and Yan Yan. Masksam: Towards auto-prompt sam with mask classifi- cation for medical image segmentation. arXiv preprint arXiv:2403.14103, 2024. 3

    arXiv 2024

  2. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ah- mad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 ,

    arXiv

  3. [2]

    Medical image segmentation review: The suc- cess of u-net

    Reza Azad, Ehsan Khodapanah Aghdam, Amelie Rauland, Yiwei Jia, Atlas Haddadi Avval, Afshin Bozorgpour, Sanaz Karimijafarbigloo, Joseph Paul Cohen, Ehsan Adeli, and Dorit Merhof. Medical image segmentation review: The suc- cess of u-net. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024. 2

    work page 2024

  4. [3]

    Uni- verseg: Universal medical image segmentation

    Victor Ion Butoi, Jose Javier Gonzalez Ortiz, Tianyu Ma, Mert R Sabuncu, John Guttag, and Adrian V Dalca. Uni- verseg: Universal medical image segmentation. In Proceed- ings of the IEEE/CVF International Conference on Com- puter Vision, pages 21438–21451, 2023. 1

    work page 2023

  5. [4]

    Uni- verseg: Universal medical image segmentation

    Victor Ion Butoi, Jose Javier Gonzalez Ortiz, Tianyu Ma, Mert R Sabuncu, John Guttag, and Adrian V Dalca. Uni- verseg: Universal medical image segmentation. In Proceed- ings of the IEEE/CVF International Conference on Com- puter Vision, pages 21438–21451, 2023. 2

    work page 2023

  6. [5]

    Swin-unet: Unet-like pure transformer for medical image segmentation

    Hu Cao, Yueyue Wang, Joy Chen, Dongsheng Jiang, Xi- aopeng Zhang, Qi Tian, and Manning Wang. Swin-unet: Unet-like pure transformer for medical image segmentation. In European conference on computer vision, pages 205–218. Springer, 2022. 2

    work page 2022

  7. [6]

    Seg- ment anything in 3d with nerfs

    Jiazhong Cen, Zanwei Zhou, Jiemin Fang, Wei Shen, Lingxi Xie, Dongsheng Jiang, Xiaopeng Zhang, Qi Tian, et al. Seg- ment anything in 3d with nerfs. Advances in Neural Infor- mation Processing Systems, 36:25971–25990, 2023. 2

    2023

  8. [7]

    Transunet: Rethinking the u-net architec- ture design for medical image segmentation through the lens of transformers

    Jieneng Chen, Jieru Mei, Xianhang Li, Yongyi Lu, Qihang Yu, Qingyue Wei, Xiangde Luo, Yutong Xie, Ehsan Adeli, Yan Wang, et al. Transunet: Rethinking the u-net architec- ture design for medical image segmentation through the lens of transformers. Medical Image Analysis, 97:103280, 2024. 2

    2024

Show all 48 references
  1. [8]

    Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolu- tion, and fully connected crfs

    Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolu- tion, and fully connected crfs. IEEE transactions on pattern analysis and machine intelligence, 40(4):834–...

    2017

  2. [9]

    Adaptformer: Adapting vision transformers for scalable visual recogni- tion

    Shoufa Chen, Chongjian Ge, Zhan Tong, Jiangliu Wang, Yibing Song, Jue Wang, and Ping Luo. Adaptformer: Adapting vision transformers for scalable visual recogni- tion. Advances in Neural Information Processing Systems , 35:16664–16678, 2022. 1

    2022

  3. [10]

    Recent advances and clin- ical applications of deep learning in medical image analysis

    Xuxin Chen, Ximin Wang, Ke Zhang, Kar-Ming Fung, Theresa C Thai, Kathleen Moore, Robert S Mannel, Hong Liu, Bin Zheng, and Yuchen Qiu. Recent advances and clin- ical applications of deep learning in medical image analysis. Medical image analysis, 79:102444, 2022. 1

    2022

  4. [11]

    Masked-attention mask transformer for universal image segmentation

    Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexan- der Kirillov, and Rohit Girdhar. Masked-attention mask transformer for universal image segmentation. In Proceed- ings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1290–1299, 2022. 5

    2022

  5. [12]

    Sam-med2d

    Junlong Cheng, Jin Ye, Zhongying Deng, Jianpin Chen, Tianbin Li, Haoyu Wang, Yanzhou Su, Ziyan Huang, Ji- long Chen, Lei Jiang, et al. Sam-med2d. arXiv preprint arXiv:2308.16184, 2023. 3, 5, 6, 7

    2023 arXiv

  6. [13]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020. 2, 3

    2010 arXiv

  7. [14]

    Training like a medical resident: Context-prior learning toward universal medical image segmentation

    Yunhe Gao. Training like a medical resident: Context-prior learning toward universal medical image segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, pages 11194–11204, 2024. 2

    2024

  8. [15]

    One model is all you need: multi-task learning enables simultaneous histology image segmentation and classifica- tion

    Simon Graham, Quoc Dang Vu, Mostafa Jahanifar, Shan E Ahmed Raza, Fayyaz Minhas, David Snead, and Nasir Ra- jpoot. One model is all you need: multi-task learning enables simultaneous histology image segmentation and classifica- tion. Medical Image Analysis, 83:102685, 2023. 1, 6

    2023

  9. [16]

    Unetr: Transformers for 3d med- ical image segmentation

    Ali Hatamizadeh, Yucheng Tang, Vishwesh Nath, Dong Yang, Andriy Myronenko, Bennett Landman, Holger R Roth, and Daguang Xu. Unetr: Transformers for 3d med- ical image segmentation. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 574–58...

    2022

  10. [17]

    The kits21 chal- lenge: Automatic segmentation of kidneys, renal tumors, and renal cysts in corticomedullary-phase ct

    Nicholas Heller, Fabian Isensee, Dasha Trofimova, Resha Tejpaul, Zhongchen Zhao, Huai Chen, Lisheng Wang, Alex Golts, Daniel Khapun, Daniel Shats, et al. The kits21 chal- lenge: Automatic segmentation of kidneys, renal tumors, and renal cysts in corticomedullary-phase ct. arXi...

    2023 arXiv

  11. [18]

    nnu-net: a self-configuring method for deep learning-based biomedical image segmen- tation

    Fabian Isensee, Paul F Jaeger, Simon AA Kohl, Jens Pe- tersen, and Klaus H Maier-Hein. nnu-net: a self-configuring method for deep learning-based biomedical image segmen- tation. Nature methods, 18(2):203–211, 2021. 6, 7

    2021

  12. [19]

    Amos: A large-scale abdominal multi-organ benchmark for versatile medical image segmentation

    Yuanfeng Ji, Haotian Bai, Chongjian Ge, Jie Yang, Ye Zhu, Ruimao Zhang, Zhen Li, Lingyan Zhanng, Wanling Ma, Xi- ang Wan, et al. Amos: A large-scale abdominal multi-organ benchmark for versatile medical image segmentation. Ad- vances in neural information processing systems, 3...

    2022

  13. [20]

    Segment anything in high qual- ity

    Lei Ke, Mingqiao Ye, Martin Danelljan, Yu-Wing Tai, Chi- Keung Tang, Fisher Yu, et al. Segment anything in high qual- ity. Advances in Neural Information Processing Systems, 36,

  14. [21]

    Adam: A method for stochastic opti- mization

    Diederik P Kingma. Adam: A method for stochastic opti- mization. arXiv preprint arXiv:1412.6980, 2014. 6

    2014 arXiv

  15. [22]

    Segment any- thing

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer White- head, Alexander C Berg, Wan-Yen Lo, et al. Segment any- thing. In Proceedings of the IEEE/CVF International Con- ference on Computer Vision, pages 4015–4026, 202...

    2023

  16. [23]

    Miccai multi-atlas la- beling beyond the cranial vault–workshop and challenge

    Bennett Landman, Zhoubing Xu, J Igelsias, Martin Styner, Thomas Langerak, and Arno Klein. Miccai multi-atlas la- beling beyond the cranial vault–workshop and challenge. In Proc. MICCAI Multi-Atlas Labeling Beyond Cranial Vault—Workshop Challenge, page 12, 2015. 5, 7

    2015

  17. [24]

    Universal and extensible language-vision models for organ segmentation and tumor detection from abdominal computed tomography

    Jie Liu, Yixiao Zhang, Kang Wang, Mehmet Can Yavuz, Xiaoxi Chen, Yixuan Yuan, Haoliang Li, Yang Yang, Alan 9 Yuille, Yucheng Tang, et al. Universal and extensible language-vision models for organ segmentation and tumor detection from abdominal computed tomography. Medical Imag...

    2024

  18. [25]

    Swin transformer: Hierarchical vision transformer using shifted windows

    Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pages 10012–10022, 2021. 2

    2021

  19. [26]

    Fully convolutional networks for semantic segmentation

    Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In Pro- ceedings of the IEEE conference on computer vision and pat- tern recognition, pages 3431–3440, 2015. 2

    2015

  20. [27]

    Image segmenta- tion using text and image prompts

    Timo L ¨uddecke and Alexander Ecker. Image segmenta- tion using text and image prompts. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 7086–7096, 2022. 1

    2022

  21. [28]

    Segment anything in medical images

    Jun Ma, Yuting He, Feifei Li, Lin Han, Chenyu You, and Bo Wang. Segment anything in medical images. Nature Communications, 15(1):654, 2024. 1, 2, 5, 6, 7

    2024

  22. [29]

    U-mamba: Enhancing long-range dependency for biomedical image segmentation

    Jun Ma, Feifei Li, and Bo Wang. U-mamba: Enhancing long-range dependency for biomedical image segmentation. arXiv preprint arXiv:2401.04722, 2024. 6, 7

    2024 arXiv

  23. [30]

    Sam 2: Segment anything in images and videos

    Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman R¨adle, Chloe Rolland, Laura Gustafson, et al. Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408.00714, 2024. 6

    2024 arXiv

  24. [31]

    Segment anything, from space? In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 8355–8365, 2024

    Simiao Ren, Francesco Luzi, Saad Lahrichi, Kaleb Kas- saw, Leslie M Collins, Kyle Bradbury, and Jordan M Malof. Segment anything, from space? In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 8355–8365, 2024. 2

    2024

  25. [32]

    U- net: Convolutional networks for biomedical image segmen- tation

    Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U- net: Convolutional networks for biomedical image segmen- tation. In Medical image computing and computer-assisted intervention–MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, pa...

    2015

  26. [33]

    Attention is all you need

    A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017. 2, 4

    2017

  27. [34]

    Medical sam adapter: Adapting seg- ment anything model for medical image segmentation.arXiv preprint arXiv:2304.12620, 2023

    Junde Wu, Wei Ji, Yuanpei Liu, Huazhu Fu, Min Xu, Yanwu Xu, and Yueming Jin. Medical sam adapter: Adapting seg- ment anything model for medical image segmentation.arXiv preprint arXiv:2304.12620, 2023. 3, 6, 7

    2023 arXiv

  28. [37]

    Uniseg: A prompt-driven universal segmenta- tion model as well as a strong representation learner

    Yiwen Ye, Yutong Xie, Jianpeng Zhang, Ziyang Chen, and Yong Xia. Uniseg: A prompt-driven universal segmenta- tion model as well as a strong representation learner. In In- ternational Conference on Medical Image Computing and Computer-Assisted Intervention , pages 508–518. Springer,

  29. [38]

    Inpaint anything: Segment anything meets image inpainting

    Tao Yu, Runseng Feng, Ruoyu Feng, Jinming Liu, Xin Jin, Wenjun Zeng, and Zhibo Chen. Inpaint anything: Segment anything meets image inpainting. arXiv preprint arXiv:2304.06790, 2023. 2

    2023 arXiv

  30. [39]

    Modality-aware mutual learning for multi-modal medical image segmenta- tion

    Yao Zhang, Jiawei Yang, Jiang Tian, Zhongchao Shi, Cheng Zhong, Yang Zhang, and Zhiqiang He. Modality-aware mutual learning for multi-modal medical image segmenta- tion. In Medical Image Computing and Computer Assisted Intervention–MICCAI 2021: 24th International Conference, S...

    2021

  31. [41]

    Sam-sp: Self-prompting makes sam great again

    Chunpeng Zhou, Kangjie Ning, Qianqian Shen, Sheng Zhou, Zhi Yu, and Haishuai Wang. Sam-sp: Self-prompting makes sam great again. arXiv preprint arXiv:2408.12364, 2024. 3

    2024 arXiv

  32. [42]

    nn- former: V olumetric medical image segmentation via a 3d transformer

    Hong-Yu Zhou, Jiansen Guo, Yinghao Zhang, Xiaoguang Han, Lequan Yu, Liansheng Wang, and Yizhou Yu. nn- former: V olumetric medical image segmentation via a 3d transformer. IEEE Transactions on Image Processing, 2023. 6, 7

    2023

  33. [43]

    Unet++: Redesigning skip connections to exploit multiscale features in image segmen- tation

    Zongwei Zhou, Md Mahfuzur Rahman Siddiquee, Nima Tajbakhsh, and Jianming Liang. Unet++: Redesigning skip connections to exploit multiscale features in image segmen- tation. IEEE transactions on medical imaging, 39(6):1856– 1867, 2019. 5

    2019

  34. [44]

    Text promptable surgical instru- ment segmentation with vision-language models

    Zijian Zhou, Oluwatosin Alabi, Meng Wei, Tom Ver- cauteren, and Miaojing Shi. Text promptable surgical instru- ment segmentation with vision-language models. Advances in Neural Information Processing Systems, 36:28611–28623,

  35. [45]

    Segment everything everywhere all at once

    Xueyan Zou, Jianwei Yang, Hao Zhang, Feng Li, Linjie Li, Jianfeng Wang, Lijuan Wang, Jianfeng Gao, and Yong Jae Lee. Segment everything everywhere all at once. Advances in Neural Information Processing Systems, 36, 2024. 1 10 SEG-SAM: Semantic-Guided SAM for Unified Medical Im...

    2024

  36. [46]

    6.1) as well as the text template we have used (Sec

    Appendix In the supplementary material, we provide the additional ablation study of our method (Sec. 6.1) as well as the text template we have used (Sec. 6.2). Training setup is the same to the main paper. 6.1. More Ablation Study To further validate the effectiveness of our m...

  37. [47]

    Texture: [Description]

  38. [48]

    Shape: [Description]

  39. [49]

    1 Method Components Med2D SAWDpmt SAWDunpmt SAWDsem T2VSE CMSA-Loss DSC mDSC Baseline 65.45 - BaselineSem 64.56 65.11

    Positional: [Description]. 1 Method Components Med2D SAWDpmt SAWDunpmt SAWDsem T2VSE CMSA-Loss DSC mDSC Baseline 65.45 - BaselineSem 64.56 65.11

  40. [50]

    The ablation study of our method (point prompts) on Med2D-16M datasets

    ✓ ✓ ✓ 71.70 73.67 SEG-SAM ✓ ✓ ✓ ✓ 73.15 75.28 Table 8. The ablation study of our method (point prompts) on Med2D-16M datasets. We take the vanilla SAM as our baseline for binary segmentation, while introducing an additional classification token in SAM’s decoder as another base...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.