Pith. sign in

REVIEW 4 major objections 5 minor 78 references

MedUP: Awakening Unified Understanding and Perception in Medical Vision-Language Models

T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Masks become discrete tokens in the language model's vocabulary, allowing a single medical vision-language model to answer questions, segment structures, and interpret regions without external decoders.

desk verdict A coherent medical mask-token VLM with a large new corpus, but the main 'outperforms all' comparison mixes protocols and may share training datasets with the benchmark, so the headline claim is not yet supported. read the letter →

arxiv 2608.10635 v1 pith:F7XW7ZQ6 submitted 2026-08-11 cs.CV cs.AI

Yuan Wang , Hualiang Wang , Yixin Chen , Songtao Jiang , Shujian Gao , Jiaming Lin , Siming Fu , Jian Wu
show 1 more author
Zuozhu Liu
This is my paper · ORCID
classification cs.CVcs.AI
keywords medicalvision-languagemodelUniMedTokmasktokenizationtext-guidedsegmentationregion-groundedunderstandingvisualquestionansweringSeg-CoTvector-quantizedautoencoder
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

MedUP is designed to show that a medical vision-language model can handle both understanding and perception in one autoregressive network if segmentation masks are turned into short sequences of discrete tokens. The paper introduces UniMedTok, a frozen image-conditioned mask tokenizer that compresses each mask into two 256-entry codebook tokens, and then trains the VLM on a 1.84M-instance corpus mixing medical VQA, text-guided segmentation, region-grounded understanding, and reasoning-augmented segmentation. On the new UniMed-Bench, MedUP reports higher scores than native, agentic, and dual-decoder baseline families on all three tasks while remaining competitive with specialist segmentors that receive oracle visual prompts. The paper's intended lesson is that treating regions as language removes the representation gap created by coordinate strings or external segmentation modules.

What carries the argument

The load-bearing mechanism is UniMedTok, a medical mask tokenizer that acts as a native region-language interface. It is an image-conditioned vector-quantized autoencoder with the MT256×2 scheme: two non-shared codebooks of 256 entries each, trained in Stage 1 by mask reconstruction and then frozen. Stage 2 expands the VLM vocabulary with a start token, an end token, and 512 mask code tokens, so the serialization operator $S(M)$ turns any mask into a four-token span. Mask-as-input appends that span to the user prompt, while mask-as-output has the VLM generate the span and the frozen tokenizer decode it back to a dense mask. This mechanism carries the argument because all three tasks are reduced to the same autoregressive next-token objective over text and mask tokens, with no trainable segmentation head.

What would settle it

A decisive check would be to list the dataset-level overlap between UniMed-Train and UniMed-Bench, and to train the strongest overlay-based baseline on the same token-based supervision before evaluating it with overlays; if the overlap is substantial, or if the baseline's region-grounded exact match jumps toward 78–82%, the claimed superiority would not be native unification.

Watch

Extended reading notes

Core claim

The central discovery the authors are trying to establish is that a mask tokenizer can live inside the language model's vocabulary and make perception a form of language modeling. UniMedTok encodes a mask, conditioned on the medical image, into an ordered two-code pair $q=[c_1,c_2]$ from 256-entry codebooks, and with start and end tokens this becomes a four-token text span. The same span can be generated as output for text-guided segmentation or inserted into the prompt for region-grounded understanding, so all tasks collapse into next-token prediction in one shared space. After two-stage training on 1.84M instances, the paper reports that MedUP beats native, agentic, and dual-decoder baselines on all UniMed-Bench tasks and remains competitive with specialist medical segmentors across modalities.

Load-bearing premise

The results depend on the benchmark being truly held out from the training corpus and on comparing MedUP with token-based region references against baselines with overlaid masks measuring model quality rather than protocol, especially since the paper's own protocol study moves exact match from 49.8% to 78.5–81.8%.

Editorial extensions

If this is right

  • A single medical VLM can handle both image-level questions and pixel-precise localization, removing the need for an external segmentation tool or a separate dense decoder for text-guided masking.
  • Region-grounded understanding is more effective when the region is given as mask tokens rather than as a visual overlay: the paper's protocol study reports exact match rising from 49.8% to 78.5–81.8%.
  • Round-trip filtering of mask supervision matters: removing low-fidelity mask-token training cases improves text-guided segmentation mean Dice by 7.1 to 9.3 points.
  • Segmentation-oriented chain-of-thought (Seg-CoT) fits naturally into the same token space, so masks can be produced with an intermediate anatomical and localization rationale within one generation.
  • Both grounded tasks improve with more Stage-2 training data in the tested scale range, suggesting that larger region-language corpora are a direct route to better grounded Med-VLMs.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper, a decisive next experiment is to train a non-token baseline on the same token-based supervision and evaluate it with overlaid masks; the paper's protocol table suggests the protocol alone can move exact match by about 29 points, so this comparison would separate interface benefit from model quality.
  • Because both the training corpus and benchmark live in the same 80-dataset family, an external-transfer split on datasets seen in neither stage would test whether the mask-language interface generalizes beyond familiar datasets.
  • The same mask-token recipe could extend to other dense medical outputs, such as landmark heatmaps, lesion counting, or hierarchical anatomy decomposition, though the paper only demonstrates binary mask prediction.
  • The paper's limitations section notes the compact two-code representation and offline-only evaluation, so interactive refinement and very small or irregular structures remain open rather than established.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes MedUP, a medical vision-language model that unifies text-guided segmentation, region-grounded understanding, and medical VQA within a single autoregressive framework. The core contribution is UniMedTok, a frozen image-conditioned tokenizer that encodes segmentation masks as two discrete tokens in the LLM vocabulary, allowing masks to be interleaved with text. The authors construct a large training corpus (UniMed-Train, 1.84M instances) and a three-task benchmark (UniMed-Bench), and report that MedUP outperforms native, agentic, and dual-decoder Med-VLMs across all tasks while remaining competitive with specialist segmentors. The central claims depend on the validity of the benchmark comparison and the claimed benefits of the mask-token interface and Seg-CoT reasoning.

Significance. The idea of representing medical regions as discrete tokens in the language modeling space is timely and aligns with recent general-domain mask-token methods (e.g., SAMTok, HiMTok). If the reported results were properly controlled, the paper would provide a useful medical-domain instantiation, a large corpus, and a benchmark that could advance unified perception-understanding modeling. The paper also ships a large corpus and explicit two-stage training pipeline. However, the current evaluation contains protocol and data-overlap confounds that prevent the headline claims from being verified. The central contribution is plausible and potentially valuable, but the evidence as presented is not yet convincing.

major comments (4)
  1. [Table 1 and Table 4] The main comparison for region-grounded understanding is not apples-to-apples. Table 1 reports MedUP under the v2_tokens protocol while all baselines are evaluated under v1_masks, as stated in the table footnote and in Section 4.1. Table 4 shows that on the same backbones, the protocol alone changes exact match from 49.8 (v1_masks) to 78.5 (MedUP-Q) and 81.8 (MedUP-H), a gap of roughly 30 points. The reported margins over baselines (up to 81.8 vs 4.4) are therefore largely protocol-driven. The claim in the abstract that MedUP 'outperforms native, agentic, and dual-decoder Med-VLMs across all tasks' is not supported by this comparison, because the region-grounded capability is not measured under matched conditions. Please compare MedUP and baselines under both protocols, or clearly restrict the claim to v1_masks, and discuss the protocol effect explicitly.
  2. [Sections 3.1 and 3.2] The held-out status of UniMed-Bench is not established. Section 3.1 states that the mask-centric training streams are built from '80+1 medical segmentation datasets,' while Section 3.2 describes UniMed-Bench as 'built from the same 80 datasets' and calls it held-out. If the benchmark draws images or datasets from the same family used to construct Stage-2 training and to train the Stage-1 tokenizer, then the segmentation and region-grounded results reflect distribution overlap rather than generalization. The paper must specify whether the split is at the dataset level, the image level, or the patient level, and must confirm that no training images appear in UniMed-Bench. Without this clarification, the benchmark numbers cannot be interpreted as measuring generalization.
  3. [Section 4.6, Abstract, Conclusion] The contribution of Seg-CoT is not quantitatively supported. The abstract and conclusion state that Seg-CoT improves text-to-mask generation through intermediate reasoning, but Section 4.6 explicitly says 'We leave additional ablations such as round-trip filtering, token budget, and Seg-CoT training effects for future versions once the corresponding experimental evidence is included.' No ablation compares MedUP with and without Seg-CoT. Either provide the missing ablation or remove the claim that Seg-CoT is an empirically validated contribution.
  4. [Section 2.3 and Appendix C.1] The role of the tokenizer in bounding segmentation performance should be quantified. The tokenizer is trained on masks from the same dataset family used for Stage-2 training and for evaluation, and it is frozen when decoding all generated mask tokens. The end-to-end segmentation numbers therefore depend on the tokenizer's reconstruction fidelity on exactly those datasets. Reporting the tokenizer's reconstruction Dice/IoU on UniMed-Bench would clarify how much of the reported segmentation performance is attributable to the VLM versus the tokenizer, and would assess the circularity risk that the tokenizer simply memorizes masks from the training family.
minor comments (5)
  1. [Tables 1 and 3] The aggregation metric for Dice is inconsistent: Table 1 reports macro Dice while Table 3 reports micro Dice. Please define both clearly and indicate which is used for the headline claims.
  2. [Appendix E, Tables 7-10] The appendix tables are explicitly selected to show only rows where MedUP achieves the best score. This selection is biased and should be replaced by full results or clearly labeled as illustrative subsets.
  3. [References] Several references (e.g., Aho and Ullman 1972, APA 1983, Chandra et al. 1981, Gusfield 1997) appear unrelated to the medical vision-language content and should be removed or replaced with relevant citations.
  4. [Equation (3)] The offset of 256 for the second codebook token is mentioned only in the equation. Please add a sentence in the main text explaining that the 512 mask tokens are split into two groups of 256.
  5. [Appendix D] The Seg-CoT examples show reasoning traces with mask tokens, but the displayed overlays do not show the decoded masks. Including decoded mask visualizations would help the reader verify that the generated tokens correspond to meaningful regions.

Circularity Check

2 steps flagged · score 5.0 of 10

Region-grounded comparison mixes evaluation protocols and shares its construction with training, so the across-tasks superiority claim is not independently established.

  1. other [Table 1 caption and Section 4.4 / Table 4]
    "For baselines without native mask tokens, Region-Grounded Understanding is evaluated under v1_masks. ... v1_masks exposes the target region visually, whereas v2_tokens uses discrete mask tokens as the region reference. ... Qwen-based: v1_masks 49.8, v2_tokens 78.5; Hulu-based: v1_masks 49.8, v2_tokens 81.8."

    Table 1 reports MedUP's Region-Grounded Understanding exact match as 78.5 (MedUP-Q) and 81.8 (MedUP-H) under v2_tokens, while all baselines are scored under v1_masks with EM between 0.0 and 4.4. The paper's own Table 4 shows that changing only the region-presentation protocol on the identical MedUP models moves EM from 49.8 to 78.5 (Qwen-based) and from 49.8 to 81.8 (Hulu-based), a +28.7/+32.0 shift. Because the baselines are not evaluated under v2_tokens, the large cross-model margin in Table 1 is substantially an artifact of the evaluation interface rather than an equivalent measurement of capability; the claimed 'outperforms native, agentic, and dual-decoder Med-VLMs across all tasks' is therefore not isolated from the protocol choice that only MedUP receives.

  2. other [Section 3.1 and Section 3.2]
    "The two mask-centric streams are constructed from 80+1 medical segmentation datasets using the frozen Stage-1 tokenizer ... Region-Grounded Understanding is built from the same 80 datasets, with 218,244 v2_tokens samples and 219,257 v1_masks samples."

    UniMed-Bench is introduced as a 'held-out benchmark,' but both the Stage-2 mask-centric training streams and the benchmark's segmentation and region-grounded tasks are built from the same 80-dataset family, using the same frozen Stage-1 tokenizer to serialize masks into token spans. No dataset-level split is specified to separate the benchmark from the training construction. As a result, the token sequences and category labels the model is asked to produce at evaluation are generated by the same tokenizer and drawn from the same dataset taxonomy that produced the Stage-2 supervision; the measured 'prediction' is an in-distribution reproduction of the training construction process rather than an independent test of generalization.

full rationale

The architectural derivation of MedUP is largely self-contained: UniMedTok is a vector-quantized mask autoencoder, and the Stage-2 formulation reduces all tasks to next-token prediction over a shared text-mask vocabulary. That derivation is not circular, and there is no load-bearing self-citation chain or uniqueness theorem being imported. However, the central empirical claim that MedUP 'outperforms native, agentic, and dual-decoder Med-VLMs across all tasks' rests on UniMed-Bench numbers that are partly produced by the evaluation protocol itself. Table 1 compares MedUP under v2_tokens with baselines under v1_masks, and Table 4 quantifies that this protocol difference alone accounts for +28.7/+32.0 EM on identical backbones. Additionally, Section 3.2 says the benchmark is built from 'the same 80 datasets' as the mask-centric training streams of Section 3.1, with no dataset-level split stated, so the region-grounded and segmentation evaluations may be in-distribution with respect to the training construction. The frozen Stage-1 tokenizer is also used both to create Stage-2 supervision targets and to decode outputs at test time, which means the segmentation numbers are bounded by the tokenizer's own reconstruction fidelity; this is a recognized design property rather than a derivation collapse, but it reinforces that the benchmark does not independently test the claimed unification. Section 4.6 further defers Seg-CoT ablations, so the Seg-CoT contribution currently has no quantitative support. Because Medical VQA is measured on external benchmarks (SLAKE, PathVQA, VQA-RAD) and text-guided segmentation is compared with specialist segmentors, some independent content remains; the circularity is partial and concentrated in the headline region-grounded comparison.

Assumptions & free parameters 4 free parameters · 4 assumptions · 1 invented entities

The central results rest on hand-chosen tokenizer capacity, unspecified round-trip filtering ratios, a self-built evaluation protocol, and the assumption that training and test datasets are separate. The token interface itself is an invented representation with no independent external evidence.

free parameters (4)
  • MT256x2 token configuration = codebook size 256, depth 2
    Chosen by hand rather than fit to data; it determines mask reconstruction fidelity and therefore sets an upper bound on segmentation performance.
  • Round-trip filtering keep ratios
    Dataset-level downsampling ratios based on reconstruction Dice/IoU are not specified; they affect training composition and the reported filtering gains.
  • Stage-2 corpus mixture = 902,648 / 902,648 / 27,738 / 4,000
    The relative weights of the four supervision streams are design choices; mask-centric streams dominate roughly 98% of training and shape model behavior.
  • Seg-CoT budget = 4,000 samples
    The reasoning-augmented stream is tiny and no ablation shows whether it contributes; it is an untested design choice.
assumptions (4)
  • ad hoc to paper A two-code discrete tokenization can faithfully represent arbitrary medical region masks across 80 datasets.
    Stage 1 assumes MT256x2 captures small, irregular, or subtle structures; the Limitations section concedes these may not be well represented.
  • ad hoc to paper UniMed-Bench is held out from UniMed-Train at the dataset level.
    Section 3.1 builds training streams from 80+1 segmentation datasets while Section 3.2 builds the benchmark from the same 80-dataset family, making dataset-level separation unclear.
  • ad hoc to paper v2_tokens and v1_masks are comparable protocols for cross-model evaluation.
    Table 1 assigns v2_tokens to MedUP and v1_masks to baselines; Table 4 shows the protocol alone changes EM by about 30 points, so this premise is violated.
  • domain assumption Seg-CoT reasoning supervision improves text-to-mask prediction.
    No quantitative experiment isolates Seg-CoT; Section 4.6 states its ablation is left to future versions.
invented entities (1)
  • UniMedTok discrete mask tokens (<|mt_0000|> to <|mt_0511|>)
    purpose: Represent region masks as language-compatible discrete tokens in the LLM vocabulary for both mask-as-input and mask-as-output.
    These tokens are defined entirely by the authors' tokenizer and benchmark; there is no external falsifiable handle outside the paper's own evaluation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MedUP: Awakening Unified Understanding and Perception in Medical Vision-Language Models." pith.science (2026). https://pith.science/paper/F7XW7ZQ6

@misc{pith2026260810635,
  author       = {Pith},
  title        = {Pith review of: MedUP: Awakening Unified Understanding and Perception in Medical Vision-Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/F7XW7ZQ6}},
  note         = {Machine review of arXiv:2608.10635}
}
read the original abstract

Medical Vision-Language Models (Med-VLMs) excel at verbalizing visual content, yet precise visual perception, segmentation, and grounding remain challenging. Existing approaches either verbalize regions as coordinate strings or rely on external modules that decouple perception from understanding, creating representation gaps for region-language alignment. We present MedUP, a Med-VLM that natively unifies perception and understanding within a shared token space. At its core lies UniMedTok, a region tokenizer that encodes masks as discrete tokens in the LLM vocabulary, enabling the model to seamlessly interleave mask tokens with text. We curate UniMed-Train, a 1.84M-instance corpus spanning text-guided segmentation, region-grounded understanding, medical VQA and CoT-based segmentation, and introduce UniMed-Bench for unified evaluation. Extensive experiments show that MedUP outperforms native, agentic, and dual-decoder Med-VLMs across all tasks while remaining competitive with specialist segmentors, demonstrating the strong potential of unified understanding and perception modeling.

Figures

Figures reproduced from arXiv: 2608.10635 by the authors.

Figure 1
Figure 1. Prior medical VLMs either lack native text [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of MedUP. MedUP is built on UniMedTok, a native mask-token interface for grounded medical vision-language modeling. Stage 1 learns a medical mask tokenizer that converts region masks into compact discrete tokens, and Stage 2 trains the VLM on four supervision streams from UniMed-Train: Medical VQA, Text-Guided Segmentation, Region-Grounded Understanding, and Seg-CoT. The same interface supports both mask-as… view at source ↗
Figure 3
Figure 3. Overview of the unified grounded medical framework and the UniMed corpus. The framework unifies segmentation, region understanding, medical VQA, and Seg-CoT reasoning through shared mask tokens across seven imaging modalities. construction loss is used in this stage; cross-task transfer is induced entirely by next-token predic￾tion over mixed text-mask sequences. For Seg-CoT specifically, the target is written as a … view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Effect of training data scale on Text-Guided Segmentation and Region-Grounded Understanding. (A) Increasing the training data scale from 10% to 100% consistently improves weighted Dice and weighted IoU in text-guided segmentation, with a 0.099 gain in weighted Dice. (B…
Figure 5
Figure 5. Figure 5: isolates the contribution of our round-trip filtering strategy. Without filtering, tokenizer recon￾struction errors introduce noisy supervision into Stage-2 mask generation, especially on datasets with small, irregular, or semantically ambiguous regions. After filterin…
Figure 6
Figure 6. Figure 6: Overview of UniMed-Train and UniMed-Bench. UniMed-Train combines four supervision streams for Stage-2 training, while UniMed-Bench provides held-out evaluation across Medical VQA, Text-Guided Segmentation, and Region-Grounded Understanding. region mask into a compact M…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

78 extracted references · 41 canonical work pages

  1. [1]

    Aho and Jeffrey D

    Alfred V. Aho and Jeffrey D. Ullman. 1972. The Theory of Parsing, Translation and Compiling, volume 1. Prentice-Hall, Englewood Cliffs, NJ

    1972

  2. [2]

    American Psychological Association . 1983. Publications Manual. American Psychological Association, Washington, DC

    1983

  3. [3]

    Rie Kubota Ando and Tong Zhang. 2005. A framework for learning predictive structures from multiple tasks and unlabeled data. Journal of Machine Learning Research, 6:1817--1853

    2005

  4. [4]

    Galen Andrew and Jianfeng Gao. 2007. Scalable training of L1 -regularized log-linear models. In Proceedings of the 24th International Conference on Machine Learning, pages 33--40

    2007

  5. [6]

    Hu Cao, Yueyue Wang, Joy Chen, Dongsheng Jiang, Xiaopeng Zhang, Qi Tian, and Manning Wang. 2023. https://doi.org/10.1007/978-3-031-25066-8_9 Swin-unet: Unet-like pure transformer for medical image segmentation . In Computer Vision -- ECCV 2022 Workshops, pages 205--218. Springer

    doi:10.1007/978-3-031-25066-8_9 2023

  6. [8]

    Junying Chen, Chi Gui, Ruyi Ouyang, Anningzhe Gao, Shunian Chen, Guiming Hardy Chen, Xidong Wang, Ruifei Zhang, Zhenyang Cai, Ke Ji, Guangjun Yu, Xiang Wan, and Benyou Wang. 2024. https://arxiv.org/abs/2406.19280 Huatuogpt-vision, towards injecting medical visual knowledge into multimodal llms at scale . Preprint, arXiv:2406.19280

    arXiv 2024

  7. [9]

    Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. 2023. https://arxiv.org/abs/2306.15195 Shikra: Unleashing multimodal llm's referential dialogue magic . Preprint, arXiv:2306.15195

    arXiv 2023

  8. [10]

    Dan Gusfield. 1997. Algorithms on Strings, Trees and Sequences. Cambridge University Press, Cambridge, UK

    1997

Show all 78 references
  1. [11]

    Roth, and Daguang Xu

    Ali Hatamizadeh, Yucheng Tang, Vishwesh Nath, Dong Yang, Andriy Myronenko, Bennett Landman, Holger R. Roth, and Daguang Xu. 2022. Unetr: Transformers for 3d medical image segmentation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 574--584

    2022

  2. [12]

    Xiaoshuang Huang, Lingdong Shen, Jia Liu, Fangxin Shang, Hongxiang Li, Haifeng Huang, and Yehui Yang. 2025 a . https://doi.org/10.1609/aaai.v39i4.32394 Towards a multimodal large language model with pixel-level insight for biomedicine . Proceedings of the AAAI Conference on Ar...

    2025 doi

  3. [13]

    Yu Huang, Zelin Peng, Yichen Zhao, Piao Yang, Xiaokang Yang, and Wei Shen. 2025 b . https://arxiv.org/abs/2506.10465 Medseg-r: Reasoning segmentation in medical images with multimodal large language models . Preprint, arXiv:2506.10465

    2025 arXiv

  4. [14]

    Jaeger, Simon A

    Fabian Isensee, Paul F. Jaeger, Simon A. A. Kohl, Jens Petersen, and Klaus H. Maier-Hein. 2021. nnu-net: a self-configuring method for deep learning-based biomedical image segmentation. Nature Methods, 18(2):203--211

    2021

  5. [17]

    Yankai Jiang, Qiaoru Li, Binlu Xu, Haoran Sun, Chao Ding, Junting Dong, Yuxiang Cai, Xuhong Zhang, and Jianwei Yin. 2026. https://arxiv.org/abs/2601.03054 Ibisagent: Reinforcing pixel-level visual reasoning in mllms for universal biomedical object referring and segmentation . ...

    2026 arXiv

  6. [18]

    Berg, Wan-Yen Lo, Piotr Dollar, and Ross Girshick

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollar, and Ross Girshick. 2023. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer V...

    2023

  7. [19]

    Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. 2024. Lisa: Reasoning segmentation via large language model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9579--9589

    2024

  8. [20]

    Binxu Li, Tiankai Yan, Yuanting Pan, Jie Luo, Ruiyang Ji, Jiayuan Ding, Zhe Xu, Shilong Liu, Haoyu Dong, Zihao Lin, and 1 others. 2024. Mmedagent: Learning to use medical tools with multi-modal agent. In Findings of the Association for Computational Linguistics: EMNLP 2024, pa...

    2024

  9. [21]

    Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon, and Jianfeng Gao. 2023. Llava-med: Training a large language-and-vision assistant for biomedicine in one day. In Advances in Neural Information Processing Systems, vo...

    2023

  10. [23]

    Jiaxiang Liu, Yuan Wang, Jiawei Du, Joey Tianyi Zhou, and Zuozhu Liu. 2024. Medcot: Medical chain of thought via hierarchical expert. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 17371--17389

    2024

  11. [24]

    Lingxiao Luo, Bingda Tang, Xuanzhong Chen, Rong Han, and Ting Chen. 2025. Vividmed: Vision language model with versatile visual grounding for medicine. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics...

    2025

  12. [25]

    Jun Ma, Yuting He, Feifei Li, Lin Han, Chenyu You, and Bo Wang. 2024. https://doi.org/10.1038/s41467-024-44824-z Segment anything in medical images . Nature Communications, 15(1):654

    2024 doi

  13. [26]

    Michael Moor, Qian Huang, Shirley Wu, Michihiro Yasunaga, Yash Dalmia, Jure Leskovec, Cyril Zakka, Eduardo Pontes Reis, and Pranav Rajpurkar. 2023. Med-flamingo: a multimodal medical few-shot learner. In Proceedings of the 3rd Machine Learning for Health Symposium, volume 225 ...

    2023

  14. [27]

    Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, Qixiang Ye, and Furu Wei. 2024. Kosmos-2: Grounding multimodal large language models to the world. In The Twelfth International Conference on Learning Representations

    2024

  15. [28]

    Tetreault

    Mohammad Sadegh Rasooli and Joel R. Tetreault. 2015. http://arxiv.org/abs/1503.06733 Yara parser: A fast and accurate dependency parser . Computing Research Repository, arXiv:1503.06733. Version 2

    2015 arXiv

  16. [29]

    Olaf Ronneberger, Philipp Fischer, and Thomas Brox. 2015. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention, pages 234--241. Springer

    2015

  17. [30]

    Andrew Sellergren, Sahar Kazemzadeh, Tiam Jaroensri, Atilla Kiraly, Madeleine Traverse, Timo Kohlberger, Shawn Xu, Fayaz Jamil, Cian Hughes, Charles Lau, and 1 others. 2026. https://arxiv.org/abs/2507.05201 Medgemma technical report . Preprint, arXiv:2507.05201

    2026 arXiv

  18. [31]

    Qinyue Tong, Ziqian Lu, Jun Liu, Yangming Zheng, and Zheming Lu. 2025. https://arxiv.org/abs/2504.11008 Medisee: Reasoning-based pixel-level perception in medical images . Preprint, arXiv:2504.11008

    2025 arXiv

  19. [32]

    Quoc-Huy Trinh, Minh-Van Nguyen, Jun Zeng, Debesh Jha, and Ulas Bagci. 2026. https://arxiv.org/abs/2505.11872 Prs-med: Position reasoning segmentation in medical imaging . Preprint, arXiv:2505.11872

    2026

  20. [33]

    Guoxin Wang, Jun Zhao, Xinyi Liu, Yanbo Liu, Xuyang Cao, Chao Li, Zhuoyun Liu, Qintian Sun, Fangru Zhou, Haoqiang Xing, and Zhenhong Yang. 2025 a . https://arxiv.org/abs/2509.19090 Citrus-v: Advancing medical foundation models with unified medical image grounding for clinical ...

    2025

  21. [34]

    Tao Wang, Changxu Cheng, Lingfeng Wang, Senda Chen, and Wuyue Zhao. 2025 b . Himtok: Learning hierarchical mask tokens for image segmentation with large multimodal model. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 23267--23278

    2025

  22. [35]

    Yuan Wang, Shujian Gao, Jiaxiang Liu, Songtao Jiang, Xia Haoxiang, Xiaotian Zhang, Zhaolu Kang, Yemin Wang, and Zuozhu Liu. 2026. Beyond n-grams: A hierarchical reward learning framework for clinically-aware medical report generation. In Proceedings of the AAAI Conference on A...

    2026

  23. [36]

    Yuan Wang, Jiaxiang Liu, Shujian Gao, Bin Feng, Zhihang Tang, Xiaotang Gai, Jian Wu, and Zuozhu Liu. 2025 c . V2t-cot: From vision to text chain-of-thought for medical reasoning and diagnosis. In International Conference on Medical Image Computing and Computer-Assisted Interve...

    2025

  24. [38]

    Chi, Quoc V

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems, volume 35, pages 24824--24837

    2022

  25. [39]

    Chaoyi Wu, Xiaoman Zhang, Ya Zhang, Hui Hui, Yanfeng Wang, and Weidi Xie. 2025 a . https://doi.org/10.1038/s41467-025-62385-7 Towards generalist foundation model for radiology by leveraging web-scale 2d&3d medical data . Nature Communications, 16:7866

    2025 doi

  26. [40]

    Linshan Wu, Yuxiang Nie, Sunan He, Jiaxin Zhuang, Luyang Luo, Tao Li, Zhuoyao Xie, Dexuan Chen, Yinghua Zhao, Neeraj Mahboobani, Varut Vardhanabhuti, Ronald Cheong Kin Chan, Yifan Peng, Pranav Rajpurkar, and Hao Chen. 2025 b . https://arxiv.org/abs/2504.21336 Unibiomed: A univ...

    2025

  27. [41]

    Davison, Hui Ren, Jing Huang, Chen Chen, Yuyin Zhou, Sunyang Fu, Wei Liu, Tianming Liu, Xiang Li, Yong Chen, Lifang He, and 4 others

    Kai Zhang, Rong Zhou, Eashan Adhikarla, Zhiling Yan, Yixin Liu, Jun Yu, Zhengliang Liu, Xun Chen, Brian D. Davison, Hui Ren, Jing Huang, Chen Chen, Yuyin Zhou, Sunyang Fu, Wei Liu, Tianming Liu, Xiang Li, Yong Chen, Lifang He, and 4 others. 2024. https://doi.org/10.1038/s41591...

    2024 doi

  28. [42]

    Yikang Zhou, Tao Zhang, Dengxian Gong, Yuanzheng Wu, Ye Tian, Haochen Wang, Haobo Yuan, Jiacong Wang, Lu Qi, Hao Fei, Anran Wang, Zhuochen Wang, Yujing Wang, Cheng Chen, Shunping Ji, and Xiangtai Li. 2026. https://arxiv.org/abs/2601.16093 Samtok: Representing any mask with two...

    2026

  29. [43]

    Aho and Jeffrey D

    Alfred V. Aho and Jeffrey D. Ullman , title =. 1972

    1972

  30. [44]

    arXiv preprint arXiv:2502.09838 , year=

    Healthgpt: A medical large vision-language model for unifying comprehension and generation via heterogeneous knowledge adaptation , author=. arXiv preprint arXiv:2502.09838 , year=

    arXiv

  31. [45]

    Findings of the Association for Computational Linguistics: EMNLP 2024 , pages=

    Mmedagent: Learning to use medical tools with multi-modal agent , author=. Findings of the Association for Computational Linguistics: EMNLP 2024 , pages=

    2024

  32. [46]

    Publications Manual , year = "1983", publisher =

    1983

  33. [47]

    Chandra and Dexter C

    Ashok K. Chandra and Dexter C. Kozen and Larry J. Stockmeyer , year = "1981", title =. doi:10.1145/322234.322243

    1981

  34. [48]

    Scalable training of

    Andrew, Galen and Gao, Jianfeng , booktitle=. Scalable training of

  35. [49]

    Dan Gusfield , title =. 1997

    1997

  36. [50]

    Tetreault , title =

    Mohammad Sadegh Rasooli and Joel R. Tetreault , title =. Computing Research Repository , volume =. 2015 , url =

    2015

  37. [51]

    A Framework for Learning Predictive Structures from Multiple Tasks and Unlabeled Data , Volume =

    Ando, Rie Kubota and Zhang, Tong , Issn =. A Framework for Learning Predictive Structures from Multiple Tasks and Unlabeled Data , Volume =. Journal of Machine Learning Research , Month = dec, Numpages =

  38. [52]

    Medical Image Computing and Computer-Assisted Intervention , pages=

    U-Net: Convolutional Networks for Biomedical Image Segmentation , author=. Medical Image Computing and Computer-Assisted Intervention , pages=. 2015 , organization=

    2015

  39. [53]

    Nature Methods , volume=

    nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation , author=. Nature Methods , volume=

  40. [54]

    Computer Vision -- ECCV 2022 Workshops , pages=

    Swin-Unet: Unet-like Pure Transformer for Medical Image Segmentation , author=. Computer Vision -- ECCV 2022 Workshops , pages=. 2023 , publisher=

    2022

  41. [55]

    Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages=

    UNETR: Transformers for 3D Medical Image Segmentation , author=. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages=

  42. [56]

    Proceedings of the IEEE/CVF International Conference on Computer Vision , pages=

    Segment Anything , author=. Proceedings of the IEEE/CVF International Conference on Computer Vision , pages=

  43. [57]

    Nature Communications , volume=

    Segment Anything in Medical Images , author=. Nature Communications , volume=. 2024 , doi=

    2024

  44. [58]

    Advances in Neural Information Processing Systems , volume=

    LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day , author=. Advances in Neural Information Processing Systems , volume=. 2023 , note=

    2023

  45. [59]

    Proceedings of the 3rd Machine Learning for Health Symposium , pages=

    Med-Flamingo: a Multimodal Medical Few-shot Learner , author=. Proceedings of the 3rd Machine Learning for Health Symposium , pages=. 2023 , volume=

    2023

  46. [60]

    Nature Medicine , volume=

    A generalist vision-language foundation model for diverse biomedical tasks , author=. Nature Medicine , volume=. 2024 , doi=

    2024

  47. [61]

    2024 , eprint=

    HuatuoGPT-Vision, Towards Injecting Medical Visual Knowledge into Multimodal LLMs at Scale , author=. 2024 , eprint=

    2024

  48. [62]

    2025 , eprint=

    Hulu-Med: A Transparent Generalist Model towards Holistic Medical Vision-Language Understanding , author=. 2025 , eprint=

    2025

  49. [63]

    2026 , eprint=

    MedGemma Technical Report , author=. 2026 , eprint=

    2026

  50. [64]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

    LISA: Reasoning Segmentation via Large Language Model , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

  51. [65]

    Proceedings of the AAAI Conference on Artificial Intelligence , volume=

    Towards a Multimodal Large Language Model with Pixel-Level Insight for Biomedicine , author=. Proceedings of the AAAI Conference on Artificial Intelligence , volume=. 2025 , doi=

    2025

  52. [66]

    Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , pages=

    Medcot: Medical chain of thought via hierarchical expert , author=. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , pages=

    2024

  53. [67]

    arXiv preprint arXiv:2503.18968 , year=

    Medagent-pro: Towards evidence-based multi-modal medical diagnosis via reasoning agentic workflow , author=. arXiv preprint arXiv:2503.18968 , year=

    arXiv

  54. [68]

    2025 , eprint=

    Citrus-V: Advancing Medical Foundation Models with Unified Medical Image Grounding for Clinical Reasoning , author=. 2025 , eprint=

    2025

  55. [69]

    2025 , eprint=

    UniBiomed: A Universal Foundation Model for Grounded Biomedical Image Interpretation , author=. 2025 , eprint=

    2025

  56. [70]

    2025 , eprint=

    MediSee: Reasoning-based Pixel-level Perception in Medical Images , author=. 2025 , eprint=

    2025

  57. [71]

    2026 , eprint=

    PRS-Med: Position Reasoning Segmentation in Medical Imaging , author=. 2026 , eprint=

    2026

  58. [72]

    2025 , eprint=

    MedSeg-R: Reasoning Segmentation in Medical Images with Multimodal Large Language Models , author=. 2025 , eprint=

    2025

  59. [73]

    2026 , eprint=

    IBISAgent: Reinforcing Pixel-Level Visual Reasoning in MLLMs for Universal Biomedical Object Referring and Segmentation , author=. 2026 , eprint=

    2026

  60. [74]

    2026 , eprint=

    SAMTok: Representing Any Mask with Two Words , author=. 2026 , eprint=

    2026

  61. [75]

    Proceedings of the IEEE/CVF International Conference on Computer Vision , pages=

    HiMTok: Learning Hierarchical Mask Tokens for Image Segmentation with Large Multimodal Model , author=. Proceedings of the IEEE/CVF International Conference on Computer Vision , pages=

  62. [76]

    Nature Communications , volume=

    Towards generalist foundation model for radiology by leveraging web-scale 2D&3D medical data , author=. Nature Communications , volume=. 2025 , doi=

    2025

  63. [77]

    Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies , pages=

    VividMed: Vision Language Model with Versatile Visual Grounding for Medicine , author=. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies , pages=

    2025

  64. [78]

    Advances in Neural Information Processing Systems , volume=

    Chain-of-Thought Prompting Elicits Reasoning in Large Language Models , author=. Advances in Neural Information Processing Systems , volume=

  65. [79]

    2023 , eprint=

    Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic , author=. 2023 , eprint=

    2023

  66. [80]

    The Twelfth International Conference on Learning Representations , year=

    Kosmos-2: Grounding Multimodal Large Language Models to the World , author=. The Twelfth International Conference on Learning Representations , year=

  67. [81]

    arXiv preprint arXiv:2511.21631 , year=

    Qwen3-vl technical report , author=. arXiv preprint arXiv:2511.21631 , year=

    arXiv

  68. [82]

    Proceedings of the AAAI Conference on Artificial Intelligence , volume=

    Beyond n-grams: A hierarchical reward learning framework for clinically-aware medical report generation , author=. Proceedings of the AAAI Conference on Artificial Intelligence , volume=

  69. [83]

    International Conference on Medical Image Computing and Computer-Assisted Intervention , pages=

    V2t-cot: From vision to text chain-of-thought for medical reasoning and diagnosis , author=. International Conference on Medical Image Computing and Computer-Assisted Intervention , pages=. 2025 , organization=

    2025

  70. [84]

    arXiv preprint arXiv:2510.08668 , year=

    Hulu-med: A transparent generalist model towards holistic medical vision-language understanding , author=. arXiv preprint arXiv:2510.08668 , year=

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.