Pith. sign in

REVIEW 4 major objections 6 minor 127 references

ClinFusion: A Vision-Centric Multimodal LLM System for Holistic Medical Understanding

T0 review · 4 major / 6 minor · reviewed 2026-07-31 · grok-4.5

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A cascaded multi-encoder vision stack plus region-grounded scoring lets a medical multimodal model match radiologists more closely than prior open systems and several proprietary APIs.

desk verdict Solid systems paper with a real multi-encoder+native-3D design and a useful radiologist study; flagship report SOTA numbers are soft until train/test decontamination is shown. read the letter →

arxiv 2607.24743 v2 pith:C5CBTD7Y submitted 2026-07-27 cs.CV cs.AIcs.CL

Hangjie Yuan , Yichen Qian , Zhiwei Tang , Xianzhe Xu , Lirong Wu , Sicheng Yang , Jinwang Wang , Pengju Wang
show 16 more authors
Zhitao Zeng Yizeng Han Yan Xing Shengxuan Luo Tao Feng Qing Xie Weigen Yao Yi Yang Zuozhu Liu Jiasheng Tang Shaocheng Wang Jitao Wang Jiahong Dong Weihua Chen Feng Xu Fan Wang
This is my paper · ORCID
classification cs.CVcs.AIcs.CL
keywords multimodallargelanguagemodelsmedicalimagingcompositionalvisionencoderCaSLFusion3DCT/MRIunderstandingreportgenerationevaluationinstructionfollowingagentictooluse
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

ClinFusion argues that medical multimodal models fail in the clinic mainly because they see poorly and are scored poorly. A single vision backbone cannot absorb heterogeneous 2D and native 3D medical images, and standard report metrics reward wording rather than factual, region-level diagnosis. The authors build a compositional vision encoder that starts from a strong base vision transformer and progressively fuses specialist 2D features, then anchors native 3D features to those 2D representations through Cascade Spatial-Aware Locality Fusion. They also introduce MedIF-Bench for medical instruction-following and an RoI-grounded report protocol that conditions generation on clinical indication and anatomical focus, then scores matched, missed, and hallucinated findings with an LLM judge. On a large suite of 2D/3D VQA, report, instruction-following, and text medical tasks, ClinFusion leads most open medical models and many proprietary multimodal comparisons; six radiologists ranking 300 cases put its reports first, and the RoI metric correlates best with their judgment. Agentic retrieval and specialist perception tools further lift performance in realistic workflows.

What carries the argument

Cascade Spatial-Aware Locality Fusion (CaSL Fusion): an asymmetric, cascaded local cross-attention operator that keeps a foundational well-aligned vision representation as the query backbone, enriches it from specialist 2D encoders in a spatial neighborhood, and, for volumes, uses 2D-anchored depth-aware fusion so native 3D features align through those 2D anchors.

What would settle it

A blinded multi-radiologist study on held-out cases where RoI context is independently verified for leakage, claim labels are audited against expert adjudication, and ClinFusion no longer ranks first or the RoI metric no longer best correlates with expert rankings would overturn the central clinical claims.

Watch

Extended reading notes

Core claim

Holistic medical understanding is achieved by treating the problem as vision-centric: a cascaded compositional encoder with CaSL Fusion unifies diverse 2D and native 3D medical images into a progressively enriched, language-aligned representation, and RoI-grounded evaluation plus MedIF-Bench make instruction-following and report quality measurable in clinically meaningful, factual terms—yielding SOTA open-model results, competitive proprietary multimodal performance, top radiologist rankings, and the automatic metric most aligned with experts.

Load-bearing premise

That clinical context pulled from ground-truth reports without leaking findings, plus an LLM judge labeling matched, missed, and hallucinated claims, is a fair enough yardstick of report quality and model ranking.

Editorial extensions

If this is right

  • Medical MLLMs should keep a strong base encoder and cascade specialist and native 3D features rather than forcing all volumes into 2D slice stacks or a single monolithic encoder.
  • Report generation benchmarks should condition on indication and anatomical focus and score factual matched/missed/hallucinated findings instead of only lexical or closed-schema entity overlap.
  • Instruction-following must be measured after medical fine-tuning; format failure alone can make knowledge look weaker than it is.
  • Agentic retrieval and specialist perception tools can raise both text and multimodal clinical utility without replacing the core vision stack.
  • Radiologist ranking studies can validate which automatic metrics actually track clinical quality.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If RoI extraction becomes standard, public leaderboards may reorder models that today look strong only under context-free or surface-matching scores.
  • The 2D-anchored 3D design suggests future modalities (e.g., dynamic ultrasound or multi-phase MRI) could plug in as additional specialists without full re-alignment of the language model.
  • Preserving instruction-following through medical adaptation may matter as much as raw VQA points for electronic-health-record integration.
  • Hospital-scale pseudo-reports from segmentation tools imply that volumetric supervision can grow faster than expert-written reports if quality control holds.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper presents ClinFusion, a medical MLLM built on Qwen3-VL (8B/32B) with a compositional vision encoder: specialist 2D encoders (DINOv2, ConvNeXt) and a native 3D encoder (PE-3D) are fused into the Qwen ViT representation via a cascaded local cross-attention operator (CaSL Fusion), including a 2D-anchored depth-aware variant for volumetric data. A five-stage progressive training recipe on 22.2M curated samples is described. On the evaluation side, the paper introduces MedIF-Bench (900 regex-verified instruction-following items) and an RoI-grounded report-generation protocol: clinical indication and areas of focus are extracted from the ground-truth report with a leakage-blocking prompt, the model generates a focused report, and an LLM judge (GPT-4.1) decomposes findings into matched/missed/hallucinated sets yielding P/R/F1. Headline claims: SOTA on 20/24 benchmarks vs. open-source medical MLLMs, better than GPT-5.2/Gemini-3-Flash on 13/16 multimodal benchmarks, top-ranked reports in a blinded six-radiologist study (300 cases), and the highest expert correlation (τ=0.511) among eleven automatic metrics. An agentic RAG + perception-tool extension gives further gains. Code and weights are released.

Significance. If the results hold, the paper makes three useful contributions: (i) a compositional 2D/3D vision encoder with a concrete, ablated fusion operator (CaSL) that beats parallel fusion and global cross-attention under controlled comparisons; (ii) a clinically motivated report-generation protocol whose design choices (context conditioning, judge-based claim decomposition) are directly ablated against RadGraph-F1, including a length-bias stress test; and (iii) MedIF-Bench, released with code. Particularly creditable are the unified re-benchmarking of all baselines under one kit, the blinded six-radiologist study on 300 cases with inter-annotator agreement reporting, the architecture ablations, and the release of model weights (8B/32B) and code. The 'strongest correlation with expert judgment' claim is supported by a real human study rather than asserted. The agentic extension is incremental but honestly framed as an extension.

major comments (4)
  1. [§2.2, §4.3; Extended Data Tables 1–2] The flagship report-generation and VQA margins are measured on benchmarks whose source datasets appear in the training corpus, and the manuscript contains no decontamination statement. Extended Data Table 1 lists IU-Xray and open-i (Stage 1), CheXpert Plus, PMC-VQA, SLAKE, VQA-RAD, PathVQA and their recaptions (Stage 3), CT-Rate-report and AMOS-MM-report (Stage 4), and 3D-RAD (Stage 5) as training data; Extended Data Table 2 then evaluates on CheXpert-Plus (200 cases), IU-Xray (296), CT-Rate Report (3,039), AMOS-MM Report (400), 3D-RAD, PMC-VQA, SLAKE, PathVQA, and VQA-RAD, with OmniMedVQA partially subsuming SLAKE/VQA-RAD/PathVQA. If training ingested full datasets (or recaptions/pseudo-labels derived from them) without restricting to official train splits and excluding evaluation IDs, the model may have seen the reference reports against which the RoI-grounded F1 is computed, inflating
  2. [§4.2.3, Extended Data Fig. 6] The RoI-grounded protocol conditions report generation on clinical context extracted from the ground-truth report itself, and the only leakage safeguard is a prompt-level 'CRITICAL SECURITY CONSTRAINT' (Extended Data Fig. 6a) with no quantitative audit. Two concrete risks: (i) the extracted 'Areas of Focus' enumerate the anatomical systems the GT discusses, which is itself signal about where findings live; (ii) nothing verifies that GPT-4.1's extraction never smuggles finding content (e.g., the worked example's indication 'known or suspected infectious disease' already biases toward consolidation-type findings present in that GT). Because the same extracted context is given to all compared models the comparison is internally fair, but the absolute F1 numbers — and the claim that this protocol is 'clinically aligned' — depend on leakage being negligible. Please add an audit: e.g., human o
  3. [§2.3, §2.4.1, Fig. 5b–c] The metric-validation argument is partially circular: the authors introduce the RoI-grounded F1, use it as the primary report metric, and then conclude it 'best reflects expert clinical judgment' (§2.3, Fig. 5b). The six-radiologist study is a genuine mitigating strength, but the correlation analysis (Kendall's τ = 0.511, Top-1 accuracy 55.5%) is computed over only four systems (ClinFusion+agent, ClinFusion, Gemini-3-Flash, Hulu-Med), two of which are the authors' own model; with such few items and system-level rankings, discriminating among eleven metrics is noisy, and the per-case τ distribution (Fig. 5c) needs confidence intervals or a bootstrap test to support 'strongest correlation'. Relatedly, the LLM judge's TP/FN/FP labels are never validated against human extraction: please report a human spot-check of the judge's claim decomposition on a sample (per-category precision/recall of
  4. [§2.3, Fig. 5a] In the radiologist study (Fig. 5a) significance is reported only for ClinFusion+Agent vs. Hulu-Med and vs. Gemini-3-Flash on the Overall dimension; the bracket in the caption mentions 'n.s.' without saying which comparison it refers to (presumably standalone ClinFusion vs. ClinFusion+Agent, or a per-dimension test). Since the paper's top-line claim is 'highest-ranked reports,' please clarify all pairwise tests (standalone vs. baselines, standalone vs. agentic) across the three dimensions, and state whether the agentic system's tool outputs (which include external segmentation/classification models) were available to radiologists as evidence or folded into the report text — this affects what exactly was being ranked.
minor comments (6)
  1. [§4.1.1–4.1.2, Eqs. (4), (6), (10)] Cross-reference errors in the Methods: §4.1.1 ('CaSL Fusion with stochastic residual regularization') refers to 'the residual connection in the fusion block (equation (10))', but the residual connection is Eq. (4); Eq. (10) is defined later in §4.1.2. Similarly §4.1.2 says 'residual connections defined in equation (10)' while defining Eq. (10) itself — presumably Eq. (4)/(6) is meant.
  2. [Extended Data Figs. 1, 4] Internal editing artifacts remain in the Extended Data: Chinese annotations such as '2D3D Case 合并（最终选择使⽤）', '⾄威的合并图1', and '下⾯这个被弃⽤了' ('the one below is deprecated') appear in/around Extended Data Figs. 1 and 4. These should be removed before publication.
  3. [§2.3, Fig. 5a] Terminology inconsistency: §2.3 defines the third ranking dimension as 'Clinical Utility', while Fig. 5a and its caption use 'Operability'. Please unify.
  4. [§2.1.2] Reproducibility of proprietary-model comparisons: API snapshot dates/versions for GPT-5.2, Gemini-3-Flash, and Claude-Sonnet-4.5 are not stated, nor is how 3D volumes were provided to them (slice count, montaging). Given the '13 of 16' claim against proprietary models, these details should be documented.
  5. [Extended Data Fig. 6a] Extended Data Fig. 6a's example GT report is internally contradictory ('bilateral minimal pleural effusion... No pleural or pericardial effusion was detected'). If this reflects source-data noise, a sentence on how noisy references affect the F1 protocol would strengthen the evaluation section; a cleaner example would help the reader.
  6. [Abstract, §2.2] The abstract and §2.2 state '20 out of 24 benchmarks' without identifying the four losses; a compact win/loss breakdown (possibly in Supplementary Tables) would make the claim auditable.

Circularity Check

0 steps flagged · score 0.0 of 10

No derivation-circularity: empirical MLLM claims rest on external benchmarks and a blinded radiologist study, not on results true by construction.

full rationale

ClinFusion is an empirical systems paper (compositional vision encoder + CaSL Fusion, MedIF-Bench, RoI-grounded report metric, agentic tools). Its load-bearing claims are comparative performance numbers and a six-radiologist blinded ranking on 300 cases, not a first-principles derivation in which outputs equal fitted or definitional inputs. Architecture choices are supported by ablations against alternative fusion/encoders on the same protocol; SOTA tables compare against independent open-source and proprietary models under a unified kit. The RoI-grounded F1 is author-proposed and used as a primary report metric, but its “strongest correlation with experts” claim is an empirical rank-correlation against external radiologist consensus (vs. eleven other metrics), not a quantity forced by how the metric is defined—and ClinFusion’s clinical advantage is separately corroborated by the blinded human ranking. No self-definitional identity, fitted-parameter-as-prediction, load-bearing self-cited uniqueness theorem, or renamed known law appears in the argument chain. Train/test overlap risks (if any) are contamination/correctness issues outside this circularity rubric. Finding: no significant circularity.

Assumptions & free parameters 5 free parameters · 6 assumptions · 4 invented entities

This is an empirical systems paper. The claim rests on standard deep-learning practice, medical-data assumptions, and several paper-specific modeling and evaluation choices rather than a formal derivation. The largest external dependencies are the Qwen3-VL stack, chosen specialist encoders, synthetic/pseudo labels, and LLM judges.

free parameters (5)
  • CaSL local window sizes (k=3 DINOv2/3D; k=5 ConvNeXt) = k=3 / k=5
    Hand-chosen locality widths that define how much specialist context each foundation token can see; ablated only coarsely via mechanism comparisons.
  • Stochastic residual drop probability p_drop = 0.1
    Training-only Bernoulli gate to force use of specialist paths; default 0.1 without extensive sensitivity analysis in the main text.
  • 2D anchor slice count S and 3D depth tokens D_vol = S=4, D_vol=4
    Controls how much of a volume is represented via 2D anchors versus native 3D features (S=4, D_vol=4 from 32-frame inputs).
  • Progressive training schedule (stages, epochs, LR, batch sizes) = LR 1e-5; stages 1–5 epochs 1/1/3/1/3; batch 128–256
    Curriculum and optimization hyperparameters that materially affect whether fusion modules and the 3D encoder learn useful features.
  • LLM-as-judge model choice and prompts for open VQA and RoI factuality = gpt-4.1-2025-04-14
    Primary scoring machinery for open answers and report F1 depends on a specific proprietary judge (gpt-4.1-2025-04-14) and long prompts.
assumptions (6)
  • domain assumption Cross-attention over projected, spatially resized encoder maps can enrich a language-aligned foundation ViT without destroying vision-language alignment.
    Core premise of CaSL Fusion and DeepStack integration (Methods 4.1); supported by ablations but not theoretically guaranteed.
  • domain assumption A small set of 2D slices plus a pre-aligned 3D encoder suffices to transfer 2D semantics into volumetric understanding.
    2D-anchored depth-aware fusion (Methods 4.1.2); 3D-off ablation shows drops but does not prove clinical sufficiency.
  • ad hoc to paper Radiology reports can be fairly scored by restricting generation to RoIs inferred from the reference report and counting finding-level TP/FN/FP with an LLM judge.
    Defines the paper’s preferred report metric (Methods 4.2.3); central to report SOTA and expert-correlation claims.
  • ad hoc to paper Format compliance on MedIF-Bench is a valid prerequisite signal for clinical usability and benchmark evaluability.
    Introduced to explain degraded medical fine-tunes and low scores of models like MedGemma on other suites (§2.2.3).
  • domain assumption Segmentation-tool pseudo-reports and dual-VLM dense captions are good enough supervision for perception gains at scale.
    Data synthesis pipelines (Section 4.3 / Supp S1) underpin much of the 3D and caption training signal.
  • standard math Standard transformer training, contrastive 3D pre-alignment, and benchmark accuracy/F1 comparisons are appropriate evidence for ‘holistic medical understanding.’
    Ordinary ML evaluation assumptions used throughout Results; no formal proof obligations.
invented entities (4)
  • CaSL Fusion (Cascade Spatial-Aware Locality Fusion), including 2D-anchored depth-aware variant independent evidence
    purpose: Progressively fuse specialist 2D and native 3D features into a foundation ViT representation with local spatial bias.
    Named operator is the architectural centerpiece; independent evidence is empirical ablations and downstream benchmarks, not external theory.
  • MedIF-Bench independent evidence
    purpose: Measure medical instruction/format following separately from medical correctness.
    New 900-sample benchmark constructed by the authors; usefulness depends on regex tasks reflecting real deployment needs.
  • RoI-grounded report generation evaluation independent evidence
    purpose: Context-condition report generation and score matched/missed/hallucinated abnormalities via LLM judge.
    Paper-defined metric used for primary report tables; partially validated by radiologist correlation study.
  • ClinFusion agentic tool-use extension (hybrid RAG + perception experts) independent evidence
    purpose: Add retrieval and specialist vision tools on top of the base MLLM for verifiable clinical workflows.
    System-level wrapper rather than a new physical entity; gains reported mainly as add-on experiments.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ClinFusion: A Vision-Centric Multimodal LLM System for Holistic Medical Understanding." pith.science (2026). https://pith.science/paper/C5CBTD7Y

@misc{pith2026260724743,
  author       = {Pith},
  title        = {Pith review of: ClinFusion: A Vision-Centric Multimodal LLM System for Holistic Medical Understanding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/C5CBTD7Y}},
  note         = {Machine review of arXiv:2607.24743}
}
read the original abstract

Multimodal large language models (MLLMs) hold immense potential to revolutionize clinical practice, yet deploying them in the medical domain is fundamentally a vision-centric challenge: models must absorb knowledge from heterogeneous 2D and 3D medical images, and evaluation protocols must align with radiologists' clinical practice and provide an accurate, fine-grained and factualness-driven assessment. In this paper, we introduce ClinFusion, a vision-centric MLLM designed for holistic medical understanding that systematically addresses these limitations. We propose a compositional and cascaded vision encoder architecture featuring a Cascade Spatial-Aware Locality Fusion operator that unifies diverse 2D and native 3D medical image understanding within a fused encoder. We further introduce a vision-grounded evaluation framework, including MedIF-Bench for instruction-following assessment and a region-of-interest-grounded method for clinically aligned and factualness-driven report generation evaluation. We show that ClinFusion sets a new state-of-the-art across a comprehensive suite of 2D and 3D multimodal medical benchmarks---spanning visual question answering, report generation, and instruction following---as well as textual medical tasks, outperforming leading open-source medical MLLMs (\textit{e.g.}, Hulu-Med, Lingshu) on 20 out of 24 benchmarks and demonstrating multimodal capabilities better than powerful proprietary models such as GPT-5.2 and Gemini-3-Flash on 13 out of 16 benchmarks, and can be further augmented with agentic tool use for retrieval-augmented and tool-assisted clinical workflows. A blinded evaluation by board-certified radiologists confirms that ClinFusion produces the highest-ranked reports, and validates our RoI-grounded metric as achieving the strongest correlation with expert judgment among all automatic evaluation metrics examined.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

127 extracted references · 5 canonical work pages

  1. [1]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report.arXiv preprint arXiv:2502.13923, 2025

    arXiv 2025

  2. [2]

    Qwen3-vl technical report.arXiv preprint arXiv:2511.21631, 2025

    Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, Wenbin Ge, Zhifang Guo, Qidong Huang, Jie Huang, Fei Huang, Binyuan Hui, Shutong Jiang, Zhaohai Li, Mingsheng Li, Mei Li, Kaixin Li, Zicheng Lin, Junyang Lin, Xuejing Liu, Jiawei Liu, Chenglong Liu, Yang Liu, Dayiheng Liu, Shixuan ...

    arXiv 2025

  3. [3]

    Visual instruction tuning.Advances in neural information processing systems, 36:34892–34916, 2023

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning.Advances in neural information processing systems, 36:34892–34916, 2023

    2023

  4. [4]

    Gpt-4o system card.arXiv preprint arXiv:2410.21276, 2024

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card.arXiv preprint arXiv:2410.21276, 2024

    arXiv 2024

  5. [5]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 26296–26306, 2024

    2024

  6. [6]

    Lingshu: A generalist foundation model for unified multimodal medical understanding and reasoning.arXiv preprint arXiv:2506.07044, 2025

    Weiwen Xu, Hou Pong Chan, Long Li, Mahani Aljunied, Ruifeng Yuan, Jianyu Wang, Chenghao Xiao, Guizhen Chen, Chaoqun Liu, Zhaodonghui Li, et al. Lingshu: A generalist foundation model for unified multimodal medical understanding and reasoning.arXiv preprint arXiv:2506.07044, 2025. 20

    arXiv 2025

  7. [7]

    Hulu-med: A transparent generalist model towards holistic medical vision-language understanding.arXiv preprint arXiv:2510.08668, 2025

    Songtao Jiang, Yuan Wang, Sibo Song, Tianxiang Hu, Chenyi Zhou, Bin Pu, Yan Zhang, Zhibo Yang, Yang Feng, Joey Tianyi Zhou, et al. Hulu-med: A transparent generalist model towards holistic medical vision-language understanding.arXiv preprint arXiv:2510.08668, 2025

    arXiv 2025

  8. [8]

    Medgemma technical report.arXiv preprint arXiv:2507.05201, 2025

    Andrew Sellergren, Sahar Kazemzadeh, Tiam Jaroensri, Atilla Kiraly, Madeleine Traverse, Timo Kohlberger, Shawn Xu, Fayaz Jamil, Cían Hughes, Charles Lau, et al. Medgemma technical report.arXiv preprint arXiv:2507.05201, 2025

    arXiv 2025

Show all 127 references
  1. [10]

    Medvlm-r1: Incentivizing medical reasoning capability of vision-language models (vlms) via reinforcement learning

    Jiazhen Pan, Che Liu, Junde Wu, Fenglin Liu, Jiayuan Zhu, Hongwei Bran Li, Chen Chen, Cheng Ouyang, and Daniel Rueckert. Medvlm-r1: Incentivizing medical reasoning capability of vision-language models (vlms) via reinforcement learning. InInternational Conference on Medical Ima...

    2025

  2. [11]

    Med- r1: Reinforcement learning for generalizable medical reasoning in vision-language models.arXiv preprint arXiv:2503.13939, 2025

    Yuxiang Lai, Jike Zhong, Ming Li, Shitian Zhao, Yuheng Li, Konstantinos Psounis, and Xiaofeng Yang. Med- r1: Reinforcement learning for generalizable medical reasoning in vision-language models.arXiv preprint arXiv:2503.13939, 2025

    2025

  3. [12]

    Huatuogpt-vision, towards injecting medical visual knowledge into multimodal llms at scale.arXiv preprint arXiv:2406.19280, 2024

    Junying Chen, Chi Gui, Ruyi Ouyang, Anningzhe Gao, Shunian Chen, Guiming Hardy Chen, Xidong Wang, Ruifei Zhang, Zhenyang Cai, Ke Ji, et al. Huatuogpt-vision, towards injecting medical visual knowledge into multimodal llms at scale.arXiv preprint arXiv:2406.19280, 2024

    2024 arXiv

  4. [13]

    A generalist vision–language foundation model for diverse biomedical tasks.Nature Medicine, 30(11):3129–3141, 2024

    Kai Zhang, Rong Zhou, Eashan Adhikarla, Zhiling Yan, Yixin Liu, Jun Yu, Zhengliang Liu, Xun Chen, Brian D Davison, Hui Ren, et al. A generalist vision–language foundation model for diverse biomedical tasks.Nature Medicine, 30(11):3129–3141, 2024

    2024

  5. [14]

    Omn- imedvqa: A new large-scale comprehensive evaluation benchmark for medical lvlm

    Yutao Hu, Tianbin Li, Quanfeng Lu, Wenqi Shao, Junjun He, Yu Qiao, and Ping Luo. Omn- imedvqa: A new large-scale comprehensive evaluation benchmark for medical lvlm. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 22170– 22183,...

    2024

  6. [15]

    Slake: A semantically-labeled knowledge- enhanced dataset for medical visual question answering

    Bo Liu, Li-Ming Zhan, Li Xu, Lin Ma, Yan Yang, and Xiao-Ming Wu. Slake: A semantically-labeled knowledge- enhanced dataset for medical visual question answering. In2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI), pages 1650–1654, 2021. URLhttps://ieeexplore...

    2021

  7. [16]

    Generalist foundation models from a multimodal dataset for 3d computed tomography.Nature Biomedical Engineering, pages 1–19, 2026

    Ibrahim Ethem Hamamci, Sezgin Er, Chenyu Wang, Furkan Almas, Ayse Gulnihan Simsek, Sevval Nil Esirgun, Irem Dogan, Omer Faruk Durugol, Benjamin Hou, Suprosanna Shit, et al. Generalist foundation models from a multimodal dataset for 3d computed tomography.Nature Biomedical Engi...

    2026

  8. [17]

    Measuring massive multitask language understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. InInternational Conference on Learning Representations,

  9. [18]

    Medxpertqa: Benchmarking expert-level medical reasoning and understanding

    Yuxin Zuo, Shang Qu, Yifei Li, Zhang-Ren Chen, Xuekai Zhu, Ermo Hua, Kaiyan Zhang, Ning Ding, and Bowen Zhou. Medxpertqa: Benchmarking expert-level medical reasoning and understanding. InInternational Conference on Machine Learning, pages 80961–80990. PMLR, 2025

    2025

  10. [19]

    Pubmedqa: A dataset for biomedical research question answering

    Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. Pubmedqa: A dataset for biomedical research question answering. InProceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natura...

    2019

  11. [20]

    Towards a holistic framework for multimodal LLM in 3D brain CT radiology report generation.Nature Communications, 16(1):2258, 2025

    Cheng-Yi Li, Kao-Jung Chang, Cheng-Fu Yang, Hsin-Yu Wu, Wenting Chen, Hritik Bansal, Ling Chen, Yi-Ping Yang, Yu-Chun Chen, Shih-Pin Chen, et al. Towards a holistic framework for multimodal LLM in 3D brain CT radiology report generation.Nature Communications, 16(1):2258, 2025....

    2025 doi

  12. [21]

    Med3dvlm: An efficient vision-language model for 3d medical image analysis.IEEE Journal of Biomedical and Health Informatics, 2025

    Yu Xin, Gorkem Can Ates, Kuang Gong, and Wei Shao. Med3dvlm: An efficient vision-language model for 3d medical image analysis.IEEE Journal of Biomedical and Health Informatics, 2025

    2025

  13. [22]

    Towards generalist foundation model for radiology by leveraging web-scale 2d&3d medical data.Nature Communications, 16(1):7866, 2025

    Chaoyi Wu, Xiaoman Zhang, Ya Zhang, Hui Hui, Yanfeng Wang, and Weidi Xie. Towards generalist foundation model for radiology by leveraging web-scale 2d&3d medical data.Nature Communications, 16(1):7866, 2025. 21

    2025

  14. [23]

    Chaunzwa, Simon Bernatz, Ahmed Hosny, Raymond H

    Suraj Pai, Dennis Bontempi, Ibrahim Hadzic, Vasco Prudente, Mateo Sokač, Tafadzwa L. Chaunzwa, Simon Bernatz, Ahmed Hosny, Raymond H. Mak, Nicolai J. Birkbak, and Hugo J. W. L. Aerts. Foundation model for cancer imaging biomarkers.Nature Machine Intelligence, 6(3):354–367, 202...

    2024 doi

  15. [24]

    Eagle: Exploring the design space for multimodal llms with mixture of encoders

    Min Shi, Fuxiao Liu, Shihao Wang, Shijia Liao, Subhashree Radhakrishnan, Yilin Zhao, De-An Huang, Hongxu Yin, Karan Sapra, Yaser Yacoob, et al. Eagle: Exploring the design space for multimodal llms with mixture of encoders. InInternational Conference on Learning Representation...

    2025

  16. [25]

    Eagle-2: Faster inference of language models with dynamic draft trees

    Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. Eagle-2: Faster inference of language models with dynamic draft trees. InProceedings of the 2024 conference on empirical methods in natural language processing, pages 7421–7432, 2024

    2024

  17. [26]

    Cambrian-1: A fully open, vision-centric exploration of multimodal llms.Advances in Neural Information Processing Systems, 37:87310–87356, 2024

    Peter Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Adithya Jairam Vedagiri IYER, Sai Charitha Akula, Shusheng Yang, Jihan Yang, Manoj Middepogu, Ziteng Wang, et al. Cambrian-1: A fully open, vision-centric exploration of multimodal llms.Advances in Neural Information Processin...

    2024

  18. [27]

    Mini-gemini: Mining the potential of multi-modality vision language models.IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025

    Yanwei Li, Yuechen Zhang, Chengyao Wang, Zhisheng Zhong, Yixin Chen, Ruihang Chu, Shaoteng Liu, and Jiaya Jia. Mini-gemini: Mining the potential of multi-modality vision language models.IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025

    2025

  19. [28]

    Prismatic vlms: Investigating the design space of visually-conditioned language models

    Siddharth Karamcheti, Suraj Nair, Ashwin Balakrishna, Percy Liang, Thomas Kollar, andDorsa Sadigh. Prismatic vlms: Investigating the design space of visually-conditioned language models. InForty-first International Conference on Machine Learning, 2024

    2024

  20. [29]

    Feast your eyes: Mixture- of-resolution adaptation for multimodal large language models

    Gen Luo, Yiyi Zhou, Yuxin Zhang, Xiawu Zheng, Xiaoshuai Sun, and Rongrong Ji. Feast your eyes: Mixture- of-resolution adaptation for multimodal large language models. InInternational Conference on Learning Representations, volume 2025, pages 84491–84506, 2025

    2025

  21. [30]

    Rouge: A package for automatic evaluation of summaries

    Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. InText summarization branches out, pages 74–81, 2004

    2004

  22. [31]

    Meteor: An automatic metric for mt evaluation with improved correlation with human judgments

    Satanjeev Banerjee and Alon Lavie. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. InProceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pages 65–72, 2005

    2005

  23. [32]

    Cider: Consensus-based image description evaluation

    Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus-based image description evaluation. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 4566–4575, 2015

    2015

  24. [33]

    Weinberger, and Yoav Artzi

    Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. InInternational Conference on Learning Representations, 2020

    2020

  25. [34]

    Improving the factual correctness of radiology report generation with semantic rewards

    Jean-Benoit Delbrouck, Pierre Chambon, Christian Bluethgen, Emily Tsai, Omar Almusa, and Curtis Langlotz. Improving the factual correctness of radiology report generation with semantic rewards. InFindings of the Association for Computational Linguistics: EMNLP 2022, pages 4348...

    2022

  26. [35]

    Ratescore: A metric for radiology report generation

    Weike Zhao, Chaoyi Wu, Xiaoman Zhang, Ya Zhang, Yanfeng Wang, and Weidi Xie. Ratescore: A metric for radiology report generation. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 15004–15019, 2024

    2024

  27. [36]

    Green: Generative radiology report evaluation and error notation

    Sophie Ostmeier, Justin Xu, Zhihong Chen, Maya Varma, Louis Blankemeier, Christian Bluethgen, Arne Edward Michalson Md, Michael Moseley, Curtis Langlotz, Akshay S Chaudhari, et al. Green: Generative radiology report evaluation and error notation. InFindings of the association ...

    2024

  28. [37]

    Jianing Qiu, Kyle Lam, Guohao Li, Amish Acharya, Tien Yin Wong, Ara Darzi, Wu Yuan, and Eric J. Topol. Llm-based agentic systems in medicine and healthcare.Nature Machine Intelligence, 6(12):1418–1420, 2024

    2024

  29. [38]

    A survey of llm-based agents in medicine: How far are we from baymax?Findings of the Association for Computational Linguistics: ACL 2025, pages 10345–10359, 2025

    Wenxuan Wang, Zizhan Ma, Zheng Wang, Chenghan Wu, Jiaming Ji, Wenting Chen, Xiang Li, and Yixuan Yuan. A survey of llm-based agents in medicine: How far are we from baymax?Findings of the Association for Computational Linguistics: ACL 2025, pages 10345–10359, 2025

    2025

  30. [39]

    Medagent- pro: Towards evidence-based multi-modal medical diagnosis via reasoning agentic workflow.arXiv preprint arXiv:2503.18968, 2025

    Ziyue Wang, Junde Wu, Linghan Cai, Chang Han Low, Xihong Yang, Qiaxuan Li, and Yueming Jin. Medagent- pro: Towards evidence-based multi-modal medical diagnosis via reasoning agentic workflow.arXiv preprint arXiv:2503.18968, 2025. 22

    2025 arXiv

  31. [40]

    Mmedagent: Learning to use medical tools with multi-modal agent

    Binxu Li, Tiankai Yan, Yuanting Pan, Jie Luo, Ruiyang Ji, Jiayuan Ding, Zhe Xu, Shilong Liu, Haoyu Dong, Zihao Lin, et al. Mmedagent: Learning to use medical tools with multi-modal agent. InFindings of the Association for Computational Linguistics: EMNLP 2024, pages 8745–8760, 2024

    2024

  32. [41]

    Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models.arXiv preprint arXiv:2504.10479, 2025

    Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, et al. Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models.arXiv preprint arXiv:2504.10479, 2025

    2025 arXiv

  33. [42]

    Bimedix2: Bio-medical expert lmm for diverse medical modalities

    Sahal Shaji Mullappilly, Mohammed Irfan Kurpath, Sara Pieri, Saeed Yahya Alseiari, Shanavas Cholakkal, Khaled Aldahmani, Fahad Khan, Rao Anwer, Salman Khan, Timothy Baldwin, et al. Bimedix2: Bio-medical expert lmm for diverse medical modalities. InFindings of the association f...

    2025

  34. [43]

    Healthgpt: A medical large vision-language model for unifying comprehension and generation via heterogeneous knowledge adaptation

    Tianwei Lin, Wenqiao Zhang, Sijing Li, Yuqian Yuan, Binhe Yu, Haoyuan Li, Wanggui He, Hao Jiang, Mengze Li, Song Xiaohui, et al. Healthgpt: A medical large vision-language model for unifying comprehension and generation via heterogeneous knowledge adaptation. InInternational C...

    2025

  35. [44]

    Gsco: Towards generalizable ai in medicine via generalist-specialist collaboration

    Sunan He, Yuxiang Nie, Hongmei Wang, Shu Yang, Yihui Wang, Zhiyuan Cai, Zhixuan Chen, Yingxue Xu, Luyang Luo, Huiling Xiang, et al. Gsco: Towards generalizable ai in medicine via generalist-specialist collaboration. arXiv preprint arXiv:2404.15127, 2024

    2024 arXiv

  36. [45]

    Ryutaro Tanno, David G. T. Barrett, Andrew Sellergren, Sumedh Ghaisas, Sumanth Dathathri, Abigail See, Johannes Welbl, Charles Lau, Tao Tu, Shekoofeh Azizi, et al. Collaboration between clinicians and vision–language models in radiology report generation.Nature Medicine, 31(2)...

    2025 doi

  37. [47]

    3d-rad: A comprehensive 3d radiology med-vqa dataset with multi-temporal analysis and diverse diagnostic tasks.arXiv preprint arXiv:2506.11147, 2025

    Xiaotang Gai, Jiaxiang Liu, Yichen Li, Zijie Meng, Jian Wu, and Zuozhu Liu. 3d-rad: A comprehensive 3d radiology med-vqa dataset with multi-temporal analysis and diverse diagnostic tasks.arXiv preprint arXiv:2506.11147, 2025

    2025

  38. [48]

    Amos: A large-scale abdominal multi-organ benchmark for versatile medical image segmentation.Advances in neural information processing systems, 35:36722–36732, 2022

    Yuanfeng Ji, Haotian Bai, Chongjian Ge, Jie Yang, Ye Zhu, Ruimao Zhang, Zhen Li, Lingyan Zhanng, Wanling Ma, Xiang Wan, et al. Amos: A large-scale abdominal multi-organ benchmark for versatile medical image segmentation.Advances in neural information processing systems, 35:367...

    2022

  39. [49]

    Bleu: a method for automatic evaluation of machine translation

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. InProceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318, 2002

    2002

  40. [50]

    RadGraph-XL: A large-scale expert-annotated dataset for entity and relation extraction from radiology reports

    Jean-Benoit Delbrouck, Pierre Chambon, Zhihong Chen, Maya Varma, Andrew Johnston, Louis Blankemeier, Dave Van Veen, Tan Bui, Steven Truong, and Curtis Langlotz. RadGraph-XL: A large-scale expert-annotated dataset for entity and relation extraction from radiology reports. In Lu...

    2024

  41. [51]

    Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity.Journal of Machine Learning Research, 23(120):1–39, 2022

    William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity.Journal of Machine Learning Research, 23(120):1–39, 2022

    2022

  42. [52]

    Position: Compositional generative modeling: A single model is not all you need

    Yilun Du and Leslie Pack Kaelbling. Position: Compositional generative modeling: A single model is not all you need. InInternational Conference on Machine Learning, pages 11721–11732. PMLR, 2024

    2024

  43. [53]

    A convnet for the 2020s.Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022

    Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s.Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022

    2022

  44. [54]

    Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V. Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel HAZIZA, Francisco Massa, Alaaeldin El-Nouby, Mido Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po-Yao Huang, Shang-Wen Li, Ishan Misra, Michael Rabba...

    2024

  45. [55]

    Exploring scalable medical image encoders beyond text supervision.Nature Machine Intelligence, 7(1):119–130, 2025

    Fernando Pérez-García, Harshita Sharma, Sam Bond-Taylor, Kenza Bouzid, Valentina Salvatelli, Maximilian Ilse, Shruthi Bannur, Daniel C Castro, Anton Schwaighofer, Matthew P Lungren, et al. Exploring scalable medical image encoders beyond text supervision.Nature Machine Intelli...

    2025

  46. [56]

    A multimodal biomedical foundation model trained from fifteen million image–text pairs.Nejm Ai, 2(1):AIoa2400640, 2025

    Sheng Zhang, Yanbo Xu, Naoto Usuyama, Hanwen Xu, Jaspreet Bagga, Robert Tinn, Sam Preston, Rajesh Rao, Mu Wei, Naveen Valluri, et al. A multimodal biomedical foundation model trained from fifteen million image–text pairs.Nejm Ai, 2(1):AIoa2400640, 2025

    2025

  47. [57]

    Visionzip: Longer is better but not necessary in vision language models

    Senqiao Yang, Yukang Chen, Zhuotao Tian, Chengyao Wang, Jingyao Li, Bei Yu, and Jiaya Jia. Visionzip: Longer is better but not necessary in vision language models. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 19792–19802, 2025

    2025

  48. [58]

    Sparsevlm: Visual token sparsification for efficient vision-language model inference

    Yuan Zhang, Chun-Kai Fan, Junpeng Ma, Wenzhao Zheng, Tao Huang, Kuan Cheng, Denis A Gudovskiy, Tomoyuki Okuno, Yohei Nakata, Kurt Keutzer, et al. Sparsevlm: Visual token sparsification for efficient vision-language model inference. InInternational Conference on Machine Learnin...

    2025

  49. [59]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. InNeurIPS, pages 5998–6008, 2017

    2017

  50. [60]

    Deepstack: Deeply stacking visual tokens is surprisingly simple and effective for lmms.Advances in Neural Information Processing Systems, 37:23464–23487, 2024

    Lingchen Meng, Jianwei Yang, Rui Tian, Xiyang Dai, Zuxuan Wu, Jianfeng Gao, and Yu-Gang Jiang. Deepstack: Deeply stacking visual tokens is surprisingly simple and effective for lmms.Advances in Neural Information Processing Systems, 37:23464–23487, 2024

    2024

  51. [61]

    Perception encoder: The best visual embeddings are not at the output of the network.arXiv preprint arXiv:2504.13181, 2025

    Daniel Bolya, Po-Yao Huang, Peize Sun, Jang Hyun Cho, Andrea Madotto, Chen Wei, Tengyu Ma, Jiale Zhi, Jathushan Rajasegaran, Hanoona Rasheed, et al. Perception encoder: The best visual embeddings are not at the output of the network.arXiv preprint arXiv:2504.13181, 2025

    2025 arXiv

  52. [62]

    Generalized radiograph representation learning via cross-supervision between images and free-text radiology reports.Nature Machine Intelligence, 4(1):32–40, 2022

    Hong-Yu Zhou, Xiaoyu Chen, Yinghao Zhang, Ruibang Luo, Liansheng Wang, and Yizhou Yu. Generalized radiograph representation learning via cross-supervision between images and free-text radiology reports.Nature Machine Intelligence, 4(1):32–40, 2022. doi: 10.1038/s42256-021-00425-9

    2022 doi

  53. [63]

    Huatuogpt, towards taming language model to be a doctor

    Hongbo Zhang, Junying Chen, Feng Jiang, Fei Yu, Zhihong Chen, Guiming Chen, Jianquan Li, Xiangbo Wu, Zhang Zhiyi, Qingying Xiao, et al. Huatuogpt, towards taming language model to be a doctor. InFindings of the association for computational linguistics: EMNLP 2023, pages 10859...

    2023

  54. [64]

    State of what art? a call for multi-prompt llm evaluation.Transactions of the Association for Computational Linguistics, 12: 933–949, 2024

    Moran Mizrahi, Guy Kaplan, Dan Malkin, Rotem Dror, Dafna Shahaf, and Gabriel Stanovsky. State of what art? a call for multi-prompt llm evaluation.Transactions of the Association for Computational Linguistics, 12: 933–949, 2024

    2024

  55. [65]

    Surveillance, epidemiology, and end results (seer) program (www.seer.cancer.gov), 2025

    National Cancer Institute, Surveillance Research Program. Surveillance, epidemiology, and end results (seer) program (www.seer.cancer.gov), 2025. URLhttps://seer.cancer.gov/data/. SEER database; data analyzed using SEER*Stat. Visit https://seer.cancer.gov/data/

    2025

  56. [66]

    Look again, think slowly: Enhancing visual reflection in vision-language models

    Pu Jian, Junhong Wu, Wei Sun, Chen Wang, Shuo Ren, and Jiajun Zhang. Look again, think slowly: Enhancing visual reflection in vision-language models. InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 9262–9281, 2025

    2025

  57. [67]

    Meng, and Bo Zhao

    Fan Bai, Yuxin Du, Tiejun Huang, Max Q.-H. Meng, and Bo Zhao. M3d: Advancing 3d medical image analysis with multi-modal large language models.arXiv preprint arXiv:2404.00578, 2024

    2024 arXiv

  58. [68]

    Merlin: a computed tomography vision–language foundation model and dataset.Nature, pages 1–11, 2026

    Louis Blankemeier, Ashwin Kumar, Joseph Paul Cohen, Jiaming Liu, Longchao Liu, Dave Van Veen, Syed Jamal Safdar Gardezi, Hongkun Yu, Magdalini Paschali, Zhihong Chen, et al. Merlin: a computed tomography vision–language foundation model and dataset.Nature, pages 1–11, 2026

    2026

  59. [69]

    Inspect: A multimodal dataset for patient outcome prediction of pulmonary embolisms

    Shih-Cheng Huang, Zepeng Huo, Ethan Steinberg, Chia-Chun Chiang, Curtis Langlotz, Matthew Lungren, Serena Yeung, Nigam Shah, and Jason Fries. Inspect: A multimodal dataset for patient outcome prediction of pulmonary embolisms. InAdvances in Neural Information Processing System...

    2023

  60. [70]

    Vista3d: A unified segmentation foundation model for 3d medical imaging

    Yufan He, Pengfei Guo, Yucheng Tang, Andriy Myronenko, Vishwesh Nath, Ziyue Xu, Dong Yang, Can Zhao, Benjamin Simon, Mason Belue, et al. Vista3d: A unified segmentation foundation model for 3d medical imaging. InProceedings of the Computer Vision and Pattern Recognition Confer...

    2025

  61. [71]

    Totalsegmentator: robust segmentation of 104 anatomic structures in ct images.Radiology: Artificial Intelligence, 5(5):e230024, 2023

    Jakob Wasserthal, Hanns-Christian Breit, Manfred T Meyer, Maurice Pradella, Daniel Hinck, Alexander W Sauter, Tobias Heye, Daniel T Boll, Joshy Cyriac, Shan Yang, et al. Totalsegmentator: robust segmentation of 104 anatomic structures in ct images.Radiology: Artificial Intelli...

    2023

  62. [72]

    Dmqr-rag: Diverse multi-query rewriting for rag.arXiv preprint arXiv:2411.13154, 2024

    Zhicong Li, Jiahao Wang, Zhishu Jiang, Hangyu Mao, Zhongxia Chen, Jiazhen Du, Yuanxing Zhang, Fuzheng Zhang, Di Zhang, and Yong Liu. Dmqr-rag: Diverse multi-query rewriting for rag.arXiv preprint arXiv:2411.13154, 2024

    2024 arXiv

  63. [73]

    UMLS knowledge sources, release 2024aa, 2024.http://www.nlm.nih.gov/ research/umls/licensedcontent/umlsknowledgesources.html(accessed 15 July 2024)

    National Library of Medicine (US). UMLS knowledge sources, release 2024aa, 2024.http://www.nlm.nih.gov/ research/umls/licensedcontent/umlsknowledgesources.html(accessed 15 July 2024)

    2024

  64. [74]

    Building a knowledge graph to enable precision medicine

    Payal Chandak, Kexin Huang, and Marinka Zitnik. Building a knowledge graph to enable precision medicine. Scientific Data, 10(1):67, 2023. doi: 10.1038/s41597-023-01960-3

    2023 doi

  65. [75]

    Sayers, Evan E

    Eric W. Sayers, Evan E. Bolton, J. Rodney Brister, et al. Database resources of the national center for biotechnology information.Nucleic Acids Research, 52(D1):D33–D43, 2024. doi: 10.1093/nar/gkad1044

    2024 doi

  66. [76]

    StatPearls Publishing, Treasure Island, FL, 2026

    StatPearls Publishing.StatPearls [Internet]. StatPearls Publishing, Treasure Island, FL, 2026. Available from: https://www.ncbi.nlm.nih.gov/books/NBK430685/(accessed 27 February 2026)

    2026

  67. [77]

    What disease does this patient have? a large-scale open domain question answering dataset from medical exams.Applied Sciences, 11 (14):6421, 2021

    Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. What disease does this patient have? a large-scale open domain question answering dataset from medical exams.Applied Sciences, 11 (14):6421, 2021. doi: 10.3390/app11146421

    2021 doi

  68. [78]

    Xiong, Q

    G. Xiong, Q. Jin, Z. Lu, et al. Benchmarking retrieval-augmented generation for medicine. InFindings of the Association for Computational Linguistics: ACL 2024, pages 6233–6251, 2024

    2024

  69. [79]

    Multi-modal ai for opportunistic screening, staging and progression risk stratification of steatotic liver disease.Nature Communications, 17(1):1562, 2026

    Yuan Gao, Chunli Li, Wanxing Chang, Bai Du, Xianghua Ye, Yee Hui Yeo, Yingda Xia, Heng Guo, Xiaoming Zhang, Wei Liu, et al. Multi-modal ai for opportunistic screening, staging and progression risk stratification of steatotic liver disease.Nature Communications, 17(1):1562, 2026

    2026

  70. [80]

    Effective lymph nodes detection in ct scans using location debiased query selection and contrastive query representation in transformer

    Qinji Yu, Yirui Wang, Ke Yan, Haoshen Li, Dazhou Guo, Li Zhang, Na Shen, Qifeng Wang, Xiaowei Ding, Le Lu, et al. Effective lymph nodes detection in ct scans using location debiased query selection and contrastive query representation in transformer. InEuropean Conference on C...

    2024

  71. [81]

    On the limits of cross-domain generalization in automated x-ray prediction

    Joseph Paul Cohen, Mohammad Hashir, Rupert Brooks, and Hadrien Bertrand. On the limits of cross-domain generalization in automated x-ray prediction. InMedical Imaging with Deep Learning (MIDL), volume 121 ofProceedings of Machine Learning Research, pages 136–155. PMLR, 2020. U...

    2020

  72. [82]

    Kalra, and Pingkun Yan

    Zefan Yang, Xuanang Xu, Jiajin Zhang, Ge Wang, Mannudeep K. Kalra, and Pingkun Yan. Chest x-ray foundation model with global and local representations integration.IEEE Transactions on Medical Imaging, 44 (12):4787–4799, 2025. doi: 10.1109/TMI.2025.3581907

    2025

  73. [83]

    Tsai, Andrew Johnston, Cameron Olsen, Tanishq Mathew Abraham, Sergios Gatidis, Akshay S

    Zhihong Chen, Maya Varma, Jean-Benoit Delbrouck, Magdalini Paschali, Louis Blankemeier, Dave Van Veen, Jeya Maria Jose Valanarasu, Alaa Youssef, Joseph Paul Cohen, Eduardo Pontes Reis, Emily B. Tsai, Andrew Johnston, Cameron Olsen, Tanishq Mathew Abraham, Sergios Gatidis, Aksh...

    2024 arXiv

  74. [84]

    Pmc-clip: Contrastive language-image pre-training using biomedical documents

    Weixiong Lin, Ziheng Zhao, Xiaoman Zhang, Chaoyi Wu, Ya Zhang, Yanfeng Wang, and Weidi Xie. Pmc-clip: Contrastive language-image pre-training using biomedical documents. InInternational Conference on Medical Image Computing and Computer-Assisted Intervention, pages 525–536. Sp...

    2023

  75. [85]

    Friedrich

    Obioma Pelka, Svenja Koitka, Johannes Rückert, Felix Nensa, and Christoph M. Friedrich. Radiology objects in context (roco): A multimodal image dataset. InLarge-Scale Annotation of Biomedical Data and Expert Label Synthesis (LABELS) 2018, held in conjunction with MICCAI 2018, ...

    2018 doi

  76. [86]

    Seco de Herrera, et al

    Johannes Rückert, Louise Bloch, Raphael Brüngel, Ahmad Idrissi-Yaghir, Henning Schäfer, Cynthia S Schmidt, Sven Koitka, Obioma Pelka, Asma Ben Abacha, Alba G. Seco de Herrera, et al. Rocov2: Radiology objects in context version 2, an updated multimodal image dataset.Scientific...

    2024

  77. [87]

    Towards injecting medical visual knowledge into multimodal llms at scale

    Junying Chen, Chi Gui, Ruyi Ouyang, Anningzhe Gao, Shunian Chen, Guiming Hardy Chen, Xidong Wang, Zhenyang Cai, Ke Ji, Xiang Wan, and Benyou Wang. Towards injecting medical visual knowledge into multimodal llms at scale. InProceedings of the 2024 Conference on Empirical Method...

    2024 doi

  78. [88]

    Mimic-cxr, a de-identified publicly available database of chest radiographs with free-text reports.Scientific data, 6(1):317, 2019

    Alistair EW Johnson, Tom J Pollard, Seth J Berkowitz, Nathaniel R Greenbaum, Matthew P Lungren, Chih- ying Deng, Roger G Mark, and Steven Horng. Mimic-cxr, a de-identified publicly available database of chest radiographs with free-text reports.Scientific data, 6(1):317, 2019

    2019

  79. [90]

    Medicat: A dataset of medical images, captions, and textual references

    Sanjay Subramanian, Lucy Lu Wang, Ben Bogin, Sachin Mehta, Madeleine Van Zuylen, Sravanthi Parasa, Sameer Singh, Matt Gardner, and Hannaneh Hajishirzi. Medicat: A dataset of medical images, captions, and textual references. InFindings of the Association for Computational Lingu...

    2020

  80. [91]

    Irene Siragusa, Salvatore Contino, Massimo La Ciura, Rosario Alicata, and Roberto Pirrone. Medpix 2.0: a comprehensive multimodal biomedical data set for advanced ai applications with retrieval augmented generation and knowledge graphs.Data Science and Engineering, pages 1–17, 2025

    2025

  81. [92]

    Alejandro Lozano, Min Woo Sun, James Burgess, Liangyu Chen, Jeffrey J. Nirschl, Jeffrey Gu, Ivan Lopez, Josiah Aklilu, Austin Wolfgang Katzer, Collin Chiu, Anita Rau, Xiaohan Wang, Yuhui Zhang, Alfred Seunghoon Song, Robert Tibshirani, and Serena Yeung-Levy. Biomedica: An open...

    2025

  82. [93]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 26296–26306, June 2024

    2024

  83. [94]

    Molmo and pixmo: Open weights and open data for state-of-the-art vision-language models

    Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang, Jae Sung Park, Mohammadreza Salehi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, et al. Molmo and pixmo: Open weights and open data for state-of-the-art vision-language models. InProceedings of the Computer V...

    2025

  84. [95]

    Llava-med: Training a large language-and-vision assistant for biomedicine in one day

    Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon, and Jianfeng Gao. Llava-med: Training a large language-and-vision assistant for biomedicine in one day. Advances in Neural Information Processing Systems, 36:28541–28...

    2023

  85. [96]

    Quilt- llava: Visual instruction tuning by extracting localized narratives from open-source histopathology videos

    Mehmet Saygin Seyfioglu, Wisdom O Ikezogwo, Fatemeh Ghezloo, Ranjay Krishna, and Linda Shapiro. Quilt- llava: Visual instruction tuning by extracting localized narratives from open-source histopathology videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and P...

    2024

  86. [97]

    Hicks, Vajira Thambawita, Pål Halvorsen, and Michael A

    Sushant Gautam, Andrea Storås, Cise Midoglu, Steven A. Hicks, Vajira Thambawita, Pål Halvorsen, and Michael A. Riegler. Kvasir-vqa: A text-image pair gi tract dataset. InProceedings of the 2nd International Workshop on Vision-Language Models for Biomedical Applications (VLM4Bi...

    2024

  87. [98]

    MIMIC-Ext-MIMIC-CXR-VQA: A complex, diverse, and large-scale visual question answering dataset for chest x-ray images, 2024

    Seongsu Bae, Daeun Kyung, Jaehee Ryu, et al. MIMIC-Ext-MIMIC-CXR-VQA: A complex, diverse, and large-scale visual question answering dataset for chest x-ray images, 2024. PhysioNet,https://physionet.org/ content/mimic-ext-mimic-cxr-vqa/

    2024

  88. [99]

    Towards visual question answering on pathology images

    Xuehai He, Zhuo Cai, Wenlan Wei, Yichen Zhang, Luntian Mou, Eric Xing, and Pengtao Xie. Towards visual question answering on pathology images. InProceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference ...

    2021 doi

  89. [100]

    Development of a large-scale medical visual question-answering dataset.Communications Medicine, 4(1):23, 2024

    Xiaoman Zhang, Chaoyi Wu, Ziheng Zhao, Weixiong Lin, Ya Zhang, Yanfeng Wang, and Weidi Xie. Development of a large-scale medical visual question-answering dataset.Communications Medicine, 4(1):23, 2024. doi: 10.1038/s43856-024-00709-2. 26

    2024 doi

  90. [101]

    Hasan, Vivek V

    Asma Ben Abacha, Sadid A. Hasan, Vivek V. Datla, Joey Liu, Dina Demner-Fushman, and Henning Müller. VQA-Med: Overview of the medical visual question answering task at ImageCLEF 2019. InCLEF 2019 Working Notes, CEUR Workshop Proceedings, volume 2380, 2019. URLhttps://ceur-ws.or...

    2019

  91. [102]

    A dataset of clinically generated visual questions and answers about radiology images.Scientific data, 5(1):1–10, 2018

    Jason J Lau, Soumya Gayen, Asma Ben Abacha, and Dina Demner-Fushman. A dataset of clinically generated visual questions and answers about radiology images.Scientific data, 5(1):1–10, 2018. URLhttps://doi.org/ 10.1038/sdata.2018.251

    2018 doi

  92. [103]

    Gmai-vl-r1: Harnessing reinforcement learning for multimodal medical reasoning.arXiv preprint arXiv:2504.01886, 2025

    Yanzhou Su, Tianbin Li, Jiyao Liu, Chenglong Ma, Junzhi Ning, Cheng Tang, Sibo Ju, Jin Ye, Pengcheng Chen, Ming Hu, et al. Gmai-vl-r1: Harnessing reinforcement learning for multimodal medical reasoning.arXiv preprint arXiv:2504.01886, 2025

    2025 arXiv

  93. [104]

    Allava: Harnessing gpt4v-synthesized data for lite vision-language models.arXiv preprint arXiv:2402.11684, 2024

    Guiming Hardy Chen, Shunian Chen, Ruifei Zhang, Junying Chen, Xiangbo Wu, Zhiyi Zhang, Zhihong Chen, Jianquan Li, Xiang Wan, and Benyou Wang. Allava: Harnessing gpt4v-synthesized data for lite vision-language models.arXiv preprint arXiv:2402.11684, 2024

    2024 arXiv

  94. [105]

    A question-entailment approach to question answering.BMC Bioinformatics, 20(1):511, 2019

    Asma Ben Abacha and Dina Demner-Fushman. A question-entailment approach to question answering.BMC Bioinformatics, 20(1):511, 2019. doi: 10.1186/s12859-019-3119-4

    2019 doi

  95. [106]

    Huatuogpt-o1, towards medical complex reasoning with llms.arXiv preprint arXiv:2412.18925, 2024

    Junying Chen, Zhenyang Cai, Ke Ji, Xidong Wang, Wanlong Liu, Rongsheng Wang, Jianye Hou, and Benyou Wang. Huatuogpt-o1, towards medical complex reasoning with llms.arXiv preprint arXiv:2412.18925, 2024

    2024 arXiv

  96. [107]

    Apollo: A lightweight multilingual medical llm towards democratizing medical ai to 6b people.arXiv preprint arXiv:2403.03640, 2024

    Xidong Wang, Nuo Chen, Junyin Chen, Yidong Wang, Guorui Zhen, Chunxian Zhang, Xiangbo Wu, Yan Hu, Anningzhe Gao, Xiang Wan, et al. Apollo: A lightweight multilingual medical llm towards democratizing medical ai to 6b people.arXiv preprint arXiv:2403.03640, 2024

    2024 arXiv

  97. [108]

    Alpacare: Instruction-tuned large language models for medical application.arXiv preprint arXiv:2310.14558, 2023

    Xinlu Zhang, Chenxin Tian, Xianjun Yang, Lichang Chen, Zekun Li, and Linda Ruth Petzold. Alpacare: Instruction-tuned large language models for medical application.arXiv preprint arXiv:2310.14558, 2023

    2023 arXiv

  98. [109]

    Chatdoctor: A medical chat model fine-tuned on a large language model meta-AI (LLaMA) using medical domain knowledge.Cureus, 15(6): e40895, 2023

    Yunxiang Li, Zihan Li, Kai Zhang, Ruilong Dan, Steve Jiang, and You Zhang. Chatdoctor: A medical chat model fine-tuned on a large language model meta-AI (LLaMA) using medical domain knowledge.Cureus, 15(6): e40895, 2023. doi: 10.7759/cureus.40895

    2023 doi

  99. [110]

    Pmc-llama: toward building open-source language models for medicine.Journal of the American Medical Informatics Association, 31(9):1833–1843, 2024

    Chaoyi Wu, Weixiong Lin, Xiaoman Zhang, Ya Zhang, Weidi Xie, and Yanfeng Wang. Pmc-llama: toward building open-source language models for medicine.Journal of the American Medical Informatics Association, 31(9):1833–1843, 2024

    2024

  100. [111]

    Huatuogpt-ii, one-stage training for medical adaption of llms.arXiv preprint arXiv:2311.09774, 2023

    Junying Chen, Xidong Wang, Anningzhe Gao, Feng Jiang, Shunian Chen, Hongbo Zhang, et al. Huatuogpt-ii, one-stage training for medical adaption of llms.arXiv preprint arXiv:2311.09774, 2023

    2023 arXiv

  101. [112]

    Medreason: Eliciting factual medical reasoning steps in llms via knowledge graphs

    Juncheng Wu, Wenlong Deng, Xingxuan Li, Sheng Liu, Taomian Mi, Yifan Peng, Ziyang Xu, Yi Liu, Hyunjin Cho, Chang-In Choi, et al. Medreason: Eliciting factual medical reasoning steps in llms via knowledge graphs. arXiv preprint arXiv:2504.00993, 2025

    2025 arXiv

  102. [113]

    Medthoughts-8k: A medical reasoning dataset distilled from deepseek-r1.https://huggingface.co/ datasets/hw-hwei/MedThoughts-8K, 2025

    Hao Wei. Medthoughts-8k: A medical reasoning dataset distilled from deepseek-r1.https://huggingface.co/ datasets/hw-hwei/MedThoughts-8K, 2025

    2025

  103. [114]

    CMB: A comprehensive medical benchmark in Chinese

    Xidong Wang, Guiming Chen, Song Dingjie, Zhang Zhiyi, Zhihong Chen, Qingying Xiao, Junying Chen, Feng Jiang, Jianquan Li, Xiang Wan, Benyou Wang, and Haizhou Li. CMB: A comprehensive medical benchmark in Chinese. InProceedings of the 2024 Conference of the North American Chapt...

    2024 doi

  104. [115]

    Multi-scale attentive interaction networks for chinese medical question answer selection.IEEE Access, 6:74061–74071, 2018

    Sheng Zhang, Xin Zhang, Hui Wang, Lixiang Guo, and Shanshan Liu. Multi-scale attentive interaction networks for chinese medical question answer selection.IEEE Access, 6:74061–74071, 2018. doi: 10.1109/ACCESS.2018. 2883637

    2018 doi

  105. [116]

    Citrus: Leveraging expert cognitive pathways in a medical language model for advanced medical decision support.arXiv preprint arXiv:2502.18274, 2025

    Guoxin Wang, Minyu Gao, Shuai Yang, et al. Citrus: Leveraging expert cognitive pathways in a medical language model for advanced medical decision support.arXiv preprint arXiv:2502.18274, 2025

    2025 arXiv

  106. [117]

    Small language models learn enhanced reasoning skills from medical textbooks.npj Digital Medicine, 8(1):240, 2025

    Hyunjae Kim, Hyeon Hwang, Jiwoo Lee, Sihyeon Park, Dain Kim, et al. Small language models learn enhanced reasoning skills from medical textbooks.npj Digital Medicine, 8(1):240, 2025. doi: 10.1038/s41746-025-01653-8

    2025 doi

  107. [118]

    Openhermes 2.5: An open dataset of synthetic data for generalist llm assistants, 2023

    Teknium. Openhermes 2.5: An open dataset of synthetic data for generalist llm assistants, 2023. Hugging Face Datasets,https://huggingface.co/datasets/teknium/OpenHermes-2.5. 27

    2023

  108. [119]

    Radgenome-chest ct: A grounded vision-language dataset for chest ct analysis.arXiv preprint arXiv:2404.16754, 2024

    Xiaoman Zhang, Chaoyi Wu, Ziheng Zhao, Jiayu Lei, Ya Zhang, Yanfeng Wang, and Weidi Xie. Radgenome-chest ct: A grounded vision-language dataset for chest ct analysis.arXiv preprint arXiv:2404.16754, 2024

    2024 arXiv

  109. [120]

    Zenodo,https://zenodo.org/doi/10

    AMOS-MM: Abdominal multimodal analysis challenge dataset, 2024. Zenodo,https://zenodo.org/doi/10. 5281/zenodo.10992154(accessed 23 February 2026)

    2024

  110. [121]

    3d-rad: A comprehensive 3d radiology med-vqa dataset with multi-temporal analysis and diverse diagnostic tasks

    Xiaotang Gai, Jiaxiang Liu, Yichen Li, Zijie Meng, Jian Wu, and Zuozhu Liu. 3d-rad: A comprehensive 3d radiology med-vqa dataset with multi-temporal analysis and diverse diagnostic tasks. InAdvances in Neural Information Processing Systems, volume 38, 2025

    2025

  111. [122]

    Medframeqa: A multi-image medical vqa benchmark for clinical reasoning.arXiv preprint arXiv:2505.16964, 2025

    Suhao Yu, Haojin Wang, Juncheng Wu, Cihang Xie, and Yuyin Zhou. Medframeqa: A multi-image medical vqa benchmark for clinical reasoning.arXiv preprint arXiv:2505.16964, 2025

    2025

  112. [123]

    Gmai-mmbench: A comprehensive multimodal evaluation benchmark towards general medical ai.Advances in Neural Information Processing Systems, 37:94327–94427, 2024

    Jin Ye, Guoan Wang, Yanjun Li, Zhongying Deng, Wei Li, Tianbin Li, Haodong Duan, Ziyan Huang, Yanzhou Su, Benyou Wang, et al. Gmai-mmbench: A comprehensive multimodal evaluation benchmark towards general medical ai.Advances in Neural Information Processing Systems, 37:94327–94...

    2024

  113. [124]

    Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering

    Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering. In Gerardo Flores, George H Chen, Tom Pollard, Joyce C Ho, and Tristan Naumann, editors,Proceedings of the Conferenc...

    2022

  114. [125]

    Benchmarking large language models on answering and explaining challenging medical questions

    Hanjie Chen, Zhouxiang Fang, Yash Singla, and Mark Dredze. Benchmarking large language models on answering and explaining challenging medical questions. InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics...

    2025

  115. [126]

    Supergpqa: Scaling llm evaluation across 285 graduate disciplines

    Xinrun Du, Yifan Yao, Kaijing Ma, Bingli Wang, Tianyu Zheng, King Zhu, Minghao Liu, Yiming Liang, Xiaolong Jin, Zhenlin Wei, Chujie Zheng, Kaixin Deng, Shuyue Guo, Shian Jia, Sichao Jiang, Yiyan Liao, Rui Li, Qinrui Li, Sirun Li, Yizhi Li, Yunwen Li, Dehua Ma, Yuansheng Ni, Ha...

    2025

  116. [127]

    Preparing a collection of radiology examinations for distribution and retrieval.Journal of the American Medical Informatics Association, 23(2):304–310, 2016

    Dina Demner-Fushman, Marc D Kohli, Marc B Rosenman, Sonya E Shooshan, Laritza Rodriguez, Sameer Antani, George R Thoma, and Clement J McDonald. Preparing a collection of radiology examinations for distribution and retrieval.Journal of the American Medical Informatics Associati...

    2016 doi

  117. [128]

    w/ Context

    Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: visual explanations from deep networks via gradient-based localization.International journal of computer vision, 128(2):336–359, 2020. 28 Acknowledgements We ...

    2020

  118. [2021]

    URLhttps://openreview.net/forum?id=d7KBjmI3GmQ

  119. [2024]

    URLhttps://arxiv.org/abs/2405.19538

    arXiv

Pith tools

Reviewed July 31, 2026 · model on record in the stance chip above.