IMCBench: A benchmark for multimodal LLMs in Image-grounded Medical Conversations

Anchal Nema; Ashutosh Joshi; Deepak Bansal; Dilek Hakkani-Tur; Korosh Vatanparvar; Maria Xenochristou; Mohammad Abuzar Hashemi; Nivedita Wadhwa; Prasad Kasu; Prashams S Jain

arxiv: 2606.28556 · v1 · pith:CILXW3GLnew · submitted 2026-06-26 · 💻 cs.AI

IMCBench: A benchmark for multimodal LLMs in Image-grounded Medical Conversations

Maria Xenochristou , Ashutosh Joshi , Korosh Vatanparvar , Mohammad Abuzar Hashemi , Prasad Kasu , Deepak Bansal , Anchal Nema , Nivedita Wadhwa

show 5 more authors

Prashams S Jain Rebecca Abraham Will Kimbrough Dilek Hakkani-Tur Wilko Schulz-Mahlendorf

This is my paper

Pith reviewed 2026-06-30 00:44 UTC · model grok-4.3

classification 💻 cs.AI

keywords multimodal LLMsmedical conversationsbenchmarksafety evaluationimage-grounded AIclinical accuracyuncertainty in diagnosismultimodal medical AI

0 comments

The pith

Accurate clinical descriptions from multimodal medical AIs do not ensure safe patient guidance.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces IMCBench to fill gaps in existing medical AI tests by creating image-grounded, multi-turn conversations. It pairs real clinical images with synthetic patient profiles and scores models on safety, accuracy, and appropriate uncertainty across three dimensions. Evaluations of eight frontier models reveal that top performers reach only moderate overall scores and that safety declines for malignant and rare conditions. Ablations confirm that removing visual input or EHR context reduces safe guidance, while stronger models use visual features more effectively. These results show that correct clinical descriptions alone do not produce safe advice, supporting the case for multi-dimensional evaluation frameworks.

Core claim

IMCBench pairs real clinical images with synthetic patient profiles to create multi-turn conversations that test multimodal models on safety, accuracy, and uncertainty; benchmark results show no model dominates all dimensions, safety drops for malignant and rare cases, and both visual input and EHR context improve safe guidance, establishing that accurate clinical description does not guarantee safe patient guidance.

What carries the argument

IMCBench benchmark, which generates multi-turn image-grounded medical conversations scored on safety, accuracy, and uncertainty using LLM-as-Jury calibrated to clinician annotations.

If this is right

Multi-dimensional evaluation is required because single metrics like accuracy miss safety shortfalls.
Safety performance varies by case type, with consistent drops for malignant and rare conditions.
Visual input and EHR context each contribute measurably to safer model guidance.
Stronger models extract more value from visual features than weaker ones do.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The benchmark could be used to track progress as models improve their handling of uncertainty in ambiguous cases.
Deployment of these models in real settings would likely require additional guardrails or human oversight for high-risk conditions.
Extending the test set to include more diverse patient demographics might expose further safety variations.

Load-bearing premise

The synthetic patient profiles paired with real clinical images simulate realistic patient-clinician interactions sufficiently for reliable evaluation of safety, accuracy, and uncertainty.

What would settle it

Direct comparison of model outputs on the benchmark cases against real clinician responses to the same image-profile pairs would show whether safety and accuracy scores match observed clinical practice.

Figures

Figures reproduced from arXiv: 2606.28556 by Anchal Nema, Ashutosh Joshi, Deepak Bansal, Dilek Hakkani-Tur, Korosh Vatanparvar, Maria Xenochristou, Mohammad Abuzar Hashemi, Nivedita Wadhwa, Prasad Kasu, Prashams S Jain, Rebecca Abraham, Wilko Schulz-Mahlendorf, Will Kimbrough.

**Figure 1.** Figure 1: Overview of IMCBench. Dataset construction: combines public dermatology images (DDI) with synthetic EHRs (Synthea) to form clinically coherent ⟨image, EHR⟩ pairs. Scenario generation: samples clinical tasks and ⟨image, EHR⟩ pairs, along with patient personalities to produce realistic scenarios and initial queries. Conversation simulation: simulates multimodal Patient-Health AI interactions via multi-turn… view at source ↗

**Figure 2.** Figure 2: Average jury score vs. average clinician annotation before and after iter [PITH_FULL_IMAGE:figures/full_fig_p012_2.png] view at source ↗

read the original abstract

Recent advances in large language models and vision-language models have enabled reasoning over multimodal data, offering opportunities for clinical applications such as decision support and triaging. However, existing medical AI benchmarks are fragmented: some support multi-turn dialogues but lack images, while others provide multimodal inputs but focus on single-turn QA tasks. To address this gap, we introduce IMCBench, an image-grounded, multi-turn medical conversation benchmark that pairs real, publicly available clinical images with synthetic patient profiles to simulate realistic patient-clinician interactions. Each conversation is evaluated across three clinical dimensions: safety, accuracy, and appropriate use of uncertainty in diagnosis. We benchmark eight multimodal frontier models across four model families (Claude, GPT, Nova, and Llama), scoring each on a 1-5 scale using LLM-as-Jury scoring calibrated against expert clinician annotations. Our results show that Claude Opus 4.6 achieves the highest overall score (3.61), followed by Claude Sonnet 4.6 (3.30) and GPT-5.2 (3.29), though no model dominates all dimensions and safety degrades for both malignant and rare conditions ($\Delta$ = -0.27 each). Ablation studies further reveal that both visual input and EHR context contribute to safe guidance (safety drops of 0.18 and 0.23 on average when each is removed), with stronger models leveraging visual features more effectively. Together, these findings demonstrate that accurate clinical description does not guarantee safe patient guidance, motivating the need for multi-dimensional evaluation frameworks in medical AI.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

IMCBench puts real images into multi-turn medical dialogues and scores safety separately from accuracy, but the synthetic patient side undercuts how much we can trust the dissociation.

read the letter

The paper's actual contribution is a benchmark that takes real clinical images, pairs them with synthetic patient profiles, runs multi-turn conversations, and scores frontier models on safety, accuracy, and uncertainty use. The ablations are the clearest part: dropping visual input costs 0.18 on safety and dropping EHR context costs 0.23, with bigger safety drops on malignant and rare cases. Claude Opus leads overall at 3.61 but no model sweeps all three dimensions. The calibration of the LLM judge to clinician annotations is a concrete step that prior fragmented benchmarks lacked.

The soft spot is exactly the one the stress-test flags. The central claim—that accurate description does not guarantee safe guidance—requires the three dimensions to be separable in something close to real interactions. Synthetic profiles can easily produce overly cooperative or medically tidy patient turns, which would make the reported safety deltas look larger than they would be with messier real patients. The abstract gives no details on profile generation, diversity checks, or comparison to recorded clinician-patient exchanges, so the ablation results on vision and EHR are hard to interpret cleanly.

This is for groups already working on medical multimodal evaluation who need something beyond single-turn VQA. It is worth sending to peer review because the gap it targets is real and the scoring setup is at least partially grounded, but any referee will need to see the synthetic data pipeline and error bars before the safety claims can be taken as settled.

Referee Report

2 major / 2 minor

Summary. The paper introduces IMCBench, a benchmark pairing real publicly available clinical images with synthetic patient profiles to create multi-turn image-grounded medical conversations. It evaluates eight multimodal LLMs across four families (Claude, GPT, Nova, Llama) on three dimensions—safety, accuracy, and appropriate uncertainty use—via LLM-as-Jury scoring calibrated against expert clinician annotations. Results indicate Claude Opus 4.6 leads with an overall score of 3.61, followed by Claude Sonnet 4.6 (3.30) and GPT-5.2 (3.29); safety degrades by Δ = -0.27 on malignant and rare conditions; ablations show average safety drops of 0.18 and 0.23 when visual input or EHR context is removed. The central claim is that accurate clinical description does not guarantee safe patient guidance, motivating multi-dimensional evaluation frameworks.

Significance. If the dissociation between accuracy and safety holds under realistic conditions, the work is significant for medical AI evaluation by providing concrete evidence that single-dimension metrics are insufficient, especially for high-risk malignant/rare cases. The use of real images, expert-calibrated LLM jury, and quantified ablation deltas (0.18, 0.23, 0.27) are strengths that support falsifiable claims about context contributions. This could inform safer model development and benchmark design, though dependent on the fidelity of the synthetic profiles.

major comments (2)

[Abstract and benchmark construction section] Abstract and benchmark construction section: The claim that synthetic patient profiles 'simulate realistic patient-clinician interactions' is load-bearing for the central dissociation between accuracy and safety, yet the manuscript provides no quantitative validation (e.g., comparison of turn length, cooperation level, or medical literacy against real dialogues) or error analysis of profile generation; without this, the reported safety degradation (Δ = -0.27) on malignant/rare conditions cannot be confidently attributed to model behavior rather than profile artifacts.
[Ablation studies] Ablation studies: The reported safety drops (0.18 for visual removal, 0.23 for EHR removal) are interpreted as evidence that both inputs contribute to safe guidance, but the paper does not confirm that patient-side dialogue turns remain fixed and identically realistic across ablated conditions; any change in patient behavior would confound the separability of the three evaluation dimensions and weaken the multi-dimensional framework claim.

minor comments (2)

The exact prompts and few-shot examples used for the LLM-as-Jury scorer should be included in an appendix to support reproducibility of the calibrated 1-5 scores.
Table or figure reporting per-dimension scores for all eight models would clarify the statement that 'no model dominates all dimensions.'

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the detailed and constructive comments. These points help clarify the evidentiary basis for our claims about synthetic profiles and ablation design. We respond to each major comment below and indicate where revisions will be made.

read point-by-point responses

Referee: [Abstract and benchmark construction section] Abstract and benchmark construction section: The claim that synthetic patient profiles 'simulate realistic patient-clinician interactions' is load-bearing for the central dissociation between accuracy and safety, yet the manuscript provides no quantitative validation (e.g., comparison of turn length, cooperation level, or medical literacy against real dialogues) or error analysis of profile generation; without this, the reported safety degradation (Δ = -0.27) on malignant/rare conditions cannot be confidently attributed to model behavior rather than profile artifacts.

Authors: We agree that the absence of quantitative validation for the synthetic profiles is a limitation that weakens attribution of the safety degradation. The profiles were derived from real clinical image metadata and standard medical dialogue templates, but the manuscript does not include direct statistical comparisons or error analysis. In revision we will add (i) an error analysis of profile generation and (ii) quantitative comparisons of turn length, lexical diversity, and cooperation indicators against publicly available real medical dialogue corpora. These additions will be placed in the benchmark construction section and will allow readers to assess whether profile artifacts could explain the Δ = -0.27 safety drop. revision: yes
Referee: [Ablation studies] Ablation studies: The reported safety drops (0.18 for visual removal, 0.23 for EHR removal) are interpreted as evidence that both inputs contribute to safe guidance, but the paper does not confirm that patient-side dialogue turns remain fixed and identically realistic across ablated conditions; any change in patient behavior would confound the separability of the three evaluation dimensions and weaken the multi-dimensional framework claim.

Authors: The patient simulator is implemented as a fixed, independent module whose responses are generated once from the patient profile and then reused verbatim across all ablation conditions. Only the clinician model's input context (visual features or EHR) is altered; the patient turns therefore remain identical. We will add an explicit statement, together with a short pseudocode description of the ablation pipeline, in the ablation studies section to document this fixed-patient design and thereby confirm that the reported safety deltas isolate the contribution of each input modality. revision: yes

Circularity Check

0 steps flagged

No circularity: empirical benchmark with external calibration

full rationale

The paper constructs IMCBench by pairing real images with synthetic profiles and evaluates eight models on safety/accuracy/uncertainty using LLM-as-Jury scores calibrated directly against expert clinician annotations. Reported results consist of concrete empirical deltas (e.g., safety drops of 0.18/0.23 on ablations, -0.27 on malignant/rare cases) rather than any fitted parameter renamed as prediction, self-definitional equation, or load-bearing self-citation. The claim that accurate description does not guarantee safe guidance follows from the observed separation of dimension scores on externally annotated data, with no reduction of outputs to the benchmark's own inputs by construction.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

Abstract-only review limits visibility into full assumptions; primary domain assumption is that synthetic profiles plus real images produce evaluable clinical interactions.

axioms (1)

domain assumption Synthetic patient profiles paired with real clinical images can simulate realistic patient-clinician interactions for safety, accuracy, and uncertainty evaluation.
Invoked to justify benchmark construction and model scoring.

pith-pipeline@v0.9.1-grok · 5876 in / 1224 out tokens · 38790 ms · 2026-06-30T00:44:53.625526+00:00 · methodology

discussion (0)

Reference graph

Works this paper leans on

22 extracted references · 13 canonical work pages · 1 internal anchor

[1]

Arora,R.K.,Wei,J.,Hicks,R.S.,Bowman,P.,Quiñonero-Candela,J.,Tsimpourlas, F., Sharman, M., Shah, M., Vallone, A., Beutel, A., Heidecke, J., Singhal, K.: Healthbench: Evaluating large language models towards improved human health (2025),https://arxiv.org/abs/2505.08775

work page internal anchor Pith review Pith/arXiv arXiv 2025
[2]

Science Advances8(32), eabq6147 (2022).https://doi.org/10

Daneshjou, R., Vodrahalli, K., Novoa, R.A., Jenkins, M., Liang, W., Rotemberg, V., Ko, J., Swetter, S.M., Bailey, E.E., Gevaert, O., Mukherjee, P., Phung, M., Yekrang,K.,Fong,B.,Sahasrabudhe,R.,Allerup,J.A.C.,Okata-Karigane,U.,Zou, J., Chiou, A.S.: Disparities in dermatology AI performance on a diverse, curated clinical image set. Science Advances8(32), e...

2022
[3]

In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops

Groh, M., Harris, C., Soenksen, L., Lau, F., Han, R., Kim, A., Koochek, A., Badri, O.: Evaluating deep neural networks trained on clinical images in dermatology with the Fitzpatrick 17k dataset. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops. pp. 1820–1828 (2021)

2021
[4]

In: Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

Hu, Y., Li, T., Lu, Q., Shao, W., He, J., Qiao, Y., Luo, P.: OmniMedVQA: A new large-scale comprehensive evaluation benchmark for medical LVLM. In: Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 22170–22183 (2024)

2024
[5]

Li, Z., Zou, C., Ma, S., Yang, Z., Du, C., Tang, Y., Cao, Z., Zhang, N., Lai, J.H., Lin, R.S., Ni, Y., Sun, X., Xiao, J., Hou, J., Zhang, K., Han, M.: ZALM3: Zero-shot enhancement of vision-language alignment via in-context information in multi-turn multimodal medical dialogue (2024),https://arxiv.org/abs/2409.17610

work page arXiv 2024
[6]

In: Proceedings of the 31st International Con- ference on Computational Linguistics

Liu, R., Xue, K., Zhang, X., Zhang, S.: Interactive evaluation for medical LLMs via task-oriented dialogue system. In: Proceedings of the 31st International Con- ference on Computational Linguistics. pp. 4871–4896. Association for Computa- tional Linguistics, Abu Dhabi, UAE (Jan 2025),https://aclanthology.org/2025. coling-main.325/

2025
[7]

Matos, J., Chen, S., Placino, S., Li, Y., Climent Pardo, J.C., Idan, D., Tohyama, T., Restrepo, D., Nakayama, L.F., Pascual-Leone, J.M.M., Savova, G., Aerts, H., Celi, L.A., Wong, A.I., Bitterman, D.S., Gallifant, J.: WorldMedQA-V: a multi- lingual, multimodal medical examination dataset for multimodal language models evaluation. In: Findings of the Assoc...

2025
[8]

doi:10.1038/s41586-023-05881-4 (2023)

Moor, M., Banerjee, O., Abad, Z.S.H., Krumholz, H.M., Leskovec, J., Topol, E.J., Rajpurkar, P.: Foundation models for generalist medical artificial intelligence. Na- ture616(7956), 259–265 (2023).https://doi.org/10.1038/s41586-023-05881-4

work page doi:10.1038/s41586-023-05881-4 2023
[9]

IEEE Access (2025)

Öğdü, Ç.U., Arslanoğlu, K., Karaköse, M.: An adaptive multi-agent llm-based clinical decision support system integrating biomedical rag and web intelligence. IEEE Access (2025)

2025
[10]

OpenAI: Introducing ChatGPT health.https://openai.com/index/ introducing-chatgpt-health/(Jan 2026), openAI blog post

2026
[11]

Panagoulias, D.P., Virvou, M., Tsihrintzis, G.A.: Evaluating llm – generated mul- timodal diagnosis from medical images and symptom analysis (2024),https: //arxiv.org/abs/2402.01730

work page arXiv 2024
[12]

Nature639(8056), 18 M

Rao, V.M., Hla, M., Moor, M., Adithan, S., Kwak, S., Topol, E.J., Rajpurkar, P.: Multimodal generative AI for medical image interpretation. Nature639(8056), 18 M. Xenochristou et al. 888–896 (2025).https://doi.org/10.1038/s41586-025-08675-y,https://www. nature.com/articles/s41586-025-08675-y

work page doi:10.1038/s41586-025-08675-y 2025
[13]

In: Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

Shi, X., Liu, Z., Wang, C., Leng, H., Xue, K., Zhang, X., Zhang, S.: MidMed: Towards mixed-type dialogues for medical consultation. In: Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 8145–8157. Association for Computational Linguistics, Toronto, Canada (Jul 2023).https://doi.org/10.1865...

work page doi:10.18653/v1/2023.acl-long.453 2023
[14]

Nature620(7972), 172–180 (2023).https://doi.org/10.1038/ s41586-023-06291-2

Singhal, K., Azizi, S., Tu, T., Mahdavi, S.S., Wei, J., Chung, H.W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al.: Large language models encode clin- ical knowledge. Nature620(7972), 172–180 (2023).https://doi.org/10.1038/ s41586-023-06291-2

2023
[15]

Sun, S., Schubert, A., Goldgof, G.M., Sun, Z., Hartvigsen, T., Butte, A.J., Alaa, A.: Dr-LLaVA: Visual instruction tuning with symbolic clinical grounding (2024), https://arxiv.org/abs/2405.19567

work page arXiv 2024
[16]

In: Proceed- ings of the 2025 Conference on Empirical Methods in Natural Language Process- ing

Sviridov, I., Miftakhova, A., Tereshchenko, A., Zubkova, G., Blinov, P., Savchenko, A.: 3MDBench: Medical multimodal multi-agent dialogue benchmark. In: Proceed- ings of the 2025 Conference on Empirical Methods in Natural Language Process- ing. pp. 26614–26654. Association for Computational Linguistics, Suzhou, China (Nov 2025).https://doi.org/10.18653/v1...

work page doi:10.18653/v1/2025.emnlp-main.1353 2025
[17]

The HAM10000 dataset, a large collection of multi- source dermatoscopic images of common pigmented skin lesions

Tschandl, P., Rosendahl, C., Kittler, H.: The HAM10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions. Scientific Data5, 180161 (2018).https://doi.org/10.1038/sdata.2018.161

work page doi:10.1038/sdata.2018.161 2018
[18]

Nature642, 442–450 (2025).https: //doi.org/10.1038/s41586-025-08866-7

Tu, T., Palepu, A., Schaekermann, M., Saab, K., Freyberg, J., Tanno, R., Wang, A., Li, B., Amin, M., Tomasev, N., Azizi, S., Singhal, K., Cheng, Y., Hou, L., Webson, A., Kulkarni, K., Mahdavi, S.S., Semturs, C., Gottweis, J., Barral, J., Chou,K.,Corrado,G.S.,Matias,Y.,Karthikesalingam,A.,Natarajan,V.:Towards conversational diagnostic artificial intelligen...

work page doi:10.1038/s41586-025-08866-7 2025
[19]

doi: 10.1093/jamia/ocx079

Walonoski, J., Kramer, M., Nichols, J., Quina, A., Moesel, C., Hall, D., Duffett, C., Dube, K., Gallagher, T., McLachlan, S.: Synthea: An approach, method, and software mechanism for generating synthetic patients and the synthetic electronic health care record. Journal of the American Medical Informatics Association25(3), 230–238 (2018).https://doi.org/10...

work page doi:10.1093/jamia/ocx079 2018
[20]

Xu, R., Huang, Z., Wei, Y., Zhou, X., Xu, Z., Liu, T., Jiang, Z., Zhou, S.K.: Medatlas: Evaluating llms for multi-round, multi-task medical reasoning across diverse imaging modalities and clinical text (2025),https://arxiv.org/abs/2508. 10947

2025
[21]

In: Proceedings of the 6th Clinical Natural Language Pro- cessing Workshop

Yim, W.w., Ben Abacha, A., Fu, Y., Sun, Z., Xia, F., Yetisgen, M., Krallinger, M.: Overview of the MEDIQA-M3G 2024 shared task on multilingual multimodal med- ical answer generation. In: Proceedings of the 6th Clinical Natural Language Pro- cessing Workshop. pp. 581–589. Association for Computational Linguistics, Mexico City, Mexico (Jun 2024).https://doi...

work page doi:10.18653/v1/2024.clinicalnlp-1 2024
[22]

Each model is evaluated on the same 155 dermatological scenarios: 130 with clinically adequate images and 25 with syn- thetically degraded images

Zhou, J., Sun, L., Xu, Y., Liu, W., Afvari, S., Han, Z., Song, J., Ji, Y., He, X., Gao, X.: SkinCAP: A multi-modal dermatology dataset annotated with rich medical captions (2024).https://doi.org/10.48550/arXiv.2405.18004 Title Suppressed Due to Excessive Length 19 Supplementary Material 5.2 Conversation Generation We construct an evaluation dataset of 1,2...

work page doi:10.48550/arxiv.2405.18004 2024

[1] [1]

Arora,R.K.,Wei,J.,Hicks,R.S.,Bowman,P.,Quiñonero-Candela,J.,Tsimpourlas, F., Sharman, M., Shah, M., Vallone, A., Beutel, A., Heidecke, J., Singhal, K.: Healthbench: Evaluating large language models towards improved human health (2025),https://arxiv.org/abs/2505.08775

work page internal anchor Pith review Pith/arXiv arXiv 2025

[2] [2]

Science Advances8(32), eabq6147 (2022).https://doi.org/10

Daneshjou, R., Vodrahalli, K., Novoa, R.A., Jenkins, M., Liang, W., Rotemberg, V., Ko, J., Swetter, S.M., Bailey, E.E., Gevaert, O., Mukherjee, P., Phung, M., Yekrang,K.,Fong,B.,Sahasrabudhe,R.,Allerup,J.A.C.,Okata-Karigane,U.,Zou, J., Chiou, A.S.: Disparities in dermatology AI performance on a diverse, curated clinical image set. Science Advances8(32), e...

2022

[3] [3]

In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops

Groh, M., Harris, C., Soenksen, L., Lau, F., Han, R., Kim, A., Koochek, A., Badri, O.: Evaluating deep neural networks trained on clinical images in dermatology with the Fitzpatrick 17k dataset. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops. pp. 1820–1828 (2021)

2021

[4] [4]

In: Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

Hu, Y., Li, T., Lu, Q., Shao, W., He, J., Qiao, Y., Luo, P.: OmniMedVQA: A new large-scale comprehensive evaluation benchmark for medical LVLM. In: Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 22170–22183 (2024)

2024

[5] [5]

Li, Z., Zou, C., Ma, S., Yang, Z., Du, C., Tang, Y., Cao, Z., Zhang, N., Lai, J.H., Lin, R.S., Ni, Y., Sun, X., Xiao, J., Hou, J., Zhang, K., Han, M.: ZALM3: Zero-shot enhancement of vision-language alignment via in-context information in multi-turn multimodal medical dialogue (2024),https://arxiv.org/abs/2409.17610

work page arXiv 2024

[6] [6]

In: Proceedings of the 31st International Con- ference on Computational Linguistics

Liu, R., Xue, K., Zhang, X., Zhang, S.: Interactive evaluation for medical LLMs via task-oriented dialogue system. In: Proceedings of the 31st International Con- ference on Computational Linguistics. pp. 4871–4896. Association for Computa- tional Linguistics, Abu Dhabi, UAE (Jan 2025),https://aclanthology.org/2025. coling-main.325/

2025

[7] [7]

Matos, J., Chen, S., Placino, S., Li, Y., Climent Pardo, J.C., Idan, D., Tohyama, T., Restrepo, D., Nakayama, L.F., Pascual-Leone, J.M.M., Savova, G., Aerts, H., Celi, L.A., Wong, A.I., Bitterman, D.S., Gallifant, J.: WorldMedQA-V: a multi- lingual, multimodal medical examination dataset for multimodal language models evaluation. In: Findings of the Assoc...

2025

[8] [8]

doi:10.1038/s41586-023-05881-4 (2023)

Moor, M., Banerjee, O., Abad, Z.S.H., Krumholz, H.M., Leskovec, J., Topol, E.J., Rajpurkar, P.: Foundation models for generalist medical artificial intelligence. Na- ture616(7956), 259–265 (2023).https://doi.org/10.1038/s41586-023-05881-4

work page doi:10.1038/s41586-023-05881-4 2023

[9] [9]

IEEE Access (2025)

Öğdü, Ç.U., Arslanoğlu, K., Karaköse, M.: An adaptive multi-agent llm-based clinical decision support system integrating biomedical rag and web intelligence. IEEE Access (2025)

2025

[10] [10]

OpenAI: Introducing ChatGPT health.https://openai.com/index/ introducing-chatgpt-health/(Jan 2026), openAI blog post

2026

[11] [11]

Panagoulias, D.P., Virvou, M., Tsihrintzis, G.A.: Evaluating llm – generated mul- timodal diagnosis from medical images and symptom analysis (2024),https: //arxiv.org/abs/2402.01730

work page arXiv 2024

[12] [12]

Nature639(8056), 18 M

Rao, V.M., Hla, M., Moor, M., Adithan, S., Kwak, S., Topol, E.J., Rajpurkar, P.: Multimodal generative AI for medical image interpretation. Nature639(8056), 18 M. Xenochristou et al. 888–896 (2025).https://doi.org/10.1038/s41586-025-08675-y,https://www. nature.com/articles/s41586-025-08675-y

work page doi:10.1038/s41586-025-08675-y 2025

[13] [13]

In: Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

Shi, X., Liu, Z., Wang, C., Leng, H., Xue, K., Zhang, X., Zhang, S.: MidMed: Towards mixed-type dialogues for medical consultation. In: Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 8145–8157. Association for Computational Linguistics, Toronto, Canada (Jul 2023).https://doi.org/10.1865...

work page doi:10.18653/v1/2023.acl-long.453 2023

[14] [14]

Nature620(7972), 172–180 (2023).https://doi.org/10.1038/ s41586-023-06291-2

Singhal, K., Azizi, S., Tu, T., Mahdavi, S.S., Wei, J., Chung, H.W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al.: Large language models encode clin- ical knowledge. Nature620(7972), 172–180 (2023).https://doi.org/10.1038/ s41586-023-06291-2

2023

[15] [15]

Sun, S., Schubert, A., Goldgof, G.M., Sun, Z., Hartvigsen, T., Butte, A.J., Alaa, A.: Dr-LLaVA: Visual instruction tuning with symbolic clinical grounding (2024), https://arxiv.org/abs/2405.19567

work page arXiv 2024

[16] [16]

In: Proceed- ings of the 2025 Conference on Empirical Methods in Natural Language Process- ing

Sviridov, I., Miftakhova, A., Tereshchenko, A., Zubkova, G., Blinov, P., Savchenko, A.: 3MDBench: Medical multimodal multi-agent dialogue benchmark. In: Proceed- ings of the 2025 Conference on Empirical Methods in Natural Language Process- ing. pp. 26614–26654. Association for Computational Linguistics, Suzhou, China (Nov 2025).https://doi.org/10.18653/v1...

work page doi:10.18653/v1/2025.emnlp-main.1353 2025

[17] [17]

The HAM10000 dataset, a large collection of multi- source dermatoscopic images of common pigmented skin lesions

Tschandl, P., Rosendahl, C., Kittler, H.: The HAM10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions. Scientific Data5, 180161 (2018).https://doi.org/10.1038/sdata.2018.161

work page doi:10.1038/sdata.2018.161 2018

[18] [18]

Nature642, 442–450 (2025).https: //doi.org/10.1038/s41586-025-08866-7

Tu, T., Palepu, A., Schaekermann, M., Saab, K., Freyberg, J., Tanno, R., Wang, A., Li, B., Amin, M., Tomasev, N., Azizi, S., Singhal, K., Cheng, Y., Hou, L., Webson, A., Kulkarni, K., Mahdavi, S.S., Semturs, C., Gottweis, J., Barral, J., Chou,K.,Corrado,G.S.,Matias,Y.,Karthikesalingam,A.,Natarajan,V.:Towards conversational diagnostic artificial intelligen...

work page doi:10.1038/s41586-025-08866-7 2025

[19] [19]

doi: 10.1093/jamia/ocx079

Walonoski, J., Kramer, M., Nichols, J., Quina, A., Moesel, C., Hall, D., Duffett, C., Dube, K., Gallagher, T., McLachlan, S.: Synthea: An approach, method, and software mechanism for generating synthetic patients and the synthetic electronic health care record. Journal of the American Medical Informatics Association25(3), 230–238 (2018).https://doi.org/10...

work page doi:10.1093/jamia/ocx079 2018

[20] [20]

Xu, R., Huang, Z., Wei, Y., Zhou, X., Xu, Z., Liu, T., Jiang, Z., Zhou, S.K.: Medatlas: Evaluating llms for multi-round, multi-task medical reasoning across diverse imaging modalities and clinical text (2025),https://arxiv.org/abs/2508. 10947

2025

[21] [21]

In: Proceedings of the 6th Clinical Natural Language Pro- cessing Workshop

Yim, W.w., Ben Abacha, A., Fu, Y., Sun, Z., Xia, F., Yetisgen, M., Krallinger, M.: Overview of the MEDIQA-M3G 2024 shared task on multilingual multimodal med- ical answer generation. In: Proceedings of the 6th Clinical Natural Language Pro- cessing Workshop. pp. 581–589. Association for Computational Linguistics, Mexico City, Mexico (Jun 2024).https://doi...

work page doi:10.18653/v1/2024.clinicalnlp-1 2024

[22] [22]

Each model is evaluated on the same 155 dermatological scenarios: 130 with clinically adequate images and 25 with syn- thetically degraded images

Zhou, J., Sun, L., Xu, Y., Liu, W., Afvari, S., Han, Z., Song, J., Ji, Y., He, X., Gao, X.: SkinCAP: A multi-modal dermatology dataset annotated with rich medical captions (2024).https://doi.org/10.48550/arXiv.2405.18004 Title Suppressed Due to Excessive Length 19 Supplementary Material 5.2 Conversation Generation We construct an evaluation dataset of 1,2...

work page doi:10.48550/arxiv.2405.18004 2024