Pith. sign in

REVIEW 3 major objections 5 minor 60 references

MedBLINK: Probing Basic Perception in Multimodal Language Models for Medicine

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A new benchmark shows multimodal language models fail basic perceptual checks in medical imaging, scoring 65 percent where human experts score 96.4 percent.

desk verdict Useful new medical perceptual benchmark with a credible but imperfectly established human baseline; the core finding holds. read the letter →

arxiv 2508.02951 v1 pith:GA2O77XS submitted 2025-08-04 cs.AI

Mahtab Bigverdi , Wisdom Ikezogwo , Kevin Zhang , Hyewon Jeong , Mingyu Lu , Sungjae Cho , Linda Shapiro , Ranjay Krishna This is my paper
classification cs.AI
keywords perceptualbenchmarkingmultimodallanguagemodelsmedicalimagingvisualgroundingclinicalAItrustworthinesscontrastenhancementdetectiondepthestimationimageorientation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

MedBLINK is a benchmark of 1,429 multiple-choice questions across eight perceptual tasks—image orientation, contrast-enhancement detection, depth estimation, counting, and age estimation—drawn from X-ray, CT, endoscopy, ultrasound, and histopathology images. The paper argues that these are 'blink' tasks: judgments a clinician makes almost reflexively, without elaborate reasoning. On these tasks, human experts score 96.4 percent, while the best of 19 multimodal language models, including general-purpose and medical-domain systems, reaches only 65 percent. The paper's claim is that this gap reveals a missing layer of visual grounding: models can produce plausible diagnostic text without robustly seeing basic structure in medical images. If true, it would mean current models are not ready for clinical adoption and that benchmarks must test perception, not only diagnostic reasoning.

What carries the argument

The engine of the paper is the MedBLINK benchmark, a set of eight clinically meaningful multiple-choice tasks built from existing medical datasets and validated with a senior radiologist. Three tasks use colored dot prompts drawn on the image, mimicking how clinicians point at regions of interest, so models must both detect the prompt and interpret what it asks about the anatomy. The comparison anchor is a human-expert accuracy of 96.4 percent, computed from a pool of four expert annotators, against which all 19 models are scored. The benchmark's design deliberately separates perception from reasoning: each question is meant to be answerable from low-effort visual cues, so a wrong answer is evidence about what the model sees, not about what it knows.

What would settle it

Have a blinded, independent panel of radiologists, not involved in the benchmark's construction, answer the same 1,429 questions. If their accuracy falls well below 96.4 percent, or if agreement among them is poor on specific tasks, the headline human-model gap is partly an artifact of the original labels rather than a pure measure of model perception.

Watch

Extended reading notes

Core claim

On its own terms, the paper establishes that state-of-the-art multimodal models, generalist and medical-specialist alike, systematically fail at elementary perceptual judgments on medical images. The best tested model reaches 65 percent overall against 96.4 percent for human annotators, with several models near random chance on tasks requiring spatial reasoning with visual prompts. The failures are selective: GPT-4o judges natural images nearly perfectly on an orientation check but drops to 36 percent on flipped pelvic X-rays, and models that can count the colored probe dots on an image have trouble using those dots to estimate depth or structure. The paper also reports that scaling the task set, changing the image source, or varying the prompting strategy does not close the gap, which it reads as evidence that the deficit is perceptual grounding rather than dataset size or instruction format.

Load-bearing premise

The human-expert labels are treated as the gold standard, but they come from only four annotators, three of them co-authors, with no reported inter-annotator agreement, so label ambiguity could shrink the reported human-model gap.

Editorial extensions

If this is right

  • A model that cannot pass these perceptual checks should not be deployed for clinical decision support, regardless of its diagnostic benchmark scores.
  • Perceptual grounding—depth, orientation, counting, and enhancement—becomes a training target rather than an assumed side effect of medical pretraining.
  • Evaluation suites for medical AI should include 'blink' tasks as a trust layer separate from complex reasoning benchmarks.
  • Improvements on MedBLINK's five modalities are expected to transfer to related imaging modalities such as MRI, PET, fluoroscopy, and OCT, because the tasks capture general perceptual skills.
  • Prompting, few-shot examples, image resolution, and dataset scale do not fix the failure, so closing the gap requires architectural or training changes, not prompt engineering.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because GPT-4o handles natural-image orientation almost perfectly yet fails on pelvic X-rays, the paper's results suggest models rely on dataset-specific shortcuts rather than a general spatial model; a direct test would be fine-tuning on a few thousand labeled medical images to see whether orientation accuracy jumps.
  • The colored-dot experiments show a dissociation between prompt detection and prompt use: models can report where a dot is but not what it means for the structure underneath, suggesting the visual encoder and the language decoder are not sharing task-relevant spatial features.
  • One could extend MedBLINK into a low-cost sensory grounding probe for medical AI by measuring not just accuracy but consistency, such as whether the model gives the same answer when the image is rotated, cropped, or color-shifted.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces MedBLINK, a benchmark of multiple-choice questions intended to probe basic perceptual abilities of multimodal language models (MLMs) in medical imaging. It comprises eight tasks across X-ray, CT, endoscopy, histopathology, and ultrasound, and reports evaluations of 19 general-purpose and medical-domain MLMs. The headline result is that the best model reaches 65% accuracy while human experts achieve 96.4%, leading the authors to conclude that current MLMs lack the visual grounding needed for clinical adoption. The paper also includes analyses of prompting strategies, resolution effects, visual-prompt detection, and scaling behavior.

Significance. If the benchmark and the human baseline are reliable, MedBLINK addresses an important and underexplored capability: routine perceptual checks that clinicians perform reflexively and that are prerequisites for trust in AI-assisted diagnosis. The paper's strengths include the breadth of evaluated models (19 systems), the diversity of modalities and anatomical regions, the reuse of public datasets with manual review, and the inclusion of auxiliary experiments (prompting, resolution, scaling, natural-image comparison). The benchmark data are made available, which supports reproducibility. However, the central quantitative claim rests on a human evaluation that is documented only in the supplement and is currently too weak to support the headline gap.

major comments (3)
  1. [Supplementary A.2] The human evaluation is not an independent gold standard. Only four experts participated, three of whom are co-authors, and the text says each task was evaluated by at least one expert, with no inter-annotator agreement reported. Because the co-authors also helped curate the benchmark, their responses may reflect familiarity with the items rather than a clean measure of expert perception. Moreover, single-rater tasks provide no variance estimate. Since the 96.4% human accuracy is the reference against which all model comparisons are made, the authors should recruit an independent expert panel, report per-task and per-item rating counts, compute inter-annotator agreement (e.g., Cohen's kappa or Fleiss' kappa), and provide confidence intervals for per-task and overall human accuracy.
  2. [Table 7] The label distributions in Table 7 sum to 1,229 questions, not the 1,429 stated in the Abstract and in Supplementary Table 6. This is an internal inconsistency in the benchmark statistics. The authors should reconcile the stated total question count with the per-task counts, and if the table is incomplete or the total is wrong, the corrected numbers should be used consistently throughout the paper.
  3. [Section 3 and Section 5] The conclusion that models lack 'basic visual grounding' assumes that the eight tasks are genuinely perceptual. Several tasks appear to require substantial domain-specific medical knowledge, such as determining whether a CT is contrast-enhanced (Task 1), distinguishing pediatric from adult chest X-rays (Task 8), and establishing relative anatomical positions of CT slices (Task 6). Without a non-expert human baseline or another validation that these are 'blink' perception tasks rather than knowledge tasks, the benchmark does not cleanly separate perceptual ability from learned medical expertise. The authors should either provide such validation or temper the perceptual-grounding interpretation of the results.
minor comments (5)
  1. [Abstract] The abstract contains a broken hyphenation: 'contrast-enhance—are' should be 'contrast-enhanced—are'.
  2. [Table 4] The table columns labeled 'Correct' and 'Incorrect Orient.' are ambiguous. Clarify that these columns report accuracy on correctly oriented and flipped images, respectively.
  3. [Model evaluation] Model accuracies are reported as point estimates without confidence intervals or significance tests. Given the benchmark's comparative claims, report uncertainty (e.g., Wilson intervals) for the model and human accuracies.
  4. [Project page] The paper states that data are available on a project page but does not provide a URL. Add a link in the Abstract or a footnote.
  5. [Figure 14] The caption contains a formatting artifact: 'M EDBLINK' should be 'MEDBLINK'.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: MedBLINK is a direct empirical benchmark with no fitted inputs or self-citation-dependent derivation.

full rationale

This paper is an empirical benchmark construction and evaluation study. It does not contain a derivation chain, fitted model parameters, a uniqueness theorem, or an ansatz that is smuggled in via citation. The central claims are direct measurements: 19 MLMs are evaluated on a fixed multiple-choice benchmark, and the reported accuracies are observed outputs, not quantities derived from the benchmark's construction. The human benchmark is an external reference point, not an output of the model evaluation, so the 96.4% human figure is not forced by any model fit or normalization. The only nearby concern is that Supplementary Section A.2 reports a human pool of four experts, three of whom are co-authors, with each task evaluated by at least one expert and no inter-annotator agreement reported. This is a legitimate validity and independence concern about the gold standard, but it is not circularity: the human score is not derived from the models, and the model scores are not derived from the human score. Self-citations appear only in contextual related-work discussion (e.g., Quilt-LLaVA) and are not load-bearing for the benchmark's conclusions. The benchmark is self-contained against external datasets and direct model evaluations, so no circular step can be exhibited.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

This is an empirical benchmark paper; no free parameters are fitted and no new physical or conceptual entities are postulated. The central claim rests on domain assumptions about task validity, the human gold standard, and model representativeness.

assumptions (4)
  • domain assumption Human expert performance is a valid gold standard for basic perceptual tasks.
    The benchmark uses human accuracy as the reference. If human labels are imperfect, the reported gap could be misstated. This is a standard assumption in benchmark papers.
  • domain assumption The eight selected tasks are clinically meaningful and perceptually simple.
    The tasks were chosen with a senior radiologist, but there is no formal validation that they are the most important or representative perceptual checks for clinical practice.
  • domain assumption The sample of 19 models is representative of current multimodal language models.
    The paper covers a range of general and medical models, but the selection may not include all relevant state-of-the-art systems, and results could vary with newer versions.
  • domain assumption Image-level labels are unambiguous for the constructed tasks.
    Orientation and counting labels come from deliberate transformations and are objective, but contrast-enhanced detection relies on subjective interpretation of CT phases and could be less definitive.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MedBLINK: Probing Basic Perception in Multimodal Language Models for Medicine." pith.science (2026). https://pith.science/paper/GA2O77XS

@misc{pith2026250802951,
  author       = {Pith},
  title        = {Pith review of: MedBLINK: Probing Basic Perception in Multimodal Language Models for Medicine},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GA2O77XS}},
  note         = {Machine review of arXiv:2508.02951}
}
read the original abstract

Multimodal language models (MLMs) show promise for clinical decision support and diagnostic reasoning, raising the prospect of end-to-end automated medical image interpretation. However, clinicians are highly selective in adopting AI tools; a model that makes errors on seemingly simple perception tasks such as determining image orientation or identifying whether a CT scan is contrast-enhance are unlikely to be adopted for clinical tasks. We introduce Medblink, a benchmark designed to probe these models for such perceptual abilities. Medblink spans eight clinically meaningful tasks across multiple imaging modalities and anatomical regions, totaling 1,429 multiple-choice questions over 1,605 images. We evaluate 19 state-of-the-art MLMs, including general purpose (GPT4o, Claude 3.5 Sonnet) and domain specific (Med Flamingo, LLaVA Med, RadFM) models. While human annotators achieve 96.4% accuracy, the best-performing model reaches only 65%. These results show that current MLMs frequently fail at routine perceptual checks, suggesting the need to strengthen their visual grounding to support clinical adoption. Data is available on our project page.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

60 extracted references · 42 canonical work pages

  1. [1]

    MedBLINK: Probing Basic Perception in Multimodal Language Models for Medicine

    Introduction Would you trust ChatGPT if it failed to identify whether an image was upside down? For artificial intelligence (AI) systems to be adopted, they must demonstrate competence not only on complex benchmarks, but also on simple, intu- itive tasks. The same expectation holds perhaps even more critically for AI in medicine, where failures on basic p...

    work page Pith review arXiv 2025

  2. [2]

    These models combine text and image understanding and are typ- ically evaluated using visual question answering (VQA) tasks

    Related Work Multimodal Language Models in Healthcare : Medical image analysis has evolved from early computer-aided de- tection efforts [ 20, 37, 58] to recent advances in Multi- modal Language Models (MLMs) [ 3, 31, 46, 54]. These models combine text and image understanding and are typ- ically evaluated using visual question answering (VQA) tasks. Their...

    work page

  3. [3]

    Perception enables clinicians to ex- tract key visual features before engaging in more complex 2 Figure 2

    MEDBLINK Benchmark Clinical image interpretation relies on both perceptual and conceptual reasoning. Perception enables clinicians to ex- tract key visual features before engaging in more complex 2 Figure 2. MEDBLINK characterization. The benchmarks con- tain 8 tasks, ranging from tasks like enhancement detection and depth estimation to anatomical underst...

    work page

  4. [4]

    In-domain imaging properties: We explicitly test mod- els on the clinical characteristics of medical imaging in- cluding knowledge of structural asymmetries, anatomic geometric reasoning, clinical relative depth estimation, and quantification of features leveraging morphology. 3.2. MEDBLINK Curation Task 1: Image Enhancement Detection Image enhancements i...

    work page

  5. [5]

    Perceptual Tasks: In contrast to other medical mul- timodal benchmarks, we explore medical visual tasks which are seemingly simple albeit clinically significant tasks, essential for ensuring accurate diagnoses and deci- sion

    work page

  6. [6]

    Diverse and Generalizable Tasks: Our data is sourced from diverse imaging modalities and anatomic areas, and our tasks are generalizable to other modalities not covered in this benchmark

    work page

  7. [7]

    We mimic this form of prompting, by leveraging visual cues like points/dots to spatially prompt the models when answering the specified question [43, 50]

    Visual Prompting: Clinicians often focus on specific im- age regions, when reviewing or communicating findings. We mimic this form of prompting, by leveraging visual cues like points/dots to spatially prompt the models when answering the specified question [43, 50]

    work page

  8. [9]

    Current models including leading generalist and domain-specialized systems perform far below human levels on perceptual tasks that clinicians solve effortlessly (best: 65% vs

    Implications Our findings with MEDBLINK have direct implications for the design and evaluation of MLMs in medicine. Current models including leading generalist and domain-specialized systems perform far below human levels on perceptual tasks that clinicians solve effortlessly (best: 65% vs. 96.4%). This gap shows that many models lack fundamental visual g...

    work page

Show all 60 references
  1. [11]

    LLaV A 1.5 (7B) [35]

  2. [12]

    INTERN VL 2.5 [13], we leverage the 4B, 8B, 26B and 38B parameterized models

  3. [13]

    Small Specialized models We train small specialized models for some of the tasks with sizeable train sets from the original dataset used to construct the task

    LL AMA 3.2 11B [21] B.1. Small Specialized models We train small specialized models for some of the tasks with sizeable train sets from the original dataset used to construct the task. We finetune a ResNet-50 [23] model on both the age estimation and image orientation tasks. F...

  4. [26]

    Omnimedvqa: A new large- scale comprehensive evaluation benchmark for medical lvlm

    Yutao Hu, Tianbin Li, Quanfeng Lu, Wenqi Shao, Junjun He, Yu Qiao, and Ping Luo. Omnimedvqa: A new large- scale comprehensive evaluation benchmark for medical lvlm. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22170–22183, 2024. 2, 7

    2024

  5. [27]

    Gpt-4o system card

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024. 1, 2, 5, 6

    2024 arXiv

  6. [28]

    The diagnostic impact of contrast-enhanced computed tomography (cect) in evaluating lymph node involvement in colorectal cancer: a comprehen- sive review

    Akash Inamdar and Raju K Shinde. The diagnostic impact of contrast-enhanced computed tomography (cect) in evaluating lymph node involvement in colorectal cancer: a comprehen- sive review. Cureus, 16(6), 2024. 1

    2024

  7. [29]

    Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison

    Jeremy Irvin, Pranav Rajpurkar, Michael Ko, Yifan Yu, Sil- viana Ciurea-Ilcus, Chris Chute, Henrik Marklund, Behzad Haghgoo, Robyn Ball, Katie Shpanskaya, et al. Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison. In Proceedings of the AAA...

    2019

  8. [30]

    A dataset of clinically generated visual questions and answers about radiology images

    Jason J Lau, Soumya Gayen, Asma Ben Abacha, and Dina Demner-Fushman. A dataset of clinically generated visual questions and answers about radiology images. Scientific data, 5(1):1–10, 2018. 2

    2018

  9. [31]

    Llava-onevision: Easy visual task transfer, 2024

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Zi- wei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer, 2024. 2

    2024

  10. [32]

    Llava-onevision: Easy visual task transfer

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Zi- wei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326, 2024. 5, 6, 1

    2024 arXiv

  11. [33]

    Llava-med: Training a large language- and-vision assistant for biomedicine in one day

    Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon, and Jianfeng Gao. Llava-med: Training a large language- and-vision assistant for biomedicine in one day. Advances in Neural Information Processing Systems, 36:28541–28564,

  12. [34]

    Slake: A semantically-labeled knowledge- enhanced dataset for medical visual question answering

    Bo Liu, Li-Ming Zhan, Li Xu, Lin Ma, Yan Yang, and Xiao-Ming Wu. Slake: A semantically-labeled knowledge- enhanced dataset for medical visual question answering. In 2021 IEEE 18th international symposium on biomedical imag- ing (ISBI), pages 1650–1654. IEEE, 2021. 2

    2021

  13. [35]

    Improved baselines with visual instruction tuning, 2024

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning, 2024. URL https://arxiv. org/abs/2310.03744, 3(4):5, 2023. 5, 7, 1

    2024 arXiv

  14. [36]

    A spectrum evaluation benchmark for medical multi-modal large language models

    Jie Liu, Wenxuan Wang, Yihang Su, Jingyuan Huan, Wenting Chen, Yudi Zhang, Cheng-Yi Li, Kao-Jung Chang, Xiaohan Xin, Linlin Shen, et al. A spectrum evaluation benchmark for medical multi-modal large language models. arXiv preprint arXiv:2402.11217, 2024. 2

    2024 arXiv

  15. [37]

    The coding of roentgen images for computer analysis as applied to lung cancer

    Gwilym S Lodwick, Theodore E Keats, and John P Dorst. The coding of roentgen images for computer analysis as applied to lung cancer. Radiology, 81(2):185–200, 1963. 2

    1963

  16. [38]

    Lu, Bowen Chen, Drew F

    Ming Y . Lu, Bowen Chen, Drew F. K. Williamson, Richard J. Chen, Melissa Zhao, Aaron K. Chow, Kenji Ikemura, Ahrong Kim, Dimitra Pouli, Ankush Patel, Amr Soliman, Chengkuan Chen, Tong Ding, Judy J. Wang, Georg Gerber, Ivy Liang, Long Phi Le, Anil V . Parwani, Luca L. Weishaupt...

    2024

  17. [39]

    Lung-rads: pushing the limits

    Maria D Martin, Jeffrey P Kanne, Lynn S Broderick, Ella A Kazerooni, and Cristopher A Meyer. Lung-rads: pushing the limits. Radiographics, 37(7):1975–1993, 2017. 5

    1975

  18. [40]

    Towards accurate differential diagnosis with large language models

    Daniel McDuff, Mike Schaekermann, Tao Tu, Anil Palepu, Amy Wang, Jake Garrison, Karan Singhal, Yash Sharma, Shekoofeh Azizi, Kavita Kulkarni, et al. Towards accurate differential diagnosis with large language models. arXiv preprint arXiv:2312.00164, 2023. 1

    2023 arXiv

  19. [41]

    Med-flamingo: a multimodal medical few-shot learner

    Michael Moor, Qian Huang, Shirley Wu, Michihiro Yasunaga, Yash Dalmia, Jure Leskovec, Cyril Zakka, Eduardo Pontes Reis, and Pranav Rajpurkar. Med-flamingo: a multimodal medical few-shot learner. In Machine Learning for Health (ML4H), pages 353–367. PMLR, 2023. 1, 2, 5, 6

    2023

  20. [42]

    Interactions of perceptual and conceptual pro- cessing: Expertise in medical image diagnosis

    Junya Morita, Kazuhisa Miwa, Takayuki Kitasaka, Kensaku Mori, Yasuhito Suenaga, Shingo Iwano, Mitsuru Ikeda, and Takeo Ishigaki. Interactions of perceptual and conceptual pro- cessing: Expertise in medical image diagnosis. International Journal of Human-Computer Studies, 66(5)...

    2008

  21. [43]

    Beyond the hype: A dispassionate look at vision-language models in medical scenario

    Yang Nan, Huichi Zhou, Xiaodan Xing, and Guang Yang. Beyond the hype: A dispassionate look at vision-language models in medical scenario. arXiv preprint arXiv:2408.08704,

    arXiv

  22. [44]

    Video-based ai for beat-to-beat assessment of cardiac function

    David Ouyang, Bryan He, Amirata Ghorbani, Neal Yuan, Joseph Ebinger, Curtis P Langlotz, Paul A Heidenreich, Robert A Harrington, David H Liang, Euan A Ashley, et al. Video-based ai for beat-to-beat assessment of cardiac function. Nature, 580(7802):252–256, 2020. 4

    2020

  23. [45]

    Kvasir: A multi-class image dataset for computer aided gastrointestinal disease detection

    Konstantin Pogorelov, Kristin Ranheim Randel, Carsten Gri- wodz, Sigrun Losada Eskeland, Thomas de Lange, Dag Johansen, Concetto Spampinato, Duc-Tien Dang-Nguyen, Mathias Lux, Peter Thelin Schmidt, Michael Riegler, and Pål Halvorsen. Kvasir: A multi-class image dataset for com...

    2017

  24. [46]

    Learning transferable visual models from natural language supervision, 2021

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021. 2

    2021

  25. [47]

    Mul- timedeval: A benchmark and a toolkit for evaluating medical vision-language models

    Corentin Royer, Bjoern Menze, and Anjany Sekuboyina. Mul- timedeval: A benchmark and a toolkit for evaluating medical vision-language models. arXiv preprint arXiv:2402.09262,

  26. [48]

    Hematoxylin and eosin-stained whole slide image dataset annotated for skin tissue segmentation

    Anum Abdul Salam, Muhammad Zeeshan Asaf, Muham- mad Usman Akram, Noah Musolff, Samavia Khan, Bassem Rafiq, and Babar Rao. Hematoxylin and eosin-stained whole slide image dataset annotated for skin tissue segmentation. Data in Brief, 59:111306, 2025. 4

    2025

  27. [49]

    Mediconfusion: Can you trust your ai radiologist? probing the reliability 10 of multimodal medical foundation models

    Mohammad Shahab Sepehri, Zalan Fabian, Maryam Soltanolkotabi, and Mahdi Soltanolkotabi. Mediconfusion: Can you trust your ai radiologist? probing the reliability 10 of multimodal medical foundation models. arXiv preprint arXiv:2409.15477, 2024. 1, 2

    2024 arXiv

  28. [50]

    Quilt-llava: Visual instruction tuning by extracting localized narratives from open-source histopathology videos

    Mehmet Saygin Seyfioglu, Wisdom O Ikezogwo, Fatemeh Ghezloo, Ranjay Krishna, and Linda Shapiro. Quilt-llava: Visual instruction tuning by extracting localized narratives from open-source histopathology videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pa...

    2024

  29. [51]

    Diagnostic ultrasound imaging: inside out

    Thomas L Szabo. Diagnostic ultrasound imaging: inside out. Academic press, 2013. 4

    2013

  30. [52]

    Gemini 1.5: Unlocking mul- timodal understanding across millions of tokens of context

    Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking mul- timodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530, 2024. 5, 6, 7, 1

    2024 arXiv

  31. [53]

    What clinicians want: contextualizing explainable machine learning for clinical end use

    Sana Tonekaboni, Shalmali Joshi, Melissa D McCradden, and Anna Goldenberg. What clinicians want: contextualizing explainable machine learning for clinical end use. In Machine learning for healthcare conference, pages 359–380. PMLR,

  32. [54]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Am- jad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023. 2

    2023 arXiv

  33. [55]

    Corrado, Yossi Matias, Karan Singhal, Pete Florence, Alan Karthikesalingam, and Vivek Natarajan

    Tao Tu, Shekoofeh Azizi, Danny Driess, Mike Schaekermann, Mohamed Amin, Pi-Chuan Chang, Andrew Carroll, Chuck Lau, Ryutaro Tanno, Ira Ktena, Basil Mustafa, Aakanksha Chowdhery, Yun Liu, Simon Kornblith, David Fleet, Philip Mansfield, Sushant Prakash, Renee Wong, Sunny Virmani,...

    2023

  34. [56]

    Baichuan-m1: Pushing the medical capability of large language models

    Bingning Wang, Haizhou Zhao, Huozhi Zhou, Liang Song, Mingyu Xu, Wei Cheng, Xiangrong Zeng, Yupeng Zhang, Yuqi Huo, Zecheng Wang, et al. Baichuan-m1: Pushing the medical capability of large language models. arXiv preprint arXiv:2502.12671, 2025. 2

    2025 arXiv

  35. [57]

    Chestx- ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases

    Xiaosong Wang, Yifan Peng, Le Lu, Zhiyong Lu, Mo- hammadhadi Bagheri, and Ronald M Summers. Chestx- ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases. In Proceedings of the IEEE conference on...

  36. [58]

    Detection of radiographic abnor- malities in mammograms by means of optical scanning and computer analysis

    Fred Winsberg, Milton Elkin, Josiah Macy Jr, Victoria Bordaz, and William Weymouth. Detection of radiographic abnor- malities in mammograms by means of optical scanning and computer analysis. Radiology, 89(2):211–215, 1967. 2

    1967

  37. [59]

    Can gpt-4v (ision) serve medical ap- plications? case studies on gpt-4v for multimodal medical diagnosis

    Chaoyi Wu, Jiayu Lei, Qiaoyu Zheng, Weike Zhao, Weixiong Lin, Xiaoman Zhang, Xiao Zhou, Ziheng Zhao, Ya Zhang, Yanfeng Wang, et al. Can gpt-4v (ision) serve medical ap- plications? case studies on gpt-4v for multimodal medical diagnosis. arXiv preprint arXiv:2310.09909, 2023. 2

    arXiv

  38. [60]

    Towards generalist foundation model for radiology by leveraging web-scale 2d&3d medical data

    Chaoyi Wu, Xiaoman Zhang, Ya Zhang, Yanfeng Wang, and Weidi Xie. Towards generalist foundation model for radiology by leveraging web-scale 2d&3d medical data. arXiv preprint arXiv:2308.02463, 2023. 1, 5, 6

    2023 arXiv

  39. [61]

    MediConfusion [ 49] probes failure modes on visually dissimilar image pairs

    assesses trustworthiness across 16 modalities in five dimensions: trustfulness, fairness, safety, privacy, robust- ness. MediConfusion [ 49] probes failure modes on visually dissimilar image pairs. RadVUQA [ 43] highlights critical gaps in spatial, anatomic, and quantitative r...

  40. [62]

    Cares: A comprehensive benchmark of trustwor- thiness in medical vision language models

    Peng Xia, Ze Chen, Juanxi Tian, Yangrui Gong, Ruibo Hou, Yue Xu, Zhenbang Wu, Zhiyuan Fan, Yiyang Zhou, Kangyu Zhu, et al. Cares: A comprehensive benchmark of trustwor- thiness in medical vision language models. arXiv preprint arXiv:2406.06007, 2024. 2

    2024 arXiv

  41. [63]

    Medtrinity-25m: A large-scale multimodal dataset with multigranular annotations for medicine

    Yunfei Xie, Ce Zhou, Lang Gao, Juncheng Wu, Xianhang Li, Hong-Yu Zhou, Sheng Liu, Lei Xing, James Zou, Cihang Xie, et al. Medtrinity-25m: A large-scale multimodal dataset with multigranular annotations for medicine. arXiv preprint arXiv:2408.02900, 2024. 5, 6

    2024 arXiv

  42. [64]

    Depth anything: Unleashing the power of large-scale unlabeled data

    Lihe Yang, Bingyi Kang, Zilong Huang, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything: Unleashing the power of large-scale unlabeled data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10371–10381, 2024. 4

    2024

  43. [65]

    Advancing multimodal medical capabilities of gemini

    Lin Yang, Shawn Xu, Andrew Sellergren, Timo Kohlberger, Yuchen Zhou, Ira Ktena, Atilla Kiraly, Faruk Ahmed, Farhad Hormozdiari, Tiam Jaroensri, et al. Advancing multimodal medical capabilities of gemini. arXiv preprint arXiv:2405.03162, 2024. 1, 2

    2024 arXiv

  44. [66]

    Gmai-mmbench: A comprehensive multimodal evaluation benchmark towards general medical ai

    Jin Ye, Guoan Wang, Yanjun Li, Zhongying Deng, Wei Li, Tianbin Li, Haodong Duan, Ziyan Huang, Yanzhou Su, Benyou Wang, et al. Gmai-mmbench: A comprehensive multimodal evaluation benchmark towards general medical ai. In The Thirty-eight Conference on Neural Information Processi...

    2024

  45. [67]

    Pmc-vqa: Visual in- struction tuning for medical visual question answering

    Xiaoman Zhang, Chaoyi Wu, Ziheng Zhao, Weixiong Lin, Ya Zhang, Yanfeng Wang, and Weidi Xie. Pmc-vqa: Visual in- struction tuning for medical visual question answering. arXiv preprint arXiv:2305.10415, 2023. 2, 1 11 MedBLINK: Probing Basic Perception in Multimodal Language Mode...

    2023 arXiv

  46. [68]

    GPT-4 O [27] version

  47. [69]

    C LAUDE 3.5 S ONNET [1]

  48. [70]

    G EMINI 1.5 P RO [52]

  49. [71]

    QWEN 2.5 VL [6], specifically, we leverage the 3B, and 7B parameterized models

  50. [72]

    Here we use two versions as well, the 0.5B parameterized model, and 7B parameter- ized models

    LLAVA-ONEVISION [32]. Here we use two versions as well, the 0.5B parameterized model, and 7B parameter- ized models

  51. [73]

    Specifically, we prompt it with three questions and answers from PMC-VQA benchmark

    MED-FLAMINGO [41], unlike other models for MED- FLAMINGO to produce valid responses, we need to use few-shot prompting [ 49]. Specifically, we prompt it with three questions and answers from PMC-VQA benchmark

  52. [74]

    as seen in Table 8 for free-from evaluation following MediConfusion [49] setup

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.