Pith. sign in

REVIEW 3 major objections 6 minor 45 references

The paper claims that four state-of-the-art medical multimodal models fail to distinguish correct from clearly incorrect image captions, despite strong scores on conventional benchmarks.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 06:05 UTC pith:QONJZBZ4

load-bearing objection New 65k medical caption-selection benchmark; all four tested VLMs near chance, but missing human baseline leaves the core claim under-supported. the 3 major comments →

arxiv 2607.21998 v1 pith:QONJZBZ4 submitted 2026-07-24 cs.CV

Medical-Checklist: Assessing the Comprehension of Medical Images by Multimodal Models

Bannapol Limanond , Masanori Suganuma , Takayuki Okatani This is my paper
classification cs.CV
keywords medical multimodal modelsbenchmark evaluationimage-caption matchingbinary choice taskUMLSMed-VQAout-of-distributionconceptual understanding
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces Medical-Checklist, a benchmark that tests whether medical multimodal models truly understand an image and its caption by forcing a binary choice: given an image and two captions, one correct and one with a single substituted medical term, the model must pick the right one. The benchmark was built to be simple, balanced, and able to test out-of-distribution inputs, unlike existing medical VQA datasets whose distractors are clinically plausible. Evaluated on it, four state-of-the-art medical multimodal models—Med-CLIP, PMC-CLIP, M3AE, and LLaVA-Med—perform at or barely above chance (about 49–61%), despite strong results on conventional benchmarks. The paper's central claim is that high scores on existing benchmarks conceal a lack of genuine conceptual understanding of medical images.

Core claim

The paper's central claim is that none of the four evaluated medical multimodal models can pass a deliberately simple comprehension test: given a medical image and two captions, one correct and one made wrong by swapping in a single medically nonsensical term of the same category, the models cannot reliably identify the correct caption. Accuracy ranges from about 49% (chance) to 61.76%, with the best model (M3AE) still far below what the authors would expect from a system that truly understands medical images. Even after instruction-tuning a large multimodal model to follow the test's format perfectly, performance only reaches about 65–70%, still low. The result is presented as revealing a g

What carries the argument

Medical-Checklist is the central object: a set of 65,464 binary tests built from 53,556 images drawn from two public medical image-caption collections. For each test, the wrong caption is generated by extracting a medical term from the correct caption with a biomedical language model, linking it to a Unified Medical Language System (UMLS) concept ID and semantic type, then replacing it with a randomly selected term of the same semantic category—so the substitution is grammatically valid but medically implausible. This single-term substitution is the load-bearing mechanism: it isolates whether the model can associate a specific image region or feature with a specific medical concept, and it m

Load-bearing premise

The entire diagnosis rests on the assumption that every test item is genuinely answerable by someone with medical knowledge—that the original caption is correct for the image and the swapped term is clearly wrong—and this was checked only by a 100-sample manual review of term extraction, not by measuring human accuracy on the tests themselves.

What would settle it

Give a panel of physicians a random sample of, say, 200 Medical-Checklist items without the intended labels and ask them to choose the correct caption. If their agreement with the intended positive caption is substantially below 90%, the near-chance model scores would reflect unanswerable or ambiguous items, not a lack of conceptual understanding.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If the paper is correct, strong Med-VQA accuracy does not imply that a medical multimodal model knows what structures or findings are actually present in an image.
  • The benchmark's balanced design and out-of-distribution coverage provide a template for evaluating models across many medical subdomains without dataset-specific biases.
  • Because even the best model reaches only about 62%, current medical multimodal models are not reliable for tasks that require rejecting a nonsensical but grammatical description, a prerequisite for safe clinical use.
  • The result implies that training objectives such as masked image/text modeling help only slightly, and no existing objective confers the fine-grained region-to-term alignment this task exposes.
  • The near-chance performance of the conversational model despite its strong dialogue ability shows that instruction-following skill can be disconnected from perceptual grounding.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The benchmark's diagnostic power depends on the quality of the source captions; a natural next step would be measuring human accuracy on a sample of the tests, since the paper validates only term extraction and linking, not the answerability of individual items.
  • A testable extension: the paper's similarity and caption-length analyses suggest that models should improve if trained with objectives that enforce token-level image-text alignment; that prediction can be checked with a targeted training experiment.
  • The same single-substitution design could be used to probe other specialized domains or general-purpose models, since it exposes region-to-word alignment failures that multiple-choice VQA hides.
  • The paper implies that benchmark scores alone are insufficient for clinical evaluation; a plausible consequence is that medical AI claims should be accompanied by adversarial, concept-level tests before deployment.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes Medical-Checklist, a new benchmark for evaluating medical multimodal models. Each test presents an image with two captions: a positive caption (the original image caption) and a negative caption formed by replacing a single medical term with another term of the same UMLS semantic category. The task is binary choice of the correct caption. The benchmark contains 65,464 tests built from MedICat and ROCO. The authors evaluate four models (Med-CLIP, PMC-CLIP, M3AE, and LLaVA-Med), reporting accuracies between 49.7% and 61.8%, and conclude that despite strong Med-VQA performance the models lack a fundamental understanding of core medical concepts. A follow-up experiment instruction-tunes LLaVA-Med on one category, improving its accuracy to ~65–70% on the remaining categories, still below what the authors consider a passing level.

Significance. If the benchmark is validated, it offers a portable and conceptually simple probe of a basic capability—rejecting a clearly incorrect medical description—that is relevant to clinical safety. The dataset construction uses UMLS controls (CUI/TUI) to avoid synonym swaps and to keep alternative terms in the same category, and the authors make the data and code public. The paper also thoughtfully analyzes performance against caption length and replacement-term similarity. However, the central claim that the models 'lack a genuine understanding of medical concepts' rests on the assumption that every binary test is answerable by a competent human: the source caption must truly match the image and the substituted term must be unambiguously wrong for that image. The manuscript does not establish this, and the LLaVA-Med result is computed on a non-random subset of tests. These issues are load-bearing for the conclusion.

major comments (3)
  1. [Sec. III-B.4 and Sec. III-A] The central interpretation assumes each binary test is solvable: the positive caption must actually describe the image and the negative term must be clearly wrong for that image. The manual validation in Sec. III-B.4 checks only term extraction and UMLS linking on 100 samples; it does not verify caption-image agreement or the implausibility of substituted terms per image. MedICat/ROCO captions are known to be noisy, and same-category substitutions such as 'C3 level' vs. 'T12 level' in a spine CT can be plausible for a specific image. Without a human-expert accuracy measurement on a random sample of the final tests, near-chance model accuracy is equally consistent with ambiguous labels as with lack of understanding. This gap is load-bearing for the Sec. V conclusion.
  2. [Sec. IV-A and Table II] LLaVA-Med's 54.25% accuracy is computed on only 48.4% of the test set after discarding 51.6% of tests where the model did not follow the instruction. This subset may differ systematically from the full benchmark, so the score is not directly comparable with the other models' full-set scores. Report the accuracy on the full test set (e.g., treating non-compliant responses as wrong or as a sensitivity analysis) and provide confidence intervals (e.g., bootstrap) for all accuracies. Without intervals, the differences among MedCLIP (50.12%), PMC-CLIP (49.72%), and M3AE (61.76%) cannot be assessed, and statements about being 'at chance' are not statistically grounded.
  3. [Sec. IV-B] The two-step argument that the benchmark cannot be solved via text-only reasoning does not validate benchmark answerability. The first step (CLIP/M3AE process captions independently) concerns model architecture, not human solvability; the second step (MLLMs would achieve near-perfect accuracy if text-only reasoning were possible) fails because MLLMs may err for instruction-following or modality-alignment reasons, as Sec. IV-A itself documents. To support the claim that the task is answerable, the paper needs human validation or a detailed qualitative analysis of a random sample showing that positive captions match images and negative terms are unambiguously wrong.
minor comments (6)
  1. [Tables II and III] Add confidence intervals (e.g., bootstrap) to the accuracy, Brier score, and log loss columns; with 65K tests the standard errors are small, but they are needed to compare models and to support claims of chance-level performance.
  2. [Sec. IV-B] Calling M3AE's 61.76% 'barely above the chance rate of 50%' is an overstatement given the large test set; the gap is ~11.7 points, which is substantial even if not a 'passing' level. Rephrase to avoid misleading wording.
  3. [Sec. IV-C] The category 'Therapeutic Procedure' is used for instruction tuning but is omitted from Table IV; state explicitly that this category was excluded from evaluation and explain the rationale.
  4. [Throughout] The model name is inconsistently written as 'LLaVA-Med', 'LLaV A-Med', and 'LLaV A' in several places. Standardize the spelling.
  5. [Fig. 3] The 'POS' and 'NEG' labels are placed under each image but are not visually tied to the corresponding captions; improve the layout to avoid ambiguity.
  6. [Algorithm 1] The prompt template uses a placeholder for the image; clarify how the image is actually inserted (e.g., as an image token) in the implementation.

Circularity Check

0 steps flagged

No circular derivation: the paper reports direct benchmark measurements; the only weakness is an unvalidated answerability assumption, which is a validity issue, not circularity.

full rationale

The paper's load-bearing conclusion is that four medical multimodal models fail a binary caption-selection test and therefore lack genuine conceptual understanding. This is an empirical measurement on a newly constructed dataset, not a derivation from fitted parameters or from an assumed conclusion. There is no parameter fitting, no prediction that reduces to a fitted input, and no self-citation chain: the authors' own prior work is not invoked as the basis for the benchmark construction or for the conclusion. The benchmark's negative captions are generated by random same-category substitution from the source caption, so the 'incorrectness' of the negative caption is by construction a textual mismatch, but the paper's stronger claim that the positive caption correctly describes the image and the negative term is clearly wrong for that image is not itself established. Section III-B.4 reports only a 100-sample manual check of term extraction and UMLS CUI/TUI linking, not human verification that the final binary tests are answerable, and LLaVA-Med's 54.25% accuracy is computed on only 48.4% of tests after dropping instruction-following failures. These are benchmark-validity and selection-bias concerns, not circularity: they concern whether the test measures what the paper claims, not whether the result is equivalent to its inputs by construction. The measured accuracies (49-61%) neither force nor are forced by the benchmark design; they are contingent model outputs. Accordingly, the appropriate circularity score is 0.

Axiom & Free-Parameter Ledger

0 free parameters · 5 axioms · 0 invented entities

The central claim relies on four domain assumptions: correctness of source captions, answerability of the binary tests by medically knowledgeable humans, validity of UMLS term linking, and fairness of the evaluation protocol across model architectures. No fitted numerical parameters are used; the benchmark design choices are qualitative.

axioms (5)
  • domain assumption Source captions from MedICat and ROCO accurately describe their paired images.
    Every binary test inherits the correctness of the source caption; if a source caption is wrong, the positive option is wrong. Only 100 sampled term-linkings were manually checked; caption-image agreement was not validated. (Sec. III-B.3, III-B.4)
  • domain assumption A model with correct medical knowledge can reliably choose the correct caption over the substituted negative caption.
    The paper asserts this in the Introduction ('a model with correct medical knowledge should be able to select the correct caption') but provides no human-expert accuracy on the benchmark to calibrate what 'expected level of accuracy' means. (Sec. I, IV-B)
  • domain assumption UMLS CUI/TUI constraints guarantee the substituted term is a same-category, non-synonym replacement, making the negative caption clearly incorrect.
    Reliance on UMLS linking is validated on 100 random cases only; linking errors would create invalid tests. (Sec. III-B.4)
  • domain assumption The binary prompt and similarity-based scoring are fair and equivalent across CLIP, image-text matching, and MLLM architectures.
    The paper itself notes the task 'may be slightly disadvantageous for MLLMs' (footnote 1); differences in instruction-following success (LLaVA-Med dropped 51.6% of tests) show the evaluation protocol is not model-neutral. (Sec. III-C, IV-A)
  • domain assumption SciBERT and SciSpacy accurately extract all relevant medical terms from captions.
    Term extraction is a core step; if terms are missed, replacements do not occur; if non-terms are extracted, negative captions could be invalid. Only 100 samples checked. (Sec. III-B.3)

pith-pipeline@v1.3.0-alltime-deepseek · 13793 in / 15865 out tokens · 147976 ms · 2026-08-01T06:05:52.477621+00:00 · methodology

0 comments
read the original abstract

This paper introduces a new benchmark test, Medical-Checklist, for assessing medical multimodal models. The recent advancements in multimodal models have demonstrated significant potential in the field of medical vision-language tasks. However, it is becoming increasingly clear that evaluating these models' performance, whether they are applied to natural or medical images, is challenging. The critical question is whether the models can accurately understand an input image while associating it with relevant input text. To address this, Medical-Checklist imposes a binary test on the models: they are given an image and two captions, where one is correct and the other incorrect, and the model must select the correct one. The incorrect caption contains a single medical concept (word or phrase) that is inaccurately substituted from the correct caption. Although the task is simple, this simplicity enables the unified assessment of diverse multimodal models designed and learned on different principles. It also enables us to verify whether models correctly understand a wide range of medical concepts across various medical sub-domains. Medical-Checklist is designed to reduce potential biases in data and to enable evaluation of the models' ability to handle out-of-distribution inputs, which were difficult in existing datasets. When evaluating four state-of-the-art medical multimodal models with Medical-Checklist, it was revealed that despite their excellent performance in specific tasks such as Med-VQA, they may not correctly understand images, suggesting a long journey ahead for clinical application. The dataset and code will be made public upon acceptance.

Figures

Figures reproduced from arXiv: 2607.21998 by Bannapol Limanond, Masanori Suganuma, Takayuki Okatani.

Figure 1
Figure 1. Figure 1: Example tests from Medical-Checklist (left) and the overview of benchmarking medical multimodal models (right). a chest x-ray image paired with contrasting captions: “Plain x-ray of the chest revealing bilateral pleural effusion” (correct) versus “Arteriography of the chest revealing bilateral pleural effusion”(incorrect); see [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Overview of the dataset creation process. imagery combined with textual descriptions. We adopt a binary classification approach, inspired by studies in natural image understanding [24]–[26], enabling us to cover a wide array of medical concepts across various subdomains. It requires the models to choose between two captions: one accurately describes the image, and the other, referred to as a negative capti… view at source ↗
Figure 3
Figure 3. Figure 3: More qualitative examples from the Medical-Checklist. intended ‘Caption A’ or ‘Caption B.’ As a result, we have excluded the test entries where it fails—accounting for 51.6% of all tests—and only calculated the average accuracy using the remaining entries (48.4%) for its evaluation. The other three models do not have this issue and are evaluated across the entire Medical-Checklist. To deal with this issue … view at source ↗
Figure 4
Figure 4. Figure 4: Model accuracy on Medical-Checklist across caption length. 0.00-0.10 0.10-0.20 0.20-0.30 0.30-0.40 0.40-0.50 0.50-0.60 0.60-0.70 0.70-0.80 0.80-0.90 Similarity Bin 0.4 0.5 0.6 0.7 Accuracy Accuracy Across Similarity Bins MedCLIP PMC-CLIP M3AE LLaVA-Med [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Model accuracy on Medical-Checklist across term similarity [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

45 extracted references · 13 linked inside Pith

  1. [1]

    Align before fuse: Vision and language representation learning with momentum distillation,

    J. Li, R. Selvaraju, A. Gotmare, S. Joty, C. Xiong, and S. C. H. Hoi, “Align before fuse: Vision and language representation learning with momentum distillation,”Advances in neural information processing systems, 2021

    2021

  2. [2]

    Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation,

    J. Li, D. Li, C. Xiong, and S. Hoi, “Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation,” inInternational Conference on Machine Learning, 2022, pp. 12 888– 12 900

    2022

  3. [3]

    Vision-language pre-training with triple contrastive learning,

    J. Yang, J. Duan, S. Tran, Y . Xu, S. Chanda, L. Chen, B. Zeng, T. Chilimbi, and J. Huang, “Vision-language pre-training with triple contrastive learning,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 15 671–15 680

    2022

  4. [4]

    Visual instruction tuning,

    H. Liu, C. Li, Q. Wu, and Y . J. Lee, “Visual instruction tuning,” Advances in neural information processing systems, 2024

    2024

  5. [5]

    BLIP-2: bootstrapping language- image pre-training with frozen image encoders and large language models,

    J. Li, D. Li, S. Savarese, and S. Hoi, “BLIP-2: bootstrapping language- image pre-training with frozen image encoders and large language models,” inICML, 2023

    2023

  6. [6]

    Minigpt-4: Enhancing vision-language understanding with advanced large language models,

    D. Zhu, J. Chen, X. Shen, X. Li, and M. Elhoseiny, “Minigpt-4: Enhancing vision-language understanding with advanced large language models,”arXiv:2304.10592, 2023

    Pith/arXiv arXiv 2023

  7. [7]

    Learning transferable visual models from natural language supervision,

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clarket al., “Learning transferable visual models from natural language supervision,” inInternational conference on machine learning, 2021, pp. 8748–8763

    2021

  8. [8]

    Align, reason and learn: Enhancing med- ical vision-and-language pre-training with knowledge,

    Z. Chen, G. Li, and X. Wan, “Align, reason and learn: Enhancing med- ical vision-and-language pre-training with knowledge,” inProceedings of the 30th ACM International Conference on Multimedia, 2022, pp. 5152–5161

    2022

  9. [9]

    Multi-modal masked autoencoders for medical vision-and-language pre-training,

    Z. Chen, Y . Du, J. Hu, Y . Liu, G. Li, X. Wan, and T.-H. Chang, “Multi-modal masked autoencoders for medical vision-and-language pre-training,” inInternational Conference on Medical Image Computing and Computer-Assisted Intervention, 2022, pp. 679–689

    2022

  10. [10]

    Llava-med: Training a large language-and-vision assistant for biomedicine in one day,

    C. Li, C. Wong, S. Zhang, N. Usuyama, H. Liu, J. Yang, T. Naumann, H. Poon, and J. Gao, “Llava-med: Training a large language-and-vision assistant for biomedicine in one day,”arXiv:2306.00890, 2023

    Pith/arXiv arXiv 2023

  11. [11]

    Medclip: Contrastive learning from unpaired medical images and text,

    Z. Wang, Z. Wu, D. Agarwal, and J. Sun, “Medclip: Contrastive learning from unpaired medical images and text,” inEMNLP, 2022

    2022

  12. [12]

    Pmc-clip: Contrastive language-image pre-training using biomedical documents,

    W. Lin, Z. Zhao, X. Zhang, C. Wu, Y . Zhang, Y . Wang, and W. Xie, “Pmc-clip: Contrastive language-image pre-training using biomedical documents,” inMICCAI, 2023

    2023

  13. [13]

    Overview of the vqa-med task at imageclef 2021: Visual question answering and generation in the medical domain,

    A. Ben Abacha, M. Sarrouti, D. Demner-Fushman, S. A. Hasan, and H. M ¨uller, “Overview of the vqa-med task at imageclef 2021: Visual question answering and generation in the medical domain,” inProceed- ings of the CLEF 2021 Conference and Labs of the Evaluation Forum- working notes, 2021

    2021

  14. [14]

    Slake: A semantically-labeled knowledge-enhanced dataset for medical visual question answering,

    B. Liu, L.-M. Zhan, L. Xu, L. Ma, Y . Yang, and X.-M. Wu, “Slake: A semantically-labeled knowledge-enhanced dataset for medical visual question answering,” in2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI), 2021, pp. 1650–1654

    2021

  15. [15]

    Pmc-vqa: Visual instruction tuning for medical visual question answer- ing,

    X. Zhang, C. Wu, Z. Zhao, W. Lin, Y . Zhang, Y . Wang, and W. Xie, “Pmc-vqa: Visual instruction tuning for medical visual question answer- ing,”arXiv:2305.10415, 2023

    Pith/arXiv arXiv 2023

  16. [16]

    Large-scale domain-specific pre- training for biomedical vision-language processing,

    S. Zhang, Y . Xu, N. Usuyama, J. Bagga, R. Tinn, S. Preston, R. Rao, M. Wei, N. Valluri, C. Wonget al., “Large-scale domain-specific pre- training for biomedical vision-language processing,”arXiv:2303.00915, 2023

    Pith/arXiv arXiv 2023

  17. [17]

    Radiology objects in context (roco): a multimodal image dataset,

    O. Pelka, S. Koitka, J. R ¨uckert, F. Nensa, and C. M. Friedrich, “Radiology objects in context (roco): a multimodal image dataset,” in Intravascular Imaging and Computer Assisted Stenting and Large-Scale Annotation of Biomedical Data and Expert Label Synthesis, 2018, pp. 180–189

    2018

  18. [18]

    Melinda: A multimodal dataset for biomedical experiment method classification,

    T.-L. Wu, S. Singh, S. Paul, G. Burns, and N. Peng, “Melinda: A multimodal dataset for biomedical experiment method classification,” inProceedings of the AAAI Conference on Artificial Intelligence, 2021, pp. 14 076–14 084

    2021

  19. [19]

    Preparing a collection of radiology examinations for distribution and retrieval,

    D. Demner-Fushman, M. D. Kohli, M. B. Rosenman, S. E. Shooshan, L. Rodriguez, S. Antani, G. R. Thoma, and C. J. McDonald, “Preparing a collection of radiology examinations for distribution and retrieval,” Journal of the American Medical Informatics Association, 2016

    2016

  20. [20]

    Medicat: A dataset of medical images, captions, and textual references,

    S. Subramanian, L. L. Wang, S. Mehta, B. Bogin, M. van Zuylen, S. Parasa, S. Singh, M. Gardner, and H. Hajishirzi, “Medicat: A dataset of medical images, captions, and textual references,” inFindings of EMNLP, 2020

    2020

  21. [21]

    Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison,

    J. Irvin, P. Rajpurkar, M. Ko, Y . Yu, S. Ciurea-Ilcus, C. Chute, H. Mark- lund, B. Haghgoo, R. Ball, K. Shpanskayaet al., “Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison,” inProceedings of the AAAI conference on artificial intelligence, 2019, pp. 590–597

    2019

  22. [22]

    Pathvqa: 30000+ questions for medical visual question answering,

    X. He, Y . Zhang, L. Mou, E. Xing, and P. Xie, “Pathvqa: 30000+ questions for medical visual question answering,”arXiv:2003.10286, 2020

    Pith/arXiv arXiv 2003

  23. [23]

    A dataset of clinically generated visual questions and answers about radiology images,

    J. J. Lau, S. Gayen, A. Ben Abacha, and D. Demner-Fushman, “A dataset of clinically generated visual questions and answers about radiology images,”Scientific data, 2018

    2018

  24. [24]

    Vl- checklist: Evaluating pre-trained vision-language models with objects, attributes and relations,

    T. Zhao, T. Zhang, M. Zhu, H. Shen, K. Lee, X. Lu, and J. Yin, “Vl- checklist: Evaluating pre-trained vision-language models with objects, attributes and relations,”arXiv:2207.00221, 2022

    Pith/arXiv arXiv 2022

  25. [25]

    When and why vision-language models behave like bags-of-words, and what to do about it?

    M. Yuksekgonul, F. Bianchi, P. Kalluri, D. Jurafsky, and J. Zou, “When and why vision-language models behave like bags-of-words, and what to do about it?” inInternational Conference on Learning Representations, 2023

    2023

  26. [26]

    Winoground: Probing vision and language models for visio- linguistic compositionality,

    T. Thrush, R. Jiang, M. Bartolo, A. Singh, A. Williams, D. Kiela, and C. Ross, “Winoground: Probing vision and language models for visio- linguistic compositionality,” inProceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, 2022, pp. 5238–5248

    2022

  27. [27]

    Comparative analysis of gpt-4vision, gpt-4 and open source llms in clinical diagnostic accuracy: A benchmark against human expertise,

    T. Han, L. C. Adams, K. Bressem, F. Busch, L. Huck, S. Nebelung, and D. Truhn, “Comparative analysis of gpt-4vision, gpt-4 and open source llms in clinical diagnostic accuracy: A benchmark against human expertise,”medRxiv, 2023

    2023

  28. [28]

    Accu- racy of a vision-language model on challenging medical cases,

    T. Buckley, J. A. Diao, A. Rodman, and A. K. Manrai, “Accu- racy of a vision-language model on challenging medical cases,” arXiv:2311.05591, 2023

    Pith/arXiv arXiv 2023

  29. [29]

    Performance of multimodal gpt-4v on usmle with image: Potential for imaging diagnostic support with explanations,

    Z. Yang, Z. Yao, M. Tasmin, P. Vashisht, W. S. Jang, B. Wang, D. Berlowitz, and H. Yu, “Performance of multimodal gpt-4v on usmle with image: Potential for imaging diagnostic support with explanations,” medRxiv, 2023

    2023

  30. [30]

    Enhancing medical task performance in gpt-4v: A comprehensive study on prompt engineering strategies,

    P. Chen, Z. Huang, Z. Deng, T. Li, Y . Su, H. Wang, J. Ye, Y . Qiao, and J. He, “Enhancing medical task performance in gpt-4v: A comprehensive study on prompt engineering strategies,”arXiv:2312.04344, 2023

    Pith/arXiv arXiv 2023

  31. [31]

    Can gpt-4v (ision) serve medical applications? case studies on gpt-4v for multimodal medical diagnosis,

    C. Wu, J. Lei, Q. Zheng, W. Zhao, W. Lin, X. Zhang, X. Zhou, Z. Zhao, Y . Zhang, Y . Wanget al., “Can gpt-4v (ision) serve medical applications? case studies on gpt-4v for multimodal medical diagnosis,” arXiv:2310.09909, 2023

    Pith/arXiv arXiv 2023

  32. [32]

    Multimodal chatgpt for medical applications: an experimental study of gpt-4v,

    Z. Yan, K. Zhang, R. Zhou, L. He, X. Li, and L. Sun, “Multimodal chatgpt for medical applications: an experimental study of gpt-4v,” arXiv:2310.19061, 2023

    Pith/arXiv arXiv 2023

  33. [33]

    Flamingo: a visual language model for few-shot learning,

    J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y . Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds, R. Ring, E. Rutherford, S. Cabi, T. Han, Z. Gong, S. Samangooei, M. Monteiro, J. Menick, S. Borgeaud, A. Brock, A. Nematzadeh, S. Sharifzadeh, M. Binkowski, R. Barreira, O. Vinyals, A. Zisserman, and K. Simonyan, “Flamingo: a visual language mod...

    Pith/arXiv arXiv 2022

  34. [34]

    Towards generalist foundation model for radiology,

    C. Wu, X. Zhang, Y . Zhang, Y . Wang, and W. Xie, “Towards generalist foundation model for radiology,”arXiv preprint arXiv:2308.02463, 2023

    Pith/arXiv arXiv 2023

  35. [35]

    Making the most of text semantics to improve biomedical vision– language processing,

    B. Boecking, N. Usuyama, S. Bannur, D. C. Castro, A. Schwaighofer, S. Hyland, M. Wetscherek, T. Naumann, A. Nori, J. Alvarez-Valleet al., “Making the most of text semantics to improve biomedical vision– language processing,” inEuropean conference on computer vision, 2022

    2022

  36. [36]

    Vqa-med: Overview of the medical visual question answering task at imageclef 2019,

    A. Ben Abacha, S. A. Hasan, V . V . Datla, D. Demner-Fushman, and H. M ¨uller, “Vqa-med: Overview of the medical visual question answering task at imageclef 2019,” inProceedings of CLEF (Conference and Labs of the Evaluation Forum) 2019 Working Notes, 2019

    2019

  37. [37]

    Omnimedvqa: A new large-scale comprehensive evaluation benchmark for medical lvlm,

    Y . Hu, T. Li, Q. Lu, W. Shao, J. He, Y . Qiao, and P. Luo, “Omnimedvqa: A new large-scale comprehensive evaluation benchmark for medical lvlm,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2024, pp. 22 170–22 183

    2024

  38. [38]

    K-pathvqa: Knowledge-aware multimodal representation for pathology visual ques- tion answering,

    U. Naseem, M. Khushi, A. G. Dunn, and J. Kim, “K-pathvqa: Knowledge-aware multimodal representation for pathology visual ques- tion answering,”IEEE Journal of Biomedical and Health Informatics, vol. 28, no. 4, pp. 1886–1895, 2024

    2024

  39. [39]

    Gpt-4v(ision) technical work and authors,

    OpenAI, “Gpt-4v(ision) technical work and authors,” 2023. [Online]. Available: https://cdn.openai.com/contributions/gpt-4v.pdf

    2023

  40. [40]

    The unified medical language system (umls): integrat- ing biomedical terminology,

    O. Bodenreider, “The unified medical language system (umls): integrat- ing biomedical terminology,”Nucleic acids research, 2004

    2004

  41. [41]

    Scibert: Pretrained language model for scientific text,

    I. Beltagy, K. Lo, and A. Cohan, “Scibert: Pretrained language model for scientific text,” inEMNLP, 2019

    2019

  42. [42]

    ScispaCy: Fast and Robust Models for Biomedical Natural Language Processing,

    M. Neumann, D. King, I. Beltagy, and W. Ammar, “ScispaCy: Fast and Robust Models for Biomedical Natural Language Processing,” in 10 IEEE TRANSACTIONS AND JOURNALS TEMPLATE Proceedings of the 18th BioNLP Workshop and Shared Task, 2019, pp. 319–327

    2019

  43. [43]

    Mismatch quest: Visual and textual feedback for image-text misalignment,

    B. Gordon, Y . Bitton, Y . Shafir, R. Garg, X. Chen, D. Lischinski, D. Cohen-Or, and I. Szpektor, “Mismatch quest: Visual and textual feedback for image-text misalignment,”arXiv:2312.03766, 2023

    Pith/arXiv arXiv 2023

  44. [44]

    Regionclip: Region-based language-image pretraining,

    Y . Zhong, J. Yang, P. Zhang, C. Li, N. Codella, L. H. Li, L. Zhou, X. Dai, L. Yuan, Y . Liet al., “Regionclip: Region-based language-image pretraining,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 16 793–16 803

    2022

  45. [45]

    Self- alignment pretraining for biomedical entity representations,

    F. Liu, E. Shareghi, Z. Meng, M. Basaldella, and N. Collier, “Self- alignment pretraining for biomedical entity representations,” inPro- ceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Jun. 2021. Mr. Bannapol Limanondis currently pursuing a Ph.D. degree at the Gra...

    2021