Pith. sign in

REVIEW 5 major objections 7 minor 1 cited by

Med-GRIM: Enhanced Zero-Shot Medical VQA using prompt-embedded Multimodal Graph RAG

T0 review · 5 major / 7 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A low-compute, zero-shot pipeline with graph retrieval claims to beat fine-tuned medical vision-language models on dermatology VQA.

desk verdict The DermaGraph accuracy claim is a retrieval echo, but the BIND encoder results on standard VQA benchmarks are worth a second look. read the letter →

arxiv 2508.06496 v1 pith:OFOJUZVF submitted 2025-07-20 cs.CV cs.MA

Rakesh Raj Madavan , Akshat Kaimal , Hashim Faisal , Chandrakala S This is my paper · ORCID
classification cs.CVcs.MA
keywords medicalvisualquestionansweringmultimodalgraphretrieval-augmentedgenerationzero-shotlearningvision-languagemodelsdermatologydatasetsmalllanguageembeddingalignmentpromptengineering
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that a low-compute, zero-shot pipeline can outperform fine-tuned medical vision-language models on visual question answering about skin conditions. The pipeline pairs a new multimodal encoder, BIND, with a graph-structured knowledge base, DermaGraph, and small language models that retrieve and filter information before answering. If the claim holds, medical VQA no longer requires expensive fine-tuning of large VLMs: a modular retrieval-and-prompt system built from small models can match or exceed their accuracy. The paper reports 83.33 percent accuracy on DermaGraph, above LLaVA-Med and MUMC, and reports gains on standard biomedical VQA benchmarks for the BIND encoder itself.

What carries the argument

The load-bearing pieces are BIND's True Transformation Layer (TTL), a learned projection with sub-query parameters that maps image and text features into a shared space; DermaGraph, a Neo4j graph of 50 NHS dermatology conditions with multimodal node embeddings and text-similarity edges; and a two-stage filter. Stage one computes a weighted cosine similarity between the encoded input and each graph node, with lambda=0.4 balancing the text-only and multimodal terms, keeps nodes scoring at least 95 percent of the maximum plus their neighbors, and passes the survivors to stage two, where Phi-3 and Mistral-7B generate clarifying questions and assign likelihood scores, discarding conditions below 50 percent. The surviving conditions are injected into a prompt template for the final answer. The TTL is what lets the encoder separate similar conditions, and the graph's edges plus filtering are what let the system narrow a broad query to a ranked set of diagnoses.

What would settle it

Have clinicians who have not seen DermaGraph write questions about skin conditions, with some conditions held out of the graph entirely, then run Med-GRIM; if accuracy drops sharply on held-out conditions or on clinician-authored questions, the reported result is retrieval echo, not generalizable medical VQA.

Watch

Extended reading notes

Core claim

Med-GRIM's central claim is that aligning image and text in a refined joint embedding space, then retrieving from a condition graph with hybrid text-image similarity and iterative small-model filtering, yields better zero-shot medical answers than fine-tuned medical VLMs. On the DermaGraph evaluation set, the full Graph-RAG pipeline scores 83.33 percent accuracy and a Semantic-BERT score of 0.81, beating LLaVA-Med (76.7), MUMC (80.0), Med-Flamingo (46.67), and vanilla RAG (78.7). The authors attribute the gain to the True Transformation Layer in BIND, which replaces query-based compression with sub-query projections, and to the graph's inter-condition edges, which let retrieval consider similar conditions rather than a single best match.

Load-bearing premise

The evaluation questions were written from the same 50 NHS condition pages that populate the retrieval graph, so the high scores may show that the system can retrieve and restate stored text rather than reason about new medical images.

Editorial extensions

If this is right

  • Medical VQA systems can be assembled from frozen small language models and a curated knowledge graph, removing the need for per-dataset fine-tuning of large vision-language models.
  • Accuracy on dermatology questions becomes a function of graph coverage and retrieval quality, so expanding DermaGraph to more conditions, modalities, or languages should improve the system directly.
  • The two-stage filtering design offers a built-in explanation trail: the system shows the user which conditions it considered and what likelihood scores resulted, making the reasoning auditable.
  • The same graph-RAG scaffold should transfer to other medical domains, such as radiology or pathology, by replacing the condition nodes with domain-appropriate entities and edges.
  • Because the graph is unimodal-compatible, the pipeline can also answer text-only dermatology queries, which widens its use to non-image settings.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An independent evaluation is needed: because the 30 test questions were generated from the same 50 NHS condition pages stored in the graph, the reported accuracy may partly measure retrieval fidelity rather than medical reasoning; a held-out-condition test would separate the two.
  • The contribution of the visual modality is not yet isolated; comparing Med-GRIM on image-only, text-only, and image-plus-text queries with lambda varied would show how much of the 83.33 percent comes from the image encoder versus the text descriptions in the graph.
  • The small test set (30 pairs) makes the 83.33 percent figure sensitive to a few responses; reporting per-question scores or confidence intervals would strengthen the comparison against baselines.
  • If the graph is a closed set of conditions, the system cannot say 'none of the above' for an unknown skin condition; a rejection-aware variant that flags low-confidence cases for a human reviewer is a natural testable extension.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 7 minor

Summary. The paper proposes Med-GRIM, a zero-shot medical VQA pipeline that combines a BIND multimodal encoder, a graph-retrieval stage over a new DermaGraph knowledge graph, and small-language-model agents for filtering and response generation. The authors report that BIND outperforms prior VLMs on general zero-shot VQA benchmarks and on VQA-RAD/PathVQA, and that Med-GRIM with Graph-RAG reaches 83.33% accuracy on a 30-item DermaGraph evaluation, beating LLaVA-Med, MUMC, Med-Flamingo, RULE, and a vanilla RAG variant. They also present ablations of the similarity weight lambda and prompt engineering. The core claim is that a low-compute, zero-shot pipeline can match or exceed fine-tuned medical VLMs.

Significance. If the reported results were reliable, the paper would make a useful contribution: DermaGraph is a new graph-structured dermatology resource, the modular pipeline is computationally light, and the authors state that code and dataset will be released. The BIND encoder also shows competitive numbers on standard VQA benchmarks. However, the headline DermaGraph result rests on an evaluation set of 30 self-authored questions generated from the same NHS source texts stored in the retrieval graph, and the key hyperparameter lambda was tuned on that same set. This makes the central comparative claim currently unsupported. The paper's usefulness therefore depends on a proper independent evaluation, a corrected retrieval formula, and fuller architectural details.

major comments (5)
  1. [Section 4.2, Table 3] The 30-question DermaGraph evaluation is not independent of the retrieval corpus. The evaluation Q&A pairs were 'generated ... based on the collected conditions,' which are the same 50 NHS dermatology entries used to construct DermaGraph. Since Med-GRIM retrieves graph nodes and injects their text into the prompt before generating an answer, a correct answer can be obtained by copying retrieved text; hence the 83.33% result may reflect retrieval echo rather than medical visual reasoning. The comparison to LLaVA-Med, MUMC, and Med-Flamingo is also unfair because those baselines do not receive the graph content. Please re-evaluate on an external benchmark or construct DermaGraph with a strict hold-out of entire conditions not present in the graph.
  2. [Section 5, Tables 4 and 3] The similarity-weight hyperparameter lambda=0.4 was chosen by ablating on the same 30-item DermaGraph benchmark that is then used to report the headline accuracy. The reported 83.33% is therefore a test-set-tuned number. Please select hyperparameters on a separate validation set or use nested cross-validation, and report performance for the selected configuration on untouched test items.
  3. [Section 3.3, Algorithm 1] The implementation in Algorithm 1 line 5 computes 'score <- lambda * cos(E_I, G) + (1-lambda) * cos(E_I, G)', so both terms are identical and the formula cannot implement the hybrid text/multimodal similarity of Eq. (2). This is a load-bearing discrepancy because the retrieval stage is the core of Med-GRIM. Please correct the pseudocode to match Eq. (2) and clarify how text-only and multimodal embeddings are computed for graph nodes.
  4. [Table 3] With only 30 test questions, the difference between 83.33% (25/30) and 80.0% (24/30) is a single item, and no confidence intervals, standard errors, or significance tests are reported. The claim that Med-GRIM 'outperforms' MUMC and other baselines is therefore not statistically grounded. Please report interval estimates or use a larger evaluation set.
  5. [Section 3.2] BIND's True Transformation Layer is described only at a high level; no equations, layer dimensions, or training objective for the TTL are given, making the main architectural novelty non-reproducible. Please add precise specifications of the TTL architecture and its training procedure.
minor comments (7)
  1. [Abstract and Section 1] The abstract and introduction contain a placeholder link ('link') instead of the actual code/dataset URL; please provide the complete repository address.
  2. [References] Several references contain placeholder author names (e.g., 'Author2 Author1 and Author3' in [5] and [6]) or incomplete author lists; these should be corrected before publication.
  3. [Eq. (3)] The notation in Eq. (3) is confusing: M is defined as 'arg max' over nodes, but then used as a numeric value in the inequality |x - M| <= 0.05*M. Please clarify that M denotes the maximum similarity value, not a node.
  4. [Eq. (5)] There is a typo in Eq. (5): 'neightbors' should be 'neighbors'.
  5. [Table 3] The table header uses 'SoTa' and the model name is written inconsistently as 'Med-Grim' and 'Med-GRIM'; please standardize both.
  6. [Section 4.2] The paper states that 50 conditions were sourced from the NHS website but does not give the data license, the exact source URLs, or the criteria for selecting the 10-15 representative images per condition; please add these details.
  7. [Eq. (3) and Algorithm 1] The filtering threshold is expressed as '0.05*M' in Eq. (3) and '0.95*MaxScore' in Algorithm 1; these are equivalent but written differently, so please unify the notation.

Circularity Check

3 steps flagged · score 7.0 of 10

DermaGraph's 83.33% is a lambda-tuned, retrieval-echo score on a test set built from the same knowledge store; BIND's VQA-RAD/PathVQA results are in-sample.

  1. fitted input called prediction [Sec. 5 (Ablation), Table 4; Sec. 4.3; Table 3]
    "Results for varying λ values are summarized in Table 4, demonstrating the influence of this balance on model accuracy. A λ value of 0.4 was found to bear the most accurate results."

    The λ-ablation in Table 4 is run on the same DermaGraph benchmark whose accuracy Table 3 reports, and Table 3's 83.33% for Med-GRIM(Graph-RAG) uses the selected λ=0.4. The headline number is therefore a test-set-tuned statistic: the hyperparameter was chosen to maximize the exact accuracy metric later presented as evidence of superiority. This is a fitted parameter renamed as a prediction, not a held-out evaluation.

  2. self definitional [Sec. 4.2 (DermaGraph) and Sec. 3.4 (Prompt injection), Table 3]
    "To create an evaluation dataset, we generated 30 open-ended Q&A pairs based on the collected conditions. ... The final step, before addressing the user’s query, involves integrating the retrieved data into a prompt template to generate a comprehensive response. ... each red node represents a specific dermatological condition ... and connects to three child nodes that store related information fields, such as symptoms, treatments, and preventive strategies."

    The 30 test questions are generated from the same 50 NHS condition entries stored as DermaGraph nodes, and the generation step injects the retrieved node text into the prompt before answering. Whenever retrieval selects the correct node, the correct condition and its symptom/treatment text are already in the prompt, so the measured accuracy is a retrieval-echo score rather than evidence of medical visual reasoning. The baselines in Table 3 (LLaVA-Med, MUMC, Med-Flamingo) do not receive this injected text, so the comparison is open-book versus closed-book by construction.

1 more flagged steps
  1. fitted input called prediction [Sec. 4.1 (Datasets) and Table 2]
    "Additionally, to assess BIND’s performance in the biomedical domain, we incorporate biomedical VQA datasets such as VQA-RAD [47] and Path-VQA [36] for training and evaluation, as detailed in Sec. 4.4."

    This sentence explicitly states that VQA-RAD and PathVQA are used both for training and for evaluation. Table 2 then reports BIND's accuracy on those same datasets and concludes it outperforms fine-tuned medical VLMs. Reporting in-sample accuracy on the training data is not a prediction; these scores are fitted to the benchmark. This also undercuts the zero-shot framing for the biomedical evaluation.

full rationale

The main headline (Table 3) is not self-contained: λ=0.4 was tuned on the same 30-item DermaGraph set whose accuracy is reported, and those 30 questions were generated from the same NHS text that DermaGraph stores and that Med-GRIM injects into the prompt before generating an answer. The biomedical support for BIND (Table 2) is likewise in-sample, since VQA-RAD and PathVQA are listed both as training and evaluation data. These are concrete reductions to the paper's own inputs, not speculation about intent. On the other hand, BIND's general VQA results on TextVQA, ST-VQA, Doc-VQA, and InfoVQA use benchmarks not in the stated training list, and the graph/SLM design is not backed by a load-bearing self-citation chain. The paper therefore has partial, not complete, circularity.

Assumptions & free parameters 3 free parameters · 4 assumptions · 2 invented entities

The central system depends on a similarity-weight hyperparameter tuned on the same benchmark used for the headline result, on hand-set filtering thresholds, and on an assumption that the self-authored 30-question test set is a valid measure of medical VQA. The new entities (BIND and DermaGraph) are contributions, but they lack independent external evidence in this paper.

free parameters (3)
  • Lambda (similarity weight in Eq. 2) = 0.4
    Controls the balance between text-only and multimodal similarity. Chosen by ablation on the same DermaGraph evaluation set (Table 4) and then used for the headline accuracy in Table 3.
  • Candidate filtering threshold (Algorithm 1, Eq. 3) = 0.95 * MaxScore and 0.05 * M
    Used to decide which nodes plus neighbors enter the candidate set; no sensitivity analysis is reported, and the threshold is set by hand.
  • Likelihood threshold for keeping a condition (Algorithm 2) = 0.5
    Conditions with probability above 0.5 are kept; chosen by hand without validation or calibration.
assumptions (4)
  • domain assumption BIND's cosine similarity between encoded query and graph node indicates diagnostic relevance.
    Used in Eq. 1-2 and Stage 1 filtering; no evidence that cosine distance in this embedding space aligns with medical similarity.
  • domain assumption NHS-sourced descriptions and images for 50 dermatological conditions are accurate and sufficient for building the knowledge graph.
    DermaGraph construction in Sec. 4.2 assumes the curated NHS text and 10-15 images per condition represent each condition correctly.
  • domain assumption Small language models (Phi-3 and Mistral) can generate useful clarifying questions and likelihood scores from retrieved condition lists.
    Stage 2 response filtering relies on SLM medical reasoning, but no validation of the generated questions or calibration of likelihoods is provided.
  • domain assumption The 30 self-generated Q&A pairs form a valid evaluation benchmark.
    Sec. 4.2 states the pairs were generated by the authors from the collected conditions; their representativeness and difficulty are unknown.
invented entities (2)
  • BIND encoder (including True Transformation Layer)
    purpose: Refines multimodal embeddings for retrieval and downstream VQA.
    BIND is new to this paper. The only evidence is the paper's own benchmarks; no external replication or independent evaluation of the TTL mechanism is provided.
  • DermaGraph dataset
    purpose: Graph-structured multimodal knowledge base for dermatology RAG and evaluation.
    A new dataset introduced here. It is constructed from public NHS material, but its 30-item evaluation split has not been externally validated or used by other groups.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Med-GRIM: Enhanced Zero-Shot Medical VQA using prompt-embedded Multimodal Graph RAG." pith.science (2026). https://pith.science/paper/OFOJUZVF

@misc{pith2026250806496,
  author       = {Pith},
  title        = {Pith review of: Med-GRIM: Enhanced Zero-Shot Medical VQA using prompt-embedded Multimodal Graph RAG},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OFOJUZVF}},
  note         = {Machine review of arXiv:2508.06496}
}
read the original abstract

An ensemble of trained multimodal encoders and vision-language models (VLMs) has become a standard approach for visual question answering (VQA) tasks. However, such models often fail to produce responses with the detailed precision necessary for complex, domain-specific applications such as medical VQA. Our representation model, BIND: BLIVA Integrated with Dense Encoding, extends prior multimodal work by refining the joint embedding space through dense, query-token-based encodings inspired by contrastive pretraining techniques. This refined encoder powers Med-GRIM, a model designed for medical VQA tasks that leverages graph-based retrieval and prompt engineering to integrate domain-specific knowledge. Rather than relying on compute-heavy fine-tuning of vision and language models on specific datasets, Med-GRIM applies a low-compute, modular workflow with small language models (SLMs) for efficiency. Med-GRIM employs prompt-based retrieval to dynamically inject relevant knowledge, ensuring both accuracy and robustness in its responses. By assigning distinct roles to each agent within the VQA system, Med-GRIM achieves large language model performance at a fraction of the computational cost. Additionally, to support scalable research in zero-shot multimodal medical applications, we introduce DermaGraph, a novel Graph-RAG dataset comprising diverse dermatological conditions. This dataset facilitates both multimodal and unimodal querying. The code and dataset are available at: https://github.com/Rakesh-123-cryp/Med-GRIM.git

Figures

Figures reproduced from arXiv: 2508.06496 by the authors.

Figure 1
Figure 1. Conveying all the details for medical diagnoses poses significant challenges. To tackle this, we propose Med-GRIM, a novel [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Pipeline of the proposed Med-GRIM system. Med-GRIM integrates multimodal inputs—such as images and descrip [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Comparison of various VLM architectures: a)BLIP uses a small set of query embeddings to compress visual information b) LLaVA trains the LLM to understand visual cues c) BIND(Ours) learns representations through sub-query projections and the True Transformation layer(TTL) proposed. C = fgraph(Ot(I)) ∪ neightbors(fgraph(Ot(I))) (5) When processing medical data, the input often presents discrepancies, as multiple disea… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: These graph structures depict our dataset DermaGraph [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Qualitative analysis of Med-GRIM’s diagnostic flow: From symptom input and follow-up questioning to condition likelihood [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. The Path to Self-Evolving Clinical Systems: Scaling Medical Agents from Assistance to Autonomy

    cs.AI 2026-07 conditional novelty 4.5 of 10

    Medical agents should be scaled mainly by richer clinical environments and self-evolution loops, not parameter growth alone, under a three-level autonomy taxonomy.

Reference graph

Works this paper leans on

48 extracted references · 43 canonical work pages · cited by 1 Pith paper

  1. [1]

    The inception team at vqa-med 2020: Pre- trained vgg with data augmentation for medical vqa and vqg

    Aisha Al-Sadi, Al-Ayyoub M Hana’Al-Theiabat, and Mah- moud Al-Ayyoub. The inception team at vqa-med 2020: Pre- trained vgg with data augmentation for medical vqa and vqg. In CLEF (Working Notes), 2020. 2

    work page 2020

  2. [2]

    Flamingo: a visual language model for few-shot learning

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Men- sch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35:23716–23736,

  3. [3]

    Bottom-up and top-down attention for image captioning and visual question answering

    Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for image captioning and visual question answering. InIEEE Conference on Computer Vision and Pattern Recognition, 2018. 1

    work page 2018

  4. [4]

    Vqa: Visual question answering

    Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In IEEE International Con- ference on Computer Vision, 2015. 1

    work page 2015

  5. [5]

    A-okvqa: A dataset for adap- tive open knowledge visual question answering

    Author2 Author1 and Author3. A-okvqa: A dataset for adap- tive open knowledge visual question answering. In Confer- ence on Neural Information Processing Systems (NeurIPS) , pages 1234–1245, 2020. 6

    work page 2020

  6. [6]

    Ocr-vqa: A dataset for visual question answering with text in the wild

    Author2 Author1 and Author3. Ocr-vqa: A dataset for visual question answering with text in the wild. IEEE Transactions on Pattern Analysis and Machine Intelligence , 42:123–135,

    work page

  7. [7]

    Okvqa: A dataset for open knowledge visual question answering

    Soumya Basu, Sarthak Gupta, Yejin Choi, Soroush Lee, Minjae Mun, and Yujia Zhang. Okvqa: A dataset for open knowledge visual question answering. In Conference on Computer Vision and Pattern Recognition (CVPR) , pages 1546–1555, 2019. 6

    work page 2019

  8. [8]

    Veagle: Advancements in multimodal rep- resentation learning

    Rajat Chawla, Arkajit Datta, Tushar Verma, Adarsh Jha, An- mol Gautam, Ayush Vatsal, Sukrit Chaterjee, Mukunda NS, and Ishaan Bhola. Veagle: Advancements in multimodal rep- resentation learning. 3

    work page

Show all 48 references
  1. [9]

    Visualgpt: Data-efficient adaptation of pretrained language models for image captioning

    Jun Chen, Han Guo, Kai Yi, Boyang Li, and Mohamed El- hoseiny. Visualgpt: Data-efficient adaptation of pretrained language models for image captioning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18030–18040, 2022. 3

    2022

  2. [10]

    Infovqa: A dataset for text and visual question an- swering from information-rich documents

    Xin Chen, Chia-Hsiu Lee, Yao-Hung Yang, and Daryl Goh. Infovqa: A dataset for text and visual question an- swering from information-rich documents. arXiv preprint arXiv:2005.00123, 2020. 6

    2005 arXiv

  3. [11]

    Textvqa: Text-based visual question answering

    Abhishek Das, Tushar Khot, and Ashish Sabharwal. Textvqa: Text-based visual question answering. arXiv preprint arXiv:1709.08364, 2017. 6

    2017 arXiv

  4. [12]

    From local to global: A graph rag approach to query-focused sum- marization, 2025

    Darren Edge, Ha Trinh, Newman Cheng, Joshua Bradley, Alex Chao, Apurva Mody, Steven Truitt, Dasha Metropoli- tansky, Robert Osazuwa Ness, and Jonathan Larson. From local to global: A graph rag approach to query-focused sum- marization, 2025. 2

    2025

  5. [13]

    Sedigheh Eslami, Christoph Meinel, and Gerard De Melo. Pubmedclip: How much does clip benefit visual question answering in the medical domain? In Findings of the As- sociation for Computational Linguistics: EACL 2023, pages 1181–1193, 2023. 7

    2023

  6. [14]

    Prompt learning with knowledge graphs for zero- shot relation extraction

    Tianyu Gao, Xu Han, Zhiyuan Liu, Maosong Sun, and Jie Zhou. Prompt learning with knowledge graphs for zero- shot relation extraction. In Proceedings of the 2021 Confer- ence on Empirical Methods in Natural Language Processing (EMNLP), pages 2712–2722, 2021. 2

    2021

  7. [15]

    Imagebind: One embedding space to bind them all

    Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind: One embedding space to bind them all. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15180–15190, 2023. 3

    2023

  8. [16]

    Girshick

    Yash Goyal, Sarthak Pi, Parth Gupta, Mitash, and Ross B. Girshick. Vqa v2.0: A new dataset for visual question an- swering. In Conference on Computer Vision and Pattern Recognition (CVPR), pages 1–12, 2017. 6

    2017

  9. [17]

    Medalpaca–an open-source collection of medical conversational ai models and training data

    T Han, LC Adams, JM Papaioannou, P Grundmann, T Ober- hauser, A L¨oser, D Truhn, and KK Bressem. Medalpaca–an open-source collection of medical conversational ai models and training data. arxiv. arXiv preprint arXiv.2304.08247 ,

    arXiv

  10. [18]

    Bliva: A simple multimodal llm for better handling of text-rich visual questions

    Wenbo Hu, Yifan Xu, Yi Li, Weiyue Li, Zeyuan Chen, and Zhuowen Tu. Bliva: A simple multimodal llm for better handling of text-rich visual questions. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 2256– 2264, 2024. 3, 4, 6

    2024

  11. [19]

    Interpretable medical image visual question answering via multi-modal relationship graph learning.Med- ical Image Analysis, 97:103279, 2024

    Xinyue Hu, Lin Gu, Kazuma Kobayashi, Liangchen Liu, Mengliang Zhang, Tatsuya Harada, Ronald M Summers, and Yingying Zhu. Interpretable medical image visual question answering via multi-modal relationship graph learning.Med- ical Image Analysis, 97:103279, 2024. 3

    2024

  12. [20]

    Retrieval-augmented gener- ation for knowledge-intensive nlp tasks

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K¨uttler, Mike Lewis, Wen tau Yih, Tim Rockt¨aschel, Sebas- tian Riedel, and Douwe Kiela. Retrieval-augmented gener- ation for knowledge-intensive nlp tasks. In Neural Infor...

    2020

  13. [21]

    Llava-med: Training a large language- and-vision assistant for biomedicine in one day

    Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon, and Jianfeng Gao. Llava-med: Training a large language- and-vision assistant for biomedicine in one day. Advances in Neural Information Processing Systems, 36, 2024. 3, 7

    2024

  14. [22]

    Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation

    Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In Interna- tional conference on machine learning, pages 12888–12900. PMLR, 2022. 3, 6

    2022

  15. [23]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In In- ternational conference on machine learning , pages 19730– 19742. PMLR, 2023. 3, 6

    2023

  16. [24]

    Masked vision and language pre-training with uni- modal and multimodal contrastive losses for medical visual question answering

    Pengfei Li, Gang Liu, Jinlong He, Zixu Zhao, and Shenjun Zhong. Masked vision and language pre-training with uni- modal and multimodal contrastive losses for medical visual question answering. In International Conference on Medi- cal Image Computing and Computer-Assisted Inter...

    2023

  17. [25]

    Lawrence Zitnick

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollar, and C. Lawrence Zitnick. Microsoft coco: Common objects in context. Eu- ropean Conference on Computer Vision (ECCV), 1:740–755,

  18. [26]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024. 3, 6, 7

    2024

  19. [27]

    Llava-instruct 150k: A large-scale instructional dataset for visual-language models

    L Liu, S Xu, Z Zhang, Y Wang, and L Li. Llava-instruct 150k: A large-scale instructional dataset for visual-language models. arXiv preprint arXiv:2304.11852, 2023. 6

    2023 arXiv

  20. [28]

    Foundation models for generalist medi- cal artificial intelligence

    Michael Moor, Oishi Banerjee, Zahra Shakeri Hossein Abad, Harlan M Krumholz, Jure Leskovec, Eric J Topol, and Pranav Rajpurkar. Foundation models for generalist medi- cal artificial intelligence. Nature, 616(7956):259–265, 2023. 3

    2023

  21. [29]

    Med-flamingo: a multimodal medical few-shot learner

    Michael Moor, Qian Huang, Shirley Wu, Michihiro Ya- sunaga, Yash Dalmia, Jure Leskovec, Cyril Zakka, Ed- uardo Pontes Reis, and Pranav Rajpurkar. Med-flamingo: a multimodal medical few-shot learner. In Machine Learning for Health (ML4H), pages 353–367. PMLR, 2023. 7

    2023

  22. [30]

    K-pathvqa: Knowledge-aware multimodal rep- resentation for pathology visual question answering

    Usman Naseem, Matloob Khushi, Adam G Dunn, and Jin- man Kim. K-pathvqa: Knowledge-aware multimodal rep- resentation for pathology visual question answering. IEEE Journal of Biomedical and Health Informatics, 2023. 7

    2023

  23. [31]

    Overcoming data limi- tation in medical visual question answering

    Diem Nguyen, Jason Moore, Anqiang Neville, Hung Bui, Quoc-Viet Do, and William Frakes. Overcoming data limi- tation in medical visual question answering. In IEEE Inter- national Conference on Computer Vision, 2019. 2

    2019

  24. [32]

    St-vqa: Visual question answering with a focus on scene texts

    Priyesh Patel, M Gupta, and Arvind Ranjan. St-vqa: Visual question answering with a focus on scene texts. In Interna- tional Conference on Computer Vision (ICCV) , pages 324– 335, 2019. 6

    2019

  25. [33]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, ...

    2021

  26. [34]

    Docvqa: A dataset for document visual question answer- ing

    Santanu Sahu, Pooja Patel, M Gupta, and Suresh Agarwal. Docvqa: A dataset for document visual question answer- ing. In International Conference on Document Analysis and Recognition (ICDAR), pages 545–552, 2020. 6

    2020

  27. [35]

    Maivar-t: Multimodal audio-image and video action recognizer using transformers

    Muhammad Bilal Shaikh, Douglas Chai, Syed Mo- hammed Shamsul Islam, and Naveed Akhtar. Maivar-t: Multimodal audio-image and video action recognizer using transformers. In 2023 11th European Workshop on Visual Information Processing (EUVIP), pages 1–6. IEEE, 2023. 1

    2023

  28. [36]

    Sharma, A

    R. Sharma, A. Gupta, S. Marwaha, and R. Singh. Path-vqa: A visual question answering dataset for digital pathology. In IEEE International Symposium on Biomedical Imaging (ISBI), pages 123–135, 2019. 6

    2019

  29. [37]

    Multimodal few-shot learning with frozen language models

    Maria Tsimpoukelli, Jacob L Menick, Serkan Cabi, SM Es- lami, Oriol Vinyals, and Felix Hill. Multimodal few-shot learning with frozen language models. Advances in Neural Information Processing Systems, 34:200–212, 2021. 3

    2021

  30. [38]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszko- reit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, 2017. 1

    2017

  31. [39]

    Medical graph rag: Towards safe medical large language model via graph retrieval-augmented generation

    Junde Wu, Jiayuan Zhu, and Yunli Qi. Medical graph rag: Towards safe medical large language model via graph retrieval-augmented generation. CoRR, 2024. 3

    2024

  32. [40]

    Mmed-rag: Versatile multimodal rag system for medical vi- sion language models

    Peng Xia, Kangyu Zhu, Haoran Li, Tianze Wang, Weijia Shi, Sheng Wang, Linjun Zhang, James Zou, and Huaxiu Yao. Mmed-rag: Versatile multimodal rag system for medical vi- sion language models. In Neurips Safe Generative AI Work- shop 2024. 3

    2024

  33. [41]

    Rule: Reliable multimodal rag for factuality in medical vision language models

    Peng Xia, Kangyu Zhu, Haoran Li, Hongtu Zhu, Yun Li, Gang Li, Linjun Zhang, and Huaxiu Yao. Rule: Reliable multimodal rag for factuality in medical vision language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 1081–1093,

    2024

  34. [42]

    Ramm: Retrieval-augmented biomedical visual question answering with multi-modal pre-training

    Zheng Yuan, Qiao Jin, Chuanqi Tan, Zhengyun Zhao, Hongyi Yuan, Fei Huang, and Songfang Huang. Ramm: Retrieval-augmented biomedical visual question answering with multi-modal pre-training. In Proceedings of the 31st ACM International Conference on Multimedia , pages 547– 556, 2023. 3

    2023

  35. [43]

    A generalist vision–language foundation model for diverse biomedical tasks

    Kai Zhang, Rong Zhou, Eashan Adhikarla, Zhiling Yan, Yixin Liu, Jun Yu, Zhengliang Liu, Xun Chen, Brian D Davi- son, Hui Ren, et al. A generalist vision–language foundation model for diverse biomedical tasks. Nature Medicine, pages 1–13, 2024. 3

    2024

  36. [44]

    Biomedclip: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairs

    Sheng Zhang, Yanbo Xu, Naoto Usuyama, Hanwen Xu, Jaspreet Bagga, Robert Tinn, Sam Preston, Rajesh Rao, Mu Wei, Naveen Valluri, et al. Biomedclip: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairs. arXiv preprint arXiv:2303.00915,

    arXiv

  37. [45]

    Multimodal representation learning by alternating uni- modal adaptation

    Xiaohui Zhang, Jaehong Yoon, Mohit Bansal, and Huaxiu Yao. Multimodal representation learning by alternating uni- modal adaptation. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 27456–27466, 2024. 3

    2024

  38. [46]

    Minigpt-4: Enhancing vision-language understanding with advanced large language models

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mo- hamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592, 2023. 3, 6

    2023 arXiv

  39. [47]

    X. Zhu, Q. Zhang, Y . Lee, J. Lee, and P. Bhadra. Vqa-rad: A visual question answering dataset for radiology. Journal of Medical Imaging, 58:45–60, 2021. 6

    2021

  40. [48]

    Crossclr: Cross-modal contrastive learning for multi-modal video representations

    Mohammadreza Zolfaghari, Yi Zhu, Peter Gehler, and Thomas Brox. Crossclr: Cross-modal contrastive learning for multi-modal video representations. In Proceedings of the IEEE/CVF International Conference on Computer Vi- sion, pages 1450–1459, 2021. 3

    2021

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.