Pith. sign in

REVIEW 3 major objections 6 minor 58 references

Uncertainty-Driven Expert Control: Enhancing the Reliability of Medical Vision-Language Models

T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Entropy-gated expert guidance lets a 4.2B medical VLM beat 13B models.

desk verdict The headline result is inflated by answer leakage: open-ended 'expert annotations' are ground-truth answers, so the 4.2B-beats-13B claim does not hold as stated; the uncertainty and retrieval analyses still have some value. read the letter →

arxiv 2507.09209 v1 pith:IS2QF5ME submitted 2025-07-12 cs.CV

Xiao Liang , Di Wang , Zhicheng Jiao , Ronghan Li , Pengfei Yang , Quan Wang , Tat-Seng Chua This is my paper
classification cs.CV
keywords medicalvision-languagemodelsclassifier-freeguidancepredictiveentropyuncertaintyestimationvisualquestionansweringexpert-in-the-loopretrieval-augmentedgenerationinference-timereliability
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Medical vision-language models give confident-sounding but sometimes wrong answers, and fixing them usually means expensive fine-tuning. This paper claims that reliability can be improved at inference time, with no additional training, by measuring the model's predictive entropy on each answer, retrieving relevant medical knowledge, having an expert highlight the key terms, and then steering the model's token generation with classifier-free guidance toward those highlights. Applied to only the roughly five percent of answers the model is least sure about, the method outperforms 13-billion-parameter models on three medical visual question answering benchmarks while using a 4.2-billion-parameter backbone. A sympathetic reader would take the central claim to be that selective, expert-controlled guidance is a practical substitute for larger models or more training.

What carries the argument

The central object is a token-level highlight mask combined with classifier-free guidance. During generation, the prompt branch is split into a normal context and an unconditional context in which highlighted tokens are scaled nearly to zero; attention scores for highlighted tokens are boosted by a factor beta, and the next-token logits are re-weighted as gamma times the conditional distribution minus (gamma minus one) times the unconditional distribution. The entropy gate decides which answers get this treatment, and the highlight mask is what carries the expert's knowledge into the model.

What would settle it

Rerun the pipeline on open-ended questions with annotations written by clinicians who see the image but are blind to the gold answer; if accuracy then matches plain retrieval-augmented generation rather than the reported numbers, the claim that expert highlights rather than gold answers drive the gains is refuted.

Watch

Extended reading notes

Core claim

Expert-CFG establishes that unreliable MedVLM outputs can be identified by length-normalized predictive entropy, with AUC above 0.8 on VQA-RAD, SLAKE, and PathVQA, and that steering generation with classifier-free guidance conditioned on expert-highlighted key terms recovers enough accuracy to beat 13B-parameter state-of-the-art models on all three benchmarks. The paper reports that retrieval alone gives small or negative gains, while the same retrieved references plus expert highlights, when forced through CFG, produce the large improvements; the guidance resolves conflicts between retrieved knowledge and the model's internal knowledge.

Load-bearing premise

The headline result assumes that the person providing the expert annotation already knows the exact correct answer to each open-ended question, because in the experiments that correct answer is what gets passed to the model.

Editorial extensions

If this is right

  • A 4.2B-parameter MedVLM with Expert-CFG outperforms 13B models on VQA-RAD, SLAKE, and PathVQA, so model size alone is not the binding constraint on medical VQA accuracy.
  • Experts only need to annotate the top roughly five percent of high-entropy outputs, making the workload small enough for routine clinical deployment.
  • Retrieval-augmented generation without expert-controlled guidance produces limited or negative gains on these benchmarks, so aligning retrieved knowledge with the model's own priors is the crucial step.
  • Predictive entropy reliably flags wrong answers with AUC above 0.8, enabling selective intervention rather than full manual review.
  • Since the method requires no training, it can be applied to an already-deployed MedVLM at inference time.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The reported gains rely on feeding the ground-truth answer as the expert annotation for open-ended questions; with noisy or partial real-world annotations, the improvement over baseline would likely be smaller than reported.
  • Because the top-five-percent entropy gate selects the samples most likely to be wrong, part of the gain is a selection effect; comparing Expert-CFG with an equal-size random sample of annotations would isolate the guidance's true contribution.
  • The same recipe of entropy gating plus highlight-conditioned CFG could transfer to other high-stakes VLM settings, such as radiology report generation or drug-label QA, wherever a brief expert highlight is cheaper than full verification.
  • An automated detector or classifier could replace the human expert for highlight generation, turning the framework into a fully automatic reliability layer.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes Expert-CFG, a training-free expert-in-the-loop framework that aligns medical vision-language models with clinical expertise. The method first estimates predictive entropy to identify unreliable outputs, retrieves relevant medical references, asks an expert to highlight key terms, and then applies token-level classifier-free guidance with embedding and attention scaling to steer generation toward the highlighted knowledge. Experiments on VQA-RAD, SLAKE, and PathVQA with 4.2B Phi3/Phi3.5-Vision models report accuracy gains over the baseline and claim state-of-the-art performance, including outperformance of 13B models. The abstract and conclusion present this as a practical, low-cost way to improve reliability in resource-limited clinical settings.

Significance. If the reported gains were obtained under realistic expert guidance, the work would be a useful contribution: it is training-free, includes a thoughtful uncertainty-selection component with reported AUC values above 0.8, and contains a transparent analysis of retrieval hit rates that honestly shows their limitations. The authors also release code and ablate the main hyperparameters. However, the central claim is not established because the open-ended evaluation injects the ground-truth answer as the 'expert annotation' and the open-ended metric rewards copying ground-truth tokens. The large open-ended gains and the near-ceiling 100%-annotation results in Appendix G are therefore an oracle-assisted upper bound rather than evidence that realistic expert guidance closes the gap to 13B models. With the current evaluation, the headline contribution is unsupported.

major comments (3)
  1. [Section 5.2, Eq. (8), Table 1] The paper states in Section 5.2 that for open-ended questions it directly uses the ground truth as expert annotations. Under Eq. (8) with gamma=1.5, the model's next-token logits are reweighted toward the highlighted tokens, and the open-ended metric in Section 4.3 is recall of ground-truth tokens in the generated response. This means the measured gains on the open-ended rows of Table 1 (e.g., Phi3.5V-Med PathVQA open-ended +8.66, VQA-RAD +5.59) are mechanically inflated by conditioning on the answer itself. Appendix G confirms the issue: with 100% annotations, overall accuracy reaches 98.44% on VQA-RAD and 99.62% on SLAKE, values that cannot be explained by the reported retrieval hit rates of at most 56%. These numbers demonstrate answer leakage, not the effect of realistic expert knowledge. The central claim of outperforming 13B models rests on this oracle-assisted evaluation and is therefore not supported.
  2. [Figure 4 caption, Section 5.2] The Figure 4 caption states that expert annotations are 'generated automatically through entity extraction and string matching,' and Section 5.2 says the same for closed-ended questions. No actual clinician annotations are used in the quantitative experiments. The paper repeatedly emphasizes reducing the burden on medical professionals, but the reported results do not measure any human workload, nor do they evaluate annotation quality, partial annotations, or annotation errors. The framework should be evaluated with realistic, possibly noisy human or simulated expert annotations that are not the ground-truth answer, and performance should be reported as a function of annotation quality. Without this, the 'expert-in-the-loop' claim is only a proposal, not a validated system.
  3. [Section 5.2, Figure 3, Table 1] The intervention is applied only to the top 5% highest-entropy samples, and Figure 3 shows that high entropy is a strong predictor of incorrect answers (AUC > 0.8). Consequently, the intervention is concentrated on exactly the samples where the baseline is most likely to fail. The paper never reports baseline accuracy restricted to the intervened subset, nor a comparison with a random 5% subset under the same annotation budget. The reported overall gains are therefore uninterpretable as evidence that uncertainty-driven targeting itself is beneficial; the comparison conflates oracle correction on hard examples with the effect of selective intervention.
minor comments (6)
  1. [Appendix G, Table 7] The row label for the Phi3.5V 4.2B baseline appears before the column headers for alpha, beta, and gamma, which makes the table layout confusing; please reformat.
  2. [Appendix B and C] 'Expeted Ouput' should be 'Expected Output'.
  3. [Appendix B, Table 5] The reference to 'Table' is incomplete; a table number or caption is missing.
  4. [Section 5.3, Appendix H] The case studies describe the expert 'manually adding' text, but the quantitative experiments use automatic string matching; the relationship between these two protocols should be clarified, since they imply different annotation costs and different amounts of information.
  5. [Table 1] No statistical significance tests, confidence intervals, or multiple-seed results are reported, although the claimed differences between Expert-CFG and Expert-RAG are sometimes small (e.g., 2-3 points on closed-ended subsets).
  6. [Eq. (5)] Eq. (5) writes the CFG weighting as a product over tokens without explicitly normalizing at each step; the notation should be made precise to avoid confusion with the standard per-token formulation.

Circularity Check

1 steps flagged · score 7.0 of 10

Open-ended gains are oracle-assisted by construction: ground-truth answers are fed as 'expert annotations' and the open-ended metric is recall of ground-truth tokens, so the headline 4.2B-beats-13B claim rests on a circular evaluation for open-ended questions.

  1. self definitional [Section 4.3 (Experimental Setup and Evaluation), Section 5.2 (Balancing Expert Workload and Accuracy), Eq. (7)-(8) in Section 3.3, Appendix G.]
    "Following the practice of LLaV A-Med [24], we evaluate performance using accuracy for closed-set questions and recall (the ratio of ground-truth tokens appearing in the generated response) for open-set questions. ... To reduce the workload, for open-ended questions, we directly use the ground truth as expert annotations. ... The introduction of expert annotations equivalent to gold standards undoubtedly enhances the performance of MedVLM."

    For open-ended questions, the 'expert annotation' is defined to be the ground-truth answer (Section 5.2). Equation (7)-(8) then make those highlighted ground-truth tokens the conditional branch of classifier-free guidance with gamma=1.5, re-weighting the token distribution toward exactly those tokens. The open-ended evaluation metric is the ratio of ground-truth tokens appearing in the generated response (Section 4.3). Therefore the reported open-ended improvements (e.g., PathVQA open-ended 37.42 to 46.08) are not an independent prediction from expert knowledge; they are mechanically inflated by feeding the target answer into the conditional branch and measuring overlap with that target.

full rationale

The core methodology—predictive-entropy uncertainty estimation, retrieval, and classifier-free guidance—is not itself circular; those components are externally meaningful and the closed-ended results are not directly contaminated by the oracle mechanism. However, the paper's strongest claim, that a 4.2B Expert-CFG outperforms 13B state-of-the-art models, depends heavily on open-ended scores, which are the largest improvements in Table 1. Section 5.2 explicitly states that for open-ended questions the ground truth is used as the expert annotation, while Section 4.3 defines the open-ended metric as recall of ground-truth tokens in the response. Equation (8) then amplifies the probability of those ground-truth tokens via CFG. Thus the open-ended evaluation reduces, by the paper's own definitions and equations, to measuring how well the model regurgitates an answer that was inserted into its conditional branch. The near-ceiling Appendix G results with 100% annotations confirm this mechanism. Because the headline outperformance rests substantially on these oracle-assisted open-ended numbers, the central claim is partially circular, even though the framework itself has non-circular components and the closed-ended gains are modest and more credible.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the realism of the expert annotation step and on the adequacy of entropy-based selection, not on new physics or new entities. The free parameters γ, α, β, the top-k count, the CLIP threshold, and the 5% selection fraction are tuned on the evaluation sets, and the oracle annotation in the main experiment is the largest uncontrolled input.

free parameters (6)
  • γ (CFG guidance strength) = 1.5
    Controls logit emphasis on expert-highlighted tokens in Eq. (8); set by ablation in Appendix G, not theoretically derived.
  • α (embedding scaling factor) = 0.01
    Scales the unconditional branch in Eq. (7); hand-set so highlighted tokens are ignored in the unconditional context.
  • β (attention scaling factor) = 3
    Modulates attention probabilities in Eq. (9); derived from δ = log(β) + 2 and tuned in ablation.
  • High-entropy annotation fraction = 5%
    Top 5% of test samples by predictive entropy receive expert intervention; chosen as workload-accuracy balance in Section 5.2.
  • CLIP score relevance threshold = 0.6
    Documents with CLIP scores below 0.6 are considered uninformative (Appendix F); set empirically, though score distributions for hits and misses overlap heavily.
  • Top-k retrieval count = 1
    Main experiments use top-1 retrieved reference; hit rate improves up to k=4 but plateaus (Figure 5).
assumptions (4)
  • domain assumption Predictive entropy from a single greedy decoding pass is a reliable indicator of answer correctness.
    Used in Section 3.2 to gate expert intervention; validated on the same test sets with AUC > 0.8, but low-entropy errors remain (Appendix J).
  • domain assumption The LoRA fine-tuned Phi-3.5V model is a strong baseline whose token distributions can be steered by classifier-free guidance.
    Foundation of the method; baseline accuracy is shown in Table 3 but the steerability is assumed from prior work [57].
  • domain assumption GPT-4 can extract and select clinically relevant keywords from captions and queries.
    Automates expert annotations (Appendix C); recall 89.57% on VQA-RAD only, no clinical validation.
  • domain assumption BioMedCLIP and FAISS retrieval return references that are relevant enough to support corrections of uncertain answers.
    Top-1 hit rate below 56% (Table 2), so this is a weak premise for the closed-ended and retrieval-dependent parts of the system.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Uncertainty-Driven Expert Control: Enhancing the Reliability of Medical Vision-Language Models." pith.science (2026). https://pith.science/paper/IS2QF5ME

@misc{pith2026250709209,
  author       = {Pith},
  title        = {Pith review of: Uncertainty-Driven Expert Control: Enhancing the Reliability of Medical Vision-Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IS2QF5ME}},
  note         = {Machine review of arXiv:2507.09209}
}
read the original abstract

The rapid advancements in Vision Language Models (VLMs) have prompted the development of multi-modal medical assistant systems. Despite this progress, current models still have inherent probabilistic uncertainties, often producing erroneous or unverified responses-an issue with serious implications in medical applications. Existing methods aim to enhance the performance of Medical Vision Language Model (MedVLM) by adjusting model structure, fine-tuning with high-quality data, or through preference fine-tuning. However, these training-dependent strategies are costly and still lack sufficient alignment with clinical expertise. To address these issues, we propose an expert-in-the-loop framework named Expert-Controlled Classifier-Free Guidance (Expert-CFG) to align MedVLM with clinical expertise without additional training. This framework introduces an uncertainty estimation strategy to identify unreliable outputs. It then retrieves relevant references to assist experts in highlighting key terms and applies classifier-free guidance to refine the token embeddings of MedVLM, ensuring that the adjusted outputs are correct and align with expert highlights. Evaluations across three medical visual question answering benchmarks demonstrate that the proposed Expert-CFG, with 4.2B parameters and limited expert annotations, outperforms state-of-the-art models with 13B parameters. The results demonstrate the feasibility of deploying such a system in resource-limited settings for clinical use.

Figures

Figures reproduced from arXiv: 2507.09209 by the authors.

Figure 1
Figure 1. Left: Vanilla MedVLM produces unreliable responses due to lack of clinical expertise. Middle: Existing methods mostly rely on training and are costly while still lacking sufficient clinical alignment. Right: Our proposed Expert-CFG utilizes plug-and-play selective expert annotations and classifier-free guidance for effective expertise alignment. Abstract The rapid advancements in Vision Language Models (VLMs) have p… view at source ↗
Figure 2
Figure 2. Overview of our proposed Expert-CFG, including: (1) Baseline MedVLM with visual instruction tuning; (2) Uncertainty estimation through normalized predictive entropy calculation for MedVLM responses; (3) Expert-Controlled Text Generation, incorpo￾rating expert control to adjust embeddings and re-weight predictions. 3.3. Expert-Controlled Text Generation Knowledge Reference Retrieval. Although retrieval￾augmented gene… view at source ↗
Figure 3
Figure 3. Uncertainty Estimation Performance of Three Med￾VQA Benchmarks. (a) Receiver Operating Characteristic (ROC) curves and corresponding Area Under the Curve (AUC). (b) Box plots of sensitivity and specificity. sification, we find that high entropy strongly correlates with incorrect predictions. The performance metric, AUC (Area Under the Curve, used to distinguish between correct and incorrect answers), exceeds 0.8 acr… view at source ↗
Figures from the paper (12 more)
Figure 4
Figure 4. Figure 4: MedVQA accuracy across three datasets with additional expert annotations. Note that expert annotations (highlights) are generated automatically through entity extraction and string matching. 2 4 6 8 10 Top-k 0.55 0.60 0.65 0.70 0.75 Hit Rate Top-k Hit Rate of External …
Figure 5
Figure 5. Figure 5: Hit Rate of answers directly appearing in retrieved captions for Top-K retrieval. Considering that the answers to closed-ended questions, such as “yes” and “no,” do not directly appear in captions, we evaluate only open-ended questions. Feature Type VQA-RAD SLAKE PathV…
Figure 6
Figure 6. Figure 6: Examples from VQA-RAD, SLAKE, and PathVQA datasets for zero-shot predictions by Phi3.5V-Med. The probability of each token is visualized using a colormap, comparing the original output, retrieval-augmented generation (RAG) with expert annotations, and expert-controlled…
Figure 7
Figure 7. Figure 7: illustrates the GPT-4 prompt used for converting question-answer pairs into coherent and clinically relevant captions. Instruction: Combine the following pairs of questions and answers into concise sentences while maintaining clarity and relevance. Use proper grammar a…
Figure 8
Figure 8. Figure 8: GPT-4 prompt for extracting medical terms from captions. Instruction: Given a query, select the most relevant keywords from the provided list and highlight them. If multiple keywords are relevant, separate them with a semicolon (;). Example Input: Query: Is this an MRI…
Figure 9
Figure 9. Figure 9: GPT-4 prompt for highlighting relevant key medical terms based on a query. D. Discussion on Uncertainty Estimation Metric For uncertainty estimation, we selected the entropy of the generated answer. This choice was motivated by a combination of effectiveness, efficienc…
Figure 10
Figure 10. Figure 10: PathVQA Dataset: (a) Entropy Distribution and (b) Overlapping Answer Hit Rate for Top-k. female pa fat 5mm cavum vergae mediport x-ray brain ct gallstones bilateral lungs right t2-weighted mri chest not seen here diverticulitis mri ascending colon motion r frontal lob…
Figure 11
Figure 11. Figure 11: VQA-RAD Dataset: (a) Entropy Distribution and (b) Overlapping Answer Hit Rate for Top-k [PITH_FULL_IMAGE:figures/full_fig_p014_11.png]
Figure 12
Figure 12. Figure 12: Slake dataset: (a) Entropy Distribution and (b) Overlapping Answer Hit Rate for Top-k. F. Knowledge Retrieval Our knowledge retrieval process utilizes CLIP scores for a preliminary relevance assessment. We rank reference texts by their CLIP scores and select the Top-K…
Figure 13
Figure 13. Figure 13: CLIP score distributions for open-ended QA hits and misses [PITH_FULL_IMAGE:figures/full_fig_p015_13.png]
Figure 14
Figure 14. Figure 14: Zero-shot case study examples from three datasets: VQA-RAD, SLAKE, and PathVQA [PITH_FULL_IMAGE:figures/full_fig_p017_14.png]
Figure 15
Figure 15. Figure 15: Additional zero-shot chinese case study examples from three datasets: VQA-RAD, SLAKE, and PathVQA [PITH_FULL_IMAGE:figures/full_fig_p018_15.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

58 extracted references · 32 canonical work pages

  1. [1]

    Phi-3 technical report: A highly capable language model locally on your phone

    Marah Abdin, Sam Ade Jacobs, and Ammar Ahmad Awan et al. Phi-3 technical report: A highly capable language model locally on your phone. ArXiv, abs/2404.14219, 2024. 3, 5

    arXiv 2024

  2. [2]

    Flamingo: a visual language model for few-shot learning

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, and An- toine Miech et al. Flamingo: a visual language model for few-shot learning. abs/2204.14198, 2022. 2

    arXiv 2022

  3. [3]

    M3d: Advancing 3d medical image analysis with multi-modal large language models

    Fan Bai, Yuxin Du, and Tiejun Huang et al. M3d: Advancing 3d medical image analysis with multi-modal large language models. ArXiv, abs/2404.00578, 2024. 2, 6

    arXiv 2024

  4. [4]

    Qwen-vl: A frontier large vision-language model with versatile abilities

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, and Sinan Tan et al. Qwen-vl: A frontier large vision-language model with versatile abilities. 2023. 1

    work page 2023

  5. [5]

    Huatuogpt-vision, towards injecting medical visual knowledge into multimodal llms at scale

    Junying Chen, Ruyi Ouyang, Anningzhe Gao, Shunian Chen, and Guiming Hardy Chen et al. Huatuogpt-vision, towards injecting medical visual knowledge into multimodal llms at scale. ArXiv, abs/2406.19280, 2024. 2, 3, 5, 8, 12

    arXiv 2024

  6. [6]

    Cost-effective Instruction Learning for Pathology Vision and Language Analysis

    Kaitao Chen, Mianxin Liu, Fang Yan, Lei Ma, Xiaoming Shi, Lilong Wang, Xiaosong Wang, Lifeng Zhu, Zhe Wang, Mu Zhou, and Shaoting Zhang. Cost-effective instruction learning for pathology vision and language analysis. ArXiv, abs/2407.17734, 2024. 2

    work page Pith review arXiv 2024

  7. [7]

    Advancing High Resolution Vision-Language Models in Biomedicine

    Zekai Chen, Arda Pekis, and Kevin Brown. Advancing high resolution vision-language models in biomedicine. ArXiv, abs/2406.09454, 2024. 2, 3

    work page Pith review arXiv 2024

  8. [8]

    Biomedical Visual Instruction Tuning with Clinician Preference Alignment

    Hejie Cui, Lingjun Mao, Xin Liang, Jieyu Zhang, Hui Ren, Quanzheng Li, Xiang Li, and Carl Yang. Biomedical vi- sual instruction tuning with clinician preference alignment. ArXiv, abs/2406.13173, 2024. 2, 3, 6

    work page Pith review arXiv 2024

Show all 58 references
  1. [9]

    Pa-llava: A large language-vision assistant for human pathology image understanding

    Dawei Dai, Yuanhui Zhang, Long Xu, Qianlan Yang, Xiao- jing Shen, Shuyin Xia, and Guo Chang Wang. Pa-llava: A large language-vision assistant for human pathology image understanding. ArXiv, abs/2408.09530, 2024. 2

    2024 arXiv

  2. [10]

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Albert Li, Pascale Fung, and Steven C. H. Hoi. Instructblip: To- wards general-purpose vision-language models with instruc- tion tuning. ArXiv, abs/2305.06500, 2023. 2

    2023 arXiv

  3. [11]

    Pubmedclip: How much does clip benefit visual question answering in the medical domain? In Findings of the As- sociation for Computational Linguistics, 2023

    Sedigheh Eslami, Christoph Meinel, Gerard, and de Melo. Pubmedclip: How much does clip benefit visual question answering in the medical domain? In Findings of the As- sociation for Computational Linguistics, 2023. 6

    2023

  4. [12]

    Ophglm: Training an ophthalmology large language-and-vision assistant based on instructions and dialogue

    Weihao Gao, Zhuo Deng, Zhiyuan Niu, Fuju Rong, Chucheng Chen, Zheng Gong, Wenze Zhang, and Daimin Xiao et al. Ophglm: Training an ophthalmology large language-and-vision assistant based on instructions and dialogue. ArXiv, abs/2306.12174, 2023. 2

    2023 arXiv

  5. [13]

    Pefomed: Parameter efficient fine-tuning of multi- modal large language models for medical imaging

    Jinlong He, Pengfei Li, Gang Liu, Zixu Zhao, and Shenjun Zhong. Pefomed: Parameter efficient fine-tuning of multi- modal large language models for medical imaging. ArXiv, abs/2401.02797, 2024. 3

    2024 arXiv

  6. [14]

    Xing, and Pengtao Xie

    Xuehai He, Yichen Zhang, Luntian Mou, Eric P. Xing, and Pengtao Xie. Pathvqa: 30000+ questions for medical visual question answering. ArXiv, abs/2003.10286, 2020. 5, 12

    2003 arXiv

  7. [15]

    Classifier-free diffusion guidance

    Jonathan Ho. Classifier-free diffusion guidance. ArXiv, abs/2207.12598, 2022. 4

    2022 arXiv

  8. [16]

    Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen

    Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. ArXiv, abs/2106.09685, 2021. 3

    2021 arXiv

  9. [17]

    Ciem: Contrastive instruction evaluation method for better instruction tuning

    Hongyu Hu, Jiyuan Zhang, Minyi Zhao, and Zhenbang Sun. Ciem: Contrastive instruction evaluation method for better instruction tuning. ArXiv, abs/2309.02301, 2023. 2

    2023 arXiv

  10. [18]

    Opera: Alleviating hallucination in multi-modal large language models via over-trust penalty and retrospection-allocation

    Qidong Huang, Xiao wen Dong, and Pan Zhang et al. Opera: Alleviating hallucination in multi-modal large language models via over-trust penalty and retrospection-allocation. 2024 IEEE/CVF Conference on Computer Vision and Pat- tern Recognition (CVPR), pages 13418–13427, 2023. 2

    2024

  11. [19]

    Med-moe: Mixture of domain-specific ex- perts for lightweight medical vision-language models

    Songtao Jiang, Tuo Zheng, Yan Zhang, Yeying Jin, Li Yuan, and Zuozhu Liu. Med-moe: Mixture of domain-specific ex- perts for lightweight medical vision-language models. In Findings of the Association for Computational Linguistics: EMNLP, pages 3843–3860, 2024. 2, 3

    2024

  12. [20]

    Cheung et al

    Qiao Jin, Fangyuan Chen, Yiliang Zhou, Ziyang Xu, and Justin M. Cheung et al. Hidden flaws behind expert-level accuracy of multimodal gpt-4 vision in medicine. NPJ Digi- tal Medicine, 7, 2024. 2

    2024

  13. [21]

    Billion- scale similarity search with gpus

    Jeff Johnson, Matthijs Douze, and Herv ´e J ´egou. Billion- scale similarity search with gpus. IEEE Transactions on Big Data, 7:535–547, 2017. 4

    2017

  14. [22]

    Lau, Soumya Gayen, Asma Ben Abacha, and Dina Demner-Fushman

    Jason J. Lau, Soumya Gayen, Asma Ben Abacha, and Dina Demner-Fushman. A dataset of clinically generated visual questions and answers about radiology images. Scientific Data, 5, 2018. 5, 12

    2018

  15. [23]

    Mitigating object hal- lucinations in large vision-language models through visual contrastive decoding

    Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shi- jian Lu, Chunyan Miao, and Li Bing. Mitigating object hal- lucinations in large vision-language models through visual contrastive decoding. 2024 IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR), pages...

    2024

  16. [24]

    Llava-med: Training a large language- and-vision assistant for biomedicine in one day

    Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon, and Jianfeng Gao. Llava-med: Training a large language- and-vision assistant for biomedicine in one day. ArXiv, abs/2306.00890, 2023. 2, 5, 6, 8

    2023 arXiv

  17. [25]

    Self-supervised vision-language pretraining for medical visual question answering

    Pengfei Li, Gang Liu, Lin Tan, Jinying Liao, and Shen- jun Zhong. Self-supervised vision-language pretraining for medical visual question answering. ArXiv, abs/2211.13594,

    arXiv

  18. [26]

    Masked vision and language pre-training with uni- modal and multimodal contrastive losses for medical visual question answering

    Pengfei Li, Gang Liu, Jinlong He, Zixu Zhao, and Shenjun Zhong. Masked vision and language pre-training with uni- modal and multimodal contrastive losses for medical visual question answering. ArXiv, abs/2307.05314, 2023. 6

    2023 arXiv

  19. [27]

    Evaluating object hallucination in large vision-language models

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji rong Wen. Evaluating object hallucination in large vision-language models. In Conference on Empirical Methods in Natural Language Processing, 2023. 2

    2023

  20. [28]

    Divide and conquer: Isolating normal-abnormal attributes in knowledge graph-enhanced radiology report generation

    Xiao Liang, Yanlei Zhang, and Di Wang et al. Divide and conquer: Isolating normal-abnormal attributes in knowledge graph-enhanced radiology report generation. In Proceedings of the 32nd ACM International Conference on Multimedia , pages 4967–4975, 2024. 3

    2024

  21. [29]

    Slake: A semantically-labeled knowledge- enhanced dataset for medical visual question answering

    Bo Liu, Li-Ming Zhan, Li Xu, Lin Ma, Yan Fang Yang, and Xiao-Ming Wu. Slake: A semantically-labeled knowledge- enhanced dataset for medical visual question answering. 2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI), pages 1650–1654, 2021. 5, 12

    2021

  22. [30]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. InAdvances in Neural Information Processing Systems 36: Annual Conference on Neural Infor- mation Processing Systems, 2023. 1, 2, 8

    2023

  23. [31]

    A survey on hallucination in large vision-language models

    Hanchao Liu, Wenyuan Xue, Yifei Chen, Dapeng Chen, Xiu- tian Zhao, Ke Wang, Liping Hou, Rong-Zhi Li, and Wei Peng. A survey on hallucination in large vision-language models. ArXiv, abs/2402.00253, 2024. 2

    2024 arXiv

  24. [32]

    Negative object presence evaluation (nope) to measure object hallucination in vision-language models

    Holy Lovenia, Wenliang Dai, Samuel Cahyawijaya, Ziwei Ji, and Pascale Fung. Negative object presence evaluation (nope) to measure object hallucination in vision-language models. ArXiv, abs/2310.05338, 2023. 2

    2023 arXiv

  25. [33]

    Biogpt: Generative pre- trained transformer for biomedical text generation and min- ing

    Renqian Luo, Liai Sun, Yingce Xia, Tao Qin, Sheng Zhang, Hoifung Poon, and Tie-Yan Liu. Biogpt: Generative pre- trained transformer for biomedical text generation and min- ing. Briefings in bioinformatics, 2022. 2

    2022

  26. [34]

    A practical bayesian frame- work for backpropagation networks

    David John Cameron MacKay. A practical bayesian frame- work for backpropagation networks. Neural Computation, 4: 448–472, 1992. 3

    1992

  27. [35]

    Malinin and M

    A. Malinin and M. Gales. Uncertainty estimation in autore- gressive structured prediction. In ICLR, 2021. 3

    2021

  28. [36]

    Med-flamingo: a multimodal medical few-shot learner

    Michael Moor, Qian Huang, Shirley Wu, Michihiro Ya- sunaga, Cyril Zakka, Yashodhara Dalmia, Eduardo Pontes Reis, Pranav Rajpurkar, and Jure Leskovec. Med-flamingo: a multimodal medical few-shot learner. InMachine Learning for Health@NeurIPS, pages 353–367, 2023. 2

    2023

  29. [37]

    Gpt-4 technical report

    OpenAI. Gpt-4 technical report. ArXiv, abs/2303.08774,

    arXiv

  30. [38]

    Friedrich

    Obioma Pelka, Sven Koitka, Johannes R ¨uckert, Felix Nensa, and C. Friedrich. Radiology objects in con- text (roco): A multimodal image dataset. In CVII- STENT/LABELS@MICCAI, 2018. 5, 12

    2018

  31. [39]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, and Gabriel Goh et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, 2021. 5

    2021

  32. [40]

    Main page — pathology education instructional resource,, 2020

    Pathology Education Instructional Resource. Main page — pathology education instructional resource,, 2020. [Online; accessed 12-July-2023]. 5, 12

    2020

  33. [41]

    Multimedeval: A benchmark and a toolkit for evaluating medical vision-language models

    Corentin Royer, Bjoern H Menze, and Anjany Kumar Sekuboyina. Multimedeval: A benchmark and a toolkit for evaluating medical vision-language models. ArXiv, abs/2402.09262, 2024. 2

    2024

  34. [42]

    Stay on topic with classifier-free guidance

    Guillaume Sanchez, Honglu Fan, Alexander Spangher, Elad Levi, Pawan Sasanka Ammanamanchi, and Stella Bider- man. Stay on topic with classifier-free guidance. ArXiv, abs/2306.17806, 2023. 4

    2023 arXiv

  35. [43]

    Ikezogwo et al

    Mehmet Saygin Seyfioglu and Wisdom O. Ikezogwo et al. Quilt-llava: Visual instruction tuning by extracting lo- calized narratives from open-source histopathology videos. 2024 IEEE/CVF Conference on Computer Vision and Pat- tern Recognition (CVPR), pages 13183–13192, 2023. 2

    2024

  36. [44]

    Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christo- pher D. Manning. Just ask for calibration: Strategies for eliciting calibrated confidence scores from language mod- els fine-tuned with human feedback. ArXiv, abs/...

    arXiv

  37. [45]

    Open-ended medical visual ques- tion answering through prefix tuning of language models

    Tom van Sonsbeek, Mohammad Mahdi Derakhshani, and Ivona Najdenkoska et al. Open-ended medical visual ques- tion answering through prefix tuning of language models. ArXiv, abs/2303.05977, 2023. 6

    2023 arXiv

  38. [46]

    Cxpmrg-bench: Pre-training and benchmarking for x-ray medical report generation on chexpert plus dataset

    Xiao Wang, Fuling Wang, Yuehang Li, Qingchuan Ma, Shiao Wang, Bowei Jiang, Chuanfu Li, and Jin Tang. Cxpmrg-bench: Pre-training and benchmarking for x-ray medical report generation on chexpert plus dataset. ArXiv, abs/2410.00379, 2024. 2, 3

    arXiv

  39. [47]

    Towards generalist foundation model for radiol- ogy

    Chaoyi Wu, Xiaoman Zhang, Ya Zhang, Yanfeng Wang, and Weidi Xie. Towards generalist foundation model for radiol- ogy. ArXiv, abs/2308.02463, 2023. 2

    2023 arXiv

  40. [48]

    Cares: A com- prehensive benchmark of trustworthiness in medical vision language models

    Peng Xia, Ze Chen, and Juanxi Tian et al. Cares: A com- prehensive benchmark of trustworthiness in medical vision language models. ArXiv, abs/2406.06007, 2024. 2

    2024 arXiv

  41. [49]

    Mmed-rag: Versatile multimodal rag system for medical vision language models

    Peng Xia, Peng Xia, Kangyu Zhu, Haoran Li, Haoran Li, Weijia Shi, Sheng Wang, Linjun Zhang, James Zou, and Huaxiu Yao. Mmed-rag: Versatile multimodal rag system for medical vision language models. 2024. 3, 6

    2024

  42. [50]

    Rule: Reliable multimodal rag for factuality in medical vision language models

    Peng Xia, Kangyu Zhu, Haoran Li, Hongtu Zhu, Yun Li, Gang Li, Linjun Zhang, and Huaxiu Yao. Rule: Reliable multimodal rag for factuality in medical vision language models. ArXiv, abs/2407.05131, 2024. 3, 6

    2024 arXiv

  43. [51]

    Multimodal chatgpt for medi- cal applications: an experimental study of gpt-4v

    Zhiling Yan, Kai Zhang, Rong Zhou, Lifang He, Xi- ang Li, and Lichao Sun. Multimodal chatgpt for medi- cal applications: an experimental study of gpt-4v. ArXiv, abs/2310.19061, 2023. 2

    2023 arXiv

  44. [52]

    mplug-owi2: Revolutionizing multi-modal large lan- guage model with modality collaboration

    Qinghao Ye, Haiyang Xu, Jiabo Ye, Mingshi Yan, Anwen Hu, Haowei Liu, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. mplug-owi2: Revolutionizing multi-modal large lan- guage model with modality collaboration. 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition ...

    2024

  45. [53]

    Ramm: Retrieval-augmented biomedical visual question answering with multi-modal pre-training

    Zheng Yuan, Qiao Jin, and Chuanqi Tan et al. Ramm: Retrieval-augmented biomedical visual question answering with multi-modal pre-training. In Proceedings of the 31st ACM International Conference on Multimedia, 2023. 3, 7

    2023

  46. [54]

    A gener- alist vision-language foundation model for diverse biomedi- cal tasks

    Kai Zhang, Jun Yu, Zhilin Yan, and Yixin Liu et al. A gener- alist vision-language foundation model for diverse biomedi- cal tasks. Nature medicine, 2023. 2

    2023

  47. [55]

    Large-scale domain- specific pretraining for biomedical vision-language process- ing

    Shenmin Zhang, Yanbo Xu, Naoto Usuyama, Jaspreet Kaur Bagga, and Robert Tinn et al. Large-scale domain- specific pretraining for biomedical vision-language process- ing. ArXiv, abs/2303.00915, 2023. 4, 6, 12

    2023 arXiv

  48. [56]

    Pmc-vqa: Vi- sual instruction tuning for medical visual question answer- ing

    Xiaoman Zhang, Chaoyi Wu, Ziheng Zhao, Weixiong Lin, Ya Zhang, Yanfeng Wang, and Weidi Xie. Pmc-vqa: Vi- sual instruction tuning for medical visual question answer- ing. ArXiv, abs/2305.10415, 2023. 2

    2023 arXiv

  49. [57]

    Prompt highlighter: Interactive control for multi- modal llms

    Yuechen Zhang, Shengju Qian, Bohao Peng, Shu Liu, and Jiaya Jia. Prompt highlighter: Interactive control for multi- modal llms. 2024 IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition (CVPR), pages 13215–13224,

    2024

  50. [58]

    is True?

    Yiyang Zhou, Chenhang Cui, Jaehong Yoon, Linjun Zhang, Zhun Deng, Chelsea Finn, Mohit Bansal, and Huaxiu Yao. Analyzing and mitigating object hallucination in large vision-language models. ArXiv, abs/2310.00754, 2023. 2 A. Dataset Details Dataset Image Description QA-Pairs Typ...

    2023 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.