Pith. sign in

REVIEW 3 major objections 5 minor 61 references

CheXPO: Preference Optimization for Chest X-ray VLMs with Counterfactual Rationale

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that preference tuning on 30k automatically mined and counterfactually constructed preference pairs raises a chest X-ray VLM's overall accuracy from 73.20% to 79.74% (an 8.93% relative gain) and improves results on the…

desk verdict Solid, reproducible preference-optimization recipe for chest X-ray VLMs with real gains, but the unverified counterfactual rejections and missing 30k uniform baseline need attention before I'd trust the headline numbers. read the letter →

arxiv 2507.06959 v1 pith:PDBIIYXC submitted 2025-07-09 cs.CV cs.AI

Xiao Liang , Jiawei Hu , Di Wang , Zhi Ma , Lin Zhao , Ronghan Li , Bo Wan , Quan Wang This is my paper
classification cs.CVcs.AI
keywords chestX-rayVQAmedicalvision-languagemodeldirectpreferenceoptimizationcounterfactualrationalehardexamplemininghallucinationmitigationlong-taileddistributionDPO
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that preference optimization can make a chest X-ray vision-language model substantially more reliable without expert annotation, provided the preference pairs are chosen for clinical value rather than sampled at random. It introduces CheXPO, a pipeline that finds hard examples where an SFT model fails or is uncertain, expands them by semantic retrieval to counter the long-tailed distribution of findings, and builds rejected responses by replacing correct answers with clinically opposite or confusable counterfactuals. The resulting DPO tuning, trained on 30k pairs from roughly 5% of the SFT samples, lifts CheX-Phi3.5V's unified-test accuracy from 73.20% to 79.74% and improves benchmark QA on MIMIC-CXR-VQA and Medical-Diff-VQA. If correct, this means hallucination-prone medical VLMs can be aligned to radiologist-style reasoning with data generated automatically from existing reports, rather than with expensive expert feedback.

What carries the argument

The load-bearing mechanism is the counterfactual rejection rationale: for a low-confidence but correct prediction, the short answer is replaced by a clinically opposite or confusable alternative from a domain-specific rejection pool (left versus right lung, pneumonia versus pulmonary edema, mild versus severe), and a semantically coherent candidate rationale is retrieved with BioMedCLIP to serve as the rejected response in a DPO pair. This is coupled with confidence-similarity joint mining: a length-normalized log-probability threshold flags hard examples, and multi-modal cosine-similarity Top-K retrieval balances the preference set across the long-tailed distribution of findings. Together these components convert an SFT model's own failures and uncertainties into preference signals without expert labels.

What would settle it

A reader could take a random sample of the constructed preference pairs, treat each counterfactual rejected answer as a hypothesis, and ask a radiologist whether that swapped answer is actually false for that image; if a substantial fraction turn out to be true findings, the preference supervision is partly inconsistent. This audit is concrete because the rejection pools are deterministic and the paper already performs a 100-sample human evaluation of chosen rationales, so the same protocol can be applied to rejected rationales.

Watch

Extended reading notes

Core claim

The central claim is that DPO's value for medical VLMs is gated by data selection and negative construction, not by the preference objective itself. On a unified chest X-ray QA dataset of 640k expert-style rationales, SFT brings Phi-3.5V to 73.20% overall accuracy. CheXPO then identifies hard examples using length-normalized token-level log-probabilities of the short answer: wrong answers become rejected responses, and correct-but-low-confidence answers are paired with counterfactual rationales assembled from domain-specific rejection pools covering anatomy, abnormality, and severity, plus opposite answers such as left versus right lung. BioMedCLIP retrieval expands the hard set with similar questions, rationales, and images. DPO on 30k such pairs yields 79.74% accuracy, an 8.93% relative improvement over SFT, and better results than earlier medical VLMs on MIMIC-CXR-VQA (81.8% accuracy) and Medical-Diff-VQA (BLEU-4 of 0.564). The paper also reports that the preference-tuned model shows lower DPO training loss, higher confidence on previously uncertain Anatomy and Severity cases, and corrected confidently wrong Abnormality predictions.

Load-bearing premise

The counterfactual rejected answers are assumed to be genuinely false for the specific image, but they are constructed by swapping in opposite or confusable terms from hand-built rejection pools without checking the actual image, so an accidentally correct swapped answer would turn the DPO pair into self-contradictory supervision.

Editorial extensions

If this is right

  • A radiology VLM can be made more reliable with 30k preference pairs drawn from only 5% of the SFT samples, so preference alignment need not depend on expert-written feedback.
  • Gains concentrate in the three error families that dominate failures (Abnormality, Anatomy, Severity), so the method targets clinically risky cases rather than easy ones.
  • Counterfactual construction breaks the plateau that uniform and hard-only DPO hit near 10k pairs, allowing larger preference sets to keep improving performance.
  • The mining and counterfactual steps transfer across preference objectives, since DPO, SimPO, CPO, ORPO, and IPO all surpass SFT in the paper's ablations.
  • The GPT-4o rationales in the instruction dataset are grounded in MIMIC-CXR radiologist reports, so the preference signal traces back to expert-written clinical text.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same counterfactual-pool recipe should transfer to other imaging modalities whenever a set of clinically opposite or confusable labels can be enumerated; a testable extension is to swap in modality-specific rejection pools.
  • The reported probability shifts for previously uncertain cases suggest the method may improve calibration, not just accuracy; measuring expected calibration error on hard question types would test this directly.
  • Because hard examples are defined by one SFT model's failures, applying the mining loop again on the preference-tuned model could yield a second round of gains, which the paper does not test.
  • The paper audits the correctness of chosen rationales but not the falsity of rejected counterfactuals; an independent audit of rejected pairs would directly test the method's central assumption.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces CheXPO, a preference-optimization pipeline for chest X-ray vision-language models. The authors first construct a large multi-task visual instruction dataset by synthesizing GPT-4o-generated rationales over MIMIC-CXR, Chest ImaGenome, MIMIC-CXR-VQA, and Medical-Diff-VQA, and use it to supervised fine-tune Phi-3.5V with LoRA. They then apply DPO using preference pairs selected by token-level confidence analysis and BioMedCLIP-based similarity retrieval, with 'counterfactual rationales' constructed by replacing the short-answer component with tokens from anatomy, abnormality, and severity rejection pools. The paper reports an 8.93% relative accuracy improvement over the SFT baseline using 30k preference pairs (about 5% of the SFT data), and reports strong results on MIMIC-CXR-VQA and Medical-Diff-VQA.

Significance. If the claims hold, the paper makes a useful contribution: it demonstrates a data-efficient, expert-annotation-free way to apply preference optimization to medical VLMs, and the proposed hard-example mining plus counterfactual rationale recipe is portable to other medical imaging domains. The paper also contributes a large public instruction dataset and reports ablations over data size, retrieval modality, and alternative preference objectives. The main experimental gains are consistent across many settings, and the work includes a human evaluation of the generated chosen rationales. However, the central mechanism of the counterfactual rejection construction is not validated per image, and the headline 30k result is not covered by the ablation curves; these gaps need to be closed before the claims can be accepted at face value.

major comments (3)
  1. [Section 4.3, Eq. (5)] The counterfactual rejection construction never verifies that the replaced short answer is actually incorrect for the specific image. The rejection pools contain anatomically or diagnostically confusable pairs, such as 'left lung' vs. 'right lung' and 'pneumonia' vs. 'pulmonary edema'; in chest X-rays these conditions are frequently bilateral or co-occurring, so a swapped answer can be accidentally correct for a given study. When this happens, the DPO pair in Eq. (6) penalizes a valid response, directly undermining the mechanism that Table 10 and Figure 6 attribute to the counterfactual rationale component. The quality evaluation in Table 9 and Appendix F addresses the generated chosen rationales, not the constructed rejected responses. Please add an automatic per-image verification or filtering step (for example, using the MIMIC-CXR structured labels or the original report), report the estimated rate of accidentally correct rejections, and show that the gains persist after removing or correcting such pairs.
  2. [Figure 6 and Table 1] The headline result, the 8.93% relative gain over the SFT baseline, is reported for 30k DPO pairs in Table 1, but the ablation in Figure 6 stops at 20k pairs and does not include a uniform-sampling or w/o-CRej baseline at the 30k scale. As a result, the paper does not establish whether the 30k improvement comes from the proposed mining and counterfactual components or simply from using a larger preference set with a different selection strategy. Please include the 30k points for the uniform, w/o-CRej, and full CheXPO variants, or explicitly restrict the central claim to the 20k setting and present the 30k result as an additional scaling observation.
  3. [Section 6.2 / Appendix F] The presentation conflates two distinct objects: the quality evaluation in Table 9 is performed on generated dataset rationales (the chosen responses), while the counterfactual-rejected responses used in Eq. (5) are never evaluated on criteria such as factual incorrectness, semantic coherence with the image, or consistency between the replaced answer and the retrieved explanation. Because the paper's stated contribution is specifically the construction of counterfactual rejections, this asymmetry should be addressed directly rather than through the current Table 9 caption, which calls the evaluated items 'counterfactual rationales'.
minor comments (5)
  1. [Section 5.2 / Table 8] The values of gamma, sigma, and Top-K are described as 'adjusted based on the target preference dataset size,' but the paper does not report how these parameters were selected or whether they were tuned on a validation set; please clarify the selection procedure to avoid test-set contamination concerns.
  2. [Equation (5)] The notation for the constructed rejection response is unclear: the same symbol y is used for the model output, the replaced answer, and the retrieved Top-1 candidate. Please introduce distinct symbols for the original output, the modified answer, and the final rejected response.
  3. [Section 4.3] The rejection-pool replacement is described as 'randomly replace'; please state the random seed or the number of random trials, and whether the reported results are averaged over multiple seeds, because the preference data construction is stochastic.
  4. [Figure 5 and Table 2] The metric 'Error Sample Hit Rate' in Figure 5 and the 'Forwarded' column in Table 2 are not defined in the main text; please define them precisely and report the size of the query and gallery sets used for the evaluation.
  5. [Appendix G, Table 10] The win-rate metric is mentioned in the table footnote as 'the PO model's short answer over that of SFT,' but the exact evaluation protocol (ties, randomness, number of judges) is not given; please specify it.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the claimed preference-optimization gain is measured on held-out test splits against independent ground-truth answers.

full rationale

The paper's derivation chain has no load-bearing step that reduces to its own input by construction. The SFT rationales T are reused as the chosen responses in DPO (Sec. 4.3, Eq. 6), but the rejected responses are either the SFT model's own incorrect outputs or rationales retrieved after a domain-specific answer replacement (Eq. 5); the DPO loss is not algebraically equivalent to the SFT objective, and the chosen/rejected pairing is not defined in terms of the final test accuracy. The headline 8.93% relative gain (Table 1) is computed by strict string matching of short answers on the authors' unified test set, which follows the original train/validation/test splits of MIMIC-CXR-VQA, Medical-Diff-VQA, and Chest ImaGenome, and the separate benchmark results in Tables 6-7 are external comparisons. No parameter is fitted to the test set and then renamed a prediction. The paper invokes no uniqueness theorem from its own prior work, and the few cited techniques (DPO, BioMedCLIP, length normalization) are external and independently established. The most significant weakness is empirical rather than circular: the constructed 'counterfactual' rejection answers (e.g., swapping 'left lung' for 'right lung') are never verified to be false for the paired image, and Appendix F's quality evaluation (Table 9) assesses rationale correctness against reports rather than the truth of the rejection pairs. That is a data-quality and validity concern, not a self-referential reduction, so it does not raise the circularity score.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central claim rests on a chain of domain assumptions about data quality and evaluation validity. The most significant are the correctness of GPT-4o-generated rationales, the assumption that low-confidence correct answers are safe to use as chosen responses, the assumption that counterfactual replacements are wrong for the specific image, and the adequacy of strict short-answer matching as a clinical reliability metric. These are not machine-checked and are only partially validated by the reported human and automated evaluations.

free parameters (5)
  • sigma (log-prob threshold) = -0.3
    Threshold below which a correct short answer is treated as low-confidence and used to construct counterfactual rejections. Set as a hyperparameter in Section 5.2 and varied with target dataset size in Table 8.
  • gamma (stratified sampling ratio) = 0.9% to 2.7%
    Proportion of the SFT training set sampled to seed hard example mining; adjusted based on target preference dataset size (Section 5.2, Table 8).
  • Top-K (retrieval count) = 1 to 10
    Number of similar samples retrieved per hard example in Eq. (4); set based on target preference dataset size (Table 8).
  • Similarity combination weights = 1.0 each
    Weights for S_Q, S_T, S_V in Eq. (3) are fixed to 1.0, a design choice not ablated.
  • Rejection pool contents = hand-crafted terms
    Anatomy, Abnormality, and Severity rejection pools are manually defined clinical term groups; the paper does not validate that replacements are incorrect per image.
assumptions (5)
  • domain assumption GPT-4o-generated rationales derived from MIMIC-CXR reports are expert-level and clinically correct.
    Used to build the SFT training set and as chosen responses in preference pairs. A sample of 100 was human-validated (Table 9), but not all 640k.
  • domain assumption Low-confidence correct predictions (p < sigma) are genuinely correct and can be paired with synthetic counterfactual rejections.
    Section 4.1: correct but low-confidence answers are assumed to be correct, so the counterfactual replacement is a valid rejection.
  • ad hoc to paper Counterfactual answer replacements, such as left vs right lung, are incorrect for the specific image.
    Section 4.3: replacements are made without checking the substitute answer against the image; some could be clinically correct, creating noisy preference labels.
  • domain assumption BioMedCLIP cosine similarity across question, rationale, and image retrieves samples with similar failure modes.
    Section 4.2 and Figure 5 support this empirically on a 100-query validation, but the retrieval is used to expand the preference set without per-sample verification.
  • domain assumption Strict string matching on short answers is a valid proxy for clinical correctness.
    Section 5.2 and 6.3 note this limitation themselves: the SFT model sometimes reaches the right short answer with flawed reasoning, so the metric may not capture hallucination reduction.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CheXPO: Preference Optimization for Chest X-ray VLMs with Counterfactual Rationale." pith.science (2026). https://pith.science/paper/PDBIIYXC

@misc{pith2026250706959,
  author       = {Pith},
  title        = {Pith review of: CheXPO: Preference Optimization for Chest X-ray VLMs with Counterfactual Rationale},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PDBIIYXC}},
  note         = {Machine review of arXiv:2507.06959}
}
read the original abstract

Vision-language models (VLMs) are prone to hallucinations that critically compromise reliability in medical applications. While preference optimization can mitigate these hallucinations through clinical feedback, its implementation faces challenges such as clinically irrelevant training samples, imbalanced data distributions, and prohibitive expert annotation costs. To address these challenges, we introduce CheXPO, a Chest X-ray Preference Optimization strategy that combines confidence-similarity joint mining with counterfactual rationale. Our approach begins by synthesizing a unified, fine-grained multi-task chest X-ray visual instruction dataset across different question types for supervised fine-tuning (SFT). We then identify hard examples through token-level confidence analysis of SFT failures and use similarity-based retrieval to expand hard examples for balancing preference sample distributions, while synthetic counterfactual rationales provide fine-grained clinical preferences, eliminating the need for additional expert input. Experiments show that CheXPO achieves 8.93% relative performance gain using only 5% of SFT samples, reaching state-of-the-art performance across diverse clinical tasks and providing a scalable, interpretable solution for real-world radiology applications.

Figures

Figures reproduced from arXiv: 2507.06959 by the authors.

Figure 1
Figure 1. Naively adopting preference alignment for real-world clinical data makes it difficult to handle imbalanced distributions [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 1
Figure 1. Specifically, we unify multiple benchmarks [ [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Example of integrated multi-task QA dataset. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figures from the paper (12 more)
Figure 3
Figure 3. Figure 3: We first fine-tune Phi-3.5V [1] with the synthetic instruc￾tion dataset to obtain a supervised base model, CheX-Phi3.5V. The CheXPO strategy is then employed, utilizing confidence-similarity joint mining to select hard examples, followed by counterfactual rationale con…
Figure 3
Figure 3. Figure 3: Overview of our chest X-ray VLM training pipeline. Our framework leverages visual instruction tuning and preference [PITH_FULL_IMAGE:figures/full_fig_p004_3.png]
Figure 4
Figure 4. Figure 4: Comparison of distribution between SFT failure [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Retrieval Performance for SFT Failure Cases. The [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Win rates relative to the SFT baseline across dif [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: Training loss across three sampling strategies (20k [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]
Figure 8
Figure 8. Figure 8: Distribution of answer-averaged probabilities for [PITH_FULL_IMAGE:figures/full_fig_p007_8.png]
Figure 9
Figure 9. Figure 9: Representative chest X-ray cases across three major tasks (Basic-QA, Region-QA, Comparison-QA), comparing outputs [PITH_FULL_IMAGE:figures/full_fig_p008_9.png]
Figure 10
Figure 10. Figure 10: Distribution of question types in the dataset. [PITH_FULL_IMAGE:figures/full_fig_p011_10.png]
Figure 11
Figure 11. Figure 11: Example of Region-QA Data Attributes. # X-Ray Diagnosis Explanation Generator You are tasked with generating a concise radiological explanation for a chest X-ray diagnosis. Your explanation should simulate the diagnostic reasoning process of a professional radiologist…
Figure 12
Figure 12. Figure 12: Prompt Template for Basic/Region-QA [PITH_FULL_IMAGE:figures/full_fig_p015_12.png]
Figure 13
Figure 13. Figure 13: Prompt Template for Comparison-QA [PITH_FULL_IMAGE:figures/full_fig_p016_13.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

61 extracted references · 36 canonical work pages

  1. [1]

    Marah Abdin, Sam Ade Jacobs, and Ammar Ahmad Awan et al. 2024. Phi-3 Tech- nical Report: A Highly Capable Language Model Locally on Your Phone.ArXiv abs/2404.14219 (2024). https://api.semanticscholar.org/CorpusID:269293048

    arXiv 2024

  2. [2]

    Seongsu Bae, Daeun Kyung, Jaehee Ryu, Eunbyeol Cho, Gyubok Lee, Sunjun Kweon, Jungwoo Oh, Lei Ji, Eric I-Chao Chang, Tackeun Kim, and Edward Choi

    work page

  3. [3]

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng X...

    arXiv 2023

  4. [4]

    Meyer, Yuning Chai, Dennis Park, and Yong Jae Lee

    Mu Cai, Haotian Liu, Siva Karthik Mustikovela, Gregory P. Meyer, Yuning Chai, Dennis Park, and Yong Jae Lee. 2023. ViP-LLaVA: Making Large Multimodal Models Understand Arbitrary Visual Prompts.2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)(2023), 12914–12923. https: //api.semanticscholar.org/CorpusID:265551875

    work page 2023

  5. [5]

    Junying Chen, Ruyi Ouyang, Anningzhe Gao, Shunian Chen, Guiming Hardy Chen, Xidong Wang, Ruifei Zhang, Zhenyang Cai, Ke Ji, Guangjun Yu, Xiang Wan, and Benyou Wang. 2024. HuatuoGPT-Vision, Towards Injecting Medical Visual Knowledge into Multimodal LLMs at Scale. arXiv:2406.19280 [cs.CV] https://arxiv.org/abs/2406.19280

    arXiv 2024

  6. [6]

    Zhihong Chen, Yu Du, Jinpeng Hu, Yang Liu, Guanbin Li, Xiang Wan, and Tsung- Hui Chang. 2022. Multi-modal Masked Autoencoders for Medical Vision-and- Language Pre-training.ArXivabs/2209.07098 (2022). https://api.semanticscholar. org/CorpusID:252280670

    work page Pith review arXiv 2022

  7. [7]

    Yung-Sung Chuang, Yujia Xie, Hongyin Luo, Yoon Kim, James Glass, and Pengcheng He. 2023. DoLa: Decoding by Contrasting Layers Improves Fac- tuality in Large Language Models.CoRRabs/2309.03883 (2023). arXiv:2309.03883 http://arxiv.org/abs/2309.03883

    arXiv 2023

  8. [8]

    Hejie Cui, Lingjun Mao, Xin Liang, Jieyu Zhang, Hui Ren, Quanzheng Li, Xiang Li, and Carl Yang. 2024. Biomedical Visual Instruction Tuning with Clinician Preference Alignment.Advances in neural information processing systems37 (2024), 96449–96467. https://api.semanticscholar.org/CorpusID:270619462

    work page 2024

Show all 61 references
  1. [9]

    Xun Deng, Han Zhong, Rui Ai, Fuli Feng, Zheng Wang, and Xiangnan He. 2025. Less is More: Improving LLM Alignment via Preference Data Selection.CoRR abs/2502.14560 (2025). doi:10.48550/ARXIV.2502.14560 arXiv:2502.14560

    2025 doi

  2. [10]

    Sedigheh Eslami, Christoph Meinel, and Gerard De Melo. 2023. PubMedCLIP: How Much Does CLIP Benefit Visual Question Answering in the Medical Do- main?. InFindings. https://api.semanticscholar.org/CorpusID:258378275

    2023

  3. [11]

    Chengqian Gao, Haonan Li, Liu Liu, Zeke Xie, Peilin Zhao, and Zhiqiang Xu

  4. [12]

    Dennis Hein, Zhihong Chen, Sophie Ostmeier, Justin Xu, Maya Varma, Ed- uardo Pontes Reis, Arne Edward Michalson, Christian Bluethgen, Hyun Joo Shin, Curtis Langlotz, et al. 2024. Preference fine-tuning for factuality in Chest X-Ray interpretation models without human Feedback....

    2024 arXiv

  5. [13]

    Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen

    J. Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen. 2021. LoRA: Low-Rank Adaptation of Large Language Models.ArXivabs/2106.09685 (2021). https://api.semanticscholar.org/CorpusID: 235458009

    2021 arXiv

  6. [14]

    An, Mengliang Zhang, Liangchen Liu, Kazuma Kobayashi, Tatsuya Harada, Ronald M

    Xinyue Hu, Lin Gu, Qi A. An, Mengliang Zhang, Liangchen Liu, Kazuma Kobayashi, Tatsuya Harada, Ronald M. Summers, and Yingying Zhu. 2023. Expert Knowledge-Aware Image Difference Graph Representation Learning for Difference-Aware Medical Visual Question Answering.Proceedings of...

    2023

  7. [15]

    Qidong Huang, Xiaoyi Dong, Pan Zhang, Bin Wang, Conghui He, Jiaqi Wang, Dahua Lin, Weiming Zhang, and Nenghai Yu. 2024. OPERA: Alleviating Hal- lucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation. InProceedings of the IEEE/CVF Co...

    2024

  8. [16]

    Alistair E. W. Johnson, Lucas Bulgarelli, Lu Shen, Alvin Gayles, Ayad Sham- mout, Steven Horng, Tom J. Pollard, Benjamin Moody, Brian Gow, Li wei H. Lehman, Leo Anthony Celi, and Roger G. Mark. 2023. MIMIC-IV, a freely accessible electronic health record dataset.Scientific Dat...

    2023

  9. [17]

    Alistair E. W. Johnson, Tom J. Pollard, Seth J. Berkowitz, Nathaniel R. Greenbaum, Matthew P. Lungren, Chih ying Deng, Roger G. Mark, and Steven Horng. 2019. MIMIC-CXR, a de-identified publicly available database of chest radiographs with free-text reports.Scientific Data6 (20...

    2019

  10. [18]

    Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shijian Lu, Chunyan Miao, and Lidong Bing. 2024. Mitigating Object Hallucinations in Large Vision- Language Models through Visual Contrastive Decoding. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Reco...

    2024

  11. [19]

    Chenyu Li, Hao Tang, Yuxuan Li, Yanzhi Zhang, Yufeng Zhang, Jingyi Yu, and Dimitris Metaxas. 2023. Dynamic Graph Enhanced Contrastive Learning for Chest X-ray Report Generation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2026–2035

    2023

  12. [20]

    Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon, and Jianfeng Gao

  13. [21]

    Mingjie Li, Haokun Lin, Liang Qiu, Xiaodan Liang, Ling Chen, Abdulmotaleb Elsaddik, and Xiaojun Chang. 2024. Contrastive Learning with Counterfactual Ex- planations for Radiology Report Generation. InEuropean Conference on Computer Vision. https://api.semanticscholar.org/Corpu...

    2024

  14. [22]

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Vi- sual Instruction Tuning. InAdvances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023, Alic...

    2023

  15. [23]

    LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day. InAdvances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023, Alice Oh, ...

    2023

  16. [24]

    Ziyu Liu, Yuhang Zang, Xiao wen Dong, Pan Zhang, Yuhang Cao, Haodong Duan, Conghui He, Yuanjun Xiong, Dahua Lin, and Jiaqi Wang. 2024. MIA-DPO: Multi- Image Augmented Direct Preference Optimization For Large Vision-Language Models.ArXivabs/2410.17637 (2024). https://api.semant...

    2024 arXiv

  17. [25]

    Matéo Mahaut, Laura Aina, Paula Czarnowska, Momchil Hardalov, Thomas Müller, and Lluís Màrquez. 2024. Factual Confidence of LLMs: on Reliability and Robustness of Current Estimators. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Vol...

    2024 doi

  18. [26]

    Ziyu Liu, Zeyi Sun, Yuhang Zang, Xiao wen Dong, Yuhang Cao, Haodong Duan, Dahua Lin, and Jiaqi Wang. 2025. Visual-RFT: Visual Reinforcement Fine-Tuning. https://api.semanticscholar.org/CorpusID:276768091

    2025

  19. [27]

    Jong Hak Moon, HyunGyung Lee, Won Young Shin, and E. Choi. 2021. Multi- Modal Understanding and Generation for Medical Images and Text via Vision- Language Pre-Training.IEEE Journal of Biomedical and Health Informatics26 (2021), 6070–6080. https://api.semanticscholar.org/Corpu...

    2021

  20. [28]

    Michael Moor, Qian Huang, Shirley Wu, Michihiro Yasunaga, Yash Dalmia, Jure Leskovec, Cyril Zakka, Eduardo Pontes Reis, and Pranav Rajpurkar. 2023. Med- Flamingo: a Multimodal Medical Few-shot Learner. InProceedings of the 3rd Ma- chine Learning for Health Symposium (Proceedin...

    2023

  21. [29]

    Andrey Malinin and Mark John Francis Gales. 2021. Uncertainty Estimation in Autoregressive Structured Prediction. InInternational Conference on Learning Representations. https://api.semanticscholar.org/CorpusID:231895728

    2021

  22. [30]

    Philip Müller, Georgios Kaissis, and Daniel Rueckert. 2024. ChEX: Interactive Localization and Region Description in Chest X-rays. InEuropean Conference on Computer Vision. https://api.semanticscholar.org/CorpusID:269362121

    2024

  23. [31]

    Sahal Shaji Mullappilly Hisham Cholakkal Rao Muhammad Anwer Salman Khan Jorma Laaksonen Omkar Thawkar, Abdelrahman Shaker and Fahad Shahbaz Khan. 2023. XrayGPT: Chest Radiographs Summarization using Large Medical Vision-Language Models.arXiv: 2306.07971(2023)

    2023 arXiv

  24. [32]

    Tetsuro Morimura, Mitsuki Sakamoto, Yuu Jinnai, Kenshi Abe, and Kaito Air

  25. [33]

    Yassine Ouali, Adrian Bulat, Brais Martínez, and Georgios Tzimiropoulos. 2024. CLIP-DPO: Vision-Language Models as a Source of Preference for Fixing Halluci- nations in LVLMs.ArXivabs/2408.10433 (2024). https://api.semanticscholar.org/ CorpusID:271909100

    2024 arXiv

  26. [34]

    PhysioNet. [n. d.]. PhysioNet. https://physionet.org/. Accessed: 2025-04-07. ACMMM25, October 27–31, 2025, Dublin, Ireland Xiao Liang et al

    2025

  27. [35]

    Yue Qiu, Shintaro Yamamoto, Kodai Nakashima, Ryota Suzuki, Kenji Iwata, Hi- rokatsu Kataoka, and Yutaka Satoh. 2021. Describing and Localizing Multiple Changes with Transformers.2021 IEEE/CVF International Conference on Com- puter Vision (ICCV)(2021), 1951–1960. https://api.se...

    2021

  28. [36]

    OpenAI. 2024. GPT-4o System Card.arXiv preprint arXiv:2410.21276(2024). https://arxiv.org/abs/2410.21276

    2024 arXiv

  29. [37]

    Manning, and Chelsea Finn

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. 2023. Direct Preference Optimization: Your Language Model is Secretly a Reward Model.ArXivabs/2305.18290 (2023). https://api. semanticscholar.org/CorpusID:258959321

    2023 arXiv

  30. [38]

    Junrong Li Patrick Pang Rongsheng Wang, Yaofei Duan and Tao Tan. 2023. XrayGLM: The first Chinese Medical Multimodal Model that Chest Radiographs Summarization. https://github.com/WangRongsheng/XrayGLM

    2023

  31. [39]

    Guohao Sun, Can Qin, Huazhu Fu, Linwei Wang, and Zhiqiang Tao. 2024. STLLaVA-Med: Self-Training Large Language and Vision Assistant for Medi- cal Question-Answering.arXiv preprint arXiv:2406.19973(2024)

    2024 arXiv

  32. [40]

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning Transferable Visual Models From Natural Language Supervision. InInternationa...

    2021

  33. [41]

    Zhanyu Wang, Lingqiao Liu, Lei Wang, and Luping Zhou. 2023. METransformer: Radiology Report Generation by Transformer with Multiple Learnable Expert Tokens. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 11558–11567

    2023

  34. [42]

    Chaoyi Wu, Xiaoman Zhang, Ya Zhang, Yanfeng Wang, and Weidi Xie. 2023. Towards Generalist Foundation Model for Radiology by Leveraging Web-scale 2D&3D Medical Data.CoRRabs/2308.02463 (2023). arXiv:2308.02463 http: //arxiv.org/abs/2308.02463

    2023 arXiv

  35. [43]

    Wu, Nkechinyere N

    Joy T. Wu, Nkechinyere N. Agu, Ismini Lourentzou, Arjun Sharma, Joseph Alexan- der Paguio, Jasper Seth Yao, Edward Christopher Dee, William Mitchell, Satyananda Kashyap, Andrea Giovannini, Leo Anthony Celi, and Mehdi Moradi

  36. [44]

    Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. InAdvances in neural information processing systems. 3104– 3112

    2014

  37. [45]

    Zhihe Yang, Xufang Luo, Dongqi Han, Yunjian Xu, and Dongsheng Li. 2025. Mitigating Hallucinations in Large Vision-Language Models via DPO: On-Policy Data Hold the Key.ArXivabs/2501.09695 (2025). https://api.semanticscholar. org/CorpusID:275570496

    2025 arXiv

  38. [46]

    Linli Yao, Weiying Wang, and Qin Jin. 2022. Image Difference Captioning with Pre- training and Contrastive Learning. InAAAI Conference on Artificial Intelligence. https://api.semanticscholar.org/CorpusID:246680391

    2022

  39. [47]

    Rao, Mu-Hsin Wei, Naveen Valluri, Cliff Wong, Matthew P

    Sheng Zhang, Yanbo Xu, Naoto Usuyama, Jaspreet Kaur Bagga, Robert Tinn, Sam Preston, Rajesh N. Rao, Mu-Hsin Wei, Naveen Valluri, Cliff Wong, Matthew P. Lungren, Tristan Naumann, and Hoifung Poon. 2023. BiomedCLIP: a multimodal biomedical foundation model pretrained from fiftee...

    2023

  40. [48]

    Yiyang Zhou, Chenhang Cui, Rafael Rafailov, Chelsea Finn, and Huaxiu Yao. 2024. Aligning Modalities in Vision Large Language Models via Preference Fine-tuning. arXiv:2402.11411 [cs.LG] https://arxiv.org/abs/2402.11411

    2024 arXiv

  41. [49]

    Peng Xia, Kangyu Zhu, Haoran Li, Hongtu Zhu, Yun Li, Gang Li, Linjun Zhang, and Huaxiu Yao. 2024. RULE: Reliable Multimodal RAG for Factuality in Medical Vision Language Models. InProceedings of the 2024 Conference on Empirical Meth- ods in Natural Language Processing, EMNLP 2...

    2024

  42. [54]

    Kangyu Zhu, Peng Xia, Yun Li, Hongtu Zhu, Sheng Wang, and Huaxiu Yao

  43. [55]

    Closed" refers to yes/no questions, while

    MMedPO: Aligning Medical Vision-Language Models with Clinical-Aware Multimodal Preference Optimization.ArXivabs/2412.06141 (2024). https://api. semanticscholar.org/CorpusID:274597778 CheXPO: Preference Optimization for Chest X-ray VLMs with Counterfactual Rationale ACMMM25, Oc...

    2024 arXiv

  44. [56]

    Atelectasis

    Even without an extensive hyper-parameter sweep, SimPO and CPO—two alternative PO objectives—achieved accuracy compara- ble to DPO. Importantly, every evaluated PO variant exhibited a marked performance gain over the SFT baseline, underscoring the robustness of preference-opti...

    2025

  45. [60]

    question

    Ensure explanation is based solely on image observations and presents a logical diagnostic process. Based on the provided input (question, answer, and radiological findings), generate a radiological explanation following the requirements above. ## Input: { question: {question}...

    2025

  46. [61]

    Start your explanation by first stating the exact answer provided, and then follow it with your diagnostic explanation

  47. [62]

    Explain how specific imaging findings support the answer using standardized radiological language, avoid terms like' reported,' 'mentioned,'noted,' 'stated,'or'described'

  48. [63]

    Include only imaging details directly relevant to the question and diagnosis

  49. [64]

    question

    Include only imaging details directly relevant to the question and diagnosis. When showing changes, clearly attribute them to your visual comparison of both images. Based on the provided input (question, answer, and radiological findings from both current and reference images)...

  50. [2021]

    https://api.semanticscholar.org/CorpusID:235420881

    Chest ImaGenome Dataset for Clinical Reasoning.ArXivabs/2108.00316 (2021). https://api.semanticscholar.org/CorpusID:235420881

    2021 arXiv

  51. [2023]

    https: //api.semanticscholar.org/CorpusID:264590453

    EHRXQA: A Multi-Modal Question Answering Dataset for Electronic Health Records with Chest X-ray Images.ArXivabs/2310.18652 (2023). https: //api.semanticscholar.org/CorpusID:264590453

    2023 arXiv

  52. [2024]

    https://api.semanticscholar.org/CorpusID:269293711

    Filtered Direct Preference Optimization.ArXivabs/2404.13846 (2024). https://api.semanticscholar.org/CorpusID:269293711

    2024 arXiv

  53. [2025]

    https://api.semanticscholar.org/CorpusID: 276394867

    Principled Data Selection for Alignment: The Hidden Risks of Difficult Ex- amples.ArXivabs/2502.09650 (2025). https://api.semanticscholar.org/CorpusID: 276394867

    2025 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.