Pith. sign in

REVIEW 3 major objections 4 minor 1 cited by

Focus on What Matters: Enhancing Medical Vision-Language Models with Automatic Attention Alignment Tuning

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Fine-tuning redirects medical AI attention to the region that actually matters

desk verdict Useful fine-tuning recipe with a clean ablation, but test-set hyperparameter selection undermines the headline claim. read the letter →

arxiv 2505.18503 v1 pith:I76H4M62 submitted 2025-05-24 cs.CV

Aofei Chang , Le Huang , Alex James Boyd , Parminder Bhatia , Taha Kass-Hout , Cao Xiao , Fenglong Ma This is my paper
classification cs.CV
keywords medicalvision-languagemodelsattentionalignmenthallucinationmitigationweaksupervisionvisualgroundingmixture-of-expertsparameter-efficientfine-tuningquestionanswering
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes A3Tune, a fine-tuning framework that fixes a specific failure mode of medical vision-language models: their attention on an image is often spread over irrelevant background, so they hallucinate answers even when the right region is visible. The claim is that by generating prompt-aware weak region labels automatically from SAM and BioMedCLIP, then fine-tuning only the most visually-critical attention heads with a mixture-of-experts router, the model learns to focus on diagnostically relevant regions and answers more accurately. If this works, it would give a practical, annotation-free recipe for improving visual grounding in medical VQA and report generation, without needing inference-time interventions or ground-truth regions of interest.

What carries the argument

The central object is the averaged visual attention map $\bar{M}^v$ restricted to visual tokens, and the mechanism is the attention alignment loss $\mathcal{L}_{\text{align}} = \sum_{s\in S} (1 - \frac{\sum_{c\in s} \bar{M}^v_c}{\sum_{c'} \bar{M}^v_{c'}})^2$, which encourages higher attention inside each prompt-aware weak segment $s$. The argument is carried by three cooperating components: prompt-aware weak labels generated by SAM plus BioMedCLIP similarity filtering, selective tuning of the top-R visually-critical heads measured by visual attention ratio, and two Mixture-of-Experts modules (Q-MoE and K-MoE) that let the LoRA query and key updates adapt per prompt and per visual token, respectively.

What would settle it

Construct a medical VQA set where the prompt describes a region that BioMedCLIP fails to associate with the image segments but a clinician would identify, then run A3Tune and check whether attention coverage on the true region decreases or the answers become more hallucinated; a drop relative to plain LoRA fine-tuning on that set would indicate the weak labels are not reliable enough as supervision.

Watch

Extended reading notes

Core claim

The paper's central claim is that attention misalignment in medical vision-language models can be corrected at fine-tuning time, rather than at inference time, using only automatically generated weak labels. A3Tune first produces candidate segments with SAM, filters them by cosine similarity between BioMedCLIP segment embeddings and the text prompt, and then uses a mask-based energy loss to pull the averaged visual attention map toward those segments. The paper also argues that modifying all attention heads is risky, so it selects the top-R heads by visual attention ratio, and that static parameter sharing is insufficient, so it wraps the LoRA query and key updates in a Q-MoE and a token-level sparse K-MoE. Experiments across five VQA and two report generation benchmarks, with LLaVA-Med, LLaVA-Med-1.5, and HuatuoGPT-Vision-7B, show consistent gains over LoRA fine-tuning and over decoding-based hallucination mitigation baselines, together with higher coverage and intensity of attention on ground-truth regions in SLAKE.

Load-bearing premise

The framework assumes that the cosine similarity between a BioMedCLIP segment embedding and the text prompt reliably identifies the clinically correct region; if those zero-shot weak labels are noisy or miss the true region, the alignment loss can pull attention toward irrelevant areas and hurt performance.

Editorial extensions

If this is right

  • Medical VQA and report generation systems can gain accuracy and interpretability without any manual region annotations at inference time.
  • The alignment objective can serve as a drop-in regularization on top of existing parameter-efficient fine-tuning, extending to newer and stronger Med-LVLM backbones.
  • Attention-based evaluation metrics such as coverage and intensity can expose interpretability improvements that accuracy numbers alone miss.
  • The framework suggests that attention biases are a trainable property of Med-LVLMs, not only a decoding-time artifact.
  • If attention becomes more grounded, hallucination rates on region-sensitive questions about anatomy and abnormalities should drop correspondingly.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension is to apply the same alignment recipe to general-domain LVLMs, since the weak-label generation steps are not inherently medical.
  • One could test whether the improvement is driven mainly by the classifier-like separation of relevant segments or by the MoE routing; ablations already hint K-MoE matters most, but a direct comparison of alignment loss strength with and without MoE would clarify the mechanism.
  • The authors' own limitation regarding noisy weak labels implies an obvious next experiment: replace BioMedCLIP-based filtering with a stronger or prompt-conditioned segmenter, and measure whether the gap between weak and ground-truth labels shrinks.
  • The coverage and intensity metrics, being patch-level, could be refined to pixel-level evaluation if higher-resolution attention maps ever become available.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. A3Tune is a fine-tuning framework for medical vision-language models that generates prompt-aware weak segmentation labels with SAM and BioMedCLIP, selects a small set of "visually-critical" attention heads, and trains an attention-alignment loss jointly with the language-modeling objective. It augments LoRA with two Mixture-of-Experts modules, Q-MoE and K-MoE, to adapt attention computations per prompt and per visual token. The paper reports accuracy gains over LoRA and decoding-based baselines on five VQA benchmarks and two report-generation benchmarks, supported by ablations, attention-map visualizations, and a ground-truth upper-bound experiment.

Significance. If the reported gains are reliable, A3Tune offers a practical, inference-time-free recipe for reducing attention bias in Med-LVLMs without requiring manual ROI annotations at test time. The paper's strengths include evaluation on three backbones (LLaVA-Med, LLaVA-Med-1.5, and HuatuoGPT-Vision-7B), an ablation showing that the attention-alignment loss alone (without MoE) still improves over the LoRA baseline, a ground-truth upper-bound experiment in Figure 4, and a public code link. The main weakness is that the evaluation protocol does not rule out test-set overfitting through hyperparameter selection, so the headline empirical claim is not yet established as reported.

major comments (3)
  1. [Appendix A, Appendix C, Figures 5, 7, 8] The paper does not describe a held-out validation split. Appendix A (Table 6) defines only train/test partitions, while Appendix C (Table 7) reports dataset-specific lambda and epoch values, and Figures 5(a), 5(b), 7(b), and 8 select K, R, B, and lambda by choosing the point at which "performance peaks" on SLAKE. If these hyperparameters are chosen on the test folds, then Tables 1, 3, 4, and 5 are optimistically biased, and the reported advantage over LLaVA-Med+LoRA (e.g., +1.71 on SLAKE Open in Table 1) may not generalize. Please specify a clear validation protocol or nested cross-validation, and report multiple-seed results with standard deviations or confidence intervals; without these, the point-score differences cannot be distinguished from noise.
  2. [Eq. (3) and Section 5.4.1(2)] The weak-label generation is specified inconsistently. Eq. (3) selects segments whose cosine similarity exceeds an adaptive threshold tau_K, but the text and Figure 5(a) describe K as the number of segments chosen by ranking similarity, and Section 5.4.1(2) states "we select K segments that are most similar to the text prompt." Please define whether K is a count or a threshold, and state how tau_K is set; this is needed to reproduce Figure 5(a) and the main experiments.
  3. [Table 2 and Eq. (10)] The attention metrics in Table 2 are not an independent validation of attention quality, because the training objective Eq. (10) directly maximizes attention inside the weak-label masks that are also used to compute Coverage and Intensity. The VQA and report-generation gains on external benchmarks are not circular, and Figure 4's ground-truth upper-bound experiment partially addresses label quality, but the Coverage/Intensity comparison should be interpreted only as a check that training moved attention as intended. A stronger test would be to evaluate the attention metrics on held-out images or on cases where the weak labels are manually scored for correctness.
minor comments (4)
  1. [Abstract and Section 1] The phrase "a A3MOE module" should be "an A3MOE module" in both the abstract and the introduction.
  2. [Table 3, MIMIC-CXR CheXbert row] The text in Section 5.3 says A3Tune outperforms all baselines across both datasets and almost all metrics; this is accurate because on MIMIC-CXR CheXbert the PAI baseline scores 25.78 while A3Tune scores 24.93, but the exception should be stated explicitly rather than left to the reader to infer from the table.
  3. [Equations (4)-(7)] The routers in Q-MoE and K-MoE are trained without a load-balancing or auxiliary loss, and Figure 7(b) shows unstable returns for larger B; a brief discussion of expert utilization or collapse would strengthen the paper.
  4. [Eq. (8)] The notation M^v_lh[c] in Eq. (8) is not defined; M^v_lh was introduced as a submatrix, so the indexing convention for individual tokens should be stated.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the attention-alignment target comes from independent zero-shot models and the headline claims are evaluated on external benchmarks.

full rationale

A3Tune's derivation chain is a standard fine-tuning objective: Eq. (12) combines a language-modeling loss with an attention-alignment regularizer (Eq. (10)), where the target masks S are produced by an independent pipeline (SAM segments filtered by BioMedCLIP cosine similarity, Eq. (3)). The claimed improvement in attention distribution is not definitional because the evaluation metrics in Table 2 and Appendix D.2 measure overlap with ground-truth RoIs, not with the weak-label training signal; the GT-upper-bound experiment in Figure 4 explicitly separates Weak and GT labels and shows they behave differently, so Eq. (10) and the reported evaluation are not the same quantity by construction. The performance claims on medical VQA and report generation use held-out test folds and external datasets, with no parameter fitted to the reported target metric. No load-bearing self-citation chain appears in the reference list; the method builds on external tools (SAM, BioMedCLIP, LoRA) and an external cross-attention guidance idea. The evaluation-protocol concerns raised by the skeptic (test-set hyperparameter selection, absence of error bars) are empirical-validity issues, not circularity, because no equation reduces to its own input under those concerns. Overall, the central empirical claims have independent grounding, so no significant circularity is identified.

Assumptions & free parameters 5 free parameters · 4 assumptions · 1 invented entities

The central claim depends on several tuned hyperparameters and on domain assumptions about the reliability of attention maps and zero-shot weak labels. The only newly introduced component is the A3MoE module, which is evaluated solely through in-paper ablations.

free parameters (5)
  • K (number of weak labels) = 4
    Tuned on SLAKE via sweep in Figure 5a; filters SAM segments using BioMedCLIP cosine similarity.
  • R (number of visually-critical attention heads) = 128
    Tuned on SLAKE via sweep in Figure 5b; controls the subset of heads used in the alignment loss.
  • lambda (alignment loss weight) = 0.1 for SLAKE; varies per dataset (Table 7)
    Controls the trade-off between the language modeling objective and attention alignment; tuned per dataset.
  • B (top experts in K-MoE) = 3 when Ok=16; 2 when Ok=8
    Sparsity level for visual token routing in K-MoE; tuned via ablation in Figure 7b.
  • Oq and Ok (number of experts) = Oq=4, Ok=8 for X-ray; Oq=8, Ok=16 for diverse datasets
    Expert counts for Q-MoE and K-MoE; adjusted per dataset as described in Appendix C.1.
assumptions (4)
  • domain assumption Attention maps of the decoder, averaged over heads and layers, reflect the model's visual grounding.
    Used to define the visual attention ratio (Eq. 2, 8) and the alignment loss (Eq. 10).
  • domain assumption SAM and BioMedCLIP zero-shot outputs provide sufficiently accurate prompt-aware weak labels.
    Section 4.1; the entire attention supervision signal depends on these labels.
  • ad hoc to paper Heads with the highest visual attention ratio are the most important heads to tune.
    Section 4.2.2; no theoretical justification is given, only empirical validation.
  • domain assumption Adding L_align as a weighted regularization term (Eq. 12) will not destabilize the language model.
    Assumed by design; lambda is tuned per dataset to keep the model stable.
invented entities (1)
  • A3MoE (Q-MoE and K-MoE)
    purpose: Adaptive LoRA parameter routing for attention alignment across diverse prompts and visual tokens.
    New architectural module introduced in Section 4.2.1; supported only by in-paper ablations (Figure 7), with no external validation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Focus on What Matters: Enhancing Medical Vision-Language Models with Automatic Attention Alignment Tuning." pith.science (2026). https://pith.science/paper/I76H4M62

@misc{pith2026250518503,
  author       = {Pith},
  title        = {Pith review of: Focus on What Matters: Enhancing Medical Vision-Language Models with Automatic Attention Alignment Tuning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/I76H4M62}},
  note         = {Machine review of arXiv:2505.18503}
}
abstract

Medical Large Vision-Language Models (Med-LVLMs) often exhibit suboptimal attention distribution on visual inputs, leading to hallucinated or inaccurate outputs. Existing mitigation methods primarily rely on inference-time interventions, which are limited in attention adaptation or require additional supervision. To address this, we propose A$^3$Tune, a novel fine-tuning framework for Automatic Attention Alignment Tuning. A$^3$Tune leverages zero-shot weak labels from SAM, refines them into prompt-aware labels using BioMedCLIP, and then selectively modifies visually-critical attention heads to improve alignment while minimizing interference. Additionally, we introduce a A$^3$MoE module, enabling adaptive parameter selection for attention tuning across diverse prompts and images. Extensive experiments on medical VQA and report generation benchmarks show that A$^3$Tune outperforms state-of-the-art baselines, achieving enhanced attention distributions and performance in Med-LVLMs.

Figures

Figures reproduced from arXiv: 2505.18503 by the authors.

Figure 1
Figure 1. (A) Examples of medical VQA and attention maps on medical images. In this example of Brain MRI [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. (A) The overview of A3TUNE and (B) the details of the designed visual attention alignment tuning. image reports, mitigation strategies specifically de￾signed for Med-LVLMs remain largely underex￾plored. Since Med-LVLMs share the same struc￾ture and training process as general LVLMs, hal￾lucination issues are a common challenge across both. As a result, many inference-time mitigation strategies developed for LVLMs ar… view at source ↗
Figure 3
Figure 3. Motivation for using A 3MOE. The second col￾umn shows prompt-aware weak labels, with red bound￾ing boxes and green inner segments. The third column shows the attention maps generated using shared param￾eters for the Query and Key matrices. organ analysis tasks ( [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Effectiveness analysis of RoIs labels. Base is the base model, Control means adding ControlMLLM to align attention maps with ground truth labels, Weak uses weak labels, and GT uses ground truth labels. Performance Number of weak labels -- K Number of selected attention…
Figure 5
Figure 5. Figure 5: Analysis of (a) the number of selected atten [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Case study for fine-grained effectiveness analysis. The [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Analysis of (a) the contribution of different [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: Analysis of the value of λ in A 3TUNE on the SLAKE dataset. C.2 λ Selection In our final loss in Eq. (12), we use a key hyperpa￾rameter λ to balance the two loss terms. Here, we conduct an analysis to select the optimal value of λ [PITH_FULL_IMAGE:figures/full_fig_p01…
Figure 9
Figure 9. Figure 9: VQA Types where A 3TUNE outperforms LLaVA-Med + LoRA across three image modalities. without fine-tuning, where the model performs sig￾nificantly worse on report generation. For example, in [PITH_FULL_IMAGE:figures/full_fig_p015_9.png]
Figure 10
Figure 10. Figure 10: Case study on LLaVA-Med-1.5. The red box in the [PITH_FULL_IMAGE:figures/full_fig_p016_10.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. A Comprehensive Survey of Electronic Health Record Modeling: From Deep Learning Approaches to Large Language Models

    cs.LG 2025-07 reject novelty 4.0 of 10

    A survey that taxonomizes EHR modeling research into data-centric, architectural, learning-focused, multimodal, and LLM-based categories, with datasets and metrics.

Reference graph

Works this paper leans on

47 extracted references · 12 canonical work pages · cited by 1 Pith paper

  1. [1]

    Satanjeev Banerjee and Alon Lavie. 2005. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pages 65--72

    2005

  2. [2]

    Jiawei Chen, Dingkang Yang, Tong Wu, Yue Jiang, Xiaolu Hou, Mingcheng Li, Shunli Wang, Dongling Xiao, Ke Li, and Lihua Zhang. 2024 a . Detecting and evaluating medical hallucinations in large vision language models. arXiv preprint arXiv:2406.10185

    arXiv 2024

  3. [3]

    Jun Chen, Deyao Zhu, Xiaoqian Shen, Xiang Li, Zechun Liu, Pengchuan Zhang, Raghuraman Krishnamoorthi, Vikas Chandra, Yunyang Xiong, and Mohamed Elhoseiny. 2023. Minigpt-v2: large language model as a unified interface for vision-language multi-task learning. arXiv preprint arXiv:2310.09478

    arXiv 2023

  4. [4]

    Junying Chen, Ruyi Ouyang, Anningzhe Gao, Shunian Chen, Guiming Hardy Chen, Xidong Wang, Ruifei Zhang, Zhenyang Cai, Ke Ji, Guangjun Yu, Xiang Wan, and Benyou Wang. 2024 b . http://arxiv.org/abs/2406.19280 Huatuogpt-vision, towards injecting medical visual knowledge into multimodal llms at scale

    arXiv 2024

  5. [5]

    Minghao Chen, Iro Laina, and Andrea Vedaldi. 2024 c . Training-free layout control with cross-attention guidance. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 5343--5353

    work page 2024

  6. [6]

    Tsai, Andrew Johnston, Cameron Olsen, Tanishq Mathew Abraham, Sergios Gatidis, Akshay S Chaudhari, and Curtis Langlotz

    Zhihong Chen, Maya Varma, Jean-Benoit Delbrouck, Magdalini Paschali, Louis Blankemeier, Dave Van Veen, Jeya Maria Jose Valanarasu, Alaa Youssef, Joseph Paul Cohen, Eduardo Pontes Reis, Emily B. Tsai, Andrew Johnston, Cameron Olsen, Tanishq Mathew Abraham, Sergios Gatidis, Akshay S Chaudhari, and Curtis Langlotz. 2024 d . https://arxiv.org/abs/2401.12208 C...

    arXiv 2024

  7. [7]

    Yung-Sung Chuang, Yujia Xie, Hongyin Luo, Yoon Kim, James Glass, and Pengcheng He. 2023. Dola: Decoding by contrasting layers improves factuality in large language models. arXiv preprint arXiv:2309.03883

    arXiv 2023

  8. [8]

    Dina Demner-Fushman, Marc D Kohli, Marc B Rosenman, Sonya E Shooshan, Laritza Rodriguez, Sameer Antani, George R Thoma, and Clement J McDonald. 2016. Preparing a collection of radiology examinations for distribution and retrieval. Journal of the American Medical Informatics Association, 23(2):304--310

    work page 2016

Show all 47 references
  1. [9]

    Alessandro Favero, Luca Zancato, Matthew Trager, Siddharth Choudhary, Pramuditha Perera, Alessandro Achille, Ashwin Swaminathan, and Stefano Soatto. 2024. Multi-modal hallucination control by visual information grounding. In Proceedings of the IEEE/CVF Conference on Computer V...

    2024

  2. [10]

    Xuan Gong, Tianshi Ming, Xinpeng Wang, and Zhihua Wei. 2024. Damro: Dive into the attention mechanism of lvlm to reduce object hallucination. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 7696--7712

    2024

  3. [11]

    Zishan Gu, Changchang Yin, Fenglin Liu, and Ping Zhang. 2024. Medvh: Towards systematic evaluation of hallucination for large vision language models in the medical context. arXiv preprint arXiv:2407.02730

    2024 arXiv

  4. [12]

    Xuehai He, Yichen Zhang, Luntian Mou, Eric Xing, and Pengtao Xie. 2020. Pathvqa: 30000+ questions for medical visual question answering. arXiv preprint arXiv:2003.10286

    2020 arXiv

  5. [13]

    Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. https://openreview.net/forum?id=rygGQyrFvH The curious case of neural text degeneration . In International Conference on Learning Representations

    2020

  6. [14]

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685

    2021 arXiv

  7. [15]

    Yutao Hu, Tianbin Li, Quanfeng Lu, Wenqi Shao, Junjun He, Yu Qiao, and Ping Luo. 2024. Omnimedvqa: A new large-scale comprehensive evaluation benchmark for medical lvlm. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22170--22183

    2024

  8. [16]

    Qidong Huang, Xiaoyi Dong, Pan Zhang, Bin Wang, Conghui He, Jiaqi Wang, Dahua Lin, Weiming Zhang, and Nenghai Yu. 2024. Opera: Alleviating hallucination in multi-modal large language models via over-trust penalty and retrospection-allocation. In Proceedings of the IEEE/CVF Con...

    2024

  9. [17]

    Saahil Jain, Ashwin Agrawal, Adriel Saporta, Steven QH Truong, Du Nguyen Duong, Tan Bui, Pierre Chambon, Yuhao Zhang, Matthew P Lungren, Andrew Y Ng, et al. 2021. Radgraph: Extracting clinical entities and relations from radiology reports. arXiv preprint arXiv:2106.14463

    2021 arXiv

  10. [18]

    Yue Jiang, Jiawei Chen, Dingkang Yang, Mingcheng Li, Shunli Wang, Tong Wu, Ke Li, and Lihua Zhang. 2024. https://api.semanticscholar.org/CorpusID:270559982 Comt: Chain-of-medical-thought reduces hallucination in medical report generation

    2024

  11. [19]

    Alistair EW Johnson, Tom J Pollard, Nathaniel R Greenbaum, Matthew P Lungren, Chih-ying Deng, Yifan Peng, Zhiyong Lu, Roger G Mark, Seth J Berkowitz, and Steven Horng. 2019. Mimic-cxr-jpg, a large publicly available database of labeled chest radiographs. arXiv preprint arXiv:1...

    2019 arXiv

  12. [20]

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. 2023. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4015--4026

    2023

  13. [21]

    Jason J Lau, Soumya Gayen, Asma Ben Abacha, and Dina Demner-Fushman. 2018. A dataset of clinically generated visual questions and answers about radiology images. Scientific data, 5(1):1--10

    2018

  14. [22]

    Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shijian Lu, Chunyan Miao, and Lidong Bing. 2024. Mitigating object hallucinations in large vision-language models through visual contrastive decoding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Reco...

    2024

  15. [23]

    Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon, and Jianfeng Gao. 2024. Llava-med: Training a large language-and-vision assistant for biomedicine in one day. Advances in Neural Information Processing Systems, 36

    2024

  16. [24]

    Xiaoyu Liang, Jiayuan Yu, Lianrui Mu, Jiedong Zhuang, Jiaqi Hu, Yuchen Yang, Jiangnan Ye, Lu Lu, Jian Chen, and Haoji Hu. 2024. Mitigating hallucination in visual-language models via re-balancing contrastive decoding. In Chinese Conference on Pattern Recognition and Computer V...

    2024

  17. [25]

    Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74--81

    2004

  18. [26]

    Bo Liu, Li-Ming Zhan, Li Xu, Lin Ma, Yan Yang, and Xiao-Ming Wu. 2021. Slake: A semantically-labeled knowledge-enhanced dataset for medical visual question answering. In 2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI), pages 1650--1654. IEEE

    2021

  19. [27]

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2024. Visual instruction tuning. Advances in neural information processing systems, 36

    2024

  20. [28]

    Shi Liu, Kecheng Zheng, and Wei Chen. 2025. Paying more attention to image: A training-free method for alleviating hallucination in lvlms. In European Conference on Computer Vision, pages 125--140. Springer

    2025

  21. [29]

    Michael Moor, Qian Huang, Shirley Wu, Michihiro Yasunaga, Yash Dalmia, Jure Leskovec, Cyril Zakka, Eduardo Pontes Reis, and Pranav Rajpurkar. 2023. Med-flamingo: a multimodal medical few-shot learner. In Machine Learning for Health (ML4H), pages 353--367. PMLR

    2023

  22. [30]

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311--318

    2002

  23. [31]

    Saikat Roy, Tassilo Wald, Gregor Koehler, Maximilian R Rokuss, Nico Disch, Julius Holzschuh, David Zimmerer, and Klaus H Maier-Hein. 2023. Sam. md: Zero-shot medical image segmentation capabilities of the segment anything model. arXiv preprint arXiv:2304.05396

    2023 arXiv

  24. [32]

    Akshay Smit, Saahil Jain, Pranav Rajpurkar, Anuj Pareek, Andrew Y Ng, and Matthew Lungren. 2020. Combining automatic labelers and expert annotations for accurate radiology report labeling using bert. In Proceedings of the 2020 Conference on Empirical Methods in Natural Languag...

    2020

  25. [33]

    Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. https://proceedings.neurips.cc/paper_files/paper/2014/file/a14ac55a4f27472c5d894ec1c3c743d2-Paper.pdf Sequence to sequence learning with neural networks . In Advances in Neural Information Processing Systems, volume 27. Curra...

    2014

  26. [34]

    Omkar Thawkar, Abdelrahman Shaker, Sahal Shaji Mullappilly, Hisham Cholakkal, Rao Muhammad Anwer, Salman Khan, Jorma Laaksonen, and Fahad Shahbaz Khan. 2023. Xraygpt: Chest radiographs summarization using medical vision-language models. arXiv preprint arXiv:2306.07971

    2023 arXiv

  27. [35]

    Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N

    Ashish Vaswani, Noam M. Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. https://api.semanticscholar.org/CorpusID:13756489 Attention is all you need . In Neural Information Processing Systems

    2017

  28. [36]

    Hanyu Wang, Bochuan Cao, Yuanpu Cao, and Jinghui Chen. 2025. Truthflow: Truthful llm generation via representation flow correction. arXiv preprint arXiv:2502.04556

    2025 arXiv

  29. [37]

    Sangmin Woo, Donguk Kim, Jaehyuk Jang, Yubin Choi, and Changick Kim. 2024. Don't miss the forest for the trees: Attentional vision calibration for large vision language models. arXiv preprint arXiv:2405.17820

    2024 arXiv

  30. [38]

    Mingrui Wu, Xinyue Cai, Jiayi Ji, Jiale Li, Oucheng Huang, Gen Luo, Hao Fei, GUANNAN JIANG, Xiaoshuai Sun, and Rongrong Ji. 2024. https://openreview.net/forum?id=LjnDqVcrE9 Control MLLM : Training-free visual prompt learning for multimodal large language models . In The Thirty...

    2024

  31. [39]

    Peng Xia, Ze Chen, Juanxi Tian, Yangrui Gong, Ruibo Hou, Yue Xu, Zhenbang Wu, Zhiyuan Fan, Yiyang Zhou, Kangyu Zhu, et al. 2024. Cares: A comprehensive benchmark of trustworthiness in medical vision language models. arXiv preprint arXiv:2406.06007

    2024 arXiv

  32. [40]

    Lingfeng Yang, Yueze Wang, Xiang Li, Xinlong Wang, and Jian Yang. 2024. Fine-grained visual prompting. Advances in Neural Information Processing Systems, 36

    2024

  33. [41]

    Feiyang Yu, Mark Endo, Rayan Krishnan, Ian Pan, Andy Tsai, Eduardo Pontes Reis, Eduardo Kaiser Ururahy Nunes Fonseca, Henrique Min Ho Lee, Zahra Shakeri Hossein Abad, Andrew Y Ng, et al. 2023. Evaluating progress in automatic chest x-ray radiology report generation. Patterns, 4(9)

    2023

  34. [42]

    Fan Yuan, Chi Qin, Xiaogang Xu, and Piji Li. 2024. Helpd: Mitigating hallucination of lvlms by hierarchical feedback learning with vision-enhanced penalty decoding. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 1768--1785

    2024

  35. [43]

    Sheng Zhang, Yanbo Xu, Naoto Usuyama, Hanwen Xu, Jaspreet Bagga, Robert Tinn, Sam Preston, Rajesh Rao, Mu Wei, Naveen Valluri, et al. 2023. Biomedclip: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairs. arXiv preprint arXiv:2303.00915

    2023 arXiv

  36. [44]

    Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675

    2019 arXiv

  37. [45]

    Weike Zhao, Chaoyi Wu, Xiaoman Zhang, Ya Zhang, Yanfeng Wang, and Weidi Xie. 2024. Ratescore: A metric for radiology report generation. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 15004--15019

    2024

  38. [46]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before...

  39. [47]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.