Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

Mitigating Hallucinations in Multimodal LLMs via Object-aware Preference Optimization

T0 review · 3 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper shows that object hallucinations in multimodal LLMs can be sharply reduced by using the CHAIR captioning metric as a preference labeler for DPO, with only an off-the-shelf object detector as the oracle.

desk verdict CHAIR-DPO is a clean, cheap DPO recipe that cuts object hallucination substantially across two backbones; the detector-as-oracle noise is the one real soft spot and should be quantified, but the result is not just metric-chasing. read the letter →

arxiv 2508.20181 v1 pith:4GXGQHFZ submitted 2025-08-27 cs.CV cs.AIcs.CLcs.MM

Alberto Compagnoni , Davide Caffagni , Nicholas Moratelli , Lorenzo Baraldi , Marcella Cornia , Rita Cucchiara This is my paper
classification cs.CVcs.AIcs.CLcs.MM
keywords multimodalLLMobjecthallucinationdirectpreferenceoptimizationCHAIRmetricdetectionoracledataalignmentLLaVA
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that object hallucinations in multimodal large language models can be substantially reduced by recasting the standard CHAIR captioning metric as a reward for Direct Preference Optimization (DPO). Instead of relying on proprietary judge models to decide which responses are hallucinated, the method scores a pair of model-generated answers with CHAIR, using detections from an off-the-shelf object detector as the ground truth, and then fine-tunes the model to prefer the lower-scoring answer. This matters because it offers a simple, fully open-source path to hallucination alignment. Experiments on AMBER, CHAIR-MSCOCO, and Object HalBench show large drops in hallucination rates (e.g., AMBER CHAIRi from 7.6 to 3.0 on LLaVA-1.5-7B) while largely preserving general capabilities.

What carries the argument

The central object is the CHAIRi score (Eq. 1), the fraction of mentioned objects that are not detected, which is used to label winner and loser completions in DPO (Eq. 3). The detector supplies the oracle set of real objects; a synonym list aligns the model's words to MSCOCO class names. A data-filtering step removes zero-difference pairs to avoid noisy supervision.

What would settle it

Take a set of images containing objects that fall outside the MSCOCO class list or that are small, occluded, or otherwise likely to be missed by the detector. Run CHAIR-DPO fine-tuning and then measure whether the fine-tuned model mentions those genuinely present objects less frequently than the baseline model. A statistically significant drop in truthful mentions would confirm that the detector's blind spots are transferred into the model as learned suppression.

Watch

Extended reading notes

Core claim

CHAIR-DPO is a training pipeline that converts the CHAIR metric into a preference oracle. For each image and prompt, the model itself samples two candidate answers; an off-the-shelf detector (DETR with MSCOCO classes) provides the set of objects presumed present; CHAIRi is computed for each answer, with any mentioned object missing from the detection set counted as hallucinated. The answer with lower CHAIRi becomes the DPO winner and the other the loser. Fine-tuning with DPO then increases the probability of non-hallucinated completions. The paper demonstrates on three benchmarks that this procedure substantially reduces measured hallucinations on two different MLLMs, and that discarding pre

Load-bearing premise

The labels of which objects are 'hallucinated' come entirely from an off-the-shelf detector with a fixed MSCOCO vocabulary; if the detector misses objects that are actually in the image, the model is trained to suppress truthful mentions.

Editorial extensions

If this is right

  • Object hallucination in MLLMs can be mitigated without proprietary judges, lowering the barrier for aligned multimodal models.
  • The method transfers across at least two open MLLMs (LLaVA-1.5-7B and LLaVA-MORE-8B), suggesting it is architecture-agnostic.
  • The beta regularizer in DPO provides a continuous trade-off between hallucination reduction and coverage of real objects.
  • Filtering out uninformative preference pairs can cut the training set by roughly 90% while improving final hallucination metrics.
  • The small coverage penalty observed on AMBER indicates hallucination reduction comes at a modest, controllable cost to object recall.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the detector vocabulary is fixed to MSCOCO classes, the method may silently penalize truthful mentions of objects outside that vocabulary; extending the oracle to larger or open-vocabulary detectors could be a natural next step.
  • The detector's miss rate is the hidden variable: if it misses objects that are actually present, CHAIR-DPO will train the model to suppress valid details, which the paper's coverage metric may not fully expose.
  • The approach is likely orthogonal to training-free decoding fixes such as contrastive decoding or over-trust penalties, so combining CHAIR-DPO with such methods could yield further gains.
  • The same preference-labeling schema could be applied with other preference losses (e.g., IPO, KTO) or other grounding signals, such as referring-expression grounding models, to move beyond pure object lists.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes CHAIR-DPO, a preference-optimization method to reduce object hallucination in multimodal LLMs. It constructs preference pairs by sampling two responses from a reference MLLM for each visual instruction in LLaVA-Instruct-665k, detecting objects with an off-the-shelf DETR detector, and scoring each response with CHAIR_i (Eq. 1). The lower-scoring response is treated as the winner and the higher-scoring one as the loser (Eq. 2), and DPO is applied with LoRA. Pairs with equal CHAIR_i are discarded, leaving 70k/77k training samples. The method is evaluated on AMBER, CHAIR-MSCOCO, and Object HalBench, where it reports large decreases in CHAIR_i/HalRate (e.g., LLaVA-1.5-7B AMBER CHAIR_i drops from 7.6 to 3.0 at beta=0.2), with moderate Coverage reduction, plus general-benchmark results showing limited degradation. The main claimed contribution is an open-source, non-proprietary pipeline for hallucination-alignment data.

Significance. If the reported effects are robust, the contribution is practically significant: it shows that DPO with a simple, open-source reward (CHAIR_i plus a detector) can match or exceed methods that rely on GPT-4/Gemini judges on several hallucination benchmarks. The paper ships code and models, uses two backbones, includes ablations (data filtering, beta), and provides qualitative examples. The main caveat is that the preference signal and the headline evaluation metrics are closely related, so the interpretation requires additional validation before the state-of-the-art claim can be fully accepted.

major comments (3)
  1. [Sec. 3.2, Eq. (2); Sec. 4.1] The entire preference dataset is built from DETR-DC5-R101 detections restricted to the MSCOCO vocabulary. If the detector misses an object that is actually present, a correct response mentioning that object is labeled as hallucinated, and DPO is trained to suppress it. The paper reports no precision/recall analysis of this oracle on LLaVA-Instruct-665k or on the evaluation sets, and no sensitivity check (e.g., different detector, confidence threshold, or vocabulary). Since all 70k/77k training labels depend on this detector, this missing analysis is load-bearing. I recommend (i) quantifying detector recall/precision on a human-annotated sample of the training data, (ii) retraining with corrected labels or with a second detector, and (iii) reporting whether the CHAIR reductions survive.
  2. [Eq. (1), Eq. (2), Table 1] The DPO objective directly optimizes CHAIR_i, and the headline benchmarks CHAIR-MSCOCO and Object HalBench compute the same CHAIR_i/CHAIR_s with the same object-matching scheme. Therefore part of the reported gains is expected from direct reward optimization. AMBER is a partial exception because its object labels are human-annotated, but it still uses the CHAIR_i formula, and the observed Coverage drop (e.g., 51.7 to 46.6 for LLaVA-1.5-7B at beta=0.2) shows a shift toward more conservative answers. To support the claim of 'effectively diminishing hallucinations', the paper should report precision/recall jointly (e.g., F1 or a Pareto plot), and evaluate on a benchmark whose ground truth is not CHAIR/COCO-based (e.g., POPE) or provide a human study on a sample.
  3. [Appendix A.1; Tables 1-3] All experiments are single-run, and the final checkpoint is selected based on the lowest CHAIR_i on a 500-sample validation set. This selection criterion is identical to the main evaluation criterion and can inflate differences. Without multiple seeds and error bars, it is not possible to assess the variability of the reported state-of-the-art improvements. Please report mean ± std over at least 3 runs (or otherwise justify single-run reporting), and select checkpoints using a criterion not directly optimizing the headline metric.
minor comments (5)
  1. [Appendix A.2] The text says 'ground-truth object annotations come from COCO ground-truth sentences,' which is inconsistent with the standard CHAIR implementation, where COCO instance annotations define ground-truth objects. If a different protocol was used, this should be stated explicitly.
  2. [Section 4.2] 'We impute it' should be 'We attribute it'.
  3. [Table 1] The Object HalBench row for LLaVA-1.5-7B baseline contains '..' placeholders; these should be filled or explicitly marked as not reported.
  4. [Figure 3 and Section 4.2] The qualitative examples mention objects like 'glasses' and 'contrails' that are not in the COCO vocabulary. The paper should clarify that the method only addresses COCO-class object hallucinations, and that hallucinations outside this vocabulary are out of scope.
  5. [Section 4.1] The exact prompts used for evaluation benchmarks (e.g., the generative task on AMBER, CHAIR-MSCOCO, and Object HalBench) are not stated. This information is needed for reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the CHAIR family is shared between training reward and evaluation, but the object sets come from different sources (DETR detections vs. human/COCO annotations), so the reported gains are empirical transfer, not definitional equivalence.

full rationale

CHAIR-DPO labels preferences with CHAIRi computed against DETR-DC5-R101 detections restricted to the MSCOCO vocabulary (Eq. 1, Eq. 2, Sec. 3.2). The Table 1 headline metrics are CHAIRi/CHAIRs on AMBER, CHAIR-MSCOCO, and Object HalBench, whose ground truth is human-annotated or COCO ground-truth annotation based (Appendix A.2), not the DETR detections used to build the preference pairs. The evaluation is therefore not the same object set as the training reward; improvement on held-out, differently labeled data is an empirical transfer claim. The ablation, the AMBER Coverage/Cognition metrics, and the MME/SEED/MMMU/Science-QA/AI2D tables provide independent non-CHAIR evidence of behavior change. The paper's self-citations (LLaVA-MORE as a backbone, earlier captioning work in the related-work section) are not used as load-bearing justification for the CHAIR-DPO derivation or for any uniqueness/optimality claim. The reviewer-skeptic concern about DETR misses is a correctness/robustness risk: if the detector misses objects, some training labels are wrong and CHAIR-style metrics can be gamed by suppressing object mentions (see the reported Coverage drop). But noise or misspecification in the training signal does not make the held-out CHAIR evaluation equal to the training objective by construction. No equation in the paper identifies the test CHAIR score with the DPO reward, and no result is asserted solely on the authority of the authors' prior work. Hence no circular step can be exhibited, and the appropriate score is 0.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim rests on treating CHAIR as both the training reward and the evaluation quantity, and on a specific object-detector oracle for ground truth. The DPO objective is taken from Rafailov et al. (2023) and the CHAIR metric from Rohrbach et al. (2018); both are standard domain tools rather than new postulates. No new entities are introduced.

free parameters (3)
  • DPO KL coefficient beta = 0.5, 0.3, 0.2
    Central results in Tables 1 and 2 are reported for three manually chosen beta values. Lower beta yields lower hallucination rates at the cost of lower Coverage, so the choice is not neutral and is not selected on a held-out set.
  • Sampling temperature for candidate completions = 0.7
    Used to draw y1 and y2 from pi_ref in Section 4.1. This temperature controls the diversity of the two candidate answers and therefore the quality of the preference pairs; it is not swept or justified.
  • LoRA and optimization hyperparameters = rank 128, alpha 256, lr 2e-6, batch 64/16, warmup 33/145
    Standard choices that set the optimization regime. They are not varied, but they affect how strongly the model moves away from pi_ref and how much general capability is preserved.
assumptions (4)
  • domain assumption CHAIRi computed over detected MSCOCO objects is a valid measure of hallucination severity for free-form MLLM responses.
    Used to define winner and loser in Eq. 2 and to report all headline results. It presumes that object mentions not detected by the detector are hallucinations and that all detected objects are genuinely present.
  • domain assumption DETR-DC5-R101 detections, filtered to MSCOCO classes and synonym-matched, form a complete and correct description of the objects truly depicted in the image.
    Section 3.2 treats the class names of detected items as the ground-truth set. No detector is perfect, so missed objects become false hallucination labels and out-of-vocabulary objects cannot be credited. The paper does not analyze detector error rates.
  • domain assumption Sampling two answers from pi_ref at temperature 0.7 provides sufficiently informative preference pairs after filtering out zero-difference pairs.
    Preference quality depends on the diversity and quality of the two samples. The paper does not analyze sample diversity, label noise, or the effect of filtering 90% of the dataset on the distribution of retained prompts.
  • domain assumption DPO with LoRA preserves general capabilities when beta is small.
    Used to claim no catastrophic forgetting. Supported empirically for LLaVA-1.5-7B but the LLaVA-MORE-8B results show regression on MME, Science-QA, and AI2D, so the preservation is not universal.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Mitigating Hallucinations in Multimodal LLMs via Object-aware Preference Optimization." pith.science (2026). https://pith.science/paper/4GXGQHFZ

@misc{pith2026250820181,
  author       = {Pith},
  title        = {Pith review of: Mitigating Hallucinations in Multimodal LLMs via Object-aware Preference Optimization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4GXGQHFZ}},
  note         = {Machine review of arXiv:2508.20181}
}
read the original abstract

Multimodal Large Language Models (MLLMs) emerge as a unified interface to address a multitude of tasks, ranging from NLP to computer vision. Despite showcasing state-of-the-art results in many benchmarks, a long-standing issue is the tendency of MLLMs to hallucinate, that is to generate answers to the user's query that are not reflected in the visual input. In this paper, we address the problem of hallucinations as an alignment problem, seeking to steer the MLLM so that it prefers generating content without hallucinations. In contrast to recent approaches that require complicated pipelines to build synthetic preference data for alignment training, often relying on proprietary models, we capitalize on the well-known CHAIR metric, originally proposed to gauge the degree of hallucinations in image captioning. Given a pair of generated answers, we leverage CHAIR to distinguish winner and loser options (i.e., non-hallucinated and hallucinated samples) and fine-tune off-the-shelf MLLMs via Direct Preference Optimization (DPO). The resulting method, which we refer to as CHAIR-DPO, effectively diminishes the amount of hallucinated answers on several hallucination benchmarks, demonstrating the effectiveness of fine-tuning the MLLM with a CHAIR-based reward. Source code and trained models are publicly available at https://github.com/aimagelab/CHAIR-DPO.

Figures

Figures reproduced from arXiv: 2508.20181 by the authors.

Figure 1
Figure 1. Overview of CHAIR-DPO. We begin by collecting preference data for optimization. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Details of the preference data collection pipeline. First, we ask an MLLM to [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Image captioning qualitative results with LLaVA-1.5-7B and LLaVA-MORE-8B. [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Qualitative results of CHAIR-DPO with LLaVA-1.5-7B [ [PITH_FULL_IMAGE:figures/full_fig_p017_4.png]
Figure 5
Figure 5. Figure 5: Qualitative results of CHAIR-DPO with LLaVA-MORE-8B [ [PITH_FULL_IMAGE:figures/full_fig_p018_5.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Rethinking Visual Neglect: Steering via Context-Preference for MLLM Hallucination Mitigation

    cs.CL 2026-05 unverdicted novelty 7.0 of 10

    CAS mitigates object hallucinations in MLLMs by extracting two context preference vectors from designed conflict samples and applying signed residual injection at mid-early MLP layers without retraining or added latency.

Reference graph

Works this paper leans on

63 extracted references · 44 canonical work pages · cited by 1 Pith paper

  1. [1]

    Qwen-VL: A Versatile Vision-Language Model for Un- derstanding, Localization, Text Reading, and Beyond

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-VL: A Versatile Vision-Language Model for Un- derstanding, Localization, Text Reading, and Beyond. arXiv preprint arXiv:2308.12966, 2023

    arXiv 2023

  2. [2]

    Hallucination of Multimodal Large Language Models: A Survey

    Zechen Bai, Pichao Wang, Tianjun Xiao, Tong He, Zongbo Han, Zheng Zhang, and Mike Zheng Shou. Hallucination of Multimodal Large Language Models: A Survey. arXiv preprint arXiv:2404.18930, 2024

    arXiv 2024

  3. [3]

    With a Little Help from your own Past: Prototypical Memory Networks for Image Captioning

    Manuele Barraco, Sara Sarto, Marcella Cornia, Lorenzo Baraldi, and Rita Cucchiara. With a Little Help from your own Past: Prototypical Memory Networks for Image Captioning. In ICCV, 2023

    work page 2023

  4. [4]

    The Revolution of Multimodal Large Language Models: A Survey

    Davide Caffagni, Federico Cocchi, Luca Barsellotti, Nicholas Moratelli, Sara Sarto, Lorenzo Baraldi, Lorenzo Baraldi, Marcella Cornia, and Rita Cucchiara. The Revolution of Multimodal Large Language Models: A Survey. In ACL Findings, 2024

    work page 2024

  5. [5]

    End-to-End Object Detection with Transformers

    Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kir- illov, and Sergey Zagoruyko. End-to-End Object Detection with Transformers. InECCV, 2020

    work page 2020

  6. [6]

    Shikra: Unleashing Multimodal LLM’s Referential Dialogue Magic

    Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing Multimodal LLM’s Referential Dialogue Magic. arXiv preprint arXiv:2306.15195, 2023. A. COMPAGNONI ET AL. : MITIGA TING HALLUCINA TIONS IN MULTIMODAL LLMS 11

    arXiv 2023

  7. [7]

    Gonzalez, Ion Stoica, and Eric P

    Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90%* ChatGPT Quality,

    work page

  8. [8]

    DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Models

    Yung-Sung Chuang, Yujia Xie, Hongyin Luo, Yoon Kim, James Glass, and Pengcheng He. DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Models. In ICLR, 2024

    work page 2024

Show all 63 references
  1. [9]

    LLaV A-MORE: A Comparative Study of LLMs and Visual Backbones for Enhanced Visual Instruction Tuning

    Federico Cocchi, Nicholas Moratelli, Davide Caffagni, Sara Sarto, Marcella Cornia, Lorenzo Baraldi, and Rita Cucchiara. LLaV A-MORE: A Comparative Study of LLMs and Visual Backbones for Enhanced Visual Instruction Tuning. In ICCV Workshops, 2025

    2025

  2. [10]

    InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning. arXiv preprint arXiv:2305.06500, 2023

    2023 arXiv

  3. [11]

    The Llama 3 Herd of Models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al- Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The Llama 3 Herd of Models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  4. [12]

    MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models

    Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, et al. MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models. arXiv preprint arXiv:2306.13394, 2023

    2023 arXiv

  5. [13]

    OneLLM: One Framework to Align All Modalities with Language

    Jiaming Han, Kaixiong Gong, Yiyuan Zhang, Jiaqi Wang, Kaipeng Zhang, Dahua Lin, Yu Qiao, Peng Gao, and Xiangyu Yue. OneLLM: One Framework to Align All Modalities with Language. In CVPR, 2024

    2024

  6. [14]

    ORPO: Monolithic Preference Optimization without Reference Model

    Jiwoo Hong, Noah Lee, and James Thorne. ORPO: Monolithic Preference Optimization without Reference Model. In EMNLP, 2024

    2024

  7. [15]

    LoRA: Low-Rank Adaptation of Large Language Models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-Rank Adaptation of Large Language Models. arXiv preprint arXiv:2106.09685, 2021

    2021 arXiv

  8. [16]

    A Survey on Hal- lucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions

    Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, et al. A Survey on Hal- lucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions. ACM Trans. on Information Sy...

    2025

  9. [17]

    OPERA: Alleviating Hallucination in Multi- Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation

    Qidong Huang, Xiaoyi Dong, Pan Zhang, Bin Wang, Conghui He, Jiaqi Wang, Dahua Lin, Weiming Zhang, and Nenghai Yu. OPERA: Alleviating Hallucination in Multi- Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation. In CVPR, 2024

    2024

  10. [18]

    SymDPO: Boosting In-Context Learning of Large Multimodal Models with Symbol Demonstration Direct Preference Optimization

    Hongrui Jia, Chaoya Jiang, Haiyang Xu, Wei Ye, Mengfan Dong, Ming Yan, Ji Zhang, Fei Huang, and Shikun Zhang. SymDPO: Boosting In-Context Learning of Large Multimodal Models with Symbol Demonstration Direct Preference Optimization. arXiv preprint arXiv:2411.11909, 2024. 12 A. ...

    2024 arXiv

  11. [19]

    Modality- Fair Preference Optimization for Trustworthy MLLM Alignment

    Songtao Jiang, Yan Zhang, Ruizhe Chen, Yeying Jin, and Zuozhu Liu. Modality- Fair Preference Optimization for Trustworthy MLLM Alignment. arXiv preprint arXiv:2410.15334, 2024

    2024 arXiv

  12. [20]

    Deep Visual-Semantic Alignments for Generating Image Descriptions

    Andrej Karpathy and Li Fei-Fei. Deep Visual-Semantic Alignments for Generating Image Descriptions. In CVPR, 2015

    2015

  13. [21]

    A Diagram is Worth a Dozen Images

    Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. A Diagram is Worth a Dozen Images. In ECCV, 2016

    2016

  14. [22]

    Adam: A Method for Stochastic Optimization

    DP Kingma. Adam: A Method for Stochastic Optimization. In ICLR, 2014

    2014

  15. [23]

    Building and Better Understanding Vision-Language Models: Insights and Future Directions

    Hugo Laurençon, Andrés Marafioti, Victor Sanh, and Léo Tronchon. Building and Better Understanding Vision-Language Models: Insights and Future Directions. In NeurIPS Workshops, 2024

    2024

  16. [24]

    Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding

    Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shijian Lu, Chunyan Miao, and Lidong Bing. Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding. In CVPR, 2024

    2024

  17. [25]

    SEED- Bench: Benchmarking Multimodal LLMs with Generative Comprehension

    Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. SEED- Bench: Benchmarking Multimodal LLMs with Generative Comprehension. arXiv preprint arXiv:2307.16125, 2023

    2023 arXiv

  18. [26]

    Microsoft COCO: Common Objects in Context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ra- manan, Piotr Dollár, and C Lawrence Zitnick. Microsoft COCO: Common Objects in Context. In ECCV, 2014

    2014

  19. [27]

    Visual Instruction Tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual Instruction Tuning. In NeurIPS, 2023

    2023

  20. [28]

    Improved Baselines with Visual Instruction Tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved Baselines with Visual Instruction Tuning. In CVPR, 2024

    2024

  21. [29]

    Neural Baby Talk

    Jiasen Lu, Jianwei Yang, Dhruv Batra, and Devi Parikh. Neural Baby Talk. In CVPR, 2018

    2018

  22. [30]

    Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering

    Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering. In NeurIPS, 2022

    2022

  23. [31]

    Revisiting Image Captioning Training Paradigm via Direct CLIP-based Optimization

    Nicholas Moratelli, Davide Caffagni, Marcella Cornia, Lorenzo Baraldi, and Rita Cucchiara. Revisiting Image Captioning Training Paradigm via Direct CLIP-based Optimization. In BMVC, 2024

    2024

  24. [32]

    Training Language Models to Follow Instructions with Human Feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training Language Models to Follow Instructions with Human Feedback. In NeurIPS, 2022

    2022

  25. [33]

    X-InstructBLIP: A Aramework for Aligning X-Modal Instruction-Aware Representations to LLMs and Emergent Cross-Modal Reasoning

    Artemis Panagopoulou, Le Xue, Ning Yu, Junnan Li, Dongxu Li, Shafiq Joty, Ran Xu, Silvio Savarese, Caiming Xiong, and Juan Carlos Niebles. X-InstructBLIP: A Aramework for Aligning X-Modal Instruction-Aware Representations to LLMs and Emergent Cross-Modal Reasoning. arXiv prepr...

    2023 arXiv

  26. [34]

    KOSMOS-2: Grounding Multimodal Large Language Models to the World

    Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. KOSMOS-2: Grounding Multimodal Large Language Models to the World. arXiv preprint arXiv:2306.14824, 2023

    2023 arXiv

  27. [35]

    ALOHa: A New Measure for Hallucination in Captioning Models

    Suzanne Petryk, David M Chan, Anish Kachinthaya, Haodi Zou, John Canny, Joseph E Gonzalez, and Trevor Darrell. ALOHa: A New Measure for Hallucination in Captioning Models. In NAACL, 2024

    2024

  28. [36]

    Learning Transferable Visual Models from Natural Language Supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning Transferable Visual Models from Natural Language Supervision. In ICML, 2021

    2021

  29. [37]

    Direct Preference Optimization: Your Language Model is Secretly a Reward Model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct Preference Optimization: Your Language Model is Secretly a Reward Model. In NeurIPS, 2023

    2023

  30. [38]

    Zero: Memory optimizations toward training trillion parameter models

    Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimizations toward training trillion parameter models. In SC, 2020

    2020

  31. [39]

    Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fahad S. Khan. LLaV A++: Extending Visual Capabilities with LLaMA-3 and Phi-3, 2024

    2024

  32. [40]

    GLaMM: Pixel Grounding Large Multimodal Model

    Hanoona Rasheed, Muhammad Maaz, Sahal Shaji, Abdelrahman Shaker, Salman Khan, Hisham Cholakkal, Rao M Anwer, Eric Xing, Ming-Hsuan Yang, and Fahad S Khan. GLaMM: Pixel Grounding Large Multimodal Model. In CVPR, 2024

    2024

  33. [41]

    Object Hallucination in Image Captioning

    Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. Object Hallucination in Image Captioning. In EMNLP, 2018

    2018

  34. [42]

    A Comprehensive Survey of Hallucination in Large Language, Image, Video and Audio Foundation Models

    Pranab Sahoo, Prabhash Meharia, Akash Ghosh, Sriparna Saha, Vinija Jain, and Aman Chadha. A Comprehensive Survey of Hallucination in Large Language, Image, Video and Audio Foundation Models. In EMNLP Findings, 2024

    2024

  35. [43]

    Mitigating Object Hallucination in MLLMs via Data-augmented Phrase-level Alignment

    Pritam Sarkar, Sayna Ebrahimi, Ali Etemad, Ahmad Beirami, Sercan Ö Arık, and Tomas Pfister. Mitigating Object Hallucination in MLLMs via Data-augmented Phrase-level Alignment. In ICLR, 2025

    2025

  36. [44]

    Image Captioning Evaluation in the Age of Multimodal LLMs: Challenges and Future Perspectives

    Sara Sarto, Marcella Cornia, and Rita Cucchiara. Image Captioning Evaluation in the Age of Multimodal LLMs: Challenges and Future Perspectives. In IJCAI, 2025

    2025

  37. [45]

    Preference Ranking Optimization for Human Alignment

    Feifan Song, Bowen Yu, Minghao Li, Haiyang Yu, Fei Huang, Yongbin Li, and Houfeng Wang. Preference Ranking Optimization for Human Alignment. In AAAI, 2024

    2024

  38. [46]

    Generative Multimodal Models Are In-Context Learners

    Quan Sun, Yufeng Cui, Xiaosong Zhang, Fan Zhang, Qiying Yu, Yueze Wang, Yongming Rao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative Multimodal Models Are In-Context Learners. In CVPR, 2024

    2024

  39. [47]

    Emu: Generative Pretraining in Multimodality

    Quan Sun, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, Yueze Wang, Hongcheng Gao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Emu: Generative Pretraining in Multimodality. In ICLR, 2024

    2024

  40. [48]

    Chameleon: Mixed-Modal Early-Fusion Foundation Models

    Chameleon Team. Chameleon: Mixed-Modal Early-Fusion Foundation Models. arXiv preprint arXiv:2405.09818, 2024. 14 A. COMPAGNONI ET AL. : MITIGA TING HALLUCINA TIONS IN MULTIMODAL LLMS

    2024 arXiv

  41. [49]

    Diffusion Model Alignment Using Direct Preference Optimization

    Bram Wallace, Meihua Dang, Rafael Rafailov, Linqi Zhou, Aaron Lou, Senthil Purush- walkam, Stefano Ermon, Caiming Xiong, Shafiq Joty, and Nikhil Naik. Diffusion Model Alignment Using Direct Preference Optimization. In CVPR, 2024

    2024

  42. [50]

    mDPO: Conditional Preference Optimization for Multimodal Large Language Models

    Fei Wang, Wenxuan Zhou, James Y Huang, Nan Xu, Sheng Zhang, Hoifung Poon, and Muhao Chen. mDPO: Conditional Preference Optimization for Multimodal Large Language Models. In EMNLP, 2024

    2024

  43. [51]

    AMBER: An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination Evaluation

    Junyang Wang, Yuhang Wang, Guohai Xu, Jing Zhang, Yukai Gu, Haitao Jia, Jiaqi Wang, Haiyang Xu, Ming Yan, Ji Zhang, et al. AMBER: An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination Evaluation. arXiv preprint arXiv:2311.07397, 2023

    2023 arXiv

  44. [52]

    β-DPO: Direct Preference Optimization with Dynamic β

    Junkang Wu, Yuexiang Xie, Zhengyi Yang, Jiancan Wu, Jinyang Gao, Bolin Ding, Xiang Wang, and Xiangnan He. β-DPO: Direct Preference Optimization with Dynamic β. In NeurIPS, 2024

    2024

  45. [53]

    Generate, but Verify: Reducing Hallucination in Vision-Language Models with Retrospective Resampling

    Tsung-Han Wu, Heekyung Lee, Jiaxin Ge, Joseph E Gonzalez, Trevor Darrell, and David M Chan. Generate, but Verify: Reducing Hallucination in Vision-Language Models with Retrospective Resampling. arXiv preprint arXiv:2504.13169, 2025

    2025

  46. [54]

    Hallucination is Inevitable: An Innate Limitation of Large Language Models

    Ziwei Xu, Sanjay Jain, and Mohan Kankanhalli. Hallucination is Inevitable: An Innate Limitation of Large Language Models. arXiv preprint arXiv:2401.11817, 2024

    2024 arXiv

  47. [55]

    mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration

    Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan, Anwen Hu, Haowei Liu, Qi Qian, Ji Zhang, and Fei Huang. mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration. In CVPR, 2024

    2024

  48. [56]

    Woodpecker: Hallucination Correction for Multimodal Large Language Models

    Shukang Yin, Chaoyou Fu, Sirui Zhao, Tong Xu, Hao Wang, Dianbo Sui, Yunhang Shen, Ke Li, Xing Sun, and Enhong Chen. Woodpecker: Hallucination Correction for Multimodal Large Language Models. Science China Information Sciences , 67(12): 220105, 2024

    2024

  49. [57]

    RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedback

    Tianyu Yu, Yuan Yao, Haoye Zhang, Taiwen He, Yifeng Han, Ganqu Cui, Jinyi Hu, Zhiyuan Liu, Hai-Tao Zheng, Maosong Sun, et al. RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedback. In CVPR, 2024

    2024

  50. [58]

    MMMU: A Massive Multi- discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. MMMU: A Massive Multi- discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI. In CVPR, 2024

    2024

  51. [59]

    Less is More: Mitigating Multimodal Hallucina- tion from an EOS Decision Perspective

    Zihao Yue, Liang Zhang, and Qin Jin. Less is More: Mitigating Multimodal Hallucina- tion from an EOS Decision Perspective. In ACL, 2024

    2024

  52. [60]

    Direct Preference Optimization of Video Large Multimodal Models from Language Model Reward

    Ruohong Zhang, Liangke Gui, Zhiqing Sun, Yihao Feng, Keyang Xu, Yuanhan Zhang, Di Fu, Chunyuan Li, Alexander Hauptmann, Yonatan Bisk, et al. Direct Preference Optimization of Video Large Multimodal Models from Language Model Reward. arXiv preprint arXiv:2404.01258, 2024. A. CO...

    2024 arXiv

  53. [61]

    Beyond Hallucinations: Enhancing LVLMs through Hallucination-Aware Direct Preference Optimization

    Zhiyuan Zhao, Bin Wang, Linke Ouyang, Xiaoyi Dong, Jiaqi Wang, and Conghui He. Beyond Hallucinations: Enhancing LVLMs through Hallucination-Aware Direct Preference Optimization. arXiv preprint arXiv:2311.16839, 2023

    2023 arXiv

  54. [62]

    Aligning Modalities in Vision Large Language Models via Preference Fine-tuning

    Yiyang Zhou, Chenhang Cui, Rafael Rafailov, Chelsea Finn, and Huaxiu Yao. Aligning Modalities in Vision Large Language Models via Preference Fine-tuning. arXiv preprint arXiv:2402.11411, 2024. In the following, we present additional materials about CHAIR-DPO, comprising techni...

    2024 arXiv

  55. [2023]

    URL https://lmsys.org/blog/2023-03-30-vicuna/

    2023

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.