Pith. sign in

REVIEW 3 major objections 5 minor 2 cited by

OmniDPO claims that adding modality-degradation preference pairs to DPO reduces omni-modal hallucination, with average gains of 3.48% on CMM and 4.23% on AVHBench.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-05 13:16 UTC pith:JC6MAHAJ

load-bearing objection Useful new dataset and a plausible DPO extension for omni-modal hallucination, but the key evidence does not rule out a conservative response shift and the method's crucial noise parameters go unreported. the 3 major comments →

arxiv 2509.00723 v1 pith:JC6MAHAJ submitted 2025-08-31 cs.AI cs.MM

OmniDPO: A Preference Optimization Framework to Address Omni-Modal Hallucination

Junzhe Chen , Tianshu Zhang , Shiyu Huang , Yuwei Niu , Chao Sun , Rongzhou Zhang , Guanyu Zhou , Lijie Wen
show 1 more author
Xuming Hu
This is my paper
classification cs.AI cs.MM
keywords omni-modal hallucinationdirect preference optimizationmultimodal groundingaudio-video-text alignmenthallucination mitigationmodality degradationpreference datasetomni-modal large language models
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper is trying to establish that hallucination in omni-modal large language models has two complementary cures: strengthening audio-video alignment and forcing the model to condition its confidence on real perceptual evidence rather than text priors. It builds a 10k-sample preference dataset from MSRVTT videos where chosen answers incorporate audio-derived descriptions and rejected answers drop them, plus pairs in which video or audio is corrupted with Gaussian noise. The method extends Direct Preference Optimization with two modality-robustness losses that contrast the model's probability of the correct answer under full versus degraded inputs. The empirical claim is that applying OmniDPO to Qwen2.5-Omni and MiniCPM-o-2.6 lifts hallucination-focused scores by an average of 3.48% on CMM and 4.23% on AVHBench, and also improves general reasoning on MMAU and MMMU. A sympathetic reader would care because this is a training recipe that treats audio-video interaction as a first-class object, targeting a failure mode that text-only and vision-centric methods leave untouched.

Core claim

OmniDPO extends Direct Preference Optimization so that, alongside standard text-preference pairs, the model is trained on pairs where the input modality is degraded: video frames with added Gaussian noise and audio with added Gaussian noise. The modality-robustness losses L_vis and L_aud (Eqs. 4-5) compare the probability of the correct answer under the full input versus the degraded input, teaching the model to be differentially confident when the perceptual evidence is actually present. On top of this, textual preference pairs are built by generating the chosen answer from video plus an audio-derived text description (extracted with Qwen2-Audio), while the rejected answer comes from the sa

What carries the argument

The central object is the OmniDPO loss, L_OMNI = L_DPO + lambda_V L_vis + lambda_A L_aud, an extension of Direct Preference Optimization. The novel components are the modality-robustness terms: L_vis contrasts log P(Y+|V,A,T) against log P(Y+|V-,A,T), where V- is video corrupted by Gaussian noise (Eq. 1), and L_aud contrasts the full-input probability against the probability under noised audio A- (Eq. 2). These terms teach the model that its confidence should track actual perceptual evidence. The dataset OmniDPO-10k supplies the preference pairs: chosen answers are generated by extracting an audio description with Qwen2-Audio and feeding it together with the video into Qwen2.5-VL, while reje

Load-bearing premise

The modality-robustness losses assume that adding Gaussian noise to video frames and audio removes the semantic evidence those modalities carry, so the correct answer should be less probable under the corrupted input; the noise level is never specified or validated.

What would settle it

Recover the noise levels sigma_v and sigma_a from the released code, apply the same corruption to the base models, and measure whether their answers or the probability of the ground-truth answer actually change. If corrupted inputs leave the probability of the correct answer nearly unchanged, the preference loss has no grounding signal to teach and any measured gains must come from something else, such as overall confidence suppression. A second check: after OmniDPO training, verify on held-out data that P(Y+|full input) exceeds P(Y+|degraded input) by a margin that tracks how informative the

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • Because the framework needs only a preference dataset and the DPO objective, the same recipe can be applied to other omni-modal LLMs and should shift their hallucination-resistance scores similarly.
  • The 16.96% drop in the 'yes' response rate of Qwen2.5-Omni indicates the method changes affirmation behavior, not just answer wording, which is the hallmark of reducing over-confident hallucination.
  • Improvements on MMAU and MMMU show that grounding-oriented preference training does not trade away general reasoning capability.
  • Text-only DPO baselines show little gain, and vision-centric methods VCD and ICT fail on audio-video subdomains, implying that the modality-robustness terms and audio-video pairs carry the improvement.
  • The audio-video alignment pairs specifically lift the V-AL (video-audio-language) subdomain of CMM, where no baseline improves, confirming the target is the audio-video interaction gap.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The paper never specifies the noise levels sigma_v and sigma_a; if the corruption is too mild to change what the model can infer, the modality-robustness losses reduce to a confidence-calibration penalty rather than genuine grounding. A reader can test this by recovering the sigma values from the released code and checking whether corrupted inputs actually change the base models' answers.
  • The chosen answers are generated by two other models (Qwen2-Audio plus Qwen2.5-VL), so OmniDPO-10k inherits those teachers' blind spots; a model trained on this data may learn their biases rather than ground truth, and measuring against human-verified answers would clarify this.
  • The ablations show a modality interference effect: optimizing only video preferences hurts audio-related performance and vice versa. This suggests a design rule for multimodal preference data: per-modality losses need joint training, or they can unbalance the model toward the optimized modality.
  • The dataset is built only from MSRVTT clips averaging tens of seconds; generalization to longer, multi-scene videos, or to inputs where audio and video genuinely disagree, is untested and is the natural next stress test.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes OmniDPO, a DPO-based training framework for mitigating hallucination in omni-modal LLMs that process video, audio, and text. The authors construct OmniDPO-10k from MSRVTT, containing (i) text preference pairs whose chosen response is generated from Qwen2-Audio-derived audio text plus Qwen2.5-VL and whose rejected response is generated from video alone, and (ii) modality-robustness pairs in which Gaussian noise is added to video or audio and the same chosen response is treated as less likely under the corrupted input. The training objective combines standard DPO with visual and audio conditional preference losses (Eqs. 4-5). Experiments on Qwen2.5-Omni and MiniCPM-o-2.6 report improved F1 on AVHBench, improved PA/HR on CMM, and gains on MMAU/MMMU, supporting the claim of reduced omni-modal hallucination and improved reasoning.

Significance. OmniDPO addresses a real gap: preference-based hallucination mitigation for audio-visual LLMs is underexplored, and the released dataset is a potentially useful resource. The two-model evaluation, ablations, and reasoning-benchmark checks are strengths. However, the central claim rests on an unvalidated assumption about the effect of Gaussian corruption: the modality-robustness losses only improve grounding if corrupted inputs actually remove evidence for the correct answer. The reported recall drops and 'yes'-rate shifts on AVHBench are consistent with a conservative response shift rather than improved grounding. With additional validation of the corruption mechanism and a more careful reading of AVHBench, the framework could be a solid contribution; in its current form the evidence is not yet sufficient.

major comments (3)
  1. [§3, Eqs. (1)-(2); §4, Eqs. (4)-(5)] The modality-robustness losses are load-bearing, but the paper never specifies σ_v and σ_a, nor does it validate that the corrupted inputs X_{V-} and X_{A-} are semantically degraded for the preferred answer Y+. If the noise is mild, or if the intact modality is sufficient to determine Y+ (e.g., the audio in an MSRVTT clip conveys the answer even when the video is corrupted), the preference label (X,Y+) > (X_deg,Y+) is invalid; the loss then trains the model to suppress confidence in correct answers. The authors should report the noise scales, show that corrupted inputs reduce task performance or answerability, and demonstrate that the loss increases dependence on the corrupted modality rather than merely reducing confidence.
  2. [§5.2, Table 1] The AVHBench results do not clearly separate 'reduced hallucination' from 'reduced affirmative responding'. For Qwen2.5-Omni on Audio-driven Video Hallucination, OmniDPO increases F1 from 77.38 to 83.47, but Recall drops from 88.56 to 78.70 and the Yes rate from 64.44% to 44.28%; on Video-driven Audio Hallucination Recall drops from 98.78 to 94.93. These patterns are exactly what would be expected if the model becomes conservative rather than better grounded. Since the paper's central claim is improved multimodal grounding, the authors need to report precision-recall operating points, possibly ROC or calibration curves, and discuss when a 'no' is correct versus an avoidance response.
  3. [§3, Table 4] The textual preference pairs are generated entirely by teacher models without human verification. The examples in Table 4 show that rejected outputs are often plausible paraphrases that omit audio details (e.g., 'a man gestures enthusiastically while observing...' vs 'a man narrates...'), not necessarily hallucinated content. If the rejected answer is not factually wrong, LDPO may reward a particular descriptive style rather than hallucination avoidance. Please provide human judgment statistics on a sample of pairs (e.g., percentage where the rejected response is actually hallucinated) or restrict the dataset to pairs where the rejected answer contradicts the input.
minor comments (5)
  1. [§4, Eq. (3)] The ratio notation in Eq. (3) is malformed: it appears to write log Pθ(Y|X) / log Pref(Y|X) as a fraction, but the intended expression is log(Pθ(Y|X)/Pref(Y|X)). Please clarify and define the reference policy Pref.
  2. [§5.1] 'Implention Details' is a typo for 'Implementation Details'.
  3. [§5.1, §5.2] No standard deviations, confidence intervals, or number of training seeds are reported. Several reported gains are small (e.g., MiniCPM-o-2.6 on CMM), so the reader cannot assess significance.
  4. [§4, §5.1] The hyperparameters λ_V, λ_A, and β are fixed at 1.0, 1.0, and 0.1 respectively, but no sensitivity analysis is provided. Since the method's mechanism depends on the balance between LDPO and Lvis/Laud, this should be discussed.
  5. [§1] The claim that OmniDPO and OmniDPO-10k are 'the first' omni-modal hallucination mitigation method and dataset is strong. Prior audio-visual hallucination benchmarks and mitigation efforts are cited later, so the novelty claim should be qualified accordingly.

Circularity Check

0 steps flagged

No significant circularity: the central claim is an empirical result evaluated on external benchmarks, and the method's training objectives are not fitted quantities.

full rationale

The paper's central assertion—that OmniDPO improves multimodal grounding and reduces hallucination—is supported by experiments on CMM and AVHBench, both of which are external benchmarks constructed independently of the OmniDPO-10k training data (built from MSRVTT). The benchmark improvements are therefore not forced by construction. The modality-robustness losses in Eqs. 4 and 5 are training objectives that explicitly impose a preference between full and degraded inputs; they are not predictions of benchmark performance and no parameter fitted to the evaluation data is renamed as a result. The assumption that Gaussian corruption in Eqs. 1 and 2 semantically degrades the input is an unvalidated modelling assumption and a legitimate correctness concern—if the corruption is too mild or the intact modality suffices, the loss may penalize correct confidence—but this is an empirical validity issue, not a circular derivation. The only self-citation is the use of ICT [8] as a baseline, whose authors overlap with the present paper. It is not load-bearing for the proposed framework or for the main result; the baseline's behavior is an empirical comparison rather than an imported premise. The novelty claim of being 'the first' omni-modal hallucination mitigation method is a contribution claim, not an argument that reduces to self-citation. Equation 3 contains a typographical ambiguity in the reference-model ratios, but that is a correctness/notation issue, not circularity. Overall, the derivation chain is self-contained: the method is defined by its loss, and the central claim is tested against external benchmarks, so no circular step can be exhibited.

Axiom & Free-Parameter Ledger

4 free parameters · 3 axioms · 0 invented entities

The method rests on DPO as standard background, on an unverified corruption-degradation assumption, and on teacher-generated targets assumed correct. The two noise scales are literally unspecified, which is the most concrete gap in the ledger.

free parameters (4)
  • sigma_v (visual Gaussian noise scale)
    Introduced in Eq. 1 but its value is never reported anywhere in the paper; the method's behavior and the meaning of 'degraded video' depend on it.
  • sigma_a (audio Gaussian noise scale)
    Introduced in Eq. 2, value never reported; the audio-preference loss depends on this corruption level.
  • lambda_V, lambda_A = 1
    Set to 1 in Section 5.1 with no ablation or sensitivity analysis; the relative weight of the auxiliary losses is chosen by hand.
  • beta (DPO temperature) = 0.1
    Fixed at 0.1 for all runs; no tuning or justification is reported in the paper.
axioms (3)
  • standard math Bradley-Terry / DPO framework: the logistic loss over log-likelihood ratios is a valid proxy for preference alignment in omni-modal generation
    Section 4.1, Eq. 3; the paper builds on DPO without a formal derivation for multi-modal token likelihoods, e.g., whether audio and video tokens enter the model likelihood the same way as text tokens.
  • ad hoc to paper Gaussian-corrupted inputs V- and A- are semantically degraded versions of V and A, so the correct answer Y+ should have lower probability under corrupted inputs
    Eqs. 1-2, Section 3; the paper never verifies that the chosen noise levels actually reduce the model's confidence or make the input unusable. If the corruption is too mild, the preference loss teaches conservative confidence suppression rather than grounding.
  • ad hoc to paper The teacher-generated Y+ (Qwen2-Audio plus Qwen2.5-VL) is the correct answer consistent with the full input
    Section 3 and Section 4.2; the paper explicitly assumes Y+ is ground-truth or human-verified, but no human verification step is described in the data construction pipeline.

pith-pipeline@v1.4.0-alltime-deepseek-medium · 17498 in / 11340 out tokens · 130948 ms · 2026-08-05T13:16:33.917004+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of OmniDPO: A Preference Optimization Framework to Address Omni-Modal Hallucination." pith.science (2026). https://pith.science/paper/JC6MAHAJ

@misc{pith2026250900723,
  author       = {Pith},
  title        = {Pith review of: OmniDPO: A Preference Optimization Framework to Address Omni-Modal Hallucination},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JC6MAHAJ}},
  note         = {Machine review of arXiv:2509.00723}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Recently, Omni-modal large language models (OLLMs) have sparked a new wave of research, achieving impressive results in tasks such as audio-video understanding and real-time environment perception. However, hallucination issues still persist. Similar to the bimodal setting, the priors from the text modality tend to dominate, leading OLLMs to rely more heavily on textual cues while neglecting visual and audio information. In addition, fully multimodal scenarios introduce new challenges. Most existing models align visual or auditory modalities with text independently during training, while ignoring the intrinsic correlations between video and its corresponding audio. This oversight results in hallucinations when reasoning requires interpreting hidden audio cues embedded in video content. To address these challenges, we propose OmniDPO, a preference-alignment framework designed to mitigate hallucinations in OLLMs. Specifically, OmniDPO incorporates two strategies: (1) constructing text-preference sample pairs to enhance the model's understanding of audio-video interactions; and (2) constructing multimodal-preference sample pairs to strengthen the model's attention to visual and auditory information. By tackling both challenges, OmniDPO effectively improves multimodal grounding and reduces hallucination. Experiments conducted on two OLLMs demonstrate that OmniDPO not only effectively mitigates multimodal hallucinations but also significantly enhances the models' reasoning capabilities across modalities. All code and datasets will be released upon paper acceptance.

Figures

Figures reproduced from arXiv: 2509.00723 by Chao Sun, Guanyu Zhou, Junzhe Chen, Lijie Wen, Rongzhou Zhang, Shiyu Huang, Tianshu Zhang, Xuming Hu, Yuwei Niu.

Figure 1
Figure 1. Figure 1: Overview of OMNIDPO. 2 Related Work 2.1 Omni-Modal Large Language Models Building on the success of large language models (LLMs) [5, 68, 52, 67, 14], research in the multimodal domain has gained significant momentum. Omni-Modal Large Language Models utilize LLMs as their core language models, enhancing them with modality-specific tokenizers and encoders to map multimodal inputs into a shared representation… view at source ↗
Figure 2
Figure 2. Figure 2: Key statistical information of our dataset. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Performance comparison on reasoning benchmarks (MMAU [61] and MMMU [93]) after [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Case Study and Error Analysis of OMNIDPO. In [PITH_FULL_IMAGE:figures/full_fig_p009_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Key statistical information of our dataset. [PITH_FULL_IMAGE:figures/full_fig_p017_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Chain of Modality: From Static Fusion to Dynamic Orchestration in Omni-MLLMs

    cs.CV 2026-04 unverdicted novelty 7.0

    Chain of Modality dynamically orchestrates multimodal input topologies and bifurcates cognitive execution to overcome static fusion biases in Omni-MLLMs.

  2. Don't Let the Video Speak: Audio-Contrastive Preference Optimization for Audio-Visual Language Models

    cs.CV 2026-04 unverdicted novelty 7.0

    Audio-Contrastive Preference Optimization (ACPO) mitigates audio hallucination in AVLMs via output-contrastive and input-contrastive objectives that enforce faithful audio grounding.

Reference graph

Works this paper leans on

108 extracted references · 57 canonical work pages · cited by 2 Pith papers · 1 internal anchor

  1. [1]

    Towards Causal VQA: Revealing and Reducing Spurious Correlations by Invariant and Covariant Semantic Editing, May 2020

    Vedika Agarwal, Rakshith Shetty, and Mario Fritz. Towards Causal VQA: Revealing and Reducing Spurious Correlations by Invariant and Covariant Semantic Editing, May 2020

    2020

  2. [2]

    Agla: Mitigating object hallucinations in large vision-language models with assembly of global and local attention

    Wenbin An, Feng Tian, Sicong Leng, Jiahao Nie, Haonan Lin, QianYing Wang, Guang Dai, Ping Chen, and Shijian Lu. Agla: Mitigating object hallucinations in large vision-language models with assembly of global and local attention. arXiv preprint, 2024

    2024

  3. [3]

    Qwen2.5-vl technical report, 2025

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report, 2025

    2025

  4. [4]

    Hallucination of multimodal large language models: A survey, 2025

    Zechen Bai, Pichao Wang, Tianjun Xiao, Tong He, Zongbo Han, Zheng Zhang, and Mike Zheng Shou. Hallucination of multimodal large language models: A survey, 2025

    2025

  5. [5]

    Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D. Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Lan- guage models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877–1901, 2020

    1901

  6. [6]

    Alleviating hallucinations in large vision-language models through hallucination-induced optimization, 2024

    Beitao Chen, Xinyu Lyu, Lianli Gao, Jingkuan Song, and Heng Tao Shen. Alleviating hallucinations in large vision-language models through hallucination-induced optimization, 2024

    2024

  7. [7]

    PerturboLLaV A: Reducing Multimodal Hallucinations with Perturbative Visual Training, March 2025

    Cong Chen, Mingyu Liu, Chenchen Jing, Yizhou Zhou, Fengyun Rao, Hao Chen, Bo Zhang, and Chunhua Shen. PerturboLLaV A: Reducing Multimodal Hallucinations with Perturbative Visual Training, March 2025

    2025

  8. [8]

    Ict: Image-object cross-level trusted intervention for mitigating object hallucination in large vision-language models, 2024

    Junzhe Chen, Tianshu Zhang, Shiyu Huang, Yuwei Niu, Linfeng Zhang, Lijie Wen, and Xum- ing Hu. Ict: Image-object cross-level trusted intervention for mitigating object hallucination in large vision-language models, 2024

    2024

  9. [9]

    Kwok, Hengshuang Zhao, Xiaodan Liang, Dit-Yan Yeung, Xiao Chen, Zhenguo Li, Wei Zhang, Qun Liu, Jun Yao, Lanqing Hong, Lu Hou, and Hang Xu

    Kai Chen, Yunhao Gou, Runhui Huang, Zhili Liu, Daxin Tan, Jing Xu, Chunwei Wang, Yi Zhu, Yihan Zeng, Kuo Yang, Dingdong Wang, Kun Xiang, Haoyuan Li, Haoli Bai, Jianhua Han, Xiaohui Li, Weike Jin, Nian Xie, Yu Zhang, James T. Kwok, Hengshuang Zhao, Xiaodan Liang, Dit-Yan Yeung, Xiao Chen, Zhenguo Li, Wei Zhang, Qun Liu, Jun Yao, Lanqing Hong, Lu Hou, and H...

    2025

  10. [10]

    V AST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Dataset, October 2023

    Sihan Chen, Handong Li, Qunbo Wang, Zijia Zhao, Mingzhen Sun, Xinxin Zhu, and Jing Liu. V AST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Dataset, October 2023

    2023

  11. [11]

    Halc: Object hallucination reduction via adaptive focal-contrast decoding, June 2024

    Zhaorun Chen, Zhuokai Zhao, Hongyin Luo, Huaxiu Yao, Bo Li, and Jiawei Zhou. Halc: Object hallucination reduction via adaptive focal-contrast decoding, June 2024. http:// arxiv.org/abs/2403.00425. 10

    Pith/arXiv arXiv 2024

  12. [12]

    Videollama 2: Advancing spatial-temporal modeling and audio understanding in video-llms, 2024

    Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, and Lidong Bing. Videollama 2: Advancing spatial-temporal modeling and audio understanding in video-llms, 2024

    2024

  13. [13]

    Factool: Factuality detection in generative ai – a tool augmented framework for multi-task and multi-domain scenarios, July 2023

    I.-Chun Chern, Steffi Chern, Shiqi Chen, Weizhe Yuan, Kehua Feng, Chunting Zhou, Junxian He, Graham Neubig, and Pengfei Liu. Factool: Factuality detection in generative ai – a tool augmented framework for multi-task and multi-domain scenarios, July 2023. http: //arxiv.org/abs/2307.13528

    Pith/arXiv arXiv 2023

  14. [14]

    Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality

    Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna. lmsys. org (accessed 14 April 2023), 2(3):6, 2023

    2023

  15. [15]

    Qwen2-audio technical report, 2024

    Yunfei Chu, Jin Xu, Qian Yang, Haojie Wei, Xipin Wei, Zhifang Guo, Yichong Leng, Yuanjun Lv, Jinzheng He, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen2-audio technical report, 2024

    2024

  16. [16]

    Dola: Decoding by contrasting layers improves factuality in large language models, March

    Yung-Sung Chuang, Yujia Xie, Hongyin Luo, Yoon Kim, James Glass, and Pengcheng He. Dola: Decoding by contrasting layers improves factuality in large language models, March

  17. [17]

    Detecting hallucinations in large language models using semantic entropy

    Sebastian Farquhar, Jannik Kossen, Lorenz Kuhn, and Yarin Gal. Detecting hallucinations in large language models using semantic entropy. Nature, 630(8017):625–630, Jun 2024

    2024

  18. [18]

    Multi-modal hallucination control by visual information grounding, 2024

    Alessandro Favero, Luca Zancato, Matthew Trager, Siddharth Choudhary, Pramuditha Perera, Alessandro Achille, Ashwin Swaminathan, and Stefano Soatto. Multi-modal hallucination control by visual information grounding, 2024

    2024

  19. [19]

    Vita: Towards open-source interactive omni multimodal llm, 2024

    Chaoyou Fu, Haojia Lin, Zuwei Long, Yunhang Shen, Meng Zhao, Yifan Zhang, Shaoqi Dong, Xiong Wang, Di Yin, Long Ma, Xiawu Zheng, Ran He, Rongrong Ji, Yunsheng Wu, Caifeng Shan, and Xing Sun. Vita: Towards open-source interactive omni multimodal llm, 2024

    2024

  20. [20]

    Exploring hallucination of large multimodal models in video understanding: Benchmark, analysis and mitigation, 2025

    Hongcheng Gao, Jiashu Qu, Jingyi Tang, Baolong Bi, Yue Liu, Hongyu Chen, Li Liang, Li Su, and Qingming Huang. Exploring hallucination of large multimodal models in video understanding: Benchmark, analysis and mitigation, 2025

    2025

  21. [21]

    Imagebind: One embedding space to bind them all, 2023

    Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind: One embedding space to bind them all, 2023

    2023

  22. [22]

    Textbooks are all you need, 2023

    Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Harkirat Singh Behl, Xin Wang, Sébastien Bubeck, Ronen Eldan, Adam Tauman Kalai, Yin Tat Lee, and Yuanzhi Li. Textbooks are all you need, 2023

    2023

  23. [23]

    Detecting and preventing hallucinations in large vision language models

    Anisha Gunjal, Jihan Yin, and Erhan Bas. Detecting and preventing hallucinations in large vision language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 18135–18143, 2024

    2024

  24. [24]

    Aligned better, listen better for audio-visual large language models, 2025

    Yuxin Guo, Shuailei Ma, Shijie Ma, Xiaoyi Bao, Chen-Wei Xie, Kecheng Zheng, Tingyu Weng, Siyang Sun, Yun Zheng, and Wei Zou. Aligned better, listen better for audio-visual large language models, 2025

    2025

  25. [25]

    Onellm: One framework to align all modalities with language, 2025

    Jiaming Han, Kaixiong Gong, Yiyuan Zhang, Jiaqi Wang, Kaipeng Zhang, Dahua Lin, Yu Qiao, Peng Gao, and Xiangyu Yue. Onellm: One framework to align all modalities with language, 2025

    2025

  26. [26]

    Opera: Alleviating hallucination in multi-modal large language models via over-trust penalty and retrospection-allocation

    Qidong Huang, Xiaoyi Dong, Pan Zhang, Bin Wang, Conghui He, Jiaqi Wang, Dahua Lin, Weiming Zhang, and Nenghai Yu. Opera: Alleviating hallucination in multi-modal large language models via over-trust penalty and retrospection-allocation. arXiv preprint, March 2024

    2024

  27. [27]

    Visual hallucinations of multi-modal large language models, 2024

    Wen Huang, Hongbin Liu, Minxin Guo, and Neil Zhenqiang Gong. Visual hallucinations of multi-modal large language models, 2024. 11

    2024

  28. [28]

    Self-introspective decoding: Alleviating hallucinations for large vision-language models, 2024

    Fushuo Huo, Wenchao Xu, Zhong Zhang, Haozhao Wang, Zhicheng Chen, and Peilin Zhao. Self-introspective decoding: Alleviating hallucinations for large vision-language models, 2024

    2024

  29. [29]

    Self-Introspective Decoding: Alleviating Hallucinations for Large Vision-Language Models, March 2025

    Fushuo Huo, Wenchao Xu, Zhong Zhang, Haozhao Wang, Zhicheng Chen, and Peilin Zhao. Self-Introspective Decoding: Alleviating Hallucinations for Large Vision-Language Models, March 2025

    2025

  30. [30]

    Interpreting and editing vision-language representations to mitigate hallucinations, 2025

    Nick Jiang, Anish Kachinthaya, Suzie Petryk, and Yossi Gandelsman. Interpreting and editing vision-language representations to mitigate hallucinations, 2025

    2025

  31. [31]

    Code: Contrasting self-generated description to combat hallucination in large multi-modal models, 2024

    Junho Kim, Hyunjun Kim, Yeonju Kim, and Yong Man Ro. Code: Contrasting self-generated description to combat hallucination in large multi-modal models, 2024

    work page 2024

  32. [32]

    Ross, Bryan Seybold, and Lu Jiang

    Dan Kondratyuk, Lijun Yu, Xiuye Gu, José Lezama, Jonathan Huang, Grant Schindler, Rachel Hornung, Vighnesh Birodkar, Jimmy Yan, Ming-Chang Chiu, Krishna Somandepalli, Hassan Akbari, Yair Alon, Yong Cheng, Josh Dillon, Agrim Gupta, Meera Hahn, Anja Hauth, David Hendon, Alonso Martinez, David Minnen, Mikhail Sirotenko, Kihyuk Sohn, Xuan Yang, Hartwig Adam, ...

    work page 2024

  33. [33]

    Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation

    Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. arXiv, Apr 2023

    work page 2023

  34. [34]

    The curse of multi-modalities: Evaluating hallucinations of large multimodal models across language, visual, and audio

    Sicong Leng, Yun Xing, Zesen Cheng, Yang Zhou, Hang Zhang, Xin Li, Deli Zhao, Shijian Lu, Chunyan Miao, and Lidong Bing. The curse of multi-modalities: Evaluating hallucinations of large multimodal models across language, visual, and audio. arXiv preprint arXiv:2410.12787, 2024

    Pith/arXiv arXiv 2024

  35. [35]

    Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding, November 2023

    Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shijian Lu, Chunyan Miao, and Lidong Bing. Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding, November 2023

    work page 2023

  36. [36]

    Mitigating object hallucinations in large vision-language models through visual contrastive decoding

    Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shijian Lu, Chunyan Miao, and Li- dong Bing. Mitigating object hallucinations in large vision-language models through visual contrastive decoding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13872–13882, 2024

    work page 2024

  37. [37]

    Mitigating Hallucination for Large Vision Language Model by Inter-Modality Correlation Calibration Decoding, March 2025

    Jiaming Li, Jiacheng Zhang, Zequn Jie, Lin Ma, and Guanbin Li. Mitigating Hallucination for Large Vision Language Model by Inter-Modality Correlation Calibration Decoding, March 2025

    work page 2025

  38. [38]

    Baichuan-omni-1.5 technical report, 2025

    Yadong Li, Jun Liu, Tao Zhang, Tao Zhang, Song Chen, Tianpeng Li, Zehuan Li, Lijun Liu, Lingfeng Ming, Guosheng Dong, Da Pan, Chong Li, Yuanbo Fang, Dongdong Kuang, Mingrui Wang, Chenglin Zhu, Youwei Zhang, Hongyu Guo, Fengyu Zhang, Yuran Wang, Bowen Ding, Wei Song, Xu Li, Yuqi Huo, Zheng Liang, Shusen Zhang, Xin Wu, Shuai Zhao, Linchu Xiong, Yozhen Wu, J...

    work page 2025

  39. [39]

    Zhuowei Li, Haizhou Shi, Yunhe Gao, Di Liu, Zhenting Wang, Yuxiao Chen, Ting Liu, Long Zhao, Hao Wang, and Dimitris N. Metaxas. The Hidden Life of Tokens: Reducing Hallucination of Large Vision-Language Models via Visual Information Steering, February 2025. 12

    work page 2025

  40. [40]

    Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning, 2022

    Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Zou. Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning, 2022

    work page 2022

  41. [41]

    Langbridge: Interpreting image as a combination of language embeddings

    Jiaqi Liao, Yuwei Niu, Fanqing Meng, Hao Li, Changyao Tian, Yinuo Du, Yuwen Xiong, Dianqi Li, Xizhou Zhu, Li Yuan, et al. Langbridge: Interpreting image as a combination of language embeddings. arXiv preprint arXiv:2503.19404, 2025

    arXiv 2025

  42. [42]

    Uniworld: High-resolution semantic encoders for unified visual understanding and generation

    Bin Lin, Zongjian Li, Xinhua Cheng, Yuwei Niu, Yang Ye, Xianyi He, Shenghai Yuan, Wangbo Yu, Shaodong Wang, Yunyang Ge, et al. Uniworld: High-resolution semantic encoders for unified visual understanding and generation. arXiv preprint arXiv:2506.03147, 2025

    Pith/arXiv arXiv 2025

  43. [43]

    Mitigating hallucination in large multi-modal models via robust instruction tuning

    Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang. Mitigating hallucination in large multi-modal models via robust instruction tuning. In The Twelfth International Conference on Learning Representations, 2023

    work page 2023

  44. [44]

    A Survey on Hallucination in Large Vision-Language Models, May 2024

    Hanchao Liu, Wenyuan Xue, Yifei Chen, Dapeng Chen, Xiutian Zhao, Ke Wang, Liping Hou, Rongjun Li, and Wei Peng. A Survey on Hallucination in Large Vision-Language Models, May 2024

    work page 2024

  45. [45]

    Improved baselines with visual instruction tuning, 2023

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning, 2023

    2023

  46. [46]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in Neural Information Processing Systems, 36, 2024

    work page 2024

  47. [47]

    Reducing hallucinations in vision-language models via latent space steering, 2024

    Sheng Liu, Haotian Ye, Lei Xing, and James Zou. Reducing hallucinations in vision-language models via latent space steering, 2024

    work page 2024

  48. [48]

    Paying more attention to image: A training-free method for alleviating hallucination in lvlms, 2024

    Shi Liu, Kecheng Zheng, and Wei Chen. Paying more attention to image: A training-free method for alleviating hallucination in lvlms, 2024

    work page 2024

  49. [49]

    Ola: Pushing the Frontiers of Omni-Modal Language Model with Progressive Modality Alignment, February 2025

    Zuyan Liu, Yuhao Dong, Jiahui Wang, Ziwei Liu, Winston Hu, Jiwen Lu, and Yongming Rao. Ola: Pushing the Frontiers of Omni-Modal Language Model with Progressive Modality Alignment, February 2025

    work page 2025

  50. [50]

    Kernel language entropy: Fine-grained uncertainty quantification for llms from semantic similarities

    Alexander Nikitin, Jannik Kossen, Yarin Gal, and Pekka Marttinen. Kernel language entropy: Fine-grained uncertainty quantification for llms from semantic similarities. arXiv, May 2024

    work page 2024

  51. [51]

    On the audio hallucinations in large audio-video language models, 2024

    Taichi Nishimura, Shota Nakada, and Masayoshi Kondo. On the audio hallucinations in large audio-video language models, 2024

    work page 2024

  52. [52]

    OpenAI. GPT-4. https://openai.com/gpt-4, 2023

    work page 2023

  53. [53]

    Leyi Pan, Aiwei Liu, Shiyu Huang, Yijian Lu, Xuming Hu, Lijie Wen, Irwin King, and Philip S. Yu. Can LLM watermarks robustly prevent unauthorized knowledge distillation? CoRR, abs/2502.11598, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  54. [54]

    Leyi Pan, Aiwei Liu, Yijian Lu, Zitian Gao, Yichen Di, Lijie Wen, Irwin King, and Philip S. Yu. Waterseeker: Pioneering efficient detection of watermarked segments in large documents. In Luis Chiruzzo, Alan Ritter, and Lu Wang, editors, Findings of the Association for Computa- tional Linguistics: NAACL 2025, Albuquerque, New Mexico, USA, April 29 - May 4,...

    work page 2025

  55. [55]

    Distillation contrastive decoding: Improving llms reasoning with contrastive decoding and distillation, 2024

    Phuc Phan, Hieu Tran, and Long Phan. Distillation contrastive decoding: Improving llms reasoning with contrastive decoding and distillation, 2024

    work page 2024

  56. [56]

    Look, compare, decide: Alleviating hallucination in large vision-language models via multi-view multi-path reasoning, August

    Xiaoye Qu, Jiashuo Sun, Wei Wei, and Yu Cheng. Look, compare, decide: Alleviating hallucination in large vision-language models via multi-view multi-path reasoning, August

    work page

  57. [57]

    Manning, and Chelsea Finn

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct Preference Optimization: Your Language Model is Secretly a Reward Model, July 2024. 13

    work page 2024

  58. [58]

    http://arxiv.org/abs/2408.17150

    Pith/arXiv arXiv

  59. [59]

    Vacode: Visual augmented contrastive decoding

    Kim S., Cho B., Bae S., Ahn S., and Yun S.Y . Vacode: Visual augmented contrastive decoding. arXiv preprint arXiv:2408.05337, July 2024

    Pith/arXiv arXiv 2024

  60. [60]

    The curious case of halluci- nations in neural machine translation

    Vikas Raunak, Arul Menezes, and Marcin Junczys-Dowmunt. The curious case of halluci- nations in neural machine translation. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, pages 1172–1183, 6 2021

    work page 2021

  61. [61]

    Mmau: A massive multi-task audio understanding and reasoning benchmark, 2024

    S Sakshi, Utkarsh Tyagi, Sonal Kumar, Ashish Seth, Ramaneswaran Selvakumar, Oriol Nieto, Ramani Duraiswami, Sreyan Ghosh, and Dinesh Manocha. Mmau: A massive multi-task audio understanding and reasoning benchmark, 2024

    work page 2024

  62. [62]

    A comprehensive survey of hallucination in large language, image, video and audio foundation models, 2024

    Pranab Sahoo, Prabhash Meharia, Akash Ghosh, Sriparna Saha, Vinija Jain, and Aman Chadha. A comprehensive survey of hallucination in large language, image, video and audio foundation models, 2024

    work page 2024

  63. [63]

    Identifying untrustworthy samples: Data filtering for open-domain dialogues with bayesian optimization

    Lei Shen, Haolan Zhan, Xin Shen, Hongshen Chen, Xiaofang Zhao, and Xiaodan Zhu. Identifying untrustworthy samples: Data filtering for open-domain dialogues with bayesian optimization. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management, pages 1598–1608, 2021

    work page 2021

  64. [64]

    Arık, and Tomas Pfister

    Pritam Sarkar, Sayna Ebrahimi, Ali Etemad, Ahmad Beirami, Sercan Ö. Arık, and Tomas Pfister. Mitigating object hallucination in mllms via data-augmented phrase-level alignment, 2025

    work page 2025

  65. [65]

    Aligning large multimodal models with factually augmented rlhf, 2023

    Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang-Yan Gui, Yu-Xiong Wang, Yiming Yang, Kurt Keutzer, and Trevor Darrell. Aligning large multimodal models with factually augmented rlhf, 2023

    work page 2023

  66. [66]

    Pandagpt: One model to instruction-follow them all, 2023

    Yixuan Su, Tian Lan, Huayang Li, Jialu Xu, Yan Wang, and Deng Cai. Pandagpt: One model to instruction-follow them all, 2023

    2023

  67. [67]

    Chatglm: A family of large language models from glm-130b to glm-4 all tools

    GLM T., Zeng A., Xu B., Wang B., Zhang C., Yin D., Rojas D., Feng G., Zhao H., Lai H., and Yu H. Chatglm: A family of large language models from glm-130b to glm-4 all tools. arXiv preprint arXiv:2406.12793, June 2024

    Pith/arXiv arXiv 2024

  68. [68]

    Avhbench: A cross-modal hallucination benchmark for audio-visual large language models, 2025

    Kim Sung-Bin, Oh Hyun-Bin, JungMok Lee, Arda Senocak, Joon Son Chung, and Tae-Hyun Oh. Avhbench: A cross-modal hallucination benchmark for audio-visual large language models, 2025

    work page 2025

  69. [69]

    Investigating and mitigating the multimodal hallucination snowballing in large vision-language models

    Zhong W., Feng X., Zhao L., Li Q., Huang L., Gu Y ., Ma W., Xu Y ., and Qin B. Investigating and mitigating the multimodal hallucination snowballing in large vision-language models. arXiv preprint arXiv:2407.00569, August 2024

    Pith/arXiv arXiv 2024

  70. [70]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint, 2023

    work page 2023

  71. [71]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution

    P Wang, S Bai, S Tan, S Wang, Z Fan, J Bai, K Chen, X Liu, J Wang, W Ge, and Y Fan. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191, Sep 2024

    Pith/arXiv arXiv 2024

  72. [72]

    Huang, Nan Xu, Sheng Zhang, Hoifung Poon, and Muhao Chen

    Fei Wang, Wenxuan Zhou, James Y . Huang, Nan Xu, Sheng Zhang, Hoifung Poon, and Muhao Chen. mDPO: Conditional Preference Optimization for Multimodal Large Language Models, June 2024

    work page 2024

  73. [73]

    Freebind: Free lunch in unified multimodal space via knowledge fusion, 2024

    Zehan Wang, Ziang Zhang, Xize Cheng, Rongjie Huang, Luping Liu, Zhenhui Ye, Haifeng Huang, Yang Zhao, Tao Jin, Peng Gao, and Zhou Zhao. Freebind: Free lunch in unified multimodal space via knowledge fusion, 2024. 14

    work page 2024

  74. [74]

    Mitigating hallucinations in large vision-language models with instruction contrastive decoding, 2024

    Xintong Wang, Jingheng Pan, Liang Ding, and Chris Biemann. Mitigating hallucinations in large vision-language models with instruction contrastive decoding, 2024

    work page 2024

  75. [75]

    Don’t miss the forest for the trees: Attentional vision calibration for large vision language models, 2024

    Sangmin Woo, Donguk Kim, Jaehyuk Jang, Yubin Choi, and Changick Kim. Don’t miss the forest for the trees: Attentional vision calibration for large vision language models, 2024

    work page 2024

  76. [76]

    OmniBind: Large-scale Omni Multimodal Representation via Binding Spaces, July 2024

    Zehan Wang, Ziang Zhang, Hang Zhang, Luping Liu, Rongjie Huang, Xize Cheng, Heng- shuang Zhao, and Zhou Zhao. OmniBind: Large-scale Omni Multimodal Representation via Binding Spaces, July 2024

    work page 2024

  77. [77]

    Noiseboost: Alleviating hallucination with noise perturbation for multimodal large language models, May 2024

    Kai Wu, Boyuan Jiang, Zhengkai Jiang, Qingdong He, Donghao Luo, Shengzhi Wang, Qing- wen Liu, and Chengjie Wang. Noiseboost: Alleviating hallucination with noise perturbation for multimodal large language models, May 2024. http://arxiv.org/abs/2405.20081

    Pith/arXiv arXiv 2024

  78. [78]

    Logical closed loop: Uncovering object hallucinations in large vision-language models, June

    Junfei Wu, Qiang Liu, Ding Wang, Jinghao Zhang, Shu Wu, Liang Wang, and Tieniu Tan. Logical closed loop: Uncovering object hallucinations in large vision-language models, June

    work page

  79. [79]

    http://arxiv.org/abs/2402.11622

    Pith/arXiv arXiv

  80. [80]

    Lanp: Rethinking the impact of language priors in large vision-language models

    Zongyu Wu, Yuwei Niu, Hongcheng Gao, Minhua Lin, Zhiwei Zhang, Zhifang Zhang, Qi Shi, Yilong Wang, Sike Fu, Junjie Xu, et al. Lanp: Rethinking the impact of language priors in large vision-language models. arXiv preprint arXiv:2502.12359, 2025

    Pith/arXiv arXiv 2025

Showing first 80 references.