Pith. sign in

REVIEW 3 major objections 6 minor 84 references

Two inference-time indicators—visual attention entropy and output-image correlation—diagnose distinct failure modes, and targeted contrastive corrections triggered by them reduce hallucinations without retraining.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-07-31 23:26 UTC pith:R4HZHIS3

load-bearing objection DICA's diagnostic split is promising and the POPE/CHAIR evidence is real, but the VAE-GDC mechanism is internally inconsistent and the MME headline is tuned in-sample. the 3 major comments →

arxiv 2607.23944 v1 pith:R4HZHIS3 submitted 2026-07-27 cs.AI

DICA: Dual-Indicator Guided Contrastive Alignment in Multimodal Large Language Models

Hao Yang , Jin Wang , Xuejie Zhang This is my paper
classification cs.AI
keywords hallucination mitigationvisual attention entropyoutput-image correlationcontrastive decodingmultimodal large language modelsattention driftvisual groundinginference-time intervention
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Multimodal models hallucinate for two distinct reasons, the paper argues: their visual attention suddenly scatters to irrelevant regions (attention drift), or their output stops depending on the image and falls back on language priors (underuse of visual evidence). DICA tracks a different indicator for each—the entropy of attention over image tokens and the divergence between image-conditioned and text-only output distributions—and triggers a specific contrastive correction only when that indicator crosses a threshold. The claim is that this targeted, indicator-driven intervention outperforms methods that treat all hallucinations with one uniform fix, across object-presence, attribute, and captioning benchmarks, without retraining or architecture changes. If right, DICA gives a training-free wrapper that tells you which of two failure modes is happening and fixes that one.

Core claim

Multimodal models hallucinate, the paper argues, because of two separable failures, not one uniform problem. A sudden rise in Visual Attention Entropy (VAE), the Shannon entropy of attention over image tokens, marks attention drift: the model's focus spreads to irrelevant regions. A drop in Output Image Correlation (OIC), the KL divergence between the image-conditioned output distribution and the text-only prior, marks underuse of visual evidence: the model answers from language priors. The paper claims that watching both indicators during decoding and activating a contrastive correction only when one crosses its threshold restores grounding: VAE-GDC subtracts a descriptive-path hidden state

What carries the argument

The key machinery is the pairing of two information-theoretic indicators with two contrastive-decoding corrections. Visual Attention Entropy (VAE) is the Shannon entropy of the attention distribution restricted to image tokens; high VAE means the model's gaze is diffuse or drifting. Output Image Correlation (OIC) is the KL divergence between the output distribution conditioned on image plus text and the distribution conditioned on text alone; low OIC means the answer is being generated from language priors. The VAE-Guided Drift Correction (VAE-GDC) blends normal image-token hidden states against descriptive-path hidden states with a weight set by Jensen–Shannon divergence; the OIC-Guided Gro

Load-bearing premise

The drift correction assumes that subtracting one diffuse, high-entropy hidden state from another sharpens attention, even though the formula reduces to the original hidden state when the two compared states are nearly identical.

What would settle it

For cases where α_d is near 1, compute the corrected hidden state from Eq. 12 and measure its Visual Attention Entropy against the uncorrected state; if VAE does not fall (or log-probability of the correct answer does not recover) while H_img ≈ H_img_desc, then VAE-GDC is not performing the correction attributed to it.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Attention drift and underuse of visual evidence can be separately detected at inference time, so hallucinations can be attributed to a mechanism rather than treated as one uniform failure.
  • A training-free decoder wrapper can improve accuracy and F1 on object-presence and attribute benchmarks by intervening only when an indicator crosses a threshold.
  • Removing either correction degrades performance, with VAE-GDC the more frequently triggered and more foundational of the two.
  • Because no weights change, the method transfers to newer or different multimodal backbones and still yields gains, as the paper's supplementary experiments claim.
  • The corrected trajectories show VAE falling and OIC rising, consistent with restoration of visual grounding rather than a simple output-lengthening bias.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The paper leaves implicit that OIC alone could serve as a cheap hallucination detector: if a drop in image-dependence precedes erroneous tokens, outputs could be flagged without any correction.
  • The same indicator pair could gate other interventions—re-running the visual encoder, restarting from an earlier layer, or asking the model to re-read the image—rather than only the two contrastive corrections tested.
  • The paper's own limitations section concedes that fixed thresholds need manual adjustment per model and that perceptual blind spots cannot be fixed; an adaptive threshold variant is the natural next test.
  • If the two failure modes are genuinely separable, error analysis should show two distinguishable clusters of hallucination cases; that taxonomy is directly testable on existing hallucination data.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes DICA, a training-free inference-time wrapper for multimodal large language models. It introduces two indicators: Visual Attention Entropy (VAE), the Shannon entropy of attention over image tokens, and Output Image Correlation (OIC), the KL divergence between image-conditioned and text-only output distributions. A spike in VAE triggers VAE-Guided Drift Correction (Eq. 12), which linearly combines normal-path and descriptive-path image-token hidden states; a drop in OIC triggers OIC-Guided Grounding Restoration (Eq. 19), which contrasts final-layer and intermediate-layer logits. The method is evaluated on POPE, CHAIR, and MME with LLaVA-1.5, Qwen-VL, and Qwen3, reporting consistent gains over VCD, OPERA, M3ID, AvisC, and MoD. The paper also includes ablations, threshold sensitivity, trigger statistics, case studies, and an efficiency comparison.

Significance. If the central mechanism were internally consistent, this would be a useful contribution: a diagnosis-driven alternative to uniform contrastive decoding, with separate corrections for attention drift and underuse of visual evidence. The paper has clear strengths: the code is released, the two indicators are cleanly defined, the evaluation covers multiple backbones and benchmarks, both components are ablated, and an additional Qwen3 generalization check is provided. However, the central VAE-GDC update in Eq. (12) is not derived in the appendices and contradicts the paper's own description in D.3.3, and the MME headline is in-sample because the thresholds are tuned on MME. These issues are load-bearing: they underdetermine the claim that indicator-driven contrastive alignment causes the reported improvements.

major comments (3)
  1. [§D.3.3, fixed-α experiment] The load-bearing claim is that when α_d≈1, VAE-GDC 'triggers strong contrastive decoding' because the Normal and Descriptive paths become unusually similar during attention drift. But Eq. (12) gives H_new = H_img + α_d(H_img − H_img_desc). If the two paths are similar (H_img ≈ H_img_desc), the correction term vanishes and H_new ≈ H_img, i.e., VAE-GDC is a no-op in exactly the regime asserted to be most corrective. Subtracting two similar high-entropy hidden-state vectors does not by itself sharpen attention or recover a grounded prediction. Since the Table 1 ablation attributes a sizable drop to removing VAE-GDC, the implemented update must deviate from Eq. (12), or the gains come from OIC-GGR / the extra descriptive forward pass. Please provide the exact implemented update and either prove that Eq. (12) sharpens attention when α_d≈1 or revise the mechanism and its description. Appendice
  2. [§D.3.3, fixed-α experiment] The fixed-α experiment is also inconsistent with Eq. (12). The text states that as α increases toward 1, 'the influence of the Descriptive path grows' and CHAIR hallucination rates rise, concluding that small α is best. In Eq. (12), however, increasing α from 0 to 1 does not move the output toward the Descriptive path; it amplifies the Normal path (coefficient 1+α) and subtracts the Descriptive path (coefficient −α). The reported trend suggests that the authors may have evaluated a convex combination such as H_new = (1−α)H_img + αH_img_desc, or some other formula. Please clarify which update was actually tested; otherwise the fixed-α analysis cannot validate the adaptive α_d in Eqs. (11)–(12).
  3. [§4.1 and §D.3.2; Table 3] The thresholds α_VAE=0.8 and α_OIC=0.5 are selected by grid search on MME (D.3.2), and Table 3 then reports DICA's MME score as a main result. This makes the MME benchmark in-sample for two free hyperparameters; the headline 'consistently outperforms' on MME is partly a fitted value. Please either tune on a development set disjoint from the evaluation benchmarks, report MME under a proper validation/test protocol, or explicitly relegate MME to a sensitivity analysis rather than a headline benchmark. POPE and CHAIR are less affected because the thresholds were not tuned on them, but the manuscript should state this.
minor comments (6)
  1. [§D.1] Equation numbering reuses (12)–(14) from the main text for the CHAIR metrics; renumber the appendix equations.
  2. [Eq. (11)] JSD is defined for probability distributions, but Z and Z_desc are described as logits. Specify that a softmax is applied before computing JSD, or define the distributions involved.
  3. [Eq. (19)] H^L_img and H^{l*}_img are called 'vocabulary logits', yet H_new is later treated as a hidden state. Clarify whether the correction is applied to logits or hidden states and how it is fed back into the next layer.
  4. [Table 2] On Qwen-VL, recall drops from 31.0 (sampling) to 27.0 (DICA) while output length increases. The statement that DICA 'preserves reasonable recall' needs quantitative support or a caveat.
  5. [Figure 4] The figure contains duplicated and misspelled labels ('OIC > OIC >', 'Chosse best suitable layer'); please clean up the diagram.
  6. [§2.3] Minor language issues: 'the answer become unstable and decline' should be 'the answer becomes unstable and declines'; also standardize spacing in VAE/OIC throughout.

Circularity Check

2 steps flagged

Partial circularity: MME thresholds are tuned on MME then the same MME score is reported, and the VAE-GDC correction is definitionally a no-op in the regime D.3.3 says it acts; POPE/CHAIR provide independent grounding.

specific steps
  1. fitted input called prediction [Appendix D.3.2 and Table 3 (Section 4.2)]
    "The sensitivity of DICA to the drift and grounding thresholds αV AEand αOIC is evaluated through a grid search on the MME benchmark... With αV AE= 0.8 and αOIC = 0.5, interventions are activated only when substantial attention drift or insufficient visual grounding occurs... resulting in the best performance of 661.53."

    The same MME total (661.5 in Table 3) that DICA reports as its headline MME result is obtained after a grid search on MME for the thresholds. The threshold choice is therefore fitted to the target benchmark, and the MME score is a selected maximum rather than an independent prediction. MME cannot be used as evidence of 'consistently outperforms' without a separate validation split; the external support must come from POPE/CHAIR, which use fixed thresholds.

  2. self definitional [Eq. 11–12 and D.3.3]
    "αd = clip(1−JSD(Z∥Zdesc),0,1)... H new = (1+αd)H img −αd H img desc... Values in [0.9, 1.0) arise during attention drift: both paths yield diffuse, high-entropy distributions due to lost visual focus, making their outputs unusually similar. The near unity αd triggers strong contrastive decoding to recover a grounded prediction."

    By Eq. 12, H_new = H_img + α_d(H_img − H_img_desc). In the high-α_d regime described in D.3.3, the Normal and Descriptive paths are 'unusually similar,' so H_img−H_img_desc≈0 and the contrastive term vanishes: H_new≈H_img. The claimed 'strong contrastive decoding' is a no-op by construction exactly when α_d≈1. The paper defines the correction strength through similarity, then asserts the opposite behavior; the mechanism reduces to its own input (the uncorrected H_img) in its trigger regime.

full rationale

DICA's two indicators are defined non-circularly from model internals (Shannon entropy over visual attention; KL divergence of image-conditioned vs text-only output distributions), and the empirical trigger/correction story is tested against external benchmarks. The POPE, CHAIR and Qwen3 results with fixed thresholds are independent evidence, so the method is not wholly circular. However, two genuine reductions appear. First, the thresholds α_VAE=0.8 and α_OIC=0.5 are selected by grid search on MME and the same optimized MME total is then reported as a headline result, making that benchmark partly a fitted value. Second, the VAE-GDC correction (Eq. 12) is defined so that its contrastive term scales as α_d(H_img−H_img_desc); D.3.3 claims this term is strongest when the two distributions are nearly identical, but in that case the term is ≈0 by construction. Thus the central causal claim for VAE-GDC (a VAE spike triggers a strong corrective contrast) is contradicted by the paper's own equation in the very regime it invokes. The Fig. 5 demonstration that DICA lowers VAE and raises OIC is a goal-directed check, not an independent test, but Table 4 (1.3% R→W, 37.4% W→R) does provide independent behavioral validation. No load-bearing self-citation chain is present: the authors' prior works appear only in related-work/hallucination context and are not used to justify the DICA mechanism. Overall: partial circularity, bounded by external POPE/CHAIR grounding — score 4.

Axiom & Free-Parameter Ledger

2 free parameters · 4 axioms · 0 invented entities

The method rests on two fitted thresholds (tuned on MME), a causal link between attention entropy / output-image KL and hallucination, and an ad hoc assumption about the Descriptive path's contrastive usefulness. No new physical entities are proposed.

free parameters (2)
  • alpha_VAE threshold = 0.8
    Grid-searched on MME (D.3.2) to maximize MME total score; the same MME result is then reported in Table 3, making MME partly in-sample.
  • alpha_OIC threshold = 0.5
    Grid-searched on MME together with alpha_VAE (D.3.2); selected for best MME total score.
axioms (4)
  • domain assumption Increasing visual attention entropy indicates attention drift and predicts hallucination; decreasing OIC indicates underuse of visual evidence and predicts hallucination.
    This causal link is asserted in Section 3 and used to trigger corrections; the appendices only prove entropy/KL inequalities, not the link to hallucination.
  • ad hoc to paper The Descriptive path's hidden states provide a useful contrast for removing attention-drift contamination (Eq. 12).
    Introduced ad hoc for VAE-GDC; D.3.3's explanation that similar diffuse outputs yield strong correction is inconsistent with Eq. 12.
  • domain assumption The coarse-to-fine human attention pattern is the normative target for MLLM inference.
    Motivates the method; cited from cognitive science (Navon 1977), but not established that MLLM internal attention should mimic humans.
  • standard math Standard properties of Shannon entropy, KL divergence, and JSD.
    Used throughout formulas.

pith-pipeline@v1.3.0-alltime-deepseek · 31201 in / 14259 out tokens · 128892 ms · 2026-07-31T23:26:34.676662+00:00 · methodology

0 comments
read the original abstract

Human visual reasoning typically follows a coarse-to-fine attention process, starting from global scene understanding and gradually focusing on question-relevant regions. However, multimodal large language models may deviate from this pattern due to attention drift and the underutilization of visual evidence, which can lead to hallucinations. To mitigate these issues, this study proposes a Dual-Indicator Guided Contrastive Alignment (DICA), which tracks two information-theoretic indicators during inference: Visual Attention Entropy (VAE), which reflects the concentration of visual attention, and Output Image Correlation (OIC), which measures the dependence of generated outputs on the visual input. An abnormal increase in VAE or a decrease in OIC corresponds to different failure modes, which trigger targeted contrastive alignment to restore visual grounding. Experimental results across multiple benchmarks demonstrate that DICA consistently outperforms existing approaches and substantially reduces hallucinations, highlighting the effectiveness of indicator-driven intervention in improving multimodal inference reliability. The code is publicly available at https://github.com/BGWH123/DICA/.

Figures

Figures reproduced from arXiv: 2607.23944 by Hao Yang, Jin Wang, Xuejie Zhang.

Figure 1
Figure 1. Figure 1: An example of attention drift and underuse of [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Top: Evolution of VAE across layer phases. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Evolution of answer log-probabilities and attention maps across layers. [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: The overall architecture of DICA. captures richer fine-grained details (Li et al., 2023a; Peng et al., 2023; Liu et al., 2023b) and maintains a more uniform attention distribution across regions. Based on this property, we perform contrastive de￾coding directly on hidden states to mitigate the effect of high-entropy contamination. Specifically, let Himg and H img desc denote the image-token hidden states i… view at source ↗
Figure 6
Figure 6. Figure 6: Trigger frequency and depth distribution of [PITH_FULL_IMAGE:figures/full_fig_p008_6.png] view at source ↗
Figure 5
Figure 5. Figure 5: The impact of DICA on VAE and OIC within [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗
Figure 7
Figure 7. Figure 7: Top: VAE changes across layers phase on DICA. Down: OIC change across layers phase on DICA. achieves the lowest hallucination rates among all methods, demonstrating its effectiveness in ground￾ing model outputs in visual evidence. Notably, DICA maintains strong performance in both the 512-token and 64-token settings, indicating its ro￾bustness regardless of whether standard or com￾pressed visual representa… view at source ↗
Figure 8
Figure 8. Figure 8: The upper figure corresponds to thresholds [PITH_FULL_IMAGE:figures/full_fig_p015_8.png] view at source ↗
Figure 10
Figure 10. Figure 10: Performance with fixed α on CHAIR. task-specific Normal path, and the low αd prevents unnecessary correction, preserving fluency. Con￾versely, values in [0.9, 1.0) arise during attention drift: both paths yield diffuse, high-entropy distri￾butions due to lost visual focus, making their out￾puts unusually similar. The near unity αd triggers strong contrastive decoding to recover a grounded prediction. The … view at source ↗
Figure 11
Figure 11. Figure 11: The change of βd on POPE. 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 635 640 645 650 655 660 665 Value d =661.53 [PITH_FULL_IMAGE:figures/full_fig_p018_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Performance with fixed β on MME. mark. As shown in [PITH_FULL_IMAGE:figures/full_fig_p018_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Evolution of answer log-probabilities and attention maps across layers applying DICA. [PITH_FULL_IMAGE:figures/full_fig_p019_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Case study of generative on CHAIR. We compare responses generated by sampling, VCD, M3ID, AvisC, [PITH_FULL_IMAGE:figures/full_fig_p021_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: Case study of generative on CHAIR. We compare responses generated by sampling, VCD, M3ID, AvisC, [PITH_FULL_IMAGE:figures/full_fig_p022_15.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

84 extracted references · 1 canonical work pages

  1. [1]

    1977 , issn =

    Forest before trees: The precedence of global features in visual perception , journal =. 1977 , issn =. doi:https://doi.org/10.1016/0010-0285(77)90012-3 , url =

    work page doi:10.1016/0010-0285(77)90012-3 1977

  2. [2]

    Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , year =

    Evaluating Object Hallucination in Large Vision-Language Models , author =. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , year =. doi:10.18653/v1/2023.emnlp-main.20 , url =

    doi:10.18653/v1/2023.emnlp-main.20 2023

  3. [3]

    Microsoft

    Lin, Tsung-Yi and Maire, Michael and Belongie, Serge and Hays, James and Perona, Pietro and Ramanan, Deva and Doll. Microsoft. Computer Vision -- ECCV 2014 , year =

    2014

  4. [4]

    European conference on computer vision , pages=

    A-okvqa: A benchmark for visual question answering using world knowledge , author=. European conference on computer vision , pages=. 2022 , organization=

    2022

  5. [5]

    Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

    Gqa: A new dataset for real-world visual reasoning and compositional question answering , author=. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

  6. [6]

    arXiv preprint arXiv:2306.13394 , volume=

    Mme: A comprehensive evaluation benchmark for multimodal large language models , author=. arXiv preprint arXiv:2306.13394 , volume=

    Pith/arXiv arXiv

  7. [7]

    Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

    Improved baselines with visual instruction tuning , author=. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

  8. [8]

    arXiv preprint arXiv:2308.12966 , year=

    Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, and Reasoning , author=. arXiv preprint arXiv:2308.12966 , year=

    Pith/arXiv arXiv

  9. [9]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

    Mitigating object hallucinations in large vision-language models through visual contrastive decoding , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

  10. [10]

    Computer Vision and Image Understanding , volume=

    Human attention in visual question answering: Do humans and deep networks look at the same regions? , author=. Computer Vision and Image Understanding , volume=. 2017 , publisher=

    2017

  11. [11]

    arXiv preprint arXiv:2502.01969 , year=

    Mitigating object hallucinations in large vision-language models via attention calibration , author=. arXiv preprint arXiv:2502.01969 , year=

    arXiv

  12. [12]

    arXiv preprint arXiv:2505.21472 , year=

    Mitigating Hallucination in Large Vision-Language Models via Adaptive Attention Calibration , author=. arXiv preprint arXiv:2505.21472 , year=

    Pith/arXiv arXiv

  13. [13]

    arXiv preprint arXiv:2306.14565 , year=

    Mitigating hallucination in large multi-modal models via robust instruction tuning , author=. arXiv preprint arXiv:2306.14565 , year=

    Pith/arXiv arXiv

  14. [14]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

    Rlhf-v: Towards trustworthy mllms via behavior alignment from fine-grained correctional human feedback , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

  15. [15]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

    Vip-llava: Making large multimodal models understand arbitrary visual prompts , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

  16. [16]

    arXiv preprint arXiv:2501.02699 , year=

    EAGLE: Enhanced Visual Grounding Minimizes Hallucinations in Instructional Multimodal Models , author=. arXiv preprint arXiv:2501.02699 , year=

    Pith/arXiv arXiv

  17. [17]

    Looking Beyond Text: Reducing Language Bias in Large Vision-Language Models via Multimodal Dual-Attention and Soft-Image Guidance

    Zhao, Haozhe and Si, Shuzheng and Chen, Liang and Zhang, Yichi and Sun, Maosong and Chang, Baobao and Zhang, Minjia. Looking Beyond Text: Reducing Language Bias in Large Vision-Language Models via Multimodal Dual-Attention and Soft-Image Guidance. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. 2025

    2025

  18. [18]

    arXiv preprint arXiv:2510.09741 , year=

    Constructive distortion: Improving mllms with attention-guided image warping , author=. arXiv preprint arXiv:2510.09741 , year=

    Pith/arXiv arXiv

  19. [19]

    2025 , eprint=

    v1: Learning to Point Visual Tokens for Multimodal Grounded Reasoning , author=. 2025 , eprint=

    2025

  20. [20]

    2025 , eprint=

    CausalVLBench: Benchmarking Visual Causal Reasoning in Large Vision-Language Models , author=. 2025 , eprint=

    2025

  21. [21]

    On learning to summarize with large language models as references (2023) , author=

    2023

  22. [22]

    Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , pages=

    Lifelong knowledge editing for llms with retrieval-augmented continuous prompt learning , author=. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , pages=

    2024

  23. [23]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

    Hallucination augmented contrastive learning for multimodal large language model , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

  24. [24]

    arXiv preprint arXiv:2402.17811 , year=

    Truthx: Alleviating hallucinations by editing large language models in truthful space , author=. arXiv preprint arXiv:2402.17811 , year=

    Pith/arXiv arXiv

  25. [25]

    Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    Parameter-aware contrastive knowledge editing: Tracing and rectifying based on critical transmission paths , author=. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

  26. [26]

    Advances in Neural Information Processing Systems , volume=

    Towards unified multimodal editing with enhanced knowledge collaboration , author=. Advances in Neural Information Processing Systems , volume=

  27. [27]

    arXiv preprint arXiv:2405.11613 , year=

    Decoding by Contrasting Knowledge: Enhancing LLMs' Confidence on Edited Facts , author=. arXiv preprint arXiv:2405.11613 , year=

    Pith/arXiv arXiv

  28. [28]

    Advances in neural information processing systems , volume=

    Visual instruction tuning , author=. Advances in neural information processing systems , volume=

  29. [29]

    Shannon, C. E. , journal=. A mathematical theory of communication , year=

  30. [30]

    International conference on machine learning , pages=

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models , author=. International conference on machine learning , pages=. 2023 , organization=

    2023

  31. [31]

    arXiv preprint arXiv:2306.14824 , year=

    Kosmos-2: Grounding multimodal large language models to the world , author=. arXiv preprint arXiv:2306.14824 , year=

    Pith/arXiv arXiv

  32. [32]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

    Multi-modal hallucination control by visual information grounding , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

  33. [33]

    Findings of the Association for Computational Linguistics: ACL 2025 , pages=

    Don't miss the forest for the trees: Attentional vision calibration for large vision language models , author=. Findings of the Association for Computational Linguistics: ACL 2025 , pages=

    2025

  34. [34]

    arXiv preprint arXiv:2505.17061 , year=

    Mixture of Decoding: An Attention-Inspired Adaptive Decoding Strategy to Mitigate Hallucinations in Large Vision-Language Models , author=. arXiv preprint arXiv:2505.17061 , year=

    Pith/arXiv arXiv

  35. [35]

    URL https://arxiv

    Interpreting context look-ups in transformers: Investigating attention-mlp interactions, 2024 , author=. URL https://arxiv. org/abs/2402.15055 , year=

    Pith/arXiv arXiv 2024

  36. [36]

    Object Hallucination in Image Captioning , author =. Proc. EMNLP , year =. doi:10.18653/v1/D18-1437 , url =

    doi:10.18653/v1/d18-1437

  37. [37]

    , journal=

    Lin, J. , journal=. Divergence measures based on the Shannon entropy , year=

  38. [38]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

    Opera: Alleviating hallucination in multi-modal large language models via over-trust penalty and retrospection-allocation , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

  39. [39]

    arXiv preprint arXiv:2302.13971 , year=

    Llama: Open and efficient foundation language models , author=. arXiv preprint arXiv:2302.13971 , year=

    Pith/arXiv arXiv

  40. [40]

    Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing , pages=

    Attention-guided self-reflection for zero-shot hallucination detection in large language models , author=. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing , pages=

    2025

  41. [41]

    Bert: Pre-training of deep bidirectional transformers for language understanding , author=. Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers) , pages=

    2019

  42. [42]

    Advances in neural information processing systems , volume=

    Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks , author=. Advances in neural information processing systems , volume=

  43. [43]

    arXiv preprint arXiv:1908.07490 , year=

    Lxmert: Learning cross-modality encoder representations from transformers , author=. arXiv preprint arXiv:1908.07490 , year=

    Pith/arXiv arXiv 1908

  44. [44]

    European conference on computer vision , pages=

    Uniter: Universal image-text representation learning , author=. European conference on computer vision , pages=. 2020 , organization=

    2020

  45. [45]

    arXiv preprint arXiv:2304.10592 , year=

    Minigpt-4: Enhancing vision-language understanding with advanced large language models , author=. arXiv preprint arXiv:2304.10592 , year=

    Pith/arXiv arXiv

  46. [46]

    Advances in neural information processing systems , volume=

    Instructblip: Towards general-purpose vision-language models with instruction tuning , author=. Advances in neural information processing systems , volume=

  47. [47]

    Findings of the Association for Computational Linguistics: ACL 2024 , pages=

    Aligning large multimodal models with factually augmented rlhf , author=. Findings of the Association for Computational Linguistics: ACL 2024 , pages=

    2024

  48. [48]

    arXiv preprint arXiv:2310.00754 , year=

    Analyzing and mitigating object hallucination in large vision-language models , author=. arXiv preprint arXiv:2310.00754 , year=

    Pith/arXiv arXiv

  49. [49]

    ACM computing surveys , volume=

    Survey of hallucination in natural language generation , author=. ACM computing surveys , volume=. 2023 , publisher=

    2023

  50. [50]

    arXiv preprint arXiv:2310.01469 , year=

    Llm lies: Hallucinations are not bugs, but features as adversarial examples , author=. arXiv preprint arXiv:2310.01469 , year=

    Pith/arXiv arXiv

  51. [51]

    Computational Linguistics , pages=

    Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models , author=. Computational Linguistics , pages=. 2025 , publisher=

    2025

  52. [52]

    arXiv preprint arXiv:2404.00971 , year=

    Exploring and evaluating hallucinations in llm-powered code generation , author=. arXiv preprint arXiv:2404.00971 , year=

    arXiv

  53. [53]

    arXiv preprint arXiv:2401.11817 , year=

    Hallucination is inevitable: An innate limitation of large language models , author=. arXiv preprint arXiv:2401.11817 , year=

    Pith/arXiv arXiv

  54. [54]

    Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages=

    Let there be a clock on the beach: Reducing object hallucination in image captioning , author=. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages=

  55. [55]

    arXiv preprint arXiv:2305.10355 , year=

    Evaluating object hallucination in large vision-language models , author=. arXiv preprint arXiv:2305.10355 , year=

    Pith/arXiv arXiv

  56. [56]

    International Conference on Multimedia Modeling , pages=

    Mitigating fine-grained hallucination by fine-tuning large vision-language models with caption rewrites , author=. International Conference on Multimedia Modeling , pages=. 2024 , organization=

    2024

  57. [57]

    Science China Information Sciences , volume=

    Woodpecker: Hallucination correction for multimodal large language models , author=. Science China Information Sciences , volume=. 2024 , publisher=

    2024

  58. [58]

    Proceedings of the 32nd ACM International Conference on Multimedia , pages=

    Combating visual question answering hallucinations via robust multi-space co-debias learning , author=. Proceedings of the 32nd ACM International Conference on Multimedia , pages=

  59. [59]

    Proceedings of the AAAI Conference on Artificial Intelligence , volume=

    Detecting and preventing hallucinations in large vision language models , author=. Proceedings of the AAAI Conference on Artificial Intelligence , volume=

  60. [60]

    arXiv preprint arXiv:2402.19103 , year=

    Whispers that shake foundations: Analyzing and mitigating false premise hallucinations in large language models , author=. arXiv preprint arXiv:2402.19103 , year=

    Pith/arXiv arXiv

  61. [61]

    Advances in Neural Information Processing Systems , volume=

    Inference-time intervention: Eliciting truthful answers from a language model , author=. Advances in Neural Information Processing Systems , volume=

  62. [62]

    arXiv preprint arXiv:2111.08276 , year=

    Multi-grained vision language pre-training: Aligning texts with visual concepts , author=. arXiv preprint arXiv:2111.08276 , year=

    Pith/arXiv arXiv

  63. [63]

    Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

    Exposing and mitigating spurious correlations for cross-modal retrieval , author=. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

  64. [64]

    arXiv preprint arXiv:2307.03987 , year=

    A stitch in time saves nine: Detecting and mitigating hallucinations of llms by validating low-confidence generation , author=. arXiv preprint arXiv:2307.03987 , year=

    Pith/arXiv arXiv

  65. [65]

    Findings of the association for computational linguistics: ACL 2024 , pages=

    Chain-of-verification reduces hallucination in large language models , author=. Findings of the association for computational linguistics: ACL 2024 , pages=

    2024

  66. [66]

    arXiv preprint arXiv:2402.11622 , year=

    Logical closed loop: Uncovering object hallucinations in large vision-language models , author=. arXiv preprint arXiv:2402.11622 , year=

    Pith/arXiv arXiv

  67. [67]

    Proceedings of the 61st annual meeting of the association for computational linguistics (volume 1: Long papers) , pages=

    Contrastive decoding: Open-ended text generation as optimization , author=. Proceedings of the 61st annual meeting of the association for computational linguistics (volume 1: Long papers) , pages=

  68. [68]

    arXiv preprint arXiv:2309.03883 , year=

    Dola: Decoding by contrasting layers improves factuality in large language models , author=. arXiv preprint arXiv:2309.03883 , year=

    Pith/arXiv arXiv

  69. [69]

    arXiv preprint arXiv:2311.00233 , year=

    Instructive decoding: Instruction-tuned large language models are self-refiner from noisy instructions , author=. arXiv preprint arXiv:2311.00233 , year=

    Pith/arXiv arXiv

  70. [70]

    Proceedings of the Computer Vision and Pattern Recognition Conference , pages=

    Ibd: Alleviating hallucinations in large vision-language models via image-biased decoding , author=. Proceedings of the Computer Vision and Pattern Recognition Conference , pages=

  71. [71]

    arXiv preprint arXiv:2502.06130 , year=

    Self-correcting decoding with generative feedback for mitigating hallucinations in large vision-language models , author=. arXiv preprint arXiv:2502.06130 , year=

    Pith/arXiv arXiv

  72. [72]

    arXiv preprint arXiv:2408.02032 , year=

    Self-introspective decoding: Alleviating hallucinations for large vision-language models , author=. arXiv preprint arXiv:2408.02032 , year=

    Pith/arXiv arXiv

  73. [73]

    2013 , publisher=

    Eye movements and vision , author=. 2013 , publisher=

    2013

  74. [74]

    Nature reviews neuroscience , volume=

    Computational modelling of visual attention , author=. Nature reviews neuroscience , volume=. 2001 , publisher=

    2001

  75. [75]

    The Annals of Mathematical Statistics , volume=

    On information and sufficiency , author=. The Annals of Mathematical Statistics , volume=. 1951 , publisher=

    1951

  76. [76]

    Mitigating Hallucinations in Multi-modal Large Language Models via Image Token Attention-Guided Decoding , author=. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) , pages=

    2025

  77. [77]

    CCF International Conference on Natural Language Processing and Chinese Computing , pages=

    EVL-MCoT: Enhanced Vision-Language Multi-CoT for Harmful Meme Detection , author=. CCF International Conference on Natural Language Processing and Chinese Computing , pages=. 2025 , organization=

    2025

  78. [78]

    2025 IEEE International Conference on Multimedia and Expo (ICME) , pages=

    Enhanced Multimodal Chain-of-Thought with Visual Self-Contrastive Distillation , author=. 2025 IEEE International Conference on Multimedia and Expo (ICME) , pages=. 2025 , organization=

    2025

  79. [79]

    Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024) , pages=

    Enhancing semantics in multimodal chain of thought via soft negative sampling , author=. Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024) , pages=

    2024

  80. [80]

    Proceedings of the AAAI Conference on Artificial Intelligence , volume=

    Vision-aware multimodal prompt tuning for uploadable multi-source few-shot domain adaptation , author=. Proceedings of the AAAI Conference on Artificial Intelligence , volume=

Showing first 80 references.