Pith. sign in

REVIEW 4 major objections 8 minor 3 cited by

Cracking the Code of Hallucination in LVLMs with Vision-aware Head Divergence

T0 review · 4 major / 8 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that hallucinated tokens in LVLM outputs correspond to low Vision-aware Head Divergence scores, and that amplifying the most visually-sensitive attention heads during decoding reduces hallucination more effectively than…

desk verdict A solid training-free decoding fix for LVLM object hallucination, with a clean metric and real gains; the main caveat is tuning transparency, not the mechanism. read the letter →

arxiv 2412.13949 v3 pith:GZLV7HWD submitted 2024-12-18 cs.CL cs.CV

Jinghan He , Kuan Zhu , Haiyun Guo , Junfeng Fang , Zhenglin Hua , Yuheng Jia , Ming Tang , Tat-Seng Chua
show 1 more author
Jinqiao Wang
This is my paper · ORCID
classification cs.CLcs.CV
keywords hallucinationlargevision-languagemodelsattentionheadsvision-awareheaddivergencelanguagebiasdecoding-timeinterventionobjecttraining-freemitigation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that hallucinations in large vision-language models can be traced to a small set of attention heads that carry visual information, and that scaling up those heads during decoding reduces hallucination without retraining. It introduces Vision-aware Head Divergence (VHD), the Euclidean distance between an attention head's output with and without image input, as a way to identify which heads are visually sensitive. Aggregating the top VHD scores per token produces Token-VHD (T-VHD), which the paper finds is lower for hallucinated words and sentences than for grounded ones. Based on this, the Vision-aware Head Reinforcement (VHR) method selects the most visually sensitive heads per sample and amplifies their outputs by a fixed factor, improving hallucination benchmarks over existing training-free decoding methods. A sympathetic reading is that this establishes a causal, internal-mechanism route to reducing hallucination, not just an output-level patch.

What carries the argument

The load-bearing object is the Vision-aware Head Divergence (VHD) score, defined as the Euclidean distance between an attention head's output when the image is included and when it is removed, holding the text prefix and generation history fixed. Token-VHD (T-VHD) aggregates the top-k VHD scores across layers for one prediction step, serving as a token-level proxy for visual grounding. Vision-aware Head Reinforcement (VHR) uses these scores to pick, per layer, the half of heads with the highest VHD and multiplies their outputs by α=2, after zeroing out heads whose high VHD reflects an activation surge when the image is removed. The theoretical support is a proposition that amplifying one head's output increases the cosine similarity between the FFN input and that head's component, so the scaling operation reorients the layer output toward the vision-aware direction.

What would settle it

A direct test would be to compute VHD on a model whose image encoder has been deliberately blinded or whose vision tokens are shuffled: if hallucination rates stay unchanged while VHD scores remain high, the metric is not tracking visual content. Conversely, a stronger test is to take tokens labeled hallucinated in a held-out benchmark and check whether their T-VHD scores are consistently below the median of correct tokens; showing even one model or benchmark where hallucinated tokens have high T-VHD would break the claimed correspondence.

Watch

Extended reading notes

Core claim

The paper's central claim is that a minority of attention heads in LVLMs are genuinely vision-aware, while most heads rely on language priors, and that hallucinated tokens are exactly those for which the language-prior heads dominate. The evidence is the T-VHD distribution: on a 500-image CHAIR sample, hallucinated object words and sentences containing them have significantly lower T-VHD scores than correct ones. The paper further claims that intervening on this mechanism—selecting the top half of heads by VHD per layer, zeroing out outliers that respond negatively to image removal, and multiplying their outputs by a factor α=2—reorients the attention output toward the vision-aware component. Across InstructBLIP-7b, LLaVA-1.5-7b, and LLaVA-NeXT-7b, this reduces CHAIR and improves POPE F1 compared with greedy, beam, DoLa, VCD, OPERA, CODE, and EAH. The stated implication is that language bias, not insufficient visual encoding, is a primary driver of hallucination, and that a per-sample, training-free head-scaling intervention is sufficient to counteract it.

Load-bearing premise

The whole metric stands on the assumption that removing the image from the input while keeping the same text prefix is a clean counterfactual that isolates what each attention head learns from the image; if the image-free run produces a different internal state or a different decoding trajectory, the VHD values measure artifacts of the comparison rather than visual awareness.

Editorial extensions

If this is right

  • If the claim holds, hallucination can be reduced by a per-sample rescaling of a small set of attention heads, with no training, no external knowledge, and negligible decoding overhead.
  • The T-VHD score could serve as a token-level diagnostic for when a model is about to generate ungrounded content, enabling early stopping or contrastive adjustment at the step where grounding drops.
  • The finding suggests that vision-language alignment is concentrated in a few heads across layers, which may guide future fine-tuning or pruning toward those heads rather than treating all parameters uniformly.
  • Because VHR is applied layer-by-layer, its reinforcement changes the internal state of deeper layers; the paper's ablations indicate that both the second layer and the last layers contribute, possibly through distinct mechanisms.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper leaves implicit whether VHR's improvement compounds with contrastive decoding methods, since VHR modifies head states while contrastive methods modify logits, and the two could operate on different failure modes.
  • The counterfactual in the VHD definition compares image-present with image-absent decoding, which may conflate the effect of the image on the current step with the effect of the image on the KV cache of earlier tokens; an alternative text-only-prefix baseline could isolate the per-step contribution.
  • If the vision-aware heads identified by VHD generalize across datasets but not across image distributions, the per-sample adaptation could be replaced by a cheap classifier predicting which heads to amplify from the image representation alone.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 8 minor

Summary. This paper studies object hallucination in large vision-language models (LVLMs) at the level of individual attention heads. It introduces Vision-aware Head Divergence (VHD), the Euclidean distance between a head's output with and without the image input at a given decoding step, and Token-VHD (T-VHD), an aggregation of top-k VHD scores across layers. The authors report that only a small number of heads show large VHD, that hallucinated words and sentences tend to have lower T-VHD than grounded ones, and that this reflects reliance on language priors. They then propose Vision-aware Head Reinforcement (VHR), a training-free decoding intervention that selects the most vision-sensitive heads at the first decoding step and scales their outputs by a factor alpha, with outlier filtering, applied layer by layer. The method is evaluated on InstructBLIP-7b, LLaVA-1.5-7b, and LLaVA-NeXT-7b using CHAIR, POPE, and LLaVA-Bench (In-the-Wild), with ablations and a theoretical proposition (Proposition 1) claiming that scaling a head's output reorients the post-attention representation toward that head's direction.

Significance. If its claims hold, this paper provides a concrete, per-sample, interpretable handle on hallucination: a diagnostic metric (T-VHD) that flags tokens with weak visual grounding, and a training-free intervention (VHR) that improves grounding by amplifying a small set of vision-sensitive heads. The empirical design is a genuine strength: results are reported on three LVLM families, CHAIR numbers are averaged over five random splits with standard deviations, ablations cover adaptive selection, outlier removal, reinforced layers, and the scale factor, the inference-time overhead is analyzed, and a code link is provided. The central hypothesis is falsifiable and the proposed correlation, if confirmed with proper statistics, would be a useful probing tool. The main gaps are that the VHD counterfactual is not position-controlled, that the key hyperparameters are selected on the same CHAIR benchmark used for the headline results, and that the proof of Proposition 1 is not rigorous as written; all three are addressable within the scope of a revision.

major comments (4)
  1. [§3.1, Eq. (4); Algorithm 1] The counterfactual in Eq. (4) removes the image tokens x_V entirely, but in the three evaluated LVLMs the image tokens occupy the first positions of the input sequence, so the text-only forward pass places the prompt tokens, y_<t, and the BOS token at different absolute positions and removes the image-token keys/values that text tokens attend to; VHD therefore measures sensitivity to visual content jointly with sensitivity to sequence position and length. Because RoPE-based text-to-text attention is shift-invariant, the positional effect is weaker than a purely absolute-position analysis would suggest, but it is not zero, as it affects the BOS boundary and any text-to-image attention, and the two runs also differ in total sequence length. This confound is load-bearing: Algorithm 1 (step 3) selects the heads to reinforce from VHD, and Eq. (5) with Figure 3 builds the hallucination analysis on the same quantity; in addition, y_<t is generated under image-conditioned decoding, so the text-only pass is queried with an out-of-distribution prefix. Please add a position-controlled counterfactual that keeps the same number of tokens in both passes, for example by replacing the vision tokens with zeroed or neutral embeddings, and report whether the selected head sets and the T-VHD distributions in Figure 3 are preserved; also state how the prefix-distribution issue is handled.
  2. [§4.4, §4.6; Tables 1, 5, 6] The free parameters of VHR—the scale factor alpha, the reinforced layer set L_r, the top-k in Eq. (5) (whose value is never reported), and the outlier thresholds in Eq. (6)—are chosen by ablating on the CHAIR benchmark (Figure 5 and Tables 5 and 6), and the same CHAIR benchmark is then used for the headline results of Table 1; with no held-out validation split, the reported CHAIR improvements are inflated by selection on the test set. This is the main threat to the empirical claim that VHR outperforms the baselines. Please either fix the hyperparameters on a validation split of the COCO/CHAIR data before computing Table 1, or report the full sensitivity grid and demonstrate that the ordering against baselines is stable across it; the POPE and LLaVA-Bench results, which were not used for tuning, should be foregrounded as the out-of-sample evidence.
  3. [§3.3, Appendix B, Eq. (13)] The proof of Proposition 1 in Appendix B is not valid as written: the step bounding the difference from below replaces the negative term -<x, y_hat>/(||x|| ||y_hat||) with '+1', but that term is at least -1, not at least +1, so the displayed lower bound (||y_hat||-||x||)/||x+y_hat|| + 1 does not follow; moreover, the statement is false in degenerate cases such as x = -mu*y with 0 < alpha-1 < mu/||y||, where the cosine similarity remains at -1 (and x + y_hat can be the zero vector). The intended geometric claim is nevertheless plausible: for x not parallel to y, the function lambda -> cos(x+lambda*y, y) is non-decreasing in lambda >= 0 by Cauchy-Schwarz, with strict increase unless x and y are parallel. Please restate the proposition with the required non-degeneracy conditions and replace the faulty inequality with a correct argument.
  4. [§3.2, Algorithm 1] Algorithm 1 and the surrounding text leave unclear whether the text-only pass (the one without the image) is also reinforced layer by layer. The sentence 'the previous layers are already reinforced when calculating the VHD scores for a given layer' suggests both passes are modified, but Eq. (4) defines VHD on the original unmodified model, so for layers after the first, the quantity computed at inference would not be the VHD defined in Eq. (4); if instead the text-only pass is left untouched, the two passes are asymmetric and the selection at layer l mixes visual sensitivity with the effect of prior-layer reinforcement. Please specify exactly which forward passes are modified, and provide a small experiment comparing the layer-by-layer head selections with the selections from the clean two-pass computation defined by Eq. (4).
minor comments (8)
  1. [Table 2] The POPE results are reported as F1 means over the three splits without standard deviations or significance tests, and several margins over the strongest baseline are very small (e.g., 85.47 vs. 85.45 for LLaVA-1.5); the claim in §4.5 that VHR outperforms all other decoding methods across all LVLMs needs variance or paired-test support.
  2. [Table 3] The LLaVA-Bench results come from a single GPT-4o evaluation without per-item scores or variance, and the prompt template in Table 7 fixes the order of Assistant 1 and Assistant 2; please describe the order randomization and report item-level statistics.
  3. [Figure 3] The T-VHD comparison in Figure 3 is described as providing statistical evidence, but no test statistic, sample size, or effect size is reported; please add a two-sample test (e.g., Mann-Whitney U) for both the sentence-level and word-level comparisons.
  4. [Eq. (5)] The value of k in the top-k operator of Eq. (5) is never stated in the implementation details; please report it.
  5. [Table 6] The alpha=4 row (CHAIRS 3.64, CHAIRI 2.01, length 144.54) is dismissed as an anomaly, but no explanation is given; please clarify what the model generates in that regime (e.g., repetitive or degenerate text) so that it is clear why the metric improvement is not meaningful.
  6. [Appendix C, Table 5] The text alternates between 'the second layer' and 'layer1' for the same setting; please make the layer indexing consistent.
  7. [Throughout] The model names appear both as 'LLaVA-1.5' and 'LLaV A-1.5' (with an internal space) in different places; please unify the notation.
  8. [Figure 2, §3.1] The claim that only a few heads are vision-aware is illustrated with a single generation step in Figure 2; reporting the distribution of the number of high-VHD heads across samples and steps would make the claim quantitative.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: VHD is an independent internal-state measurement, VHR is tested on external benchmarks, and the geometric proposition is proved from the MHA decomposition.

full rationale

The paper's central derivation chain is not circular. VHD (Eq. 4) is defined directly as the Euclidean distance between the output of an attention head with and without image input; hallucination labels do not enter the definition. T-VHD (Eq. 5) is an aggregation of top-k VHD values, again label-free. VHR selects heads by VHD and scales their outputs by a fixed factor; Proposition 1 is proved algebraically from the decomposition of the MHA output and the RMSNorm expression, and it does not presuppose the empirical claim that hallucinated tokens have lower T-VHD. The CHAIR, POPE, and LLaVA-Bench evaluations are external benchmarks. Although some VHR hyperparameters (e.g., alpha, number of reinforced layers) are ablated on CHAIR, they are not fitted to reproduce the T-VHD-hallucination correlation, and the CHAIR improvements are reported as benchmark results rather than as a prediction derived from the fitted values. The only same-author citation (He et al., 2024) appears in a list of prior attention-head identification methods and is not load-bearing. The positional-shift confound in the image-removal counterfactual, noted by the skeptical reader, is a validity or robustness concern about what VHD measures, not a circularity in which a result is equivalent to its input by construction.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim rests on three domain assumptions: that image removal is a valid counterfactual for measuring visual sensitivity, that Euclidean head-output distance captures visual awareness, and that amplifying high-VHD heads improves visual grounding. No new entities are introduced; the hyperparameters alpha, reinforced layers, and outlier thresholds are set by ablations.

free parameters (4)
  • alpha (head scaling factor) = 2
    Chosen via ablation on CHAIR (Table 6) as a balance between hallucination reduction and output stability; the paper notes alpha=3 gives lower CHAIR but alpha=4 causes anomalies.
  • Reinforced layers L_r = Second layer plus last 14 layers for LLaVA-1.5 and LLaVA-NeXT; last 18 layers for InstructBLIP
    Selected through ablations on CHAIR (Table 5, Figure 5, Appendix C); the paper notes that reinforcing the second layer does not universally improve performance.
  • Top-k value for T-VHD aggregation = not specified
    Equation 5 uses topki(VHD_{l,i}, k) but k is never stated; T-VHD is used for analysis rather than as the deployed head-selection rule.
  • Outlier thresholds (mean + standard deviation) = Per-layer mean plus one standard deviation for VHD and for head norm without image
    Equation 6 zeroes VHD values exceeding these thresholds; a fixed statistical choice adopted by the authors without a separate tuning split.
assumptions (4)
  • domain assumption Euclidean distance between head outputs is a meaningful divergence measure for visual sensitivity.
    VHD is defined as Euclidean distance in Equation 4; the paper does not justify that this distance isolates visual information from other distribution shifts.
  • domain assumption Image removal is a valid counterfactual for isolating visual context.
    Equation 4 compares runs with and without xV while conditioning on the same y<t, but y<t was generated with the image, so the text-only continuation is off-distribution.
  • domain assumption Scaling head outputs reorients Z_l toward the reinforced head, and this direction is the one that improves grounding.
    Proposition 1 is a geometric statement about cosine similarity; it does not establish that Z_{l,h} is the correct direction for reducing hallucination.
  • standard math Standard linear algebra and the RMSNorm structure of the model.
    Appendix B partitions W^O and uses cosine properties; these are standard mathematical facts.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Cracking the Code of Hallucination in LVLMs with Vision-aware Head Divergence." pith.science (2026). https://pith.science/paper/GZLV7HWD

@misc{pith2026241213949,
  author       = {Pith},
  title        = {Pith review of: Cracking the Code of Hallucination in LVLMs with Vision-aware Head Divergence},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GZLV7HWD}},
  note         = {Machine review of arXiv:2412.13949}
}
read the original abstract

Large vision-language models (LVLMs) have made substantial progress in integrating large language models (LLMs) with visual inputs, enabling advanced multimodal reasoning. Despite their success, a persistent challenge is hallucination-where generated text fails to accurately reflect visual content-undermining both accuracy and reliability. Existing methods focus on alignment training or decoding refinements but primarily address symptoms at the generation stage without probing the underlying causes. In this work, we investigate the internal mechanisms driving hallucination in LVLMs, with an emphasis on the multi-head attention module. Specifically, we introduce Vision-aware Head Divergence (VHD), a metric that quantifies the sensitivity of attention head outputs to visual context. Based on this, our findings reveal the presence of vision-aware attention heads that are more attuned to visual information; however, the model's overreliance on its prior language patterns is closely related to hallucinations. Building on these insights, we propose Vision-aware Head Reinforcement (VHR), a training-free approach to mitigate hallucination by enhancing the role of vision-aware attention heads. Extensive experiments demonstrate that our method achieves superior performance compared to state-of-the-art approaches in mitigating hallucinations, while maintaining high efficiency with negligible additional time overhead.

Figures

Figures reproduced from arXiv: 2412.13949 by the authors.

Figure 1
Figure 1. (a) An example indicating the connection be [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 3
Figure 3. Relationship between T-VHD scores and hal [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figure 4
Figure 4. The illustration of the proposed VHD metric and the VHR approach to mitigate hallucinations in LVLM. [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗
Figures from the paper (6 more)
Figure 5
Figure 5. Figure 5: Results of VHR applied across different num [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Comparison of inference time for different [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: An example of VHR eliminating the halluci [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: An example indicating the connection be [PITH_FULL_IMAGE:figures/full_fig_p011_8.png]
Figure 9
Figure 9. Figure 9: More examples of VHR eliminating the hallucinated content. [PITH_FULL_IMAGE:figures/full_fig_p013_9.png]
Figure 10
Figure 10. Figure 10: More examples of the VHD scores during different generation steps of different samples. [PITH_FULL_IMAGE:figures/full_fig_p013_10.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Reinforced Lifelong Editing for Language Models

    cs.CL 2025-02 conditional novelty 7.0 of 10

    RLEdit trains a hypernetwork to edit LLM parameters over long knowledge sequences by maximizing a trajectory-level reward, and reports strong accuracy and large speedups versus existing editing methods.

  2. Role-Break in Attention Heads: Understanding and Detecting Hallucinations in VLMs

    cs.CV 2026-07 conditional novelty 6.0 of 10

    Hallucinated tokens in vision-language models are detectable from each attention head's deviation from its own faithful source-allocation role, reaching ~93% AUROC with a linear probe.

  3. Capturing Gaze Shifts for Guidance: Cross-Modal Fusion Enhancement for VLM Hallucination Mitigation

    cs.CV 2025-10 conditional novelty 6.0 of 10

    Tracking positive shifts in visual attention over information-rich query words yields a saliency map that, when used to boost visual and query attention during decoding, reduces object hallucination on CHAIR, POPE, an...

Reference graph

Works this paper leans on

33 extracted references · 4 canonical work pages · cited by 3 Pith papers

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang. 2025. An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models. In European Conference on Computer Vision, pages 19--35. Springer

    work page 2025

  4. [4]

    Yung-Sung Chuang, Yujia Xie, Hongyin Luo, Yoon Kim, James Glass, and Pengcheng He. 2023. Dola: Decoding by contrasting layers improves factuality in large language models. arXiv preprint arXiv:2309.03883

    arXiv 2023

  5. [5]

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. 2023. https://arxiv.org/abs/2305.06500 Instructblip: Towards general-purpose vision-language models with instruction tuning . Preprint, arXiv:2305.06500

    arXiv 2023

  6. [6]

    Junfeng Fang, Zac Bi, Ruipeng Wang, Houcheng Jiang, Yuan Gao, Kun Wang, An Zhang, Jie Shi, Xiang Wang, and Tat-Seng Chua. 2024 a . Towards neuron attributions in multi-modal large language models. Advances in Neural Information Processing Systems, 37:122867--122890

    work page 2024

  7. [7]

    Junfeng Fang, Houcheng Jiang, Kun Wang, Yunshan Ma, Shi Jie, Xiang Wang, Xiangnan He, and Tat-Seng Chua. 2024 b . Alphaedit: Null-space constrained knowledge editing for language models. arXiv preprint arXiv:2410.02355

    arXiv 2024

  8. [8]

    Sreyan Ghosh, Chandra Kiran Reddy Evuru, Sonal Kumar, Utkarsh Tyagi, Oriol Nieto, Zeyu Jin, and Dinesh Manocha. 2024. Visual description grounding reduces hallucinations and boosts reasoning in lvlms. arXiv preprint arXiv:2405.15683

    arXiv 2024

Show all 33 references
  1. [9]

    Xuan Gong, Tianshi Ming, Xinpeng Wang, and Zhihua Wei. 2024. Damro: Dive into the attention mechanism of lvlm to reduce object hallucination. arXiv preprint arXiv:2410.04514

    2024

  2. [10]

    Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2017. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6904--6913

    2017

  3. [11]

    Jinghan He, Haiyun Guo, Kuan Zhu, Zihan Zhao, Ming Tang, and Jinqiao Wang. 2024. Seekr: Selective attention-guided knowledge retention for continual learning of large language models. arXiv preprint arXiv:2411.06171

    2024 arXiv

  4. [12]

    Qidong Huang, Xiaoyi Dong, Pan Zhang, Bin Wang, Conghui He, Jiaqi Wang, Dahua Lin, Weiming Zhang, and Nenghai Yu. 2024. Opera: Alleviating hallucination in multi-modal large language models via over-trust penalty and retrospection-allocation. In Proceedings of the IEEE/CVF Con...

    2024

  5. [13]

    Junho Kim, Hyunjun Kim, Yeonju Kim, and Yong Man Ro. 2024. Code: Contrasting self-generated description to combat hallucination in large multi-modal models. arXiv preprint arXiv:2406.01920

    2024 arXiv

  6. [14]

    Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shijian Lu, Chunyan Miao, and Lidong Bing. 2024. Mitigating object hallucinations in large vision-language models through visual contrastive decoding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Reco...

    2024

  7. [15]

    Kenneth Li, Oam Patel, Fernanda Vi \'e gas, Hanspeter Pfister, and Martin Wattenberg. 2024. Inference-time intervention: Eliciting truthful answers from a language model. Advances in Neural Information Processing Systems, 36

    2024

  8. [16]

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. 2023. Evaluating object hallucination in large vision-language models. arXiv preprint arXiv:2305.10355

    2023 arXiv

  9. [17]

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll \'a r, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In Computer Vision--ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, ...

    2014

  10. [18]

    Hanchao Liu, Wenyuan Xue, Yifei Chen, Dapeng Chen, Xiutian Zhao, Ke Wang, Liping Hou, Rongjun Li, and Wei Peng. 2024 a . A survey on hallucination in large vision-language models. arXiv preprint arXiv:2402.00253

    2024 arXiv

  11. [19]

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2024 b . Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296--26306

    2024

  12. [20]

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. 2024 c . https://llava-vl.github.io/blog/2024-01-30-llava-next/ Llava-next: Improved reasoning, ocr, and world knowledge

    2024

  13. [21]

    Yulei Niu, Kaihua Tang, Hanwang Zhang, Zhiwu Lu, Xian-Sheng Hua, and Ji-Rong Wen. 2021. Counterfactual vqa: A cause-effect look at language bias. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12700--12710

    2021

  14. [22]

    Letitia Parcalabescu and Anette Frank. 2024. Do vision & language decoders use images and text equally? how self-consistent are their explanations? arXiv preprint arXiv:2404.18624

    2024 arXiv

  15. [23]

    Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. 2018. Object hallucination in image captioning. arXiv preprint arXiv:1809.02156

    2018 arXiv

  16. [24]

    John Tabak. 2014. Geometry: the language of space and form. Infobase Publishing

    2014

  17. [25]

    Shukang Yin, Chaoyou Fu, Sirui Zhao, Tong Xu, Hao Wang, Dianbo Sui, Yunhang Shen, Ke Li, Xing Sun, and Enhong Chen. 2023. Woodpecker: Hallucination correction for multimodal large language models. arXiv preprint arXiv:2310.16045

    2023 arXiv

  18. [26]

    Junwei You, Haotian Shi, Zhuoyu Jiang, Zilin Huang, Rui Gan, Keshu Wu, Xi Cheng, Xiaopeng Li, and Bin Ran. 2024. V2x-vlm: End-to-end v2x cooperative autonomous driving through large vision-language models. arXiv preprint arXiv:2408.09251

    2024 arXiv

  19. [27]

    Qinan Yu, Jack Merullo, and Ellie Pavlick. 2023. Characterizing mechanisms for factual recall in language models. arXiv preprint arXiv:2310.15910

    2023 arXiv

  20. [28]

    Tianyu Yu, Yuan Yao, Haoye Zhang, Taiwen He, Yifeng Han, Ganqu Cui, Jinyi Hu, Zhiyuan Liu, Hai-Tao Zheng, Maosong Sun, et al. 2024. Rlhf-v: Towards trustworthy mllms via behavior alignment from fine-grained correctional human feedback. In Proceedings of the IEEE/CVF Conference...

    2024

  21. [29]

    Xiaofeng Zhang, Yihao Quan, Chaochen Gu, Chen Shen, Xiaosong Yuan, Shaotian Yan, Hao Cheng, Kaijie Wu, and Jieping Ye. 2024. Seeing clearly by layer two: Enhancing attention heads to alleviate hallucination in lvlms. arXiv preprint arXiv:2411.09968

    2024 arXiv

  22. [30]

    Zhiyuan Zhao, Bin Wang, Linke Ouyang, Xiaoyi Dong, Jiaqi Wang, and Conghui He. 2023. Beyond hallucinations: Enhancing lvlms through hallucination-aware direct preference optimization. arXiv preprint arXiv:2311.16839

    2023 arXiv

  23. [31]

    Yiyang Zhou, Chenhang Cui, Jaehong Yoon, Linjun Zhang, Zhun Deng, Chelsea Finn, Mohit Bansal, and Huaxiu Yao. 2023. Analyzing and mitigating object hallucination in large vision-language models. arXiv preprint arXiv:2310.00754

    2023 arXiv

  24. [32]

    Zhenhong Zhou, Haiyang Yu, Xinghua Zhang, Rongwu Xu, Fei Huang, Kun Wang, Yang Liu, Junfeng Fang, and Yongbin Li. 2024. On the role of attention heads in large language model safety. arXiv preprint arXiv:2410.13708

    2024 arXiv

  25. [33]

    Lanyun Zhu, Deyi Ji, Tianrun Chen, Peng Xu, Jieping Ye, and Jun Liu. 2024. Ibd: Alleviating hallucinations in large vision-language models via image-biased decoding. arXiv preprint arXiv:2402.18476

    2024 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.