Pith. sign in

REVIEW 5 major objections 5 minor 16 cited by

Seeing Clearly by Layer Two: Enhancing Attention Heads to Alleviate Hallucination in LVLMs

T0 review · 5 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that hallucinations in vision-language models track the density of attention sinks over image tokens, and that copying the densest early-layer attention head across its layer reduces hallucination without retraining.

desk verdict A simple training-free attention-broadcast trick that cuts CHAIR scores across LVLMs; the mechanism claim needs a random-head control and the mask equation looks like a zero-matrix typo. read the letter →

arxiv 2411.09968 v1 pith:TQVV4L6W submitted 2024-11-15 cs.CV cs.AI

Xiaofeng Zhang , Yihao Quan , Chaochen Gu , Chen Shen , Xiaosong Yuan , Shaotian Yan , Hao Cheng , Kaijie Wu
show 1 more author
Jieping Ye
This is my paper · ORCID
classification cs.CVcs.AI
keywords hallucinationmultimodallargelanguagemodelsattentionsinkvisionheadstraining-freeinferenceobjectvision-language
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that hallucinations in multimodal large language models are tied to how attention is distributed across image tokens in the early transformer layers. When early-layer attention heads spread their scores densely over many image tokens, models are less likely to invent objects; when attention collapses into a few sparse columns, hallucination becomes more likely. The proposed intervention, EAH, finds the head with the densest vision sink in a shallow layer and replaces every other head in that layer with that head's attention map. If right, this turns an interpretability observation into a training-free fix that lowers object-hallucination rates across several model families and even improves some general vision-language benchmarks.

What carries the argument

The key machinery is the vision-sink measure combined with a broadcast operation. A vision sink is defined as a column in the image-token range whose average attention score exceeds $\beta$; the paper counts such columns per head in a shallow layer and classifies heads with many such columns as dense vision sink heads. EAH takes the head with the maximum count and overwrites every other head in that layer with its attention map, forcing the layer to follow one high-density attention pattern. This operation carries the whole argument because the claimed effect depends on whether homogenizing early-layer image attention is safe and beneficial.

What would settle it

A direct test would be to broadcast a randomly chosen head, or the head with the fewest vision sinks, instead of the densest one; if hallucination drops just as much under those alternatives, density is not what matters. A complementary test would be to apply EAH to images deliberately chosen to contain many small, spatially spread objects and check whether recall of those objects falls, which would show the homogenization sacrifices coverage rather than simply sharpening vision.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is a consistent relationship between the density of vision sinks in shallow-layer attention heads and hallucination, plus an intervention that exploits it. A vision sink is a column within the image-token range of an attention map whose average score exceeds a threshold $\beta$; a dense vision sink head is one in which many such columns appear. The paper reports that hallucinating outputs activate fewer dense vision sink heads in layers 1 and 2, while non-hallucinating outputs activate more. EAH then selects the head with the highest vision-sink count and broadcasts its attention map to all heads in that layer, using $A[i][j] = A[i][n]$ for every head $j$. The authors claim this dense, homogenized early-layer attention carries image information forward and reduces hallucination on POPE and CHAIR, with additional gains on several general benchmarks and on LLM factuality tasks.

Load-bearing premise

The load-bearing premise is that overwriting every attention head in the first two layers with one selected head's attention map preserves enough information for downstream layers, so that the forced consensus pattern helps more than the destroyed multi-head diversity hurts.

Editorial extensions

If this is right

  • Applying EAH to layer 2 with $\beta=0.002$ and the top-1 head improves CHAIR$_S$ and CHAIR$_I$ on LLaVA-1.5-7B relative to greedy search while keeping caption length comparable (36.4/9.9 versus 47.0/13.8).
  • The method transfers across model families including LLaVA, Shikra, MiniGPT-4, InstructBLIP, Qwen-VL, Mini-Gemini, and InternVL, with larger gains reported for models using MLP or Linear projectors.
  • EAH also produces consistent accuracy gains on GSM8K and TruthfulQA for LLaMA-3.1, Ministral-8B, Qwen-2-7B, and Qwen-2.5-7B, suggesting the underlying LLM pattern is shared.
  • Ablation results show that broadcasting to all 32 heads works best and that applying EAH in deeper layers (3, 4, 16, 32) degrades performance, supporting the claim that only shallow layers should be modified.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the mechanism is right, the density signal could be computed per token during generation to select which head to broadcast dynamically, rather than fixing the layer and head in advance as the paper does.
  • The aggressive overwrite implies that multi-head diversity in early layers is largely redundant for vision grounding; a testable extension is whether those heads could be pruned entirely instead of merely copied over.
  • The reported gains on pure LLM factuality tasks suggest the attention-sink pattern may be inherited from the base language model rather than learned by the vision projector, a distinction the paper leaves open.
  • A stress test with images containing many small, distributed objects would show whether the homogenized attention pattern trades hallucination reduction for missed visual details.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes a training-free, plug-and-play method, Enhancing Attention Heads (EAH), to reduce object hallucinations in large vision-language models (LVLMs). The authors analyze attention maps and observe that shallow layers contain dense 'vision sinks' (columns with high average attention over image tokens) and that hallucination tokens tend to be associated with fewer such heads. EAH identifies, within a chosen shallow layer, the head with the densest vision sink and broadcasts that head's attention map to all other heads in that layer (Algorithm 1, Eq. 12). They report CHAIR and POPE improvements on LLaVA-1.5-7B, plus additional experiments on other LVLMs and LLMs.

Significance. The paper addresses an important problem, hallucination in LVLMs, and proposes a simple, training-free intervention that requires no parameter updates, which is attractive for practical deployment. The observation that attention patterns in shallow layers correlate with hallucination is interesting and potentially useful for future interpretability work. A strength is the breadth of models tested (seven LVLMs and four LLMs). However, the current evidence does not yet establish the paper's causal mechanism: the missing control experiments, the incorrect mask definition, and the lack of statistical rigor all limit the significance of the reported gains. If the control experiments confirm that dense vision sinks—rather than mere attention homogenization—are responsible, the method would be a meaningful contribution; as written, the support is incomplete.

major comments (5)
  1. [Section 3.2, Eq. (5)-(6), Eq. (8)] The mask matrix is defined as M = eye(r,c) - diag(1). For a square attention map (r = c), this equals the zero matrix. With M = 0, the left-hand side of Eq. (6) and Eq. (8) is identically 0, so no column can ever exceed any positive threshold β, contradicting the existence of the vision sinks that the paper reports. This is a central definitional error that prevents replication. Please correct the mask definition (likely M = ones(r,c) - eye(r,c)) and verify all subsequent equations.
  2. [Section 3.3, Algorithm 1, Eq. (12)] The central causal claim—that broadcasting the head with the densest vision sink reduces hallucination—is not tested. All ablations vary layer, threshold, or the number of broadcast heads, but none compare EAH to an equally aggressive control that broadcasts a randomly chosen head, the head with the fewest vision sinks, or the average of all heads. Without such controls, the observed CHAIR improvements could be due to the homogenization of early-layer attention (a diversity-reducing operation) rather than the specific dense-sink property. Please add these controls and report their CHAIR and POPE numbers.
  3. [Section 4.4.3, Table 6] The hyperparameters β, layer index, and top-N are selected directly on the CHAIR evaluation benchmark, and the same CHAIR numbers are then reported as the main result in Table 1. This selection procedure risks overfitting to the evaluation set. The paper provides no held-out validation, cross-validation, or confidence intervals. Please report results on a separate validation split or provide statistical significance tests (e.g., repeated runs with different seeds) to show that the improvements are not artifacts of tuning.
  4. [Section 4.4.2, Table 5] Applying EAH to pure text LLMs (LLaMA-3.1, Ministral, Qwen-2, Qwen-2.5) is conceptually unclear: the vision sink definition depends on an image token range [s, e] and an image token count (e.g., 576), but these quantities are undefined for text-only inputs. The paper does not explain how the method is adapted to LLMs, what token range is used, or what the broadcast operation means without image tokens. Please either clarify the adaptation or remove this section, since it currently undermines the paper's focus and reproducibility.
  5. [Section 4.2, Table 1 and Appendix A.1] EAH reduces CHAIR CS from 47.0 to 36.4 but also lowers recall from 76.6 to 73.9, and POPE F1 remains unchanged at 85.7. The appendix acknowledges a roughly 4-point recall drop. This pattern is consistent with a diversity-reducing regularizer that makes the model more conservative in its captions, rather than with a mechanism that specifically improves visual grounding. The paper should analyze whether the gains are driven by reduced output diversity (e.g., by measuring caption length, lexical diversity, and recall on a per-category basis) and should report the standard deviations across the evaluation set.
minor comments (5)
  1. [Equations (1), (6), (8)] The symbol "Pr" in the equations appears to be a summation (Σ), not a probability. Please replace it with \sum for clarity.
  2. [Section 4.4.1] The text refers to "Table 8" for the generalization study on other LVLMs, but the corresponding table is numbered Table 4 in the paper. Please fix the cross-reference.
  3. [Table 7] The row for Layer=32, Threshold=0.008 lists a threshold value that does not appear in the other rows for layer 32; likely a typo for 0.002. Please correct.
  4. [Appendix A.3.1] The phrase "layer 1.2" likely means "layers 1–2" or "layers 1 and 2". Please clarify.
  5. [General] The word "significant" is used to describe performance improvements without any statistical significance testing. Please either add significance tests or replace the word with "substantial" or similar.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the vision-sink ranking is geometric and externally validated; the CHAIR-tuned hyperparameters and missing broadcast control are correctness risks, not reduction-by-construction.

full rationale

This paper's central chain is observational, not derivational. Vision sinks are defined geometrically in Eqs. 6-7 as image-range attention columns whose mean score exceeds a threshold beta; dense-vision-sink heads are then correlated with hallucination-free tokens (Sec. 3.1, Figs. 3-4) using external CHAIR/POPE labels, and EAH (Eqs. 11-12) broadcasts the attention map of the head with the maximum sink count. No step reduces by construction: the vision-sink definition contains no reference to hallucination labels or to CHAIR/POPE scores, and the outcome of the broadcast is an empirical measurement that could have failed. The closest concern is hyperparameter selection: layer=2, beta=0.002, and N=1 are chosen on the CHAIR benchmark itself (Sec. 4.4.3), and the same CHAIR numbers are then reported as the headline result (Table 1), which inflates but does not force the gain - deeper layers are worse than the baseline (Table 7), so the improvement is not entailed by the search. The causal claim that dense vision sinks, rather than attention homogenization, drive the gains is under-supported because no control broadcasts a random, sparse-sink, or averaged head; that is a missing-control confound for the mechanism, not a circular reduction. Self-citations are minor and not load-bearing: Ref. [45] (overlapping authors) supplies the Grad-CAM visualization tool and Ref. [36] (a co-author) is related-work comparison. Independent evidence is substantial: the early-layer observation is credited to external FastV work, and fixed-hyperparameter gains on other LVLMs (Table 4), LLMs (Table 5), and general benchmarks (Tables 2-3) are genuine held-out tests. The paper also discloses its own 4-point recall drop (Appendix A.1), consistent with a diversity-reducing intervention rather than a forced identity. Verdict: no significant circularity (score 2), with the CHAIR tuning and missing-control issues flagged as correctness risks.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

No new particles, forces, or entities are introduced; the dense vision sink head is a classification of existing attention heads. The main load-bearing inputs are the threshold beta, the chosen layer, the top-1 head count, and the fixed image token range, plus the assumption that early-layer attention maps can be safely homogenized.

free parameters (4)
  • beta (vision sink threshold) = 0.002
    Tuned on CHAIR in Tables 6 and A.3.2; controls whether a column counts as a vision sink.
  • layer L = 2
    Tuned on CHAIR; layer 2 with beta=0.002 gives the best CHAIRs and CHAIRi, while deeper layers degrade performance.
  • top-N broadcast heads = 1
    Ablation in Table 8 shows broadcasting the top-1 head to all 32 heads is best; other N values perform worse.
  • image token range [s,e] = 36 to 611 on LLaVA-1.5
    Fixed to LLaVA-1.5's 576 image tokens; the paper does not state how s and e are set for Shikra, MiniGPT-4, InternVL, Qwen-VL, or Mini-Gemini.
assumptions (3)
  • domain assumption Overwriting every attention head in layers 0 to 2 with one head's map is a safe intervention.
    EAH sets A[i][j] = A[i][n] for all heads in Eq. (12); the paper provides no analysis of information loss or of destroyed multi-head diversity.
  • domain assumption The observed correlation between dense vision sinks and fewer hallucinations is causal and intervention-relevant.
    Figures 3 and 4 are correlational; the design of EAH assumes that forcing denser sinks will reduce hallucinations rather than simply accompanying them.
  • ad hoc to paper Hyperparameters chosen on CHAIR transfer to other benchmarks and models.
    Tables 6 to 8 tune beta, layer, and head count on CHAIR; these exact values are then used to claim generalization in Tables 1 to 5.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Seeing Clearly by Layer Two: Enhancing Attention Heads to Alleviate Hallucination in LVLMs." pith.science (2026). https://pith.science/paper/TQVV4L6W

@misc{pith2026241109968,
  author       = {Pith},
  title        = {Pith review of: Seeing Clearly by Layer Two: Enhancing Attention Heads to Alleviate Hallucination in LVLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TQVV4L6W}},
  note         = {Machine review of arXiv:2411.09968}
}
read the original abstract

The hallucination problem in multimodal large language models (MLLMs) remains a common issue. Although image tokens occupy a majority of the input sequence of MLLMs, there is limited research to explore the relationship between image tokens and hallucinations. In this paper, we analyze the distribution of attention scores for image tokens across each layer and head of the model, revealing an intriguing and common phenomenon: most hallucinations are closely linked to the pattern of attention sinks in the self-attention matrix of image tokens, where shallow layers exhibit dense attention sinks and deeper layers show sparse attention sinks. We further analyze the attention heads of different layers and find that heads with high-density attention sink in the image part play a positive role in alleviating hallucinations. In this paper, we propose a training-free method named \textcolor{red}{\textbf{E}}nhancing \textcolor{red}{\textbf{A}}ttention \textcolor{red}{\textbf{H}}eads (EAH), an approach designed to enhance the convergence of image tokens attention sinks in the shallow layers. EAH identifies the attention head that shows the vision sink in a shallow layer and extracts its attention matrix. This attention map is then broadcast to other heads in the layer, thereby strengthening the layer to pay more attention to the image itself. With extensive experiments, EAH shows significant hallucination-mitigating performance on different MLLMs and metrics, proving its effectiveness and generality.

Figures

Figures reproduced from arXiv: 2411.09968 by the authors.

Figure 1
Figure 1. We found a common phenomenon through the attention [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Definition of dense vision sink head and its layer-wise [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Relationship between text tokens and the average pro [PITH_FULL_IMAGE:figures/full_fig_p002_3.png] view at source ↗
Figures from the paper (16 more)
Figure 4
Figure 4. Figure 4: (a) A example of distribution of dense vision head [PITH_FULL_IMAGE:figures/full_fig_p003_4.png]
Figure 5
Figure 5. Figure 5: Differences in analysis perspectives between Label [PITH_FULL_IMAGE:figures/full_fig_p004_5.png]
Figure 6
Figure 6. Figure 6: The structure of Enhancing Attention Head. [PITH_FULL_IMAGE:figures/full_fig_p005_6.png]
Figure 7
Figure 7. Figure 7: The Grad-CAM results of LLaVA1.5 and LLaVA1.5 af [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: Generation time of a single response. in the attention weights at layer 1.2 during inference. In comparison, the other methods inevitably introduce addi￾tional computational overhead. A.3.2. Qualitative Experiment of Thresholds and Layers [PITH_FULL_IMAGE:figures/full…
Figure 9
Figure 9. Figure 9: The generalization results on the datasets for the seven multimodal models (MMEs) demonstrate that EAH combined with [PITH_FULL_IMAGE:figures/full_fig_p012_9.png]
Figure 10
Figure 10. Figure 10: The flow map of EAH which aims to find the head with the densest attention sink in the 32 layers and broadcast the attention [PITH_FULL_IMAGE:figures/full_fig_p013_10.png]
Figure 11
Figure 11. Figure 11: The flow map of EAH which aims to find the head with the densest attention sink in the 32 layers and broadcast the attention [PITH_FULL_IMAGE:figures/full_fig_p014_11.png]
Figure 12
Figure 12. Figure 12: The flow map of EAH which aims to find the head with the densest attention sink in the 32 layers and broadcast the attention [PITH_FULL_IMAGE:figures/full_fig_p015_12.png]
Figure 13
Figure 13. Figure 13: The flow map of EAH which aims to find the head with the densest attention sink in the 32 layers and broadcast the attention [PITH_FULL_IMAGE:figures/full_fig_p016_13.png]
Figure 14
Figure 14. Figure 14: The flow map of EAH which aims to find the head with the densest attention sink in the 32 layers and broadcast the attention [PITH_FULL_IMAGE:figures/full_fig_p017_14.png]
Figure 15
Figure 15. Figure 15: The flow map of EAH which aims to find the head with the densest attention sink in the 32 layers and broadcast the attention [PITH_FULL_IMAGE:figures/full_fig_p018_15.png]
Figure 16
Figure 16. Figure 16: The flow map of EAH which aims to find the head with the densest attention sink in the 32 layers and broadcast the attention [PITH_FULL_IMAGE:figures/full_fig_p019_16.png]
Figure 17
Figure 17. Figure 17: Results of LLaVA1.5 with EAH, EAH can significantly reduce hallucinations while maintaining the original sentence length. [PITH_FULL_IMAGE:figures/full_fig_p019_17.png]
Figure 18
Figure 18. Figure 18: Results of Intern-VL with EAH, EAH can significantly reduce hallucinations while maintaining the original sentence length. [PITH_FULL_IMAGE:figures/full_fig_p020_18.png]
Figure 19
Figure 19. Figure 19: Results of Shikra with EAH, EAH can significantly reduce hallucinations while maintaining the original sentence length. [PITH_FULL_IMAGE:figures/full_fig_p020_19.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 16 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Visual Attention Never Fades: Selective Progressive Attention ReCalibration for Detailed Image Captioning in Multimodal Large Language Models

    cs.CV 2025-02 conditional novelty 7.0 of 10

    SPARC selectively and progressively reinforces attention to relevant image tokens during decoding, improving both precision and recall in detailed image captioning compared to baselines and prior hallucination-mitigat...

  2. Cracking the Code of Hallucination in LVLMs with Vision-aware Head Divergence

    cs.CL 2024-12 conditional novelty 7.0 of 10

    Hallucination in LVLMs correlates with low sensitivity of attention heads to image input, and scaling up the most image-sensitive heads reduces it without training.

  3. The Ebb and Flow of Multimodal Focus: Scheduling Visual Relay Windows for Grounded VLM Reasoning

    cs.AI 2026-07 conditional novelty 6.5 of 10

    Visual evidence in VLMs flows through a depth-wise Visual Relay Window; scheduling that window with the lightweight TRACE controller yields +4.33 points on grounding benchmarks and +3.05 on MathVista across four open-...

  4. Role-Break in Attention Heads: Understanding and Detecting Hallucinations in VLMs

    cs.CV 2026-07 conditional novelty 6.0 of 10

    Hallucinated tokens in vision-language models are detectable from each attention head's deviation from its own faithful source-allocation role, reaching ~93% AUROC with a linear probe.

  5. ST-Veto: Spatio-Temporal Token Veto for Diffusion MLLMs via Taylor Prediction and Visual Grounding

    cs.AI 2026-07 conditional novelty 6.0 of 10

    ST-Veto improves reasoning in diffusion MLLMs by vetoing temporally unstable tokens and tokens with weak image grounding, swapping in safer near-boundary candidates.

  6. METEOR: Multi-Encoder Collaborative Token Pruning for Efficient Vision Language Models

    cs.CV 2025-07 conditional novelty 6.0 of 10

    METEOR is a three-stage token pruning framework that reduces visual tokens in multi-encoder MLLMs by 76% with only a 0.3% average accuracy drop.

  7. Kwai Keye-VL Technical Report

    cs.CV 2025-07 conditional novelty 6.0 of 10

    Kwai Keye-VL shows that a five-mode chain-of-thought cold-start plus mix-mode reinforcement learning can push an 8B multimodal model to strong short-video and general vision-language performance.

  8. DenseWorld-1M: Towards Detailed Dense Grounded Caption in the Real World

    cs.CV 2025-06 conditional novelty 6.0 of 10

    DenseWorld-1M provides one million images with detailed object captions, pixel masks, and spatial relations by chaining SAM, APE, RAM++, and VLMs through a three-stage labeling pipeline.

  9. Dense360: Dense Understanding from Omnidirectional Panoramas

    cs.CV 2025-06 reject novelty 6.0 of 10

    Introduces a 160K-panorama auto-annotated dataset, a dense captioning and grounding benchmark, and ERP-RoPE; fine-tuning Qwen2.5VL on the data lifts benchmark scores.

  10. Mitigating Object Hallucination via Robust Local Perception Search

    cs.CV 2025-06 conditional novelty 6.0 of 10

    A training-free decoding method that uses an MLLM's own local object descriptions as a reward prior, combined with CLIP similarity, to cut object hallucination, especially under adversarial image noise.

  11. Enhancing Multimodal Large Language Models Complex Reason via Similarity Computation

    cs.CV 2024-12 conditional novelty 6.0 of 10

    Simignore improves multimodal LLM complex question answering on ScienceQA by masking image tokens whose embeddings have low cosine similarity to the text prompt.

  12. A Comprehensive Analysis for Visual Object Hallucination in Large Vision-Language Models

    cs.CV 2025-05 conditional novelty 5.0 of 10

    The paper finds the LLM is mostly faithful given good captions, the CLIP vision encoder contributes perception errors, and the projector preserves visual information but aligns it poorly with text.

  13. Toward Modality Gap: Vision Prototype Learning for Weakly-supervised Semantic Segmentation with CLIP

    cs.CV 2024-12 conditional novelty 5.0 of 10

    Weakly supervised segmentation with CLIP improves by learning per-class prototypes in vision space instead of relying on text prototypes.

  14. MCA-LLaVA: Manhattan Causal Attention for Reducing Hallucination in Large Vision-Language Models

    cs.CV 2025-07 conditional novelty 4.0 of 10

    MCA-LLaVA reindexes image tokens by sums of mirrored 2D coordinates so instruction tokens attend across the whole image, reducing hallucination on POPE, CHAIR, and MME.

  15. AdaToken-3D: Dynamic Spatial Gating for Efficient 3D Large Multimodal-Models Reasoning

    cs.GR 2025-05 conditional novelty 4.0 of 10

    AdaToken-3D uses attention-derived contribution scores to prune spatial tokens layer by layer in 3D LMMs, achieving about 60 percent FLOPs reduction with roughly unchanged benchmark scores.

  16. First-place Solution for Streetscape Shop Sign Recognition Competition

    cs.CV 2025-01 reject novelty 2.0 of 10

    A team reports winning a street-view shop sign recognition competition with a multi-stage OCR pipeline built from known components, but provides no code, data, or rigorous ablations.

Reference graph

Works this paper leans on

46 extracted references · 11 canonical work pages · cited by 16 Pith papers

  1. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ah- mad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 ,

    arXiv

  2. [2]

    Agla: Mitigating object hallucinations in large vision- language models with assembly of global and local attention

    Wenbin An, Feng Tian, Sicong Leng, Jiahao Nie, Haonan Lin, QianYing Wang, Guang Dai, Ping Chen, and Shijian Lu. Agla: Mitigating object hallucinations in large vision- language models with assembly of global and local attention. arXiv preprint arXiv:2406.12718, 2024. 6, 7

    arXiv 2024

  3. [3]

    Qwen-vl: A frontier large vision-language model with versatile abilities

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966 , 2023. 1, 4, 7, 8

    arXiv 2023

  4. [4]

    Shikra: Unleashing multimodal llm’s ref- erential dialogue magic

    Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multimodal llm’s ref- erential dialogue magic. arXiv preprint arXiv:2306.15195 ,

    arXiv

  5. [5]

    An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models

    Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang. An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models. 18th European Conference on Computer Vision ECCV 2024, 2024. 2, 6, 7

    work page 2024

  6. [6]

    Internvl: Scaling up vision foundation mod- els and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation mod- els and aligning for generic visual-linguistic tasks. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 24185–24198, 2024. 2, 3, 4, 7, 8

    work page 2024

  7. [7]

    Halc: Object hallucination reduc- tion via adaptive focal-contrast decoding

    Zhaorun Chen, Zhuokai Zhao, Hongyin Luo, Huaxiu Yao, Bo Li, and Jiawei Zhou. Halc: Object hallucination reduc- tion via adaptive focal-contrast decoding. arXiv preprint arXiv:2403.00425, 2024. 6, 7, 11

    arXiv 2024

  8. [8]

    Dola: Decoding by con- trasting layers improves factuality in large language models

    Yung-Sung Chuang, Yujia Xie, Hongyin Luo, Yoon Kim, James Glass, and Pengcheng He. Dola: Decoding by con- trasting layers improves factuality in large language models. arXiv preprint arXiv:2309.03883, 2023. 6, 7, 11

    arXiv 2023

Show all 46 references
  1. [9]

    Training verifiers to solve math word problems

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021. 7

    arXiv

  2. [10]

    Instructblip: Towards general-purpose vision-language models with instruction tuning

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning. Advances in Neural Information Processing Systems, 36, 2024....

    2024

  3. [11]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Ab- hishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 ,

    arXiv

  4. [12]

    Blink: Multimodal large language models can see but not perceive

    Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A Smith, Wei-Chiu Ma, and Ranjay Krishna. Blink: Multimodal large language models can see but not perceive. arXiv preprint arXiv:2404.12390,

    arXiv

  5. [13]

    Multimodal-gpt: A vision and lan- guage model for dialogue with humans

    Tao Gong, Chengqi Lyu, Shilong Zhang, Yudong Wang, Miao Zheng, Qian Zhao, Kuikun Liu, Wenwei Zhang, Ping Luo, and Kai Chen. Multimodal-gpt: A vision and lan- guage model for dialogue with humans. arXiv preprint arXiv:2305.04790, 2023. 1

    2023 arXiv

  6. [14]

    Making the v in vqa matter: Elevating the role of image understanding in visual question answer- ing

    Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Ba- tra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answer- ing. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6904–6913...

    2017

  7. [15]

    Hallusionbench: An advanced diag- nostic suite for entangled language hallucination and visual illusion in large vision-language models

    Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, et al. Hallusionbench: An advanced diag- nostic suite for entangled language hallucination and visual illusion in large vision-language models. arXiv ...

    arXiv

  8. [16]

    Vizwiz grand challenge: Answering visual questions from blind people

    Danna Gurari, Qing Li, Abigale J Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3608–3617,

  9. [17]

    Opera: Alleviating hallucination in multi-modal large language models via over-trust penalty and retrospection- allocation

    Qidong Huang, Xiaoyi Dong, Pan Zhang, Bin Wang, Con- ghui He, Jiaqi Wang, Dahua Lin, Weiming Zhang, and Neng- hai Yu. Opera: Alleviating hallucination in multi-modal large language models via over-trust penalty and retrospection- allocation. In Proceedings of the IEEE/CVF Conf...

    2024

  10. [18]

    Gqa: A new dataset for real-world visual reasoning and compositional question answering

    Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition , pages 6700–6709, 2019. 6

    2019

  11. [19]

    Mistral 7b

    Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lam- ple, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023. 7, 8

    arXiv

  12. [20]

    Mitigating object hal- lucinations in large vision-language models through visual contrastive decoding

    Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shijian Lu, Chunyan Miao, and Lidong Bing. Mitigating object hal- lucinations in large vision-language models through visual contrastive decoding. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recogn...

    2024

  13. [21]

    Seed-bench: Benchmarking multi- modal llms with generative comprehension

    Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yix- iao Ge, and Ying Shan. Seed-bench: Benchmarking multi- modal llms with generative comprehension. arXiv preprint arXiv:2307.16125, 2023. 6

    2023 arXiv

  14. [22]

    Fine-tuning multimodal llms to follow zero-shot demonstrative instructions

    Juncheng Li, Kaihang Pan, Zhiqi Ge, Minghe Gao, Wei Ji, Wenqiao Zhang, Tat-Seng Chua, Siliang Tang, Han- wang Zhang, and Yueting Zhuang. Fine-tuning multimodal llms to follow zero-shot demonstrative instructions. In The Twelfth International Conference on Learning Representa- ...

    2023

  15. [23]

    Inference-time intervention: Elicit- ing truthful answers from a language model

    Kenneth Li, Oam Patel, Fernanda Vi ´egas, Hanspeter Pfister, and Martin Wattenberg. Inference-time intervention: Elicit- ing truthful answers from a language model. Advances in Neural Information Processing Systems , 36, 2024. 3, 6, 7, 11

    2024

  16. [24]

    Evaluating object hallucina- tion in large vision-language models

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucina- tion in large vision-language models. arXiv preprint arXiv:2305.10355, 2023. 6, 7

    2023 arXiv

  17. [25]

    Llama-vid: An image is worth 2 tokens in large language models

    Yanwei Li, Chengyao Wang, and Jiaya Jia. Llama-vid: An image is worth 2 tokens in large language models. arXiv preprint arXiv:2311.17043, 2023. 1

    2023 arXiv

  18. [26]

    Mini-gemini: Mining the potential of multi-modality vision language models

    Yanwei Li, Yuechen Zhang, Chengyao Wang, Zhisheng Zhong, Yixin Chen, Ruihang Chu, Shaoteng Liu, and Jiaya Jia. Mini-gemini: Mining the potential of multi-modality vision language models. arXiv preprint arXiv:2403.18814,

    arXiv

  19. [27]

    Truthfulqa: Measuring how models mimic human falsehoods

    Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprint arXiv:2109.07958, 2021. 7

    2021 arXiv

  20. [28]

    Aligning large multi-modal model with robust instruction tuning

    Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang. Aligning large multi-modal model with robust instruction tuning. arXiv preprint arXiv:2306.14565, 2023. 1

    2023 arXiv

  21. [29]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024. 1, 2, 3, 4, 6, 7, 8

    2024

  22. [30]

    Mitigating dialogue hallucination for large multi- modal models via adversarial instruction tuning

    Dongmin Park, Zhaofang Qian, Guangxing Han, and Ser- Nam Lim. Mitigating dialogue hallucination for large multi- modal models via adversarial instruction tuning. arXiv preprint arXiv:2403.10492, 2024. 1

    2024 arXiv

  23. [31]

    Object hallucination in image cap- tioning

    Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. Object hallucination in image cap- tioning. arXiv preprint arXiv:1809.02156, 2018. 6, 8

    2018 arXiv

  24. [32]

    Gemini: a family of highly capable multimodal models

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023. 1

    2023 arXiv

  25. [33]

    Qwen2.5: A party of foundation models, 2024

    Qwen Team. Qwen2.5: A party of foundation models, 2024. 7, 8

    2024

  26. [34]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth ´ee Lacroix, Bap- tiste Rozi `ere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023. 7

    2023 arXiv

  27. [35]

    Label words are an- chors: An information flow perspective for understanding in- context learning

    Lean Wang, Lei Li, Damai Dai, Deli Chen, Hao Zhou, Fan- dong Meng, Jie Zhou, and Xu Sun. Label words are an- chors: An information flow perspective for understanding in- context learning. arXiv preprint arXiv:2305.14160, 2023. 2, 4

    2023 arXiv

  28. [36]

    Dopra: Decoding over-accumulation penalization and re-allocation in specific weighting layer

    Jinfeng Wei and Xiaofeng Zhang. Dopra: Decoding over-accumulation penalization and re-allocation in specific weighting layer. Proceedings of the 32nd ACM International Conference on Multimedia, 2024. 4, 6, 7

    2024

  29. [37]

    Efficient streaming language models with attention sinks

    Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. arXiv preprint arXiv:2309.17453, 2023. 4

    2023 arXiv

  30. [38]

    Miti- gating object hallucination via concentric causal attention

    Yun Xing, Yiheng Li, Ivan Laptev, and Shijian Lu. Miti- gating object hallucination via concentric causal attention. arXiv preprint arXiv:2410.15926, 2024. 6, 7

    2024 arXiv

  31. [39]

    Qwen2 technical report

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jian- wei Zhang, Jianxin Ma, Jin Xu, Jingren Zhou, Jinze Bai, Jinzhe...

    arXiv

  32. [40]

    A survey on multimodal large language models

    Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. A survey on multimodal large language models. arXiv preprint arXiv:2306.13549 , 2023. 6, 7

    2023 arXiv

  33. [41]

    Mm-vet: Evaluating large multimodal models for integrated capabilities

    Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308.02490, 2023. 6

    2023 arXiv

  34. [42]

    Unveiling and har- nessing hidden attention sinks: Enhancing large language models without training through attention calibration

    Zhongzhi Yu, Zheng Wang, Yonggan Fu, Huihong Shi, Khalid Shaikh, and Yingyan Celine Lin. Unveiling and har- nessing hidden attention sinks: Enhancing large language models without training through attention calibration. arXiv preprint arXiv:2406.15765, 2024. 4

    2024 arXiv

  35. [43]

    Less is more: Mit- igating multimodal hallucination from an eos decision per- spective

    Zihao Yue, Liang Zhang, and Qin Jin. Less is more: Mit- igating multimodal hallucination from an eos decision per- spective. The 62nd Annual Meeting of the Association for Computational Linguistics, 2024. 6, 7

    2024

  36. [44]

    Video-llama: An instruction-tuned audio-visual language model for video un- derstanding

    Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tuned audio-visual language model for video un- derstanding. arXiv preprint arXiv:2306.02858, 2023. 1

    2023 arXiv

  37. [45]

    From redundancy to relevance: Infor- mation flow in lvlms across reasoning tasks

    Xiaofeng Zhang, Yihao Quan, Chen Shen, Xiaosong Yuan, Shaotian Yan, Liang Xie, Wenxiao Wang, Chaochen Gu, Hao Tang, and Jieping Ye. From redundancy to relevance: Infor- mation flow in lvlms across reasoning tasks. arXiv preprint arXiv:2406.06579, 2024. 8

    2024 arXiv

  38. [46]

    Minigpt-4: Enhancing vision-language understanding with advanced large language models

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mo- hamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592, 2023. 1, 2, 3, 4, 7, 8 10 A. Appendix A.1. Discussion and Limitations The results of ...

    2023 arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.