Pith. sign in

REVIEW 3 major objections 6 minor 38 references

LVLMs keep hidden neurons that flag words missing from the image

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Feed-forward neurons in LVLMs encode whether a text token is visually grounded, and a detector built on these neurons can reduce hallucination by overriding or replacing ungrounded tokens.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection A genuinely new finding about FFN neurons that respond to visually absent tokens, with a transferable detector—but the evaluation has enough selection and leakage issues that the headline numbers need a careful redo. the 3 major comments →

arxiv 2509.03025 v2 pith:KZU7VOJM submitted 2025-09-03 cs.CV cs.AI

Unveiling the Response of Large Vision-Language Models to Visually Absent Tokens

Sohee Kim , Soohyun Ryu , Joonhyung Park , Eunho Yang This is my paper
classification cs.CV cs.AI
keywords visual absencehallucination mitigationfeed-forward network neuronsvision-language modelsneuron activation analysisvisual groundingVA-QA datasetcontrastive probing
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Large vision-language models (LVLMs) often answer as if a word in the question were visible in the image when it is not, and the paper sets out to show that the model itself carries an internal signal for this failure. Examining feed-forward network (FFN) activations on contrastive image-question pairs, the authors identify a subset of neurons—Visual Absence-aware (VA) neurons—whose activation rises consistently for text tokens with no visual counterpart. They turn this into a detector: a lightweight classifier over VA-neuron activations labels each token as visually grounded or not, and the label is used to override "Yes" answers in binary QA or to roll back and replace hallucinated tokens during open-ended generation. Across several LVLMs the refinement raises accuracy on questions whose correct answer is "No" and lowers object-hallucination metrics in captions. If the mechanism is right, a model's own internals can police its visual grounding without retraining or external knowledge.

Core claim

The paper claims that LVLMs do not only hallucinate while generating; they also misread input tokens that have no visual evidence, and the model's feed-forward layers contain a content-invariant internal indicator of that absence. A sensitivity score S_VA, defined as one minus the Bhattacharyya coefficient between activation distributions for present versus absent tokens, selects neurons, concentrated in middle layers, that respond strongly to any absent token regardless of word meaning. Suppressing these neurons lowers, and amplifying them raises, accuracy on "No" questions. The authors build a detector from the selected neurons' activations and use it to refine outputs: override to "No" in

What carries the argument

Visual Absence-aware (VA) neurons—FFN neurons, mostly in middle layers, whose gated activations shift sharply when an input token has no visual counterpart—together with the sensitivity score S_VA = 1 - BC(A_pre, A_abs), where BC is the Bhattacharyya coefficient between activation histograms for visually present and absent tokens. The score identifies the neurons; a linear classifier over their activations is the VA detector; its binary labels drive answer overriding and token rollback.

Load-bearing premise

The load-bearing premise is that the activation differences measured on the VA-QA pairs come from visual absence itself, not from lexical, syntactic, or question-format differences between the paired questions; if that premise fails, the VA neurons are not absence detectors.

What would settle it

Train a detector on VA-QA and test it on contrastive pairs where the same object word is visually absent for a different reason—occlusion, blur, or removal by inpainting rather than replacement by another object. If the detector no longer separates present from absent tokens, the neurons are tracking the contrastive question format rather than visual grounding.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Binary QA: across LLaVA-v1.5, LLaVA-v1.6, mPLUG-Owl2, InstructBLIP, Qwen2-VL, and Gemma3, overriding to "No" when any question token is flagged sharply raises Acc_no, with overall accuracy usually improving or staying level.
  • Open-ended generation: when a generated token is flagged absent, rolling back one step and suppressing its logit lowers CHAIR sentence- and instance-level hallucination ratios and GPT-based severity scores, with minimal change in caption length.
  • Transfer: detectors trained on VA-QA generalize to out-of-domain binary benchmarks—POPE, R-Bench, Winoground, and binary-converted SEED-Bench—without model-specific tuning.
  • Composability: the refinement is decoding-strategy-agnostic and stacks with VCD and DoLA, further reducing hallucination metrics beyond either method alone.
  • Scale: the pattern appears in models up to Qwen2.5-VL 32B, where the same detector still cuts hallucination metrics.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If VA neurons are a general internal mechanism, the same activation-difference scoring could identify detectors for other hallucination types—attributes, spatial relations, temporal claims—and the paper's diverse-data training results suggest this direction is testable.
  • The detector's success suggests the decoder does not fully use what earlier layers know: hallucination is partly an output-alignment failure, so coupling decoding to internal detectors could be more reliable than external fact-checking alone.
  • The answer-override rule is conservative by design; in open-ended chat, a practical deployment would need confidence thresholds or task context to avoid suppressing valid answers where an absent word is functionally relevant. The paper's role-based robustness test addresses one such case, but does not fully settle the trade-off.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper studies the behavior of Large Vision-Language Models (LVLMs) when input text contains tokens that have no visual counterpart in the accompanying image. The authors construct a contrastive Visual Absence Question Answering (VA-QA) dataset from SVO-Probes, show that LVLMs are more error-prone on questions containing visually absent tokens, and identify a set of Feed-Forward Network (FFN) neurons, termed Visual Absence-aware (VA) neurons, whose activations differ between visually present and absent tokens according to a Bhattacharyya-coefficient-based sensitivity score S_VA. They then train a lightweight VA detector on the activations of high-S_VA neurons and use its predictions to refine binary question answering and open-ended generation. The method is evaluated on multiple LVLMs (LLaVA-v1.5/v1.6, mPLUG-Owl2, InstructBLIP, Qwen2-VL, Gemma3) on in-domain VA-QA and out-of-domain benchmarks (POPE, R-Bench, SEED-Bench, Winoground, CHAIR), with additional ablations in the appendix.

Significance. If the central claim holds, the paper makes a useful contribution to the interpretability and hallucination-mitigation literature: it identifies a sparse, causally relevant subset of FFN neurons that encode visual absence and shows that their activations can be used for a lightweight, training-free intervention. The paper's strengths include a carefully constructed contrastive dataset, evaluation across many LVLMs, out-of-domain benchmarks, and detailed appendices with additional ablations and qualitative results. However, the core mechanistic claim rests on a data-selection procedure that may break the intended lexical balance of VA-QA, and the causal evidence in Table 1 is suggestive but not statistically grounded. These issues are load-bearing for the claim that the identified neurons specifically encode visual absence rather than lexical identity, so the current evidence is not fully convincing. The out-of-domain results provide some support for generality, but they evaluate the full refinement pipeline rather than isolating the VA-neuron selection mechanism.

major comments (3)
  1. [§3.2, §F.2, Eq. (3)–(4), Table 2] The correct-answer filtering in §3.2 can break the pair-level lexical balance that the VA-QA construction is designed to achieve. For LLaVA-v1.5, Acc_yes ≈ 95% while Acc_no ≈ 48% (Table 2), so the curated subset used to compute A_pre and A_abs contains almost all present-token questions but fewer than half of the absent-token questions. A lexical item whose matched 'absent' question is answered incorrectly is dropped from A_abs while its 'present' counterpart remains, so the two sets in Eq. (3) are no longer matched for word identity. High S_VA may therefore reflect the particular lexical items surviving in A_abs rather than visual absence, and because Eq. (4) selects detector features using S_VA, the detector accuracy in Table 11 could reflect token identity or word frequency. The same-word control in Fig. 5 is illustrative and is not used to train or evaluate the detector, so it does n
  2. [§F.2, §5.1, Table 2] F.2 states that S_VA is computed on the entire curated subset before the 9:1 train/validation split, and that the threshold β is chosen to maximize validation accuracy. Thus the neuron selection in Eq. (4) uses validation labels, and the in-domain VA-QA numbers in Table 2 are not an independent evaluation of the detector. The out-of-domain benchmarks (POPE, R-Bench, SEED-Bench) are less affected by this leakage, but they test the full refinement pipeline, not the VA-neuron selection mechanism in isolation. Please recompute S_VA on the training split only (or use a nested selection procedure) and report detector-only accuracy on a held-out lexical-balanced set. Without this, the in-domain results cannot be taken as evidence that the detector generalizes to visual absence beyond the curated lexicon.
  3. [§3.3, Table 1] The causal ablation in Table 1 is suggestive but weak. For LLaVA-v1.5, zeroing high-S_VA neurons changes GT=No accuracy from 48.0 to 41.5 and enhancing them changes it to 50.5, while the random-neuron controls change it to 48.5 and 47.7, respectively. No confidence intervals, multiple seeds, or significance tests are reported. The effect is directionally consistent with the claim, but the magnitude is small and the random-neuron baseline is not clearly separated. Given that the paper states these neurons 'play a key role' in visual absence recognition, please provide statistical significance testing across repeated runs, per-pair analyses, or a dose-response curve over the number/strength of modified neurons.
minor comments (6)
  1. [§3.3, Eq. (3)] The number of histogram bins K used for the Bhattacharyya coefficient is not specified in the main text. Please state K and the binning scheme, and report sensitivity to K.
  2. [§3.1] The VA-QA dataset is manually constructed (600 pairs, 2400 questions) with 'distinct differences' between image pairs, but no inter-annotator agreement or verification statistics are reported. Please provide details on how the distinctness and the correctness of the general questions were validated.
  3. [Table 4] For Qwen2-VL, Ci increases from 22.6 to 23.7 under the proposed method. The text explains this by reduced caption length lowering the denominator, but this is not fully convincing. Please report the absolute number of hallucinated objects and the total number of objects mentioned, so the trade-off is transparent.
  4. [Table 3] The SEED-Bench results show substantial Acc_yes drops in some categories (e.g., LLaVA-v1.5 'Instances' drops from 95.6 to 68.1). Calling this a 'slight trade-off' understates the change. Please quantify the yes/no trade-off and discuss when the override policy is appropriate.
  5. [Fig. 5] The same-word control in Fig. 5 is described only qualitatively. Since it is central to ruling out a lexical-identity confound, please report the number of tokens used, error bars, and a quantitative same-word discrimination score (e.g., classifier accuracy on same-word present/absent pairs).
  6. [Appendix D.2, F.5] There are typos: 'weevaluatede' in D.2 and 'score shall' in F.5. Please proofread the appendix.

Circularity Check

1 steps flagged

VA-QA in-domain accuracy and VA-neuron selection are partly by construction: S_VA is computed on the same correctly-answered, lexically unbalanced curated set that later trains and evaluates the detector.

specific steps
  1. fitted input called prediction [Section 3.2 (correct-answer filter), Section 3.3 (Eq. 3), Section 4.1 (Eqs. 4-5), Section 5.1, Appendix F.2]
    "To ensure precise analysis, we restrict our examination to questions for which the model generates the correct answer... From the original VA-QA dataset of 2400 samples, we first curated a subset consisting exclusively of instances answered correctly by each model individually. This curated subset was then partitioned into training and validation sets using a 9:1 ratio."

    The correct-answer filter is applied before S_VA is computed. Since baseline Acc_yes is ~95% while Acc_no is ~48% (Table 2), the filter removes far more absent-token instances, so A_abs and A_pre are no longer matched by lexical item: a word whose 'absent' question is answered incorrectly drops out of A_abs while its 'present' counterpart remains in A_pre. S_VA (Eq. 3) is computed on this unbalanced curated set before the 9:1 split and then used (Eq. 4) to select the detector's features. The detector is trained on the same VA-QA dataset and Section 5.1 explicitly calls VA-QA 'in-domain'; no separate test split is described in F.2. Consequently, the reported VA-QA accuracy (e.g., Acc_no 48.0→77.5 for LLaVA-v1.5) and the identity of the 'VA neurons' partly restate the selection/filtering cri

full rationale

The paper's central observational claim—that specific FFN neurons respond to visual absence—is not circular in itself: it is an empirical finding supported by activation plots, intervention experiments (Table 1), cross-model consistency (Appendix A), and same-word illustrative analysis. The circularity arises specifically in the detector pipeline. S_VA is computed on all correctly answered VA-QA samples before the train/validation split, and the correct-answer filter is strongly asymmetric between present and absent questions, breaking the dataset's intended lexical balance. This means neuron selection can be driven by token identity or question difficulty rather than visual absence, and the in-domain VA-QA evaluation then reports high accuracy on data that contributed to both feature selection and training. The out-of-domain benchmarks (POPE, R-Bench, SEED-Bench, Winoground) and CHAIR results provide genuine external evidence, so the circularity is partial rather than total. Thus score 6: one or more 'predictions' reduce partly by construction, but independent content remains.

Axiom & Free-Parameter Ledger

5 free parameters · 6 axioms · 1 invented entities

The central claim rests on several domain assumptions about token semantics, dataset construction, and evaluation metrics. The main free parameters are the threshold beta, histogram bin count, ablation scope, MLP size, and rollback schedule; none of these are independently motivated beyond fitting the validation or target benchmarks.

free parameters (5)
  • S_VA threshold beta = per-model optimum in [0.3, 0.8] selected by validation accuracy
    Controls which neurons enter the detector feature vector (Eq. 4); selected on validation, so it is a fitted hyperparameter.
  • Histogram bin count K = not reported
    Used in the Bhattacharyya coefficient in Eq. 3; changes S_VA values and neuron ranking, but K is never stated.
  • Top neuron count and layer range in ablation = top-100 neurons in layers 8-16 (900 neurons, 0.255% of 352256)
    Table 1 uses this hand-chosen ablation scope; no sensitivity analysis is given for this choice.
  • MLP hidden units = 128
    F.2 states the VA detector is an MLPClassifier with one hidden layer of 128 units; this is an architecture choice not derived from the data.
  • Rollback deepening window = 5-token window, extra rollback step after two rollbacks
    F.1 introduces this decoding heuristic without ablation; it can influence open-ended generation results.
axioms (6)
  • domain assumption Gated FFN equation (Eq. 1) accurately describes the architecture of all evaluated LVLMs.
    The analysis and detector features are built from the activation vector a defined by this equation; if a model uses a different FFN variant, the scoring and features may not apply.
  • domain assumption The last token of a multi-token word represents the meaning of the whole word.
    Footnote 3 in Section 3.2; this decides which token's activation is compared across present and absent conditions.
  • domain assumption Contrastive VA-QA pairs isolate visual absence as the only systematic difference between present and absent token sets.
    Section 3.1; if the pairs differ in salience or lexical predictability, activation differences may encode those confounds.
  • standard math Histogram binning with K bins and the Bhattacharyya coefficient is a valid separation measure for activation distributions.
    Section 3.3, Eq. 3; BC measures overlap but not direction, and K is not reported.
  • domain assumption Correctly answered questions are representative of all questions when characterizing VA neurons.
    Section 3.2 restricts analysis to correct answers; if incorrect answers have different activation patterns, the 'consistent' claim is overstated.
  • domain assumption CHAIR and GPT-4o scores measure hallucination adequately for open-ended evaluation.
    Section 5.2.2 and F.5; CHAIR only covers annotated objects and GPT-4o evaluation is model-based and prompt-dependent.
invented entities (1)
  • Visual Absence-aware (VA) neurons no independent evidence
    purpose: A subset of FFN neurons whose activations are claimed to signal whether an input text token has no visual evidence in the image; used as features for the VA detector.
    The paper provides internal activation plots, ablations, and detector accuracy, but no falsifiable handle outside this paper's own datasets; the same VA-QA data is used to define, select, and evaluate them.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Unveiling the Response of Large Vision-Language Models to Visually Absent Tokens." pith.science (2026). https://pith.science/paper/KZU7VOJM

@misc{pith2026250903025,
  author       = {Pith},
  title        = {Pith review of: Unveiling the Response of Large Vision-Language Models to Visually Absent Tokens},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KZU7VOJM}},
  note         = {Machine review of arXiv:2509.03025}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Large Vision-Language Models (LVLMs) generate contextually relevant responses by jointly interpreting visual and textual inputs. However, our finding reveals they often mistakenly perceive text inputs lacking visual evidence as being part of the image, leading to erroneous responses. In light of this finding, we probe whether LVLMs possess an internal capability to determine if textual concepts are grounded in the image, and discover a specific subset of Feed-Forward Network (FFN) neurons, termed Visual Absence-aware (VA) neurons, that consistently signal the visual absence through a distinctive activation pattern. Leveraging these patterns, we develop a detection module that systematically classifies whether an input token is visually grounded. Guided by its prediction, we propose a method to refine the outputs by reinterpreting question prompts or replacing the detected absent tokens during generation. Extensive experiments show that our method effectively mitigates the models' tendency to falsely presume the visual presence of text input and its generality across various LVLMs.

Figures

Figures reproduced from arXiv: 2509.03025 by Eunho Yang, Joonhyung Park, Sohee Kim, Soohyun Ryu.

Figure 1
Figure 1. Figure 1: LVLM’s Vulnerability to Visually Absent Tokens. (a) When the input prompt contains visually absent tokens (e.g., standing), LVLM becomes confused and generates a response accordingly. (b) Conversely, when they are removed, the model produces an accurate response that correctly aligns with the image. performance in visual understanding and reason￾ing, LVLMs often suffer from hallucination, where they descri… view at source ↗
Figure 2
Figure 2. Figure 2: VA-QA Dataset Construction. For each im￾age pair differing by a single element in the 〈subject, verb, object〉, we generate yes-or-no questions. Counter￾part question includes a visually absent token, making the correct answer “No.” Additionally, we create a gen￾eral question, which does not contain visually present or absent tokens, so the model responds without bias. inputs on the LVLM’s response, we gene… view at source ↗
Figure 3
Figure 3. Figure 3: Observation on FFN Activation Patterns in Response to Visual Absence. (a) Using the VA-QA dataset, we extract FFN activations a from all L layers for visually present and absent tokens by inputting each image with its corresponding and counterpart questions. (b) Visual absence-aware neurons (i) exhibit a significant difference in activation values between visually present and absent tokens, while other neu… view at source ↗
Figure 4
Figure 4. Figure 4: Top-100 S VA across layers. The figure depicts the S VA of top-100 neurons for each layer, with scores greater than 0.4 set as 0.4 for better interpretability. for which the model generates the correct answer, as this suggests that the model has accurately un￾derstood the image within the given context. Our observation reveals that there is a specific group of FFN neurons that respond selectively to visual… view at source ↗
Figure 5
Figure 5. Figure 5: Activation Patterns of High S VA Neurons Across Varying Textual Contexts. (a) Given that middle-layer neurons exhibit high S VA, we show their activation levels for present and absent tokens. The activation level is computed by normalizing the actual activation value based on the range of Al,i. (b) Activation values of these neurons show similar patterns in cases with the same visual absence status but dif… view at source ↗
Figure 6
Figure 6. Figure 6: VA Detector-based Refinement. For each task, visually absent tokens are identified and used to guide response refinement. The strategy is tailored to the task format: answer overriding for binary QA and correction of visually unsupported content during generation. of visual absence, leading to improved accuracy on GT=No questions. Experiment 2: Are neurons with high S VA invari￾ant to input context? To ass… view at source ↗
Figure 7
Figure 7. Figure 7: Qualitative Results. Comparison between a caption generated using LLaVA-1.5 and the refined version using our method [PITH_FULL_IMAGE:figures/full_fig_p009_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Top-100 S VA across layers [PITH_FULL_IMAGE:figures/full_fig_p013_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: FFN Activation Patterns in Response to Visual Absence. This figure shows the distinct activation pattern of Visual Absence-aware neurons in response to visual absence. It demonstrates that all models possess a specific set of neurons that respond to visually absent tokens [PITH_FULL_IMAGE:figures/full_fig_p014_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: VA Neuron Activation and No Probability. For the VA-QA dataset, we visualize VA neuron activations for visually absent tokens in yes-or-no questions alongside the model’s probability of predicting “No.” Ideally, the model should answer “No” due to the absence of relevant visual information, though it often fails. Nonetheless, regardless of whether the prediction is correct or wrong, VA neurons constantly … view at source ↗
Figure 11
Figure 11. Figure 11: Effect of Score Threshold β Selection. This figure shows the accuracy of binary question answering benchmarks across various models depending on the threshold β used to select VA neurons. C Effect of Score Threshold Selection for VA Neuron Identification To examine the impact of the threshold hyperpa￾rameter β used to identify Visual Absence-aware (VA) neurons, we conduct an ablation study across multiple… view at source ↗
Figure 12
Figure 12. Figure 12: Open-ended Generation Qualitative Results [PITH_FULL_IMAGE:figures/full_fig_p020_12.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

38 extracted references · 5 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, and 1 others. 2023. Qwen technical report. arXiv preprint arXiv:2309.16609

    Pith/arXiv arXiv 2023

  4. [4]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, and 1 others. 2025. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923

    Pith/arXiv arXiv 2025

  5. [5]

    Anil Bhattacharyya. 1946. On a measure of divergence between two multinomial populations. Sankhy \=a : the indian journal of statistics , pages 401--406

    work page 1946

  6. [6]

    Yung-Sung Chuang, Yujia Xie, Hongyin Luo, Yoon Kim, James Glass, and Pengcheng He. 2023. Dola: Decoding by contrasting layers improves factuality in large language models. arXiv preprint arXiv:2309.03883

    Pith/arXiv arXiv 2023

  7. [7]

    Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. 2021. Knowledge neurons in pretrained transformers. arXiv preprint arXiv:2104.08696

    Pith/arXiv arXiv 2021

  8. [8]

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. 2023. https://arxiv.org/abs/2305.06500 Instructblip: Towards general-purpose vision-language models with instruction tuning . Preprint, arXiv:2305.06500

    Pith/arXiv arXiv 2023

  9. [9]

    Yann N Dauphin, Angela Fan, Michael Auli, and David Grangier. 2017. Language modeling with gated convolutional networks. In International conference on machine learning, pages 933--941. PMLR

    2017

  10. [10]

    Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. 2020. Transformer feed-forward layers are key-value memories. arXiv preprint arXiv:2012.14913

    Pith/arXiv arXiv 2020

  11. [11]

    Lisa Anne Hendricks and Aida Nematzadeh. 2021. Probing image-language transformers for verb understanding. arXiv preprint arXiv:2106.09141

    Pith/arXiv arXiv 2021

  12. [12]

    Qidong Huang, Xiaoyi Dong, Pan Zhang, Bin Wang, Conghui He, Jiaqi Wang, Dahua Lin, Weiming Zhang, and Nenghai Yu. 2024. Opera: Alleviating hallucination in multi-modal large language models via over-trust penalty and retrospection-allocation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13418--13427

    2024

  13. [13]

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, and 1 others. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276

    Pith/arXiv arXiv 2024

  14. [14]

    Chaoya Jiang, Haiyang Xu, Mengfan Dong, Jiaxing Chen, Wei Ye, Ming Yan, Qinghao Ye, Ji Zhang, Fei Huang, and Shikun Zhang. 2024. Hallucination augmented contrastive learning for multimodal large language model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 27036--27046

    work page 2024

  15. [15]

    Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shijian Lu, Chunyan Miao, and Lidong Bing. 2024. Mitigating object hallucinations in large vision-language models through visual contrastive decoding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13872--13882

    2024

  16. [16]

    Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. 2023 a . Seed-bench: Benchmarking multimodal llms with generative comprehension. arXiv preprint arXiv:2307.16125

    Pith/arXiv arXiv 2023

  17. [17]

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. 2023 b . Evaluating object hallucination in large vision-language models. arXiv preprint arXiv:2305.10355

    Pith/arXiv arXiv 2023

  18. [18]

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll \'a r, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In Computer vision--ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings, part v 13, pages 740--755. Springer

    2014

  19. [19]

    Hanchao Liu, Wenyuan Xue, Yifei Chen, Dapeng Chen, Xiutian Zhao, Ke Wang, Liping Hou, Rongjun Li, and Wei Peng. 2024 a . A survey on hallucination in large vision-language models. arXiv preprint arXiv:2402.00253

    Pith/arXiv arXiv 2024

  20. [20]

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2024 b . Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296--26306

    work page 2024

  21. [21]

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual instruction tuning. Advances in neural information processing systems, 36:34892--34916

    2023

  22. [22]

    Shi Liu, Kecheng Zheng, and Wei Chen. 2024 c . Paying more attention to image: A training-free method for alleviating hallucination in lvlms. In European Conference on Computer Vision, pages 125--140. Springer

    work page 2024

  23. [23]

    Pedregosa, G

    F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine learning in P ython. Journal of Machine Learning Research, 12:2825--2830

    2011

  24. [24]

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, and 1 others. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748--8763. PmLR

    2021

  25. [25]

    Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. 2018. Object hallucination in image captioning. arXiv preprint arXiv:1809.02156

    Pith/arXiv arXiv 2018

  26. [26]

    Noam Shazeer. 2020. Glu variants improve transformer. arXiv preprint arXiv:2002.05202

    Pith/arXiv arXiv 2020

  27. [27]

    Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang-Yan Gui, Yu-Xiong Wang, Yiming Yang, and 1 others. 2023. Aligning large multimodal models with factually augmented rlhf. arXiv preprint arXiv:2309.14525

    Pith/arXiv arXiv 2023

  28. [28]

    Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ram \'e , Morgane Rivi \`e re, and 1 others. 2025. Gemma 3 technical report. arXiv preprint arXiv:2503.19786

    Pith/arXiv arXiv 2025

  29. [29]

    Tristan Thrush, Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, and Candace Ross. 2022. Winoground: Probing vision and language models for visio-linguistic compositionality. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5238--5248

    2022

  30. [30]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, and 1 others. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288

    Pith/arXiv arXiv 2023

  31. [31]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30

    2017

  32. [32]

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. 2024. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. arXiv preprint arXiv:2409.12191

    Pith/arXiv arXiv 2024

  33. [33]

    Mingrui Wu, Jiayi Ji, Oucheng Huang, Jiale Li, Yuhang Wu, Xiaoshuai Sun, and Rongrong Ji. 2024. Evaluating and analyzing relationship hallucinations in large vision-language models. arXiv preprint arXiv:2406.16449

    Pith/arXiv arXiv 2024

  34. [34]

    Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan, Anwen Hu, Haowei Liu, Qi Qian, Ji Zhang, and Fei Huang. 2024. mplug-owl2: Revolutionizing multi-modal large language model with modality collaboration. In Proceedings of the ieee/cvf conference on computer vision and pattern recognition, pages 13040--13051

    work page 2024

  35. [35]

    Kening Zheng, Junkai Chen, Yibo Yan, Xin Zou, and Xuming Hu. 2024. Reefknot: A comprehensive benchmark for relation hallucination evaluation, analysis and mitigation in multimodal large language models. arXiv preprint arXiv:2408.09429

    Pith/arXiv arXiv 2024

  36. [36]

    Yiyang Zhou, Chenhang Cui, Jaehong Yoon, Linjun Zhang, Zhun Deng, Chelsea Finn, Mohit Bansal, and Huaxiu Yao. 2023. Analyzing and mitigating object hallucination in large vision-language models. arXiv preprint arXiv:2310.00754

    Pith/arXiv arXiv 2023

  37. [37]

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. 2023. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592

    Pith/arXiv arXiv 2023

  38. [38]

    Lanyun Zhu, Deyi Ji, Tianrun Chen, Peng Xu, Jieping Ye, and Jun Liu. 2024. Ibd: Alleviating hallucinations in large vision-language models via image-biased decoding. arXiv preprint arXiv:2402.18476

    Pith/arXiv arXiv 2024

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.