Pith. sign in

REVIEW 4 major objections 6 minor 57 references

Energy-Guided Decoding for Object Hallucination Mitigation

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Decoding from the hidden layer with the lowest energy score reduces object hallucination in vision-language models, improving accuracy and F1 while shrinking the yes-ratio gap.

desk verdict Simple, training-free decoding trick that cuts yes-bias, but the missing fixed-penultimate-layer baseline leaves the mechanism unresolved. read the letter →

arxiv 2507.07731 v1 pith:FZRR5JX5 submitted 2025-07-10 cs.CV

Xixi Liu , Ailin Deng , Christopher Zach This is my paper
classification cs.CV
keywords objecthallucinationvision-languagemodelsenergy-guideddecodingyes-ratiobiasearlyexitinghiddenstatesPOPEMMEbenchmark
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that a large share of object hallucination in vision-language models is a decoding bias: the language decoder over-produces "Yes" answers, and the bias grows as questions get harder. It proposes energy-guided decoding, which at each token position computes an energy score from the vocabulary-head logits of every decoder layer and decodes from the layer with the minimal score. Across POPE, MME, and MMVP benchmarks and three different vision-language models, this single-pass, hyperparameter-free rule improves accuracy and F1 over greedy decoding and narrows the gap between predicted and expected yes ratios, with average gains of 4.82% accuracy and 8.81% yes-ratio gap reduction. A reader should care because it suggests hallucination mitigation can be a byproduct of where you read out the representation, not of extra models, image edits, or training.

What carries the argument

The energy score is the central object: for a layer's hidden state $h^k_t$, the vocabulary head $H$ maps it to a logit vector and the energy is the negative LogSumExp of those logits, $E(h^k_t) = -\operatorname{LogSumExp}[H(h^k_t)]$. The mechanism is per-token early exiting, the logit-lens technique of reading out from any transformer layer, steered by the criterion that the lowest-energy layer is the most reliable representation of the grounded input. The work this does is turning hallucination mitigation into a layer-selection problem solvable in one forward pass.

What would settle it

Decode identically from a fixed penultimate layer, the layer the energy rule most often picks, across the same POPE, MME, and MMVP evaluations; if accuracy, F1, and yes-ratio gap match the energy-guided results, then the energy ranking itself adds nothing beyond early exiting.

Watch

Extended reading notes

Core claim

The central claim is that the final layer of a vision-language decoder is not the best layer to decode from when the task is visual grounding. Using the "logit lens" early-exit idea, the paper projects each decoder layer's hidden states through the learned vocabulary head to obtain logits, defines the energy of layer $k$ at token $t$ as $E(h^k_t) = -\operatorname{LogSumExp}[H(h^k_t)]$, and selects $k^* = \arg\min_k E(h^k_t)$ for decoding. Empirically the penultimate layer usually carries the minimal energy, and decoding from it produces better-calibrated Yes/No confidence, higher accuracy and F1, and a smaller yes-ratio gap than greedy decoding from the final layer on POPE, MME, and MMVP across three open-sourced vision-language models. The paper interprets the phenomenon as the language prior leaking through the final layer, which the energy criterion resists.

Load-bearing premise

The method assumes the layer with the lowest energy score at each step is the layer whose hidden states are most grounded in the image; since the energy rule mostly selects the second-to-last layer and no baseline decodes from that fixed layer, the observed gains could come from early exiting itself rather than from the energy-based ranking.

Editorial extensions

If this is right

  • Deploying energy-guided decoding costs a single forward pass, no fine-tuning, and no external models, so it can replace greedy decoding directly in autoregressive vision-language models.
  • Across POPE, MME, and MMVP, the method improves accuracy and F1 while cutting the average yes-ratio gap by 8.81% relative to greedy decoding.
  • The advantage is largest on the hardest, adversarial POPE settings, where the "Yes" bias is most severe.
  • Sentence-level caption hallucination (CHAIR_S) is reduced, though instance-level CHAIR_I degrades, a trade-off the paper reports.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the energy rule mostly selects the penultimate layer, as the paper's ablation shows, a fixed-penultimate-layer decoder may reproduce most of the gains, implying the real effect is early exiting rather than dynamic energy ranking.
  • Because the "Yes" bias transfers from text-only inputs, the same exit strategy might mitigate confirmation bias in text-only language models, not just vision-language ones.
  • The reported CHAIR_I degradation suggests the method sometimes drops correct object mentions; combining energy selection with a visual-attention check could recover instance-level recall in captioning.
  • A natural test is to vary the exit layer within a single long generation, where the optimal layer for the first token (a short yes/no answer) need not be optimal for subsequent tokens.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes 'energy-guided decoding' for large vision-language models: at each token-generation step, the method projects every decoder hidden state through the LM head, computes the energy score -LogSumExp of the resulting logit vector, and decodes from the layer with the minimal energy score. The method is motivated by an observed transfer of yes/no response bias from language-prior behavior to visual question answering, and requires no training, contrastive decoding, external models, or image perturbation. Experiments on POPE (MSCOCO/A-OKVQA/GQA), MME, MMVP, and CHAIR, with LLaVA-1.5, InstructBLIP, and mPLUG-Owl2, report accuracy and F1 improvements and yes-ratio-gap reductions on several settings, and the abstract claims consistent improvement over baselines. The paper includes layer-wise energy visualizations and a confidence-calibration analysis.

Significance. If the mechanism holds, the method is an attractive minimal-cost intervention: it is hyperparameter-free, requires a single forward pass, and is compatible with standard autoregressive decoding. Its strengths are the absence of fitted parameters, the evaluation across three architectures and several hallucination benchmarks, and the clear analysis of yes-ratio transfer and confidence asymmetry. However, the central causal claim—that the energy score identifies the most reliable layer for decoding—is not yet established, because the energy rule almost always selects the penultimate layer and no fixed-penultimate-layer control is provided. In addition, the paper's own tables contradict the abstract's claim of consistent improvement. Both issues are addressable, so the work is a plausible but currently under-supported contribution.

major comments (4)
  1. [§4.4, §A.8, Fig. 6] The paper reports that the penultimate layer has the lowest energy score and is 'predominantly utilized' for decoding, but no experiment decodes from a fixed penultimate layer or from a randomly selected earlier layer. Without such a control, the accuracy/F1/yes-gap gains in Tables 3–5 could be an early-exit effect of decoding from layer L-1 rather than an effect of the energy-based selection rule. Please add fixed-penultimate, fixed-layer (e.g., layer L-2), and random-layer decoding controls on the same benchmarks, and report the distribution of selected layers per dataset and model, to show that the energy ranking itself has predictive content beyond simply decoding from an earlier layer.
  2. [Abstract, Table 3, Table 5] The claim that the method 'consistently improves accuracy and F1 score on three VQA datasets across three commonly used VLMs' is contradicted by the paper's own results. On POPE-MSCOCO random with LLaVA-1.5, energy-guided decoding yields 87.50 accuracy and 86.22 F1 versus greedy's 89.37 and 89.33, and the yes-ratio gap worsens from 0.37 to 9.30; with InstructBLIP the corresponding numbers are 86.80/85.10/11.40 versus 90.17/89.86/3.03. On MMVP (Table 5), F1 is slightly lower than greedy for both LLaVA-1.5 (67.67 vs 67.90) and InstructBLIP (66.67 vs 68.12). Please qualify the claims to the settings where improvements actually hold, and define the scope of the 'average accuracy improvement 4.82%' (supplementary A.4 reports a different average, 4.37%, for LLaVA-1.5).
  3. [§A.1, Eq. (4), Table 5] The MMVP evaluation uses a subset of 122 image-question pairs that yield a yes/no answer after applying the POPE-like template, and the manuscript corrects two dataset answers without detailed justification. The yes-ratio gap in Eq. (4) assumes the expected yes ratio is 0.5, but nothing guarantees that the selected subset is balanced. Please report the ground-truth yes/no counts for the selected subset, state the exact selection procedure, and analyze the sensitivity of the reported gaps and accuracies to the corrected answers and to alternative subsets, including the full 300-pair MMVP benchmark.
  4. [§3.3, Eq. (3), Fig. 6] The energy score is applied to logits produced from hidden states at different layers, but only the final layer's logits are trained for use with the LM head; intermediate layers may have different logit scales and calibration. The observation that the penultimate layer is almost always chosen raises the possibility that the minimum-energy rule selects a layer by logit scale rather than by representational reliability. To support the OOD/evidence interpretation, please add a variant that normalizes the logits (e.g., by vector norm or temperature) before computing the energy, and compare the selected layers with and without normalization.
minor comments (6)
  1. [Algorithm 1] The variable named 'hidden energy' stores LogSumExp, not the negated energy of Eq. (3), and the code sorts descending while the comment refers to 'highest energy state'; both the naming and the comment should be corrected to make the sign convention unambiguous.
  2. [§4.3] There is a typo, 'accurcy', and a few other grammatical slips; please proofread the experiment section carefully.
  3. [Figure 1] The y-axis label 'Expected Yes Ratio' is misleading because the plots show the observed yes ratio; relabel it to 'Yes Ratio' and note in the caption that the green line is the expected ratio.
  4. [References] Reference [38] is cited as a blog post with no URL; since the method relies heavily on the logit lens, include a full citation with URL and access date.
  5. [§4.2, Table 2] The text states POPE has 27,000 query-answer pairs while Table 2 lists 9,000 pairs per dataset; please clarify that 27,000 is the total across the three datasets (9,000 × 3) so the reader does not read the table as inconsistent.
  6. [Table 6] CHAIR_I increases for both LLaVA-1.5 (5.4 to 9.4) and InstructBLIP (13.3 to 14.4) relative to greedy; since the paper motivates preserving open-ended generation, this regression deserves discussion in the main text rather than only in the experiment section, ideally with a fixed-penultimate-layer comparison to separate early-exit effects from energy-selection effects.

Circularity Check

0 steps flagged · score 2.0 of 10

No circular derivation: energy-guided decoding is an externally evaluated heuristic; yes-bias motivation and missing fixed-layer control affect interpretation, not circularity.

full rationale

The derivation chain is independent at every link. Section 3.3 defines the layer-wise energy score Energy(h^k_t) = -LogSumExp[H(h^k_t)] and selects k* = argmin_k Energy(h^k_t) for decoding; this selection involves no fitted parameters and is not defined in terms of the reported metrics (accuracy, F1, or the yes-ratio gap of Eq. 4). The improvement claims are supported by external benchmark comparisons in Tables 3-5, so the central result is not an input to itself. The paper states in Sec. 4.4 and A.8 that the penultimate layer is 'predominantly utilized' (lowest energy), and no fixed-penultimate-layer control is reported; that is a genuine confounding/ablation gap regarding whether early exiting rather than the energy ranking drives the gains, but it is not a circular reduction because no equation or fitted parameter equates the method's output with its target. One author-overlapping citation (CGD, Ref. [13]) is used only as a related-work baseline and is not load-bearing. The abstract's claim of 'consistently improves accuracy and F1' is contradicted by the MSCOCO rows of Table 3, but inconsistency with one's own table is a correctness issue, not circularity. Overall: no significant circularity.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The method is an empirical heuristic with no formal derivation. It relies on the logit lens assumption, on energy score as reliability measure, and on the transfer of yes-bias from text-only inputs. No invented entities are introduced. The central claim does not depend on fitted constants; the only hand-chosen elements are the prompt, max token count, and the MMVP subset.

free parameters (2)
  • MMVP evaluation subset = 122 of 300 image-question pairs
    The paper selects pairs whose answer starts with a simple yes/no and whose prompt matches POPE/MME, which changes MMVP results.
  • Prompt template and max tokens = 'Is there a {} in the image?', max_tokens=16/64
    Chosen to align with VCD/HALC evaluation; not fitted but affects yes/no measurement.
assumptions (4)
  • domain assumption Hidden states at every layer can be projected through the vocabulary head to produce valid next-token distributions (logit lens)
    Assumed in Section 3.3, citing [38]; not proven for VLMs like mPLUG-Owl2 with a modality-adaptive decoder.
  • ad hoc to paper The layer with minimal energy score yields the most reliable representation
    Stated in Section 3.3; no theoretical justification, and no ablation vs fixed penultimate layer.
  • domain assumption Yes-ratio bias transfers from text-only input to visual input
    Empirical observation in Figure 3; used to motivate the method.
  • domain assumption Balanced yes/no ratio (0.5) is the correct target for the balanced POPE and MMVP tasks
    Equation (4) assumes the expected yes ratio is 0.5 because the datasets are balanced.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Energy-Guided Decoding for Object Hallucination Mitigation." pith.science (2026). https://pith.science/paper/FZRR5JX5

@misc{pith2026250707731,
  author       = {Pith},
  title        = {Pith review of: Energy-Guided Decoding for Object Hallucination Mitigation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FZRR5JX5}},
  note         = {Machine review of arXiv:2507.07731}
}
read the original abstract

Mitigating object hallucination in large vision-language models (LVLMs) is critical to their safe deployment. Existing methods either are restricted to specific decoding methods, or demand sophisticated modifications to visual inputs, or rely on knowledge from external models. In this work, we first reveal the phenomenon that VLMs exhibit significant imbalance in the ``Yes'' ratio ( \ie, the fraction of ``Yes'' answers among the total number of questions) across three different visual question answering (VQA) datasets. Furthermore, we propose an energy-based decoding method, which dynamically selects the hidden states from the layer with minimal energy score. It is simple yet effective in reducing the bias for the yes ratio while boosting performance across three benchmarks (POPE, MME, and MMVP). Our method consistently improves accuracy and F1 score on three VQA datasets across three commonly used VLMs over several baseline methods. The average accuracy improvement is 4.82% compared to greedy decoding. Moreover, the average yes-ratio gap reduction is 8.81%, meaning the proposed method is less biased as shown in Figure 1.

Figures

Figures reproduced from arXiv: 2507.07731 by the authors.

Figure 1
Figure 1. Yes Ratio using Greedy Decoding (left) and Energy￾Guided Decoding (right) across three datasets over three settings. LLaVA-1.5 [32] is employed as the VLM backbone. The optimal yes ratio is 50% (green dashed line). not supported by the visual input. A recent survey [2] cat￾egories the hallucinations in VLMs, in particular, object￾related hallucinations into the following groups: 1) cate￾gory, where the VLM identifie… view at source ↗
Figure 2
Figure 2. An Illustration of Energy-Guided Decoding. We observe that the hidden states from the layer with minimal energy score generates more accurate responses. We also report the confidence of “ Yes” and “ No” measuring by the corresponding token probability. The right barplot shows the overall performance comparison in terms of accuracy and F1 score on GQA dataset with adversarial setting and the VLM backbone is LLaVA-1.5… view at source ↗
Figure 3
Figure 3. Transfer of “Yes” Ratio from non-visual input to visual inputs using greedy decoding. Three settings of POPE-GQA [29] are utilized including random (left column), popular (middle col￾umn), and adversarial (right column). 3.2. Empirical Yes Ratio Transfer The source of hallucination appeared in VLMs can be at￾tributed to (i) the embedded knowledge in the language decoder’s parameters (i.e., the cause of hallucination… view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Kernel Density Estimation for answers with yes (blue) and the ones with no (orange), using hidden states from the last layer for decoding (top row) and the ones with minimum energy for decoding (bottom row), respectively. The confidence is mea￾sured by the predictive p…
Figure 5
Figure 5. Figure 5: Accuracy vs. Confidence for answers with yes (left) and the ones with no (right), using hidden states from the last layer and the ones with minimum energy for decoding, respectively. The GQA dataset with adversarial setting is utilized along with greedy decoding. LLaVA…
Figure 6
Figure 6. Figure 6: Energy Score Distribution with LLaVa-1.5 [32] as the VLM backbone, and adversarial setting is utilized for evaluation. 5. Conclusion and Discussion In this work, we empirically observe a notable bias in terms of “Yes” ratio within VLMs when utilizing the hidden states …
Figure 8
Figure 8. Figure 8: Energy distribution at each layer with LLaVa-1.5 [32] as the VLM backbone. Three datasets including MSCOCO [30], A￾OKVQA [45], and GQA [23] with adversarial setting are utilized along with greedy decoding. LLaVA-1.5 [32] is utilized as the VLM backbone. and specificity…
Figure 7
Figure 7. Figure 7: Transfer of “Yes” ratio from non-visual input to visual inputs using nucleus sampling. Three settings of POPE-GQA [29] are utilized including random (left column), popular (middle col￾umn), and adversarial (right column). LLaVA-1.5 [32] is em￾ployed as the VLM backbone…
Figure 9
Figure 9. Figure 9: Accuracy vs. Confidence for answers with “Yes” (top row) and the ones with “No” (bottom row), using hidden states from the last layer and the ones with minimum energy for decoding, respectively. Three datasets including MSCOCO, A-OKVQA, and GQA with adversarial setting…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

57 extracted references · 31 canonical work pages

  1. [1]

    Openflamingo: An open-source frame- work for training large autoregressive vision-language mod- els

    Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, Jenia Jitsev, Simon Kornblith, Pang Wei Koh, Gabriel Ilharco, Mitchell Wortsman, and Ludwig Schmidt. Openflamingo: An open-source frame- work for training large autoregressive vision-language mod- els. arXiv preprint...

    arXiv 2023

  2. [2]

    Hallucination of multimodal large language models: A survey

    Zechen Bai, Pichao Wang, Tianjun Xiao, Tong He, Zongbo Han, Zheng Zhang, and Mike Zheng Shou. Hallucination of multimodal large language models: A survey. arXiv preprint arXiv:2404.18930, 2024. 1

    arXiv 2024

  3. [3]

    Eliciting latent predictions from transformers with the tuned lens

    Nora Belrose, Zach Furman, Logan Smith, Danny Halawi, Igor Ostrovsky, Lev McKinney, Stella Biderman, and Ja- cob Steinhardt. Eliciting latent predictions from transformers with the tuned lens. arXiv preprint arXiv:2303.08112, 2023. 3

    arXiv 2023

  4. [4]

    Man is to computer program- mer as woman is to homemaker? debiasing word embed- dings, 2016

    Tolga Bolukbasi, Kai-Wei Chang, James Zou, Venkatesh Saligrama, and Adam Kalai. Man is to computer program- mer as woman is to homemaker? debiasing word embed- dings, 2016. 3

    work page 2016

  5. [5]

    Your Classifier Can Be Secretly a Likelihood-Based OOD Detector

    Jirayu Burapacheep and Yixuan Li. Your classifier can be secretly a likelihood-based ood detector. arXiv preprint arXiv:2408.04851, 2024. 4

    work page Pith review arXiv 2024

  6. [6]

    Generic attention- model explainability for interpreting bi-modal and encoder- decoder transformers

    Hila Chefer, Shir Gur, and Lior Wolf. Generic attention- model explainability for interpreting bi-modal and encoder- decoder transformers. In Proceedings of the IEEE/CVF In- ternational Conference on Computer Vision (ICCV) , pages 397–406, 2021. 3

    work page 2021

  7. [7]

    Halc: Object hallucination reduc- tion via adaptive focal-contrast decoding

    Zhaorun Chen, Zhuokai Zhao, Hongyin Luo, Huaxiu Yao, Bo Li, and Jiawei Zhou. Halc: Object hallucination reduc- tion via adaptive focal-contrast decoding. ICML, 2024. 2, 3, 5, 7, 8, 12, 14, 15, 16

    work page 2024

  8. [8]

    Gonzalez, Ion Stoica, and Eric P

    Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhang- hao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yong- hao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, 2023. 5

    2023

Show all 57 references
  1. [9]

    Dola: Decoding by con- trasting layers improves factuality in large language models

    Yung-Sung Chuang, Yujia Xie, Hongyin Luo, Yoon Kim, James Glass, and Pengcheng He. Dola: Decoding by con- trasting layers improves factuality in large language models. ICLR, 2024. 1, 2, 4

    2024

  2. [10]

    Mavor-Parker, Aengus Lynch, Stefan Heimersheim, and Adri `a Garriga-Alonso

    Arthur Conmy, Augustine N. Mavor-Parker, Aengus Lynch, Stefan Heimersheim, and Adri `a Garriga-Alonso. Towards automated circuit discovery for mechanistic interpretability. In Thirty-seventh Conference on Neural Information Pro- cessing Systems, 2023. 3

    2023

  3. [11]

    Instructblip: Towards general- purpose vision-language models with instruction tuning

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general- purpose vision-language models with instruction tuning. arXiv preprint arXiv:2305.06500, 2023. 5, 6, 7, 8, 12, 15

    2023 arXiv

  4. [12]

    An- alyzing transformers in embedding space

    Guy Dar, Mor Geva, Ankit Gupta, and Jonathan Berant. An- alyzing transformers in embedding space. In Proceedings of the 61st Annual Meeting of the Association for Computa- tional Linguistics (Volume 1: Long Papers), 2023. 3, 4

    2023

  5. [13]

    Seeing is believing: Mitigating hallucination in large vision- language models via clip-guided decoding

    Ailin Deng, Zhirui Chen, and Bryan Hooi. Seeing is believing: Mitigating hallucination in large vision- language models via clip-guided decoding. arXiv preprint arXiv:2402.15300, 2024. 2, 3

    2024 arXiv

  6. [14]

    Danny Driess, Fei Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang, Yevgen Chebotar, Pierre Sermanet, Daniel Duck- worth, Sergey Levine, Vincent Vanhoucke, Karol Hausman, Marc Toussai...

    2023 arXiv

  7. [15]

    Mme: A compre- hensive evaluation benchmark for multimodal large language models

    Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. Mme: A compre- hensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306.13394, 2024. 1, 5, 6

    2024 arXiv

  8. [16]

    Transformer feed-forward layers are key-value memories

    Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers are key-value memories. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. Association for Computa- tional Linguistics, 2021. 3, 4

    2021

  9. [17]

    Transformer feed-forward layers build predictions by pro- moting concepts in the vocabulary space

    Mor Geva, Avi Caciularu, Kevin Wang, and Yoav Goldberg. Transformer feed-forward layers build predictions by pro- moting concepts in the vocabulary space. In Proceedings of the 2022 Conference on Empirical Methods in Natural Lan- guage Processing, 2022. 3, 4

    2022

  10. [18]

    Your classifier is secretly an energy based model and you should treat it like one

    Will Grathwohl, Kuan-Chieh Wang, J ¨orn-Henrik Jacobsen, David Duvenaud, Mohammad Norouzi, and Kevin Swersky. Your classifier is secretly an energy based model and you should treat it like one. ICLR, 2020. 4

    2020

  11. [19]

    Sequence transduction with recurrent neural networks

    Alex Graves. Sequence transduction with recurrent neural networks. arXiv:1211.3711, 2012. 3

    2012 arXiv

  12. [20]

    Overthinking the truth: Understanding how lan- guage models process false demonstrations

    Danny Halawi, Jean-Stanislas Denain, and Jacob Stein- hardt. Overthinking the truth: Understanding how lan- guage models process false demonstrations. arXiv preprint arXiv:2307.09476, 2023. 3

    2023 arXiv

  13. [21]

    The curious case of neural text degeneration

    Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. arXiv preprint arXiv:1904.09751, 2020. 3, 5

    1904 arXiv

  14. [22]

    Opera: Alleviating hallucination in multi- modal large language models via over-trust penalty and retrospection-allocation

    Qidong Huang, Xiaoyi Dong, Pan zhang, Bin Wang, Con- ghui He, Jiaqi Wang, Dahua Lin, Weiming Zhang, and Nenghai Yu. Opera: Alleviating hallucination in multi- modal large language models via over-trust penalty and retrospection-allocation. arXiv preprint arXiv:2311.17911 ,

    arXiv

  15. [23]

    Hudson and Christopher D

    Drew A. Hudson and Christopher D. Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. arXiv preprint arXiv:1902.09506, 2019. 2, 4, 5, 6, 12, 13

    1902 arXiv

  16. [24]

    Devils in middle layers of large vision-language models: Interpreting, detecting and mitigat- ing object hallucinations via attention lens

    Zhangqi Jiang, Junkai Chen, Beier Zhu, Tingjin Luo, Yankun Shen, and Xu Yang. Devils in middle layers of large vision-language models: Interpreting, detecting and mitigat- ing object hallucinations via attention lens. arXiv preprint arXiv:2411.16724, 2024. 2, 3

    2024 arXiv

  17. [25]

    Attention is not only a weight: Analyzing trans- formers with vector norms

    Goro Kobayashi, Tatsuki Kuribayashi, Sho Yokoi, and Ken- taro Inui. Attention is not only a weight: Analyzing trans- formers with vector norms. In Proceedings of the 2020 Con- ference on Empirical Methods in Natural Language Process- ing (EMNLP) . Association for Computational...

    2020

  18. [26]

    Mitigating object hal- lucinations in large vision-language models through visual contrastive decoding

    Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shijian Lu, Chunyan Miao, and Lidong Bing. Mitigating object hal- lucinations in large vision-language models through visual contrastive decoding. CVPR, 2024. 1, 2, 3, 5, 6, 7, 8, 12, 13, 14, 15, 16

    2024

  19. [27]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models, 2023

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models, 2023. 5

    2023

  20. [28]

    Contrastive decoding: Open-ended text genera- tion as optimization

    Xiang Lisa Li, Ari Holtzman, Daniel Fried, Percy Liang, Jason Eisner, Tatsunori Hashimoto, Luke Zettlemoyer, and Mike Lewis. Contrastive decoding: Open-ended text genera- tion as optimization. InProceedings of the 61st Annual Meet- ing of the Association for Computational Ling...

  21. [29]

    Evaluating object hallucination in large vision-language models

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. In The 2023 Conference on Empirical Methods in Natural Language Processing , 2023. 1, 4, 5, 6, 12, 13

    2023

  22. [30]

    Lawrence Zitnick, and Piotr Doll ´ar

    Tsung-Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C. Lawrence Zitnick, and Piotr Doll ´ar. Mi- crosoft coco: Common objects in context. arXiv preprint arXiv:1405.0312, 2015. 5, 6, 13

    2015 arXiv

  23. [31]

    Aligning large multi-modal model with robust instruction tuning

    Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang. Aligning large multi-modal model with robust instruction tuning. arXiv preprint arXiv:2306.14565, 2023. 8

    2023 arXiv

  24. [32]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 26296–26306, 2024. 1, 2, 4, 5, 7, 8, 12, 13, 14, 17

    2024

  25. [33]

    Grounding dino: Marrying dino with grounded pre-training for open-set object detection

    Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499, 2023. 3

    2023 arXiv

  26. [34]

    Paying more atten- tion to image: A training-free method for alleviating halluci- nation in lvlms

    Shi Liu, Kecheng Zheng, and Wei Chen. Paying more atten- tion to image: A training-free method for alleviating halluci- nation in lvlms. ECCV, 2024. 2, 3, 5, 8, 12, 14

    2024

  27. [35]

    Energy-based out-of-distribution detection

    Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. Energy-based out-of-distribution detection. In NeurIPS,

  28. [36]

    Deepseek-vl: Towards real-world vision- language understanding

    Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Hao Yang, Yaofeng Sun, Chengqi Deng, Hanwei Xu, Zhenda Xie, and Chong Ruan. Deepseek-vl: Towards real-world vision- language understanding. arXiv preprint arXiv:2403.05525,

    arXiv

  29. [37]

    Locating and editing factual associations in GPT.Ad- vances in Neural Information Processing Systems, 36, 2022

    Kevin Meng, David Bau, Alex Andonian, and Yonatan Be- linkov. Locating and editing factual associations in GPT.Ad- vances in Neural Information Processing Systems, 36, 2022. arXiv:2202.05262. 3

    2022 arXiv

  30. [38]

    interpreting gpt: the logit lens

    nostalgebraist. interpreting gpt: the logit lens. 2024. 3, 4, 8

    2024

  31. [39]

    Chatgpt, 2023

    OpenAI. Chatgpt, 2023. 1

    2023

  32. [40]

    Gpt-4(vision) technical report, 2023

    OpenAI. Gpt-4(vision) technical report, 2023. 1

    2023

  33. [41]

    Clip-dpo: Vision-language models as a source of preference for fixing hallucinations in lvlms

    Yassine Ouali, Adrian Bulat, Brais Martinez, and Georgios Tzimiropoulos. Clip-dpo: Vision-language models as a source of preference for fixing hallucinations in lvlms. arXiv preprint arXiv:2408.10433, 2024. 4

    2024 arXiv

  34. [42]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In International Con...

    2021

  35. [43]

    Linear adversarial concept erasure

    Shauli Ravfogel, Michael Twiton, Yoav Goldberg, and Ryan Cotterell. Linear adversarial concept erasure. arXiv preprint arXiv:2201.12091, 2024. 3

    2024 arXiv

  36. [44]

    Object hallucination in image cap- tioning

    Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. Object hallucination in image cap- tioning. In Empirical Methods in Natural Language Process- ing (EMNLP), 2018. 1, 5, 12

    2018

  37. [45]

    A-okvqa: A benchmark for visual question answering using world knowl- edge

    Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. A-okvqa: A benchmark for visual question answering using world knowl- edge. arXiv preprint arXiv:2206.01718, 2022. 2, 5, 6, 12, 13

    2022 arXiv

  38. [46]

    Metamorph: Multi- modal understanding and generation via instruction tuning

    Shengbang Tong, David Fan, Jiachen Zhu, Yunyang Xiong, Xinlei Chen, Koustuv Sinha, Michael Rabbat, Yann Le- Cun, Saining Xie, and Zhuang Liu. Metamorph: Multi- modal understanding and generation via instruction tuning. arXiv:2412.14164, 2024. 2

    2024 arXiv

  39. [47]

    Eyes wide shut? exploring the visual shortcomings of multimodal llms

    Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal llms. In CVPR, 2024. 1, 3, 4, 5, 6, 12

    2024

  40. [48]

    Llama 2: Open foundation and fine- tuned chat models, 2023

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Al- bert, Amjad Almahairi, Yasmine Babaei, Nikolay Bash- lykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhos- ale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fer- nandes, Jere...

    2023

  41. [49]

    Mitigating hallucinations in large vision-language models with instruction contrastive decoding

    Xintong Wang, Jingheng Pan, Liang Ding, and Chris Bie- mann. Mitigating hallucinations in large vision-language models with instruction contrastive decoding. arXiv preprint arXiv:2403.18715, 2024. 1, 2, 3

    2024 arXiv

  42. [50]

    Extrinsic hallucinations in llms

    Lilian Weng. Extrinsic hallucinations in llms. https://lilianweng.github.io/posts/2024-07-07-hallucination,

    2024

  43. [51]

    Mitigat- ing hallucination in large vision-language models via mod- ular attribution and intervention

    Tianyun Yang, Ziniu Li, Juan Cao, and Chang Xu. Mitigat- ing hallucination in large vision-language models via mod- ular attribution and intervention. In The Thirteenth Interna- tional Conference on Learning Representations, 2025. 2

    2025

  44. [52]

    mplug-owl2: Revolutionizing multi-modal large lan- guage model with modality collaboration, 2023

    Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan, Anwen Hu, Haowei Liu, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. mplug-owl2: Revolutionizing multi-modal large lan- guage model with modality collaboration, 2023. 5, 6, 7, 16

    2023

  45. [53]

    Woodpecker: Hallucination correction for multimodal large language models

    Shukang Yin, Chaoyou Fu, Sirui Zhao, Tong Xu, Hao Wang, Dianbo Sui, Yunhang Shen, Ke Li, Xing Sun, and Enhong Chen. Woodpecker: Hallucination correction for multimodal large language models. arXiv preprint arXiv:2310.16045 ,

    arXiv

  46. [54]

    Llama-adapter: Efficient fine-tuning of language models with zero-init attention

    Renrui Zhang, Jiaming Han, Chris Liu, Peng Gao, Ao- jun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, and Yu Qiao. Llama-adapter: Efficient fine-tuning of language models with zero-init attention. arXiv preprint arXiv:2303.16199, 2023. 1

    2023 arXiv

  47. [55]

    Debiasing large visual language models

    Yi-Fan Zhang, Weichen Yu, Qingsong Wen, Xue Wang, Zhang Zhang, Liang Wang, Rong Jin, and Tieniu Tan. Debiasing large visual language models. arXiv preprint arXiv:2403.05262, 2024. 2

    2024 arXiv

  48. [56]

    Yes” or “No

    Yiyang Zhou, Chenhang Cui, Jaehong Yoon, Linjun Zhang, Zhun Deng, Chelsea Finn, Mohit Bansal, and Huaxiu Yao. Analyzing and mitigating object hallucination in large vision-language models. In ICLR, 2024. 1, 4 Supplementary Material A. Experimental results In this section, we s...

    2024

  49. [2023]

    1, 2, 3, 5, 12, 14, 15, 16

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.