FADE: Mitigating Hallucinations by Reducing Language-Prior Dominance in Large Vision-Language Models

Dongxu Zhang; Fenglai Lin; Kai Tang; Lin William Cong; Shanghang Zhang; Wenya Wang; Yichen Guo; Yiding Sun

Hallucinations in vision-language models arise when feed-forward layers inject language priors that override visual evidence gathered by attention, and attenuating those layers at inference reduces the errors.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-12 10:57 UTC pith:N22IRFYJ

load-bearing objection Clean single-pass FFN attenuation that works, but the language-prior story is still correlational and the gains shrink on stronger models. the 4 major comments →

arxiv 2606.29431 v4 pith:N22IRFYJ submitted 2026-06-28 cs.AI

FADE: Mitigating Hallucinations by Reducing Language-Prior Dominance in Large Vision-Language Models

Yichen Guo , Kai Tang , Fenglai Lin , Yiding Sun , Dongxu Zhang , Wenya Wang , Lin William Cong , Shanghang Zhang This is my paper

classification cs.AI

keywords vision-language modelshallucination mitigationlanguage priorsfeed-forward networksattentiontraining-free decodingresidual streamlogit lens

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Large vision-language models often answer questions about images with fluent text that is simply wrong about what is in the picture. This paper argues the root is not that vision is ignored: attention modules keep gathering visual evidence toward the right answer. Instead, feed-forward modules at a band of mid-to-late layers inject language priors that can push the residual stream the other way, so an intermediate correct prediction drifts into a hallucinated final answer. From that diagnosis the authors propose FADE: at those critical layers, simply scale down the feed-forward contribution by a fixed factor during decoding. The change needs no training, no second forward pass, and almost no extra compute, yet it lowers hallucination rates on standard probes while leaving general perception scores intact across several model families.

Core claim

Across transformer layers of LVLMs, attention consistently aggregates visual evidence toward correct answers, while FFN modules at a critical mid-to-late band are the source of language priors that can override that evidence and produce the observed late-layer prediction drift; therefore attenuating only those FFN outputs reduces language-prior dominance and mitigates hallucination.

What carries the argument

FADE (FFN Attenuation for Decoding): at selected critical layers, replace the residual update h ← h̃ + FFN(h̃) with h ← h̃ + (1−α)·FFN(h̃), a single scalar scale applied only at those layers in one forward pass.

Load-bearing premise

The critical-layer band found by a small differential-logit analysis on one 7B model, and a single attenuation strength, can be transferred by proportional mapping to other architectures and tasks without wiping out useful knowledge stored in the same feed-forward layers.

What would settle it

If, on a held-out set of hallucinated versus correct samples, attenuating the claimed critical FFN layers either fails to reverse the late-layer drop in correct-token probability or produces larger hallucination rates than leaving those layers alone while attenuating earlier or later layers, the central mechanism claim fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

Clean single-pass FFN attenuation that works, but the language-prior story is still correlational and the gains shrink on stronger models.

read the letter

The useful thing here is a cheap, single-forward-pass intervention that is competitive with dual-pass contrastive decoding and attention re-weighting, plus a residual-stream observation that is more specific than the usual “language priors dominate” slogan.

What is actually new is the localization: on LLaVA-1.5-7B they show attention contributions stay positive toward the correct token on both correct and hallucinated samples, while FFN at a mid-to-late band (roughly 16–22) flips sign and is the main divergence point. FADE simply multiplies those FFN outputs by (1-α). That is a cleaner engineering move than VCD’s second noisy pass or VISTA’s steering vectors, and the efficiency table is honest (essentially free). They ship code, configs, and ablations across LLaVA, mPLUG-Owl2, InstructBLIP, InternVL3, and Qwen2.5/3-VL; the tables are readable and the method is easy to re-implement.

The soft spots are real but not fatal. The mechanistic claim rests on 50 POPE-Adversarial samples and a correct-direction metric that already knows the ground-truth token; it is correlational, not a causal isolation of “language prior content” versus any residual update that points the wrong way. Transfer of the critical band is done by proportional mapping and then task-specific retuning of α and layer (POPE likes ~0.6 at L18, CHAIR likes α=1.0 at L20, MME needs α~0.02). Absolute gains are often 1–2 F1 points and shrink further on the stronger modern models, where language priors appear more tightly optimized. None of this invalidates the practical result, but it means the story is still “damping the right residual updates helps” more than a fully settled mechanistic account.

This is for people who ship or evaluate training-free hallucination fixes and for anyone who wants a simple residual-stream baseline. It deserves a serious referee; the contribution is real and usable even if the causal story needs tightening. I would engage with it.

Referee Report

4 major / 0 minor

Summary. The paper argues that LVLM hallucinations arise because attention modules consistently aggregate visual evidence toward correct answers while FFN modules at critical mid-to-late layers inject language priors that can override that evidence, producing prediction drift. Using residual-stream decompositions and a differential logit-lens correct-direction metric on LLaVA-1.5-7B, the authors identify layers 16–22 as the critical band. They propose FADE, a training-free intervention that multiplies FFN outputs by (1−α) at selected layers (Eq. 5), and evaluate it on POPE, CHAIR, MME, MMHal-Bench and related suites across LLaVA-1.5 (7B/13B), mPLUG-Owl2, InstructBLIP, InternVL3-8B and Qwen2.5/3-VL. FADE is frequently best or second-best among training-free baselines, with only ~3% latency overhead.

Significance. If the mechanistic story holds, the paper supplies a more targeted account of language-prior dominance than output-level contrastive decoding and motivates a single-pass, architecture-agnostic fix with clear efficiency advantages (Table 9). The empirical method is practical, code is released, and evaluation spans multiple architectures and six benchmarks—including open-ended MMHal-Bench and next-generation models—which is a genuine strength relative to many training-free hallucination papers. Even if the language-prior attribution is only partially correct, a simple FFN attenuation recipe that preserves general capability would still be useful to practitioners.

major comments (4)

§3.3, Table 1 and Eqs. (3)–(4): The causal claim that FFN modules are the source of language priors rests on a differential logit-lens analysis of only 50 POPE-Adversarial samples of LLaVA-1.5-7B. C^(l) is positive for attention on both correct and wrong samples and negative for FFN on wrong samples, but this is correlational: the metric does not establish that the FFN contribution is language-prior content rather than any residual update that happens to point the wrong way. Without an isolation experiment (e.g., swapping or ablating FFN vs. attention residuals, or intervening on language-only vs. vision-language inputs at the same layers), the mechanistic origin claim that frames the paper remains suggestive rather than demonstrated.
§3.4 and Appendix D: Critical-layer transfer is asserted via proportional mid-to-late mapping (e.g., L18 on 32-layer 7B → L34 on 40-layer 13B), yet optimal layers and α vary substantially by task and architecture—POPE prefers α≈0.5–0.7 at mid-late layers, CHAIR prefers α=1.0 at L20, MME requires α=0.02–0.05 and sometimes early layers (Appendix B, Tables 14–18). This undermines the claim of a stable, architecture-agnostic critical band and makes FADE look closer to per-task hyperparameter search than a single mechanistic intervention. A controlled transfer protocol (fix layers from analysis only, freeze α, report zero-shot transfer) is needed to support the central narrative.
§3.1–3.4 and Limitations: The paper notes that FFNs store factual knowledge (Geva et al., 2021; Meng et al., 2022) yet does not test whether attenuating the same critical FFNs degrades useful non-visual knowledge or closed-book factual recall. Given that MME cognition and open-ended generation are sensitive to α, a load-bearing check is missing: measure knowledge/VQA degradation under the same α/layer settings used for hallucination mitigation. Without this, the claim that FADE “preserves visual evidence while suppressing language priors” is incomplete.
Tables 2–8: Gains are often small (≈0.5–2 F1 on POPE; mixed CHAIR/MME results) and sometimes FADE is only second-best or tied with greedy on stronger models (Qwen2.5/3-VL, InternVL3). The abstract and contribution bullets overstate “effective mitigation” relative to the effect sizes. Please report statistical significance or bootstrap intervals on the main tables and temper claims where improvements are within noise of greedy decoding.

Circularity Check

0 steps flagged

No definitional or constructional circularity: mechanistic analysis motivates FADE; reported gains are measured on external benchmarks independent of the 50-sample decomposition.

full rationale

The paper's derivation chain is observational then interventional, not self-definitional. Prediction drift (Fig. 3) and the differential logit-lens decomposition (Eqs. 3–4, Table 1) on 50 POPE-Adversarial samples of LLaVA-1.5-7B motivate identifying mid-to-late FFN layers as the intervention site; FADE then scales those FFN outputs by (1−α) (Eq. 5). Success is scored on held-out POPE/CHAIR/MME (and further models) metrics that are not algebraic rearrangements of the 50-sample C^(l) statistics, nor of the fitted α. Hyperparameter choice (layer band, α) is ordinary selection on the evaluation suite—selection bias, not a fitted input renamed as a prediction. Citations for residual-stream and logit-lens methodology (Elhage et al., Geva et al., Belrose et al.) are external and not load-bearing uniqueness claims by the same authors. No equation reduces a reported F1/CHAIR/MME number to a constant fixed by construction. The skeptic's concern that the FFN contribution is correlational rather than proven 'language prior' content is a correctness/mechanism issue, not circularity. Score 0 with empty steps is therefore the honest finding.

Axiom & Free-Parameter Ledger

2 free parameters · 4 axioms · 2 invented entities

The central claim rests on the residual-stream decomposition, the validity of differential logit-lens projections, the interpretation of FFN layers as key-value memories storing language priors, and two free hyperparameters (attenuation strength α and the choice of critical layers) that are selected per model and task. No new physical entities are postulated; the ‘critical-layer band’ is an empirical construct fitted on a small diagnostic set.

free parameters (2)

attenuation strength α = 0.6 (POPE LLaVA), 1.0 (CHAIR), 0.02 (MME), etc.
Scalar multiplier (1−α) applied to FFN outputs; chosen by grid search per model and task (0.02–1.0). Directly controls the reported gains.
critical layer indices = layers 16–22 band; typically layer 18 (POPE) or 20 (CHAIR)
Which transformer layers receive attenuation; identified on 50 POPE samples for LLaVA-1.5-7B then proportionally mapped to other models. Choice materially affects F1 and CHAIR scores.

axioms (4)

domain assumption Residual-stream decomposition: each layer’s contribution can be isolated as Attn then FFN additions to the residual (Elhage et al. 2021).
Invoked in §3.1–3.3 to define Δ_Attn and Δ_FFN.
domain assumption Differential logit-lens projections faithfully measure each component’s push toward correct vs incorrect tokens.
Used to construct the correct-direction metric C^(l) in §3.3; validity of logit lens on intermediate layers is assumed rather than proven for these LVLMs.
domain assumption FFN layers function as key-value memories that can store and inject language priors (Geva et al. 2021; Meng et al. 2022).
Underpins the interpretation that negative FFN contributions equal language-prior override.
ad hoc to paper Proportional mid-to-late layer mapping transfers the critical band across architectures of different depth.
Stated in §3.4 and Appendix D without independent causal verification on each model.

invented entities (2)

critical-layer band (layers 16–22 / proportional equivalents) no independent evidence
purpose: Localizes the source of language-prior dominance so that a single-component intervention can be applied.
Defined empirically from the 50-sample analysis; no independent theoretical derivation of the band boundaries.
FADE (FFN Attenuation for Decoding) independent evidence
purpose: Training-free decoding intervention that multiplies selected FFN outputs by (1−α).
The method itself is the paper’s engineering contribution; its efficacy is measured on external benchmarks.

pith-pipeline@v1.1.0-grok45 · 29161 in / 3116 out tokens · 35098 ms · 2026-07-12T10:57:44.413867+00:00 · methodology

0 comments

read the original abstract

Despite the impressive capabilities of Large Vision-Language Models (LVLMs), they remain susceptible to hallucination, generating content inconsistent with the input image. Recent studies attribute this to the dominance of language priors over visual inputs and employ contrastive decoding methods to mitigate this dominance, but the mechanistic origin remains unexplored. We investigate the information flow through each transformer layer and find that attention modules consistently aggregate visual evidence, while FFN modules at critical layers act as the source of language priors. These priors can override visual evidence, causing correct predictions in intermediate layers to drift toward incorrect outputs. Based on this insight, we propose FADE (FFN Attenuation for DEcoding), a training-free method that attenuates FFN outputs to reduce language-prior dominance. Evaluations on POPE, CHAIR, and MME benchmarks across LLaVA-1.5, mPLUG-Owl2, and InstructBLIP show that FADE effectively mitigates hallucinations while preserving inference efficiency.

Figures

Figures reproduced from arXiv: 2606.29431 by Dongxu Zhang, Fenglai Lin, Kai Tang, Lin William Cong, Shanghang Zhang, Wenya Wang, Yichen Guo, Yiding Sun.

**Figure 2.** Figure 2: Overview of our approach. Left: LVLMs suffer from hallucinations where language priors override visual evidence, causing prediction drift from correct to incorrect outputs. Middle: Our mechanistic analysis reveals that attention modules aggregate visual evidence toward correct answers, while FFN modules at critical layers introduce language priors that can override visual evidence. Right: FADE attenuates F… view at source ↗

**Figure 3.** Figure 3: P(Correct Answer) trajectories across lay [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

**Figure 4.** Figure 4: Qualitative comparison of hallucination cor [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗

**Figure 5.** Figure 5: Ablation on POPE. (a)(c) Strength sensitiv [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗

Review history (3 revisions) →

discussion (0)

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

SPARK: Susceptibility-Guided Profiling and Steering of Latent Reasoning States in Large Language Models
cs.AI 2026-07 conditional novelty 6.0

Length-controlled hidden-state susceptibility diagnoses under-activated reasoning in LLMs and guides selective test-time steering that lifts MATH-500 accuracy by roughly 2–3 points.
SeeMe: Mitigating Hallucinations in Large Vision-Language Models through Effective Visual Token Engineering
cs.CV 2026-07 conditional novelty 5.5

Restructuring visual tokens via cross-modal prune–merge–refine consistently lowers hallucination rates on MME, POPE and AMBER across four 7B LVLMs without any training.
Better Starts, Better Ends: Bootstrapped Iterative Self-Reasoning Distillation for Compressed Reasoning
cs.CL 2026-07 conditional novelty 5.0

Warm-starting on-policy self-distillation with a correctness-filtered, prompt-switched SFT bootstrap improves compression and accuracy of reasoning traces across math benchmarks.

Reference graph

Works this paper leans on

7 extracted references · 2 linked inside Pith · cited by 3 Pith papers

[1]

Nora Belrose, Igor Ostrovsky, Lev McKinney, Zach Fur- man, Logan Smith, Danny Halawi, Stella Biderman, and Jacob Steinhardt

Hallucination of multimodal large language models: A survey.Preprint, arXiv:2404.18930. Nora Belrose, Igor Ostrovsky, Lev McKinney, Zach Fur- man, Logan Smith, Danny Halawi, Stella Biderman, and Jacob Steinhardt. 2023. Eliciting latent predic- tions from transformers with the tuned lens.arXiv preprint arXiv:2303.08112. Yen-Chun Chen, Linjie Li, Licheng Yu...

Pith/arXiv arXiv 2023
[2]

Advances in Neural Information Processing Systems, 37:133571–133599

Code: Contrasting self-generated description to combat hallucination in large multi-modal models. Advances in Neural Information Processing Systems, 37:133571–133599. Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shijian Lu, Chunyan Miao, and Lidong Bing
[3]

InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13872–13882

Mitigating object hallucinations in large vision- language models through visual contrastive decod- ing. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13872–13882. Chenxi Li, Yichen Guo, Benfang Qian, Jinhao You, Kai Tang, Yaosong Du, Zonghao Zhang, and Xiande Huang. 2025a. Map: Mitigating hallucinations in lar...

arXiv 2022
[4]

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, and 1 others

Intervene-all-paths: Unified mitigation of lvlm hallucinations across alignment formats.Preprint, arXiv:2511.17254. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, and 1 others. 2021. Learning transferable visual models from natural language supervision. ...

arXiv 2021
[5]

InFindings of the Association for Computational Linguistics: ACL 2024, pages 13088–13110

Aligning large multimodal models with factu- ally augmented rlhf. InFindings of the Association for Computational Linguistics: ACL 2024, pages 13088–13110. Zhongxiang Sun, Xiaoxue Zang, Kai Zheng, Jun Xu, Xiao Zhang, Weijie Yu, Yang Song, and Han Li

2024
[6]

InInternational Conference on Learning Repre- sentations, volume 2025, pages 50250–50279

Redeep: Detecting hallucination in retrieval- augmented generation via mechanistic interpretabil- ity. InInternational Conference on Learning Repre- sentations, volume 2025, pages 50250–50279. Kai Tang, Jinhao You, Xiuqi Ge, Hanze Li, Yichen Guo, and Xiande Huang. 2025. Mitigating hallucinations via inter-layer consistency aggregation in large vision- lan...

Pith/arXiv arXiv 2025
[7]

In Proceedings of the ieee/cvf conference on computer vision and pattern recognition, pages 13040–13051

mplug-owl2: Revolutionizing multi-modal large language model with modality collaboration. In Proceedings of the ieee/cvf conference on computer vision and pattern recognition, pages 13040–13051. Tianyu Yu, Yuan Yao, Haoye Zhang, Taiwen He, Yifeng Han, Ganqu Cui, Jinyi Hu, Zhiyuan Liu, Hai-Tao Zheng, Maosong Sun, and 1 others. 2024. Rlhf-v: Towards trustwo...

arXiv 2024

[1] [1]

Nora Belrose, Igor Ostrovsky, Lev McKinney, Zach Fur- man, Logan Smith, Danny Halawi, Stella Biderman, and Jacob Steinhardt

Hallucination of multimodal large language models: A survey.Preprint, arXiv:2404.18930. Nora Belrose, Igor Ostrovsky, Lev McKinney, Zach Fur- man, Logan Smith, Danny Halawi, Stella Biderman, and Jacob Steinhardt. 2023. Eliciting latent predic- tions from transformers with the tuned lens.arXiv preprint arXiv:2303.08112. Yen-Chun Chen, Linjie Li, Licheng Yu...

Pith/arXiv arXiv 2023

[2] [2]

Advances in Neural Information Processing Systems, 37:133571–133599

Code: Contrasting self-generated description to combat hallucination in large multi-modal models. Advances in Neural Information Processing Systems, 37:133571–133599. Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shijian Lu, Chunyan Miao, and Lidong Bing

[3] [3]

InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13872–13882

Mitigating object hallucinations in large vision- language models through visual contrastive decod- ing. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13872–13882. Chenxi Li, Yichen Guo, Benfang Qian, Jinhao You, Kai Tang, Yaosong Du, Zonghao Zhang, and Xiande Huang. 2025a. Map: Mitigating hallucinations in lar...

arXiv 2022

[4] [4]

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, and 1 others

Intervene-all-paths: Unified mitigation of lvlm hallucinations across alignment formats.Preprint, arXiv:2511.17254. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, and 1 others. 2021. Learning transferable visual models from natural language supervision. ...

arXiv 2021

[5] [5]

InFindings of the Association for Computational Linguistics: ACL 2024, pages 13088–13110

Aligning large multimodal models with factu- ally augmented rlhf. InFindings of the Association for Computational Linguistics: ACL 2024, pages 13088–13110. Zhongxiang Sun, Xiaoxue Zang, Kai Zheng, Jun Xu, Xiao Zhang, Weijie Yu, Yang Song, and Han Li

2024

[6] [6]

InInternational Conference on Learning Repre- sentations, volume 2025, pages 50250–50279

Redeep: Detecting hallucination in retrieval- augmented generation via mechanistic interpretabil- ity. InInternational Conference on Learning Repre- sentations, volume 2025, pages 50250–50279. Kai Tang, Jinhao You, Xiuqi Ge, Hanze Li, Yichen Guo, and Xiande Huang. 2025. Mitigating hallucinations via inter-layer consistency aggregation in large vision- lan...

Pith/arXiv arXiv 2025

[7] [7]

In Proceedings of the ieee/cvf conference on computer vision and pattern recognition, pages 13040–13051

mplug-owl2: Revolutionizing multi-modal large language model with modality collaboration. In Proceedings of the ieee/cvf conference on computer vision and pattern recognition, pages 13040–13051. Tianyu Yu, Yuan Yao, Haoye Zhang, Taiwen He, Yifeng Han, Ganqu Cui, Jinyi Hu, Zhiyuan Liu, Hai-Tao Zheng, Maosong Sun, and 1 others. 2024. Rlhf-v: Towards trustwo...

arXiv 2024