Pith. sign in

REVIEW 5 major objections 6 minor 5 cited by

ASCD: Attention-Steerable Contrastive Decoding for Reducing Hallucination in MLLM

T0 review · 5 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that directly steering attention inside a multimodal LLM—boosting text-centric heads and damping critical visual tokens—reduces object hallucination by up to 38.2 percent without training and while improving VQA accuracy.

desk verdict The paper's experiments are broad and the attention-shift observation is useful, but Algorithm 2's positive-steering step does not implement the claimed mechanism, which undercuts the central narrative. read the letter →

arxiv 2506.14766 v3 pith:GEJ4EWA4 submitted 2025-06-17 cs.CV cs.CL

Yujun Wang , Aniri , Jinhe Bi , Soeren Pirk , Yunpu Ma This is my paper · ORCID
classification cs.CVcs.CL
keywords attentionsteeringcontrastivedecodingmultimodallargelanguagemodelshallucinationmitigationtext-centricheadsobjecttraining-free
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Multimodal large language models often invent objects that are not in the image. The paper claims that the known hallucination-reducing effects of Visual and Instruction Contrastive Decoding (VCD, ICD) come from how those methods redistribute attention, steering it away from visual tokens and toward textual tokens. If that is right, the natural fix is to steer attention directly rather than perturb the image or prompt. The proposed ASCD does exactly that, amplifying a model's own text-centric attention heads while suppressing only the most attended visual tokens, and it reports lower hallucination on POPE, CHAIR, and MMHal-Bench by up to 38.2 percent while preserving or improving accuracy on five standard VQA benchmarks. No extra training is needed, and the added runtime and memory cost is small.

What carries the argument

The central mechanism is the attention-steerable contrastive decoding equation, $p^{\text{final}}_\theta = (1+\alpha)p^{\text{pos-steered}}_\theta - \alpha p^{\text{neg-steered}}_\theta$, where both branches are produced by editing attention matrices instead of logits. Positive steering adds $\alpha_{\text{pos}}|A^{(l)}_h|$ to the attention of a small set of text-centric heads selected offline by voting over a reference image set using the ratio of text attention to visual attention. Negative steering computes per-token head-averaged attention $s(v)=\frac{1}{H}\sum_h A^{(L)}_h(v)$, selects the top-$\kappa_{\text{vis}}$ visual tokens as critical, and subtracts $\alpha_{\text{neg}}|A^{(l)}_h(v)|$ from every attention entry to those tokens in all heads. The text-centric-head selection is the load-bearing component, since the paper argues these heads are stable within a model across prompts, image sets, domain shifts, and generation lengths.

What would settle it

Run VCD or ICD while holding the model's attention distribution artificially fixed; if hallucination still drops, attention redistribution is not the causal mechanism and the motivation for direct attention steering collapses.

Watch

Extended reading notes

Core claim

ASCD's central claim is that attention redistribution is the operative mechanism behind contrastive-decoding hallucination fixes, and that deliberately manipulating attention is sufficient to reproduce and improve on those fixes. The paper first measures that VCD and ICD, and their combination, systematically lower attention on visual tokens and raise it on textual tokens, with stronger perturbations producing larger shifts. It then constructs two steered decoding branches: a positive branch that boosts the attention weights of automatically selected text-centric heads (heads whose text-to-visual attention ratio is consistently high for that model) and a negative branch that down-weights only the top-attended visual tokens. Subtracting the negative branch's logits from the positive branch's logits, with truncation, yields the final distribution. Across five backbones and three decoding schemes, the paper reports the lowest CHAIR scores and highest POPE accuracy among compared methods, plus gains on MMHal-Bench, while the reproduced VCD and ICD degrade on standard VQA benchmarks.

Load-bearing premise

The method assumes that the attention shifts seen with VCD and ICD are the cause of their hallucination reduction, not just a correlated side effect, and that directly editing attention therefore transfers the benefit.

Editorial extensions

If this is right

  • Hallucination mitigation can be treated as an attention-editing problem rather than a logit-subtraction or input-perturbation problem, so new contrastive-decoding variants can be designed as cheap attention edits.
  • One offline profiling pass per model, running a reference image set and voting on high text-to-visual attention heads, is enough to transfer the method across domains because the identified heads are stable within a model.
  • ASCD can be applied at inference time on top of existing MLLMs without retraining, and it works across greedy, nucleus, and beam search, so it could be dropped into deployed systems with negligible latency increase.
  • Unlike VCD and ICD, which the paper reproduces as degrading on MMMU, MM-VET, ScienceQA, TextVQA, and GQA, ASCD is reported to match or improve those benchmarks, meaning hallucination reduction and general VQA ability need not trade off.
  • The gains persist at larger scale (LLaVA-1.5 13B) and on a newer architecture (Qwen2.5-VL-Instruct), suggesting the mechanism is not tied to one training recipe.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the causal reading of Figure 1 is correct, then any decoding intervention that changes the visual-to-text attention balance, not just contrastive decoding, should shift hallucination rates, which could turn attention balance into a general diagnostic for hallucination propensity.
  • A testable extension would be to profile text-centric heads on a model whose hallucination rate is artificially raised or lowered by training interventions; if the same heads stay text-centric, head identity is a fixed architectural bias, while if they change, profiling may need to be repeated after fine-tuning.
  • Attention steering could combine with training-time methods: using ASCD at inference may reduce the amount of hallucination-specific data or reward modeling needed during training, since the mechanism already counteracts over-commitment to textual priors.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The manuscript proposes ASCD, a training-free decoding-time method intended to reduce object hallucination in multimodal large language models. The method first identifies "text-centric" attention heads offline from a reference set, then at generation time runs two contrastive branches: a positive-steering branch that is intended to amplify visual attention by modifying text-centric heads, and a negative-steering branch that down-weights the most-attended visual tokens. The two logit distributions are combined with contrastive decoding and truncation (Eq. 2, Algorithm 2). The paper reports that ASCD reduces hallucination on POPE, CHAIR, and MMHal-Bench by up to 38.2% across five backbones and three decoding schemes while preserving or improving accuracy on standard VQA benchmarks.

Significance. If the method worked as described, it would be a practically attractive training-free intervention: it is cheap, model-agnostic in principle, backed by a released code repository, and evaluated more broadly than many prior contrastive-decoding papers (five backbones, three decoding schemes, POPE/CHAIR/MMHal-Bench, and five VQA benchmarks). The runtime/memory measurements and the out-of-domain head-stability analysis are useful additions. However, the paper's central mechanistic claim is not supported by the supplied algorithm, and the headline CHAIR gains are obtained after tuning hyperparameters on CHAIR. The empirical evaluation therefore needs substantial revision before the central claim can be accepted.

major comments (5)
  1. [Methodology, Algorithm 2 (Positive Steering Pass)] Step 1 of Algorithm 2 specifies the positive-steering update as A(l)_h <- A(l)_h + alpha_pos |A(l)_h|, followed by normalization. If A(l)_h is the post-softmax attention matrix, all entries are nonnegative and the update is exactly a row-wise multiplication by (1+alpha_pos), which is cancelled by row normalization; the positive branch is then identical to the unsteered branch, and Eq. (2) reduces to (1+alpha)p_orig - alpha p_neg. If A(l)_h instead denotes pre-softmax scores, the update is not row-normalization invariant, but nothing in the update specifically increases attention to visual tokens; for nonnegative scores it again scales all entries by the same factor, and for negative scores it moves them by alpha|A| toward zero, which is not the 'increase vision / decrease text' behavior claimed in Fig. 2 and Section 4.2. Because both branches in Algorithm 2 either leave attention unchanged or suppress visual tokens, the contrastive subtraction's success is unexplained by the stated attention-steering mechanism. The authors should clarify which quantity A(l)_h denotes and provide a version of the positive steer that provably increases relative visual attention, together with an attention-measurement verification.
  2. [Appendix (Hyperparameters); Table 1] The appendix states that the hyperparameters were chosen 'to obtain the best result on CHAIR,' and Table 1 reports the CHAIR improvements, including the abstract's headline 'up to 38.2%.' This means the main hallucination-reduction claim is evaluated on the same benchmark used for hyperparameter selection, so the CHAIR numbers are overoptimistic and are not a valid test of the method. In addition, no standard errors or confidence intervals are reported for nucleus-sampling runs or for CHAIR, whose random image sampling introduces variance; the POPE/CHAIR tables as printed do not allow the reader to assess whether the gains are significant.
  3. [Experiments, Table 1] The reproduced VCD and ICD baselines are inconsistent with published results: in multiple rows VCD increases CHAIR relative to the original model (e.g., LLaVA-1.5 greedy: CHAIRs 56.8 vs 53.2; LLaVA-NeXT greedy: 37.2 vs 31.6), whereas the original VCD paper reports reductions. This suggests the reproduction may not match the reference implementations, and it weakens the claim that ASCD outperforms prior contrastive-decoding methods. The authors should report the original published numbers, state which exact repositories and versions were used, and release or verify the baseline code.
  4. [Figure 1 and Section 'Proactive Steering of Attention'] The motivating claim that VCD and ICD reduce hallucination 'largely attributable' to attention redistribution is inferred from correlations in Figure 1; no controlled experiment separates the attention shift from the logit-subtraction step. Because the entire design of ASCD depends on this causal reading, the authors should perform an intervention test (e.g., applying the observed attention redistribution without logit subtraction, or steering attention in the opposite direction) to establish that the attention shift is causally responsible. As it stands, Figure 1 only shows a correlation.
  5. [Experiments, Figure 6] MMHal-Bench is one of the three hallucination benchmarks used to support the abstract's claim, but the results are presented only as radar charts with no numeric values, no aggregate scores, and no error estimates. The reader cannot verify that ASCD's apparent advantage is real or compare it with the baselines quantitatively. Numeric results should be added for every sub-dimension and for the overall score.
minor comments (6)
  1. [Preliminaries, Eq. (2)] Equation (2) is written with p, but the text calls these logits; please clarify whether the contrast is applied to probabilities, log-probabilities, or logits, since this affects the truncation step in Algorithm 2.
  2. [Figure 3] The panel labeling in Figure 3 is confusing: the text refers to '(a) center,' '(b) bottom left,' and '(c) bottom right,' but the layout is not self-explanatory; please add explicit panel order and a color scale for the heatmaps.
  3. [Appendix, Table 6] The hyperparameter ablation reports only POPE; since CHAIR was the tuning target, the authors should also include CHAIR results or state why CHAIR was not swept on a held-out split.
  4. [Appendix, Table 7] Table 7 reports dashes for ASCD with SDPA and FlashAttention, yet the text claims overhead is measured under three attention backends; please clarify that the latency/memory comparison for ASCD is for the eager backend only.
  5. [Appendix, Eq. (4) and Algorithm 1] The definitions of textAttn and visAttn should specify whether attention is averaged over layers, heads, or tokens, and whether image patch tokens include all vision tokens; this is needed to reproduce the head-selection procedure.
  6. [Figure 8] The claim that the ASCD caption contains only entities that are actually present is based on a single qualitative example; please label it as illustrative rather than as quantitative evidence.

Circularity Check

2 steps flagged · score 6.0 of 10

Positive-steering update cancels to original logits by construction, and the headline CHAIR reduction is tuned on CHAIR, so the central claim is partly fitted and the positive branch does not implement the stated mechanism.

  1. fitted input called prediction [Appendix, Hyperparameters; Abstract; Table 1]
    "The experiments are conducted with the following hyperparameter settings to obtain the best result on CHAIR: κvis = 0.1, α_neg = 1.0, α = 1.0, and β = 0.1."

    The CHAIR reductions in Table 1, including the 38.2% CHAIRs drop for LLaVA-NeXT 7B beam search (34.0 -> 21.0), are produced with settings explicitly selected 'to obtain the best result on CHAIR.' The target metric is therefore both the selection objective and the reported outcome: the headline CHAIR claim is an in-sample optimum rather than an out-of-sample prediction. The same tuned settings are carried into the other reported benchmarks, so the abstract's 'up to 38.2%' number is not a held-out result.

  2. other [Algorithm 2, Step 1 (Positive Steering Pass); Eq. (2); Section 'Proactive Steering of Attention']
    "Algorithm 2: 'A(l)_h ← A(l)_h + α_pos|A(l)_h|; Normalize A(l) and continue.' Text: 'Conversely, the positive-steered branch increases attention to visual tokens or downgrades textual tokens, making the model more sensitive to the actual (albeit unexpected) color in the image.'"

    For a nonnegative attention matrix with rows summing to one, |A(l)_h| = A(l)_h, so the update is (1+α_pos)A(l)_h; row normalization then returns exactly A(l)_h. Hence the 'positive-steered' branch produces p_pos_θ identical to the original unsteered logits. Eq. (2) collapses to (1+α) original − α negative-steered, and the claimed 'increase vision attention / decrease text attention' in the positive branch is not implemented by the algorithm. Under the alternative pre-softmax reading, the update sharpens the already text-dominant entries of text-centric heads rather than promoting visual tokens. The positive-steering component of the central mechanism therefore reduces by construction to a no-op.

full rationale

The paper's contrastive formula (Eq. 2) is not circular in itself: it explicitly combines two steered logit distributions. The POPE, MMHal-Bench, and VQA results are external benchmarks, and the text-centric-head stability analysis (Fig. 3, Table 3) is an independent empirical measurement. Self-citations such as [8] and [9] appear only in related work and are not load-bearing. However, two load-bearing reductions prevent a clean non-circular verdict. First, the Appendix states that the reported hyperparameters were chosen 'to obtain the best result on CHAIR,' so the CHAIR component of the headline 'up to 38.2%' is a selected optimum rather than a prediction. Second, the positive-steering update in Algorithm 2 is invariant under row normalization for post-softmax attention weights, making p_pos_steered equal to the original logits by construction; under a pre-softmax reading it sharpens text-centric entries instead of up-weighting visual tokens. Eq. (2) therefore does not implement the claimed positive-steering mechanism. The Figure 1 causal premise (attention shift causes hallucination reduction) is an unsupported assumption rather than a circular step, so it is noted here but not scored.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

The method depends on several empirical assumptions (head stability, token criticality, causal role of attention shifts) and six tunable hyperparameters, five of which are set to optimize CHAIR. No new physical entities are introduced.

free parameters (6)
  • alpha_pos (positive steering strength) = 0.6 for LLaVA-1.5, 0.7 for LLaVA-NeXT, 0.8 for Phi2-SigLIP
    Tuned to obtain the best result on CHAIR (Appendix Hyperparameters).
  • alpha_neg (negative steering strength) = 1.0
    Tuned to obtain the best result on CHAIR (Appendix Hyperparameters).
  • alpha (contrastive weight) = 1.0
    Tuned to obtain the best result on CHAIR (Appendix Hyperparameters).
  • beta (truncation threshold) = 0.1
    Tuned to obtain the best result on CHAIR (Appendix Hyperparameters).
  • kappa_tch (number of text-centric heads) = 32 for LLaVA-1.5 and LLaVA-NeXT, 128 for Phi2-SigLIP
    Per-model values selected to optimize CHAIR performance.
  • kappa_vis (fraction of critical visual tokens) = 0.1
    Tuned to obtain the best result on CHAIR (Appendix Hyperparameters).
assumptions (4)
  • domain assumption The attention redistribution observed for VCD and ICD (lower visual attention, higher text attention) is the causal mechanism behind their hallucination reduction.
    Used in Section 'Proactive Steering of Attention' and Figure 1 to motivate directly steering attention; if correlation is spurious, the method's rationale weakens.
  • domain assumption Text-centric heads, identified on a 500-image reference set, remain stable within a model across images, generation lengths, and domains (JSD <= 0.11).
    Used in Algorithm 1 and Section 'Text-centric Heads are Model-specific'; if this stability fails for new domains, positive steering is mis-targeted.
  • domain assumption The top-attended visual tokens, as measured by s(v) in Eq. 3, are the right tokens to suppress to create an effective negative branch.
    Used in Algorithm 2 negative steering; no theoretical justification for why suppressing the most-attended tokens yields a useful contrast.
  • standard math Standard multi-head attention with softmax normalization is a valid substrate for the additive steering modifications in Algorithm 2.
    Assumes modifying attention scores before normalization preserves a valid distribution and that the two-pass logit combination via Eq. 2 is legitimate.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ASCD: Attention-Steerable Contrastive Decoding for Reducing Hallucination in MLLM." pith.science (2026). https://pith.science/paper/GEJ4EWA4

@misc{pith2026250614766,
  author       = {Pith},
  title        = {Pith review of: ASCD: Attention-Steerable Contrastive Decoding for Reducing Hallucination in MLLM},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GEJ4EWA4}},
  note         = {Machine review of arXiv:2506.14766}
}
read the original abstract

Multimodal large language models (MLLMs) frequently hallucinate by over-committing to spurious visual cues. Prior remedies-Visual and Instruction Contrastive Decoding (VCD, ICD)-mitigate this issue, yet the mechanism remains opaque. We first empirically show that their improvements systematically coincide with redistributions of cross-modal attention. Building on this insight, we propose Attention-Steerable Contrastive Decoding (ASCD), which directly steers the attention scores during decoding. ASCD combines (i) positive steering, which amplifies automatically mined text-centric heads-stable within a model and robust across domains-with (ii) negative steering, which dampens on-the-fly identified critical visual tokens. The method incurs negligible runtime and memory overhead and requires no additional training. Across five MLLM backbones and three decoding schemes, ASCD reduces hallucination on POPE, CHAIR, and MMHal-Bench by up to 38.2 percent while improving accuracy on standard VQA benchmarks, including MMMU, MM-VET, ScienceQA, TextVQA, and GQA. These results position attention steering as a simple, model-agnostic, and principled route to safer, more faithful multimodal generation.

Figures

Figures reproduced from arXiv: 2506.14766 by the authors.

Figure 1
Figure 1. Impact of VCD and ICD on attention distribution. [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. A motivating example of proactive attention steering in a visually ambiguous scenario. Top: Conversation context in [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 4
Figure 4. Illustration of positive and negative steering. Left: [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗
Figures from the paper (4 more)
Figure 5
Figure 5. Figure 5: Comparative effectiveness of selective attention [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Radar charts of MMHal-Bench results. Each axis [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: Text-centric-head frequency maps. Each pixel de [PITH_FULL_IMAGE:figures/full_fig_p011_7.png]
Figure 8
Figure 8. Figure 8: Original LLaVA-1.5 drifts completely, describing a [PITH_FULL_IMAGE:figures/full_fig_p013_8.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Dual Latent Memory for Visual Multi-agent System

    cs.AI 2026-01 conditional novelty 6.0 of 10

    L2-VMAS replaces text-based inter-agent communication in visual multi-agent systems with decoupled latent perception and thinking memories, improving accuracy by 2.7–5.4% and cutting token use by 21.3–44.8%.

  2. TECP: Token-Entropy Conformal Prediction for LLMs

    cs.CL 2025-08 reject novelty 4.0 of 10

    TECP applies split conformal prediction with token-entropy nonconformity scores to LLM question answering and reports reliable coverage, but its implementation requires the token probabilities it claims to avoid.

  3. FADE: Adversarial Concept Erasure in Flow Models

    cs.CV 2025-07 reject novelty 4.0 of 10

    FADE combines adversarial training with trajectory preservation to erase concepts from diffusion models, reporting state-of-the-art erasure on Stable Diffusion benchmarks, but the evidence is incomplete and the theore...

  4. Conformal Sets in Multiple-Choice Question Answering under Black-Box Settings with Provable Coverage Guarantees

    cs.CL 2025-08 conditional novelty 3.0 of 10

    Repeatedly sampling an LLM and using the entropy of answer frequencies yields conformal prediction sets for multiple-choice questions with empirical miscoverage near the target, and AUROC comparable to logit-based scores.

  5. Conformal P-Value in Multiple-Choice Question Answering Tasks with Provable Risk Control

    cs.CL 2025-08 reject novelty 2.0 of 10

    A p-value reformulation of split conformal prediction for LLM multiple-choice QA achieves nominal miscoverage control on MMLU and MMLU-Pro.

Reference graph

Works this paper leans on

53 extracted references · 6 canonical work pages · cited by 5 Pith papers

  1. [1]

    D.; Goswami, A.; Gunasekar, S.; Haider, E.; Hao, J.; Hewett, R

    Abdin, M.; Aneja, J.; Awadalla, H.; Awadallah, A.; Awan,A.A.;Bach,N.;Bahree,A.;Bakhtiari,A.;Bao, J.; Behl, H.; Benhaim, A.; Bilenko, M.; Bjorck, J.; Bubeck, S.; Cai, M.; Cai, Q.; Chaudhary, V.; Chen, D.; Chen, D.; Chen, W.; Chen, Y.-C.; Chen, Y.-L.; Cheng, H.; Chopra, P.; Dai, X.; Dixon, M.; Eldan, R.; Fragoso, V.; Gao, J.; Gao, M.; Gao, M.; Garg, A.; Gio...

    arXiv 2024

  2. [2]

    L.; Batra, D.; and Parikh, D

    Agrawal, A.; Lu, J.; Antol, S.; Mitchell, M.; Zitnick, C. L.; Batra, D.; and Parikh, D. 2016. VQA: Visual Question Answering. arXiv:1505.00468

    arXiv 2016

  3. [3]

    Alayrac, J.-B.; Donahue, J.; Luc, P.; Miech, A.; Barr, I.; Hasson, Y.; Lenc, K.; Mensch, A.; Millican, K.; Reynolds, M.; Ring, R.; Rutherford, E.; Cabi, S.; Han, T.; Gong, Z.; Samangooei, S.; Monteiro, M.; Menick, J.; Borgeaud, S.; Brock, A.; Nematzadeh, A.; Shar- ifzadeh, S.; Binkowski, M.; Barreira, R.; Vinyals, O.; Zisserman, A.; and Simonyan, K. 2022....

    arXiv 2022

  4. [4]

    Anderson, P.; He, X.; Buehler, C.; Teney, D.; Johnson, M.; Gould, S.; and Zhang, L. 2018. Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering. arXiv:1707.07998

    arXiv 2018

  5. [5]

    Bai, S.; Chen, K.; Liu, X.; Wang, J.; Ge, W.; Song, S.; Dang,K.;Wang,P.;Wang,S.;Tang,J.;Zhong,H.;Zhu, Y.; Yang, M.; Li, Z.; Wan, J.; Wang, P.; Ding, W.; Fu, Z.;Xu,Y.;Ye,J.;Zhang,X.;Xie,T.;Cheng,Z.;Zhang, H.; Yang, Z.; Xu, H.; and Lin, J. 2025. Qwen2.5-VL Technical Report. arXiv:2502.13923

    arXiv 2025

  6. [6]

    Bai, Z.; Wang, P.; Xiao, T.; He, T.; Han, Z.; Zhang, Z.; and Shou, M. Z. 2024. Hallucination of Multimodal LargeLanguageModels:ASurvey. arXiv:2404.18930

    arXiv 2024

  7. [7]

    Ben-Kish, A.; Yanuka, M.; Alper, M.; Giryes, R.; and Averbuch-Elor, H. 2024. Mitigating Open-Vocabulary Caption Hallucinations. arXiv:2312.03631

    arXiv 2024

  8. [8]

    Bi, J.; Wang, Y.; Chen, H.; Xiao, X.; Hecker, A.; Tresp, V.; and Ma, Y. 2024. Visual Instruction Tuning with 500x Fewer Parameters through Modal- ity Linear Representation-Steering.arXiv preprint arXiv:2412.12359

    arXiv 2024

Show all 53 references
  1. [9]

    Bi, J.; Wang, Y.; Yan, D.; Xiao, X.; Hecker, A.; Tresp, V.; and Ma, Y. 2025. Prism: Self-pruning intrinsic selection method for training-free multimodal data se- lection.arXiv preprint arXiv:2502.12119

    2025 arXiv

  2. [10]

    Brown, T. B.; Mann, B.; Ryder, N.; Subbiah, M.; Ka- plan, J.; Dhariwal, P.; Neelakantan, A.; Shyam, P.; Sastry, G.; Askell, A.; Agarwal, S.; Herbert-Voss, A.; Krueger, G.; Henighan, T.; Child, R.; Ramesh, A.; Ziegler, D. M.; Wu, J.; Winter, C.; Hesse, C.; Chen, M.; Sigler, E.;...

    2020 arXiv

  3. [11]

    Chen, H.; Li, H.; Zhang, Y.; Bi, J.; Zhang, G.; Zhang, Y.; Torr, P.; Gu, J.; Krompass, D.; and Tresp, V. 2025. FedBiP: Heterogeneous One-Shot Federated Learning with Personalized Latent Diffusion Models. InPro- ceedings of the Computer Vision and Pattern Recogni- tion Conferen...

    2025

  4. [12]

    PaLI: A Jointly-Scaled Multilingual Language-Image Model

    Chen, X.; Wang, X.; Changpinyo, S.; Piergiovanni,A.; Padlewski, P.; Salz, D.; Goodman, S.; Grycner, A.; Mustafa, B.; Beyer, L.; Kolesnikov, A.; Puigcerver, J.; Ding, N.; Rong, K.; Akbari, H.; Mishra, G.; Xue, L.; Thapliyal, A.; Bradbury, J.; Kuo, W.; Seyedhosseini, M.;Jia,C.;A...

    2023 arXiv

  5. [13]

    Chen,Z.;Wu,J.;Wang,W.;Su,W.;Chen,G.;Xing,S.; Zhong, M.; Zhang, Q.; Zhu, X.; Lu, L.; Li, B.; Luo, P.; Lu, T.; Qiao, Y.; and Dai, J. 2024. InternVL: Scaling upVisionFoundationModelsandAligningforGeneric Visual-Linguistic Tasks. arXiv:2312.14238

    2024 arXiv

  6. [14]

    Chen, Z.; Zhu, Y.; Zhan, Y.; Li, Z.; Zhao, C.; Wang, J.; and Tang, M. 2023. Mitigating Hallucination in Visual Language Models with Visual Supervision. arXiv:2311.16479

    2023 arXiv

  7. [15]

    arXiv:2305.06500

    Dai,W.;Li,J.;Li,D.;Tiong,A.M.H.;Zhao,J.;Wang, W.;Li,B.;Fung,P.;andHoi,S.2023.InstructBLIP:To- wards General-purpose Vision-Language Models with Instruction Tuning. arXiv:2305.06500

    2023 arXiv

  8. [16]

    Visual Dialog

    Das, A.; Kottur, S.; Gupta, K.; Singh, A.; Yadav, D.; Moura,J.M.F.;Parikh,D.;andBatra,D.2017. Visual Dialog. arXiv:1611.08669

    2017 arXiv

  9. [17]

    Deng, A.; Chen, Z.; and Hooi, B. 2024. See- ing is Believing: Mitigating Hallucination in Large Vision-Language Models via CLIP-Guided Decoding. arXiv:2402.15300

    2024 arXiv

  10. [18]

    Devlin, J.; Chang, M.-W.; Lee, K.; and Toutanova, K. 2019. BERT: Pre-training of Deep Bidirec- tional Transformers for Language Understanding. arXiv:1810.04805

    2019 arXiv

  11. [19]

    Huang, Q.; Dong, X.; Zhang, P.; Wang, B.; He, C.; Wang, J.; Lin, D.; Zhang, W.; and Yu, N. 2024. OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation. arXiv:2311.17911

    2024 arXiv

  12. [20]

    Huang, W.; Liu, H.; Guo, M.; and Gong, N. Z. 2024. Visual Hallucinations of Multi-modal Large Language Models. arXiv:2402.14683

    2024 arXiv

  13. [21]

    arXiv:1902.09506

    Hudson,D.A.;andManning,C.D.2019.GQA:ANew Dataset for Real-World Visual Reasoning and Compo- sitional Question Answering. arXiv:1902.09506

    2019 arXiv

  14. [22]

    Huo, F.; Xu, W.; Zhang, Z.; Wang, H.; Chen, Z.; and Zhao, P. 2024. Self-Introspective Decoding: Alleviat- ingHallucinationsforLargeVision-LanguageModels. arXiv:2408.02032

    2024 arXiv

  15. [23]

    FaithScore: Fine-grained Evaluations of Hallucinations in Large Vision-Language Models

    Jing,L.;Li,R.;Chen,Y.;andDu,X.2024. FaithScore: Fine-grained Evaluations of Hallucinations in Large Vision-Language Models. arXiv:2311.01477

    2024 arXiv

  16. [24]

    MitigatingObjectHallucinationsin Large Vision-Language Models through Visual Con- trastive Decoding

    Leng,S.;Zhang,H.;Chen,G.;Li,X.;Lu,S.;Miao,C.; andBing,L.2023. MitigatingObjectHallucinationsin Large Vision-Language Models through Visual Con- trastive Decoding. arXiv:2311.16922

    2023 arXiv

  17. [25]

    Li, J.; Li, D.; Savarese, S.; and Hoi, S. 2023. BLIP- 2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models. arXiv:2301.12597

    2023 arXiv

  18. [26]

    X.; and Wen, J.-R

    Li, Y.; Du, Y.; Zhou, K.; Wang, J.; Zhao, W. X.; and Wen, J.-R. 2023. Evaluating Object Hallucination in Large Vision-Language Models. arXiv:2305.10355

    2023 arXiv

  19. [27]

    Liu, H.; Li, C.; Li, Y.; and Lee, Y. J. 2023. Improved Baselines with Visual Instruction Tuning

    2023

  20. [28]

    Liu, H.; Li, C.; Li, Y.; Li, B.; Zhang, Y.; Shen, S.; and Lee, Y. J. 2024. LLaVA-NeXT: Improved reasoning, OCR, and world knowledge

    2024

  21. [29]

    Liu, H.; Li, C.; Wu, Q.; and Lee, Y. J. 2023. Visual Instruction Tuning

    2023

  22. [30]

    Liu, H.; Xue, W.; Chen, Y.; Chen, D.; Zhao, X.; Wang, K.; Hou, L.; Li, R.; and Peng, W. 2024. A Survey on Hallucination in Large Vision-Language Models. arXiv:2402.00253

    2024 arXiv

  23. [31]

    Liu, S.; Zheng, K.; and Chen, W. 2024. Paying More Attention to Image: A Training-Free Method for Alle- viating Hallucination in LVLMs. arXiv:2407.21771

    2024 arXiv

  24. [32]

    NegativeObjectPresenceEvaluation(NOPE) to Measure Object Hallucination in Vision-Language Models

    Lovenia,H.;Dai,W.;Cahyawijaya,S.;Ji,Z.;andFung, P.2024. NegativeObjectPresenceEvaluation(NOPE) to Measure Object Hallucination in Vision-Language Models. arXiv:2310.05338

    2024 arXiv

  25. [33]

    arXiv:2209.09513

    Lu,P.;Mishra,S.;Xia,T.;Qiu,L.;Chang,K.-W.;Zhu, S.-C.;Tafjord,O.;Clark,P.;andKalyan,A.2022.Learn toExplain:MultimodalReasoningviaThoughtChains for Science Question Answering. arXiv:2209.09513

    2022 arXiv

  26. [34]

    Raffel, C.; Shazeer, N.; Roberts, A.; Lee, K.; Narang, S.; Matena, M.; Zhou, Y.; Li, W.; and Liu, P. J. 2023. Exploring the Limits of Transfer Learning with a Uni- fied Text-to-Text Transformer. arXiv:1910.10683

    2023 arXiv

  27. [35]

    Rohrbach,A.;Hendricks,L.A.;Burns,K.;Darrell,T.; and Saenko, K. 2019. Object Hallucination in Image Captioning. arXiv:1809.02156

    2019 arXiv

  28. [36]

    BackdoorCleaningwith- out External Guidance in MLLM Fine-tuning.arXiv preprint arXiv:2505.16916

    Rong, X.; Huang, W.; Liang, J.; Bi, J.; Xiao, X.; Li, Y.;Du,B.;andYe,M.2025. BackdoorCleaningwith- out External Guidance in MLLM Fine-tuning.arXiv preprint arXiv:2505.16916

    2025 arXiv

  29. [37]

    Towards VQA Models That Can Read

    Singh,A.;Natarajan,V.;Shah,M.;Jiang,Y.;Chen,X.; Batra,D.;Parikh,D.;andRohrbach,M.2019. Towards VQA Models That Can Read. arXiv:1904.08920

    2019 arXiv

  30. [38]

    Learningtosummarizefromhumanfeedback

    Stiennon,N.;Ouyang,L.;Wu,J.;Ziegler,D.M.;Lowe, R.;Voss, C.;Radford,A.; Amodei,D.;andChristiano, P.2022. Learningtosummarizefromhumanfeedback. arXiv:2009.01325

    2022 arXiv

  31. [39]

    Sun,Z.;Shen,S.;Cao,S.;Liu,H.;Li,C.;Shen,Y.;Gan, C.;Gui,L.-Y.;Wang,Y.-X.;Yang,Y.;Keutzer,K.;and Darrell, T. 2023. Aligning Large Multimodal Models with Factually Augmented RLHF. arXiv:2309.14525

    2023 arXiv

  32. [40]

    Touvron, H.; Lavril, T.; Izacard, G.; Martinet, X.; Lachaux, M.-A.; Lacroix, T.; Rozière, B.; Goyal, N.; Hambro,E.;Azhar,F.;Rodriguez,A.;Joulin,A.;Grave, E.; and Lample, G. 2023. LLaMA: Open and Efficient Foundation Language Models. arXiv:2302.13971

    2023 arXiv

  33. [41]

    Wang, X.; Pan, J.; Ding, L.; and Biemann, C

  34. [42]

    Xiao,Z.;Wang,Z.;Ma,W.;Zhang,Y.;Shen,W.;Wang, Y.; Gong, L.; and Liu, Z. 2025. Mitigating Posterior Salience Attenuation in Long-Context LLMs with Po- sitional Contrastive Decoding. arXiv:2506.08371

    2025 arXiv

  35. [43]

    Yang, A.; Yang, B.; Hui, B.; Zheng, B.; Yu, B.; Zhou, C.; Li, C.; Li, C.; Liu, D.; Huang, F.; Dong, G.; Wei, H.; Lin, H.; Tang, J.; Wang, J.; Yang, J.; Tu, J.; Zhang, J.; Ma, J.; Yang, J.; Xu, J.; Zhou, J.; Bai, J.; He, J.; Lin, J.; Dang, K.; Lu, K.; Chen, K.; Yang, K.; Li, M....

    2024 arXiv

  36. [44]

    Yu, W.; Yang, Z.; Li, L.; Wang, J.; Lin, K.; Liu, Z.; Wang, X.; and Wang, L. 2024. MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities. arXiv:2308.02490

    2024 arXiv

  37. [45]

    MMMU: A Massive Multi-discipline Multimodal Un- derstandingandReasoningBenchmarkforExpertAGI

    Yue, X.; Ni, Y.; Zhang, K.; Zheng, T.; Liu, R.; Zhang, G.; Stevens, S.; Jiang, D.; Ren, W.; Sun, Y.; Wei, C.; Yu,B.;Yuan,R.;Sun,R.;Yin,M.;Zheng,B.;Yang,Z.; Liu,Y.;Huang,W.;Sun,H.;Su,Y.;andChen,W.2024. MMMU: A Massive Multi-discipline Multimodal Un- derstandingandReasoningBench...

    2024 arXiv

  38. [46]

    HowLanguageModelHallucinationsCan Snowball

    Zhang, M.; Press, O.; Merrill, W.; Liu, A.; and Smith, N.A.2023. HowLanguageModelHallucinationsCan Snowball. arXiv:2305.13534

    2023 arXiv

  39. [47]

    Zhao, L.; Deng, Y.; Zhang, W.; and Gu, Q

  40. [48]

    Zhou, B.; Hu, Y.; Weng, X.; Jia, J.; Luo, J.; Liu, X.; Wu, J.; and Huang, L. 2024. TinyLLaVA: A Framework of Small-scale Large Multimodal Models. arXiv:2402.14289

    2024 arXiv

  41. [49]

    arXiv:2402.08680

    Mitigating Object Hallucination in Large Vision-LanguageModelsviaClassifier-FreeGuidance. arXiv:2402.08680

    arXiv

  42. [50]

    Subsection: Text-centric Heads are Model-specific

    Zhu, L.; Ji, D.; Chen, T.; Xu, P.; Ye, J.; and Liu, J. 2024. IBD: Alleviating Hallucinations in Large Vision-Language Models via Image-Biased Decoding. arXiv:2402.18476. Appendix Text-centric Heads Experiment Settings The following setup applies to "Subsection: Text-centric He...

    2024 arXiv

  43. [51]

    Zhu, D.; Chen, J.; Shen, X.; Li, X.; and Elhoseiny, M

  44. [2023]

    arXiv:2304.10592

    MiniGPT-4: Enhancing Vision-Language Un- derstanding with Advanced Large Language Models. arXiv:2304.10592

    arXiv

  45. [2024]

    arXiv:2403.18715

    Mitigating Hallucinations in Large Vision- Language Models with Instruction Contrastive Decod- ing. arXiv:2403.18715

    arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.