{"id":"4837cd89-9bc7-4c45-a20a-a3a2af31567b","arxiv_id":"2506.14766","paper_version":3,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Directly steering attention during decoding, boosting text-centric heads and suppressing over-attended visual tokens, reduces object hallucination in multimodal LLMs by up to 38.2 percent with no retraining.","lead":"ASCD is a training-free decoding method that cuts hallucination in multimodal AI models by directly adjusting how much attention the model pays to image versus text. It matters because it offers a simple, model-agnostic reliability fix for image-grounded language models, without retraining.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Algorithm 2's positive-steering update cannot produce the claimed increase in visual attention: for post-softmax weights it cancels under normalization, and for pre-softmax scores it sharpens the existing text bias; both branches then shift attention the same way.","rationale":"The reader's weakest assumption (inferring causation from Figure 1) is a valid concern about motivation, but it is not the most load-bearing point: even if VCD/ICD's gains were not caused by attention shifts, a direct attention-steering method could still work, and the paper could simply drop the causal story. The more serious issue is that the mechanism as specified in Algorithm 2 cannot produce the positive-steering effect that the paper's central claim requires. The Positive Steering Pass updates the whole attention matrix A of selected text-centric heads by A + alpha_pos|A|. For post-softmax attention this is a no-op after normalization; for pre-softmax scores it is a magnitude-dependent sharpening that, in text-centric heads, will amplify the already-dominant text tokens. Either way it does not 'increase vision attention', and it certainly does not implement the 'increase vision / decrease text' branch shown in Figure 2. Since the Negative Steering Pass also reduces visual attention, the two contrastive branches would be expected to move logits in the same direction, so the paper's explanation for why contrastive subtraction helps is unsupported. The empirical CHAIR/POPE improvements could still be real if the code does something different from Algorithm 2; the GitHub repository makes this testable, and the absence of a correct description is exactly the kind of condition that should gate acceptance. I also note that the Hyperparameters section says parameters were chosen 'to obtain the best result on CHAIR', which is a separate threat to the headline 38.2% improvement; a verification should fix hyperparameters on a validation split. These two concerns, especially the Algorithm 2 issue, keep the verdict at 'conditional' at best: the paper should not be accepted until the public implementation is shown to implement positive steering and the positive-steering branch's attention direction is verified. This partially agrees with the reader: we both suspect the attention mechanism is under-supported, but the concrete algorithmic inconsistency is sharper than the correlational premise.","tokens_in":19555,"tokens_out":10705,"duration_ms":112541,"concrete_test":"Run the released ASCD code with hooks that record attention matrices and output logits for about 100 COCO images; apply the Positive Steering Pass exactly as written in Algorithm 2 and measure the mean attention mass on visual vs. text tokens before vs. after. If visual attention does not increase (or logits are unchanged), the described algorithm does not implement positive steering. Then re-run CHAIR/POPE with a corrected positive steering that explicitly boosts visual-token attention in the selected heads; if the gains vanish, the claimed mechanism is the wrong explanation.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's central claim is that ASCD works by directly steering attention: positive steering amplifies text-centric heads to emphasize visual content (Section 4.2, Fig. 2), and negative steering dampens critical visual tokens. The Positive Steering Pass in Algorithm 2 is specified as A(l)_h <- A(l)_h + alpha_pos|A(l)_h|, followed by normalization. This update cannot implement the claimed 'increase vision attention / decrease text attention'. If A is the post-softmax attention matrix (nonnegative, rows summing to 1), this is exactly a scalar multiplication (1+alpha_pos)A of each row, which is invariant under row normalization, so the positive branch is identical to the unsteered branch and the contrastive term degenerates. If A is the pre-softmax score matrix, adding alpha_pos times the absolute value is not a fixed scalar, but it amplifies entries with larger magnitude; in text-centric heads those entries are the text tokens, so the update sharpens the existing text bias rather than increasing visual attention. In either reading, the positive branch does not do what Figure 2 and the text claim. Note that the Negative Steering Pass also suppresses visual tokens, so both contrastive branches would shift attention in the same direction (away from visual tokens), leaving the contrastive subtraction's success unexplained by the stated mechanism. The paper's reproducible-code commitment (GitHub link) makes this checkable, but as written the core mechanism is not implemented by the algorithm.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes ASCD, a training-free decoding-time method intended to reduce object hallucination in multimodal large language models. The method first identifies \"text-centric\" attention heads offline from a reference set, then at generation time runs two contrastive branches: a positive-steering branch that is intended to amplify visual attention by modifying text-centric heads, and a negative-steering branch that down-weights the most-attended visual tokens. The two logit distributions are combined with contrastive decoding and truncation (Eq. 2, Algorithm 2). The paper reports that ASCD reduces hallucination on POPE, CHAIR, and MMHal-Bench by up to 38.2% across five backbones and three decoding schemes while preserving or improving accuracy on standard VQA benchmarks.","tokens_in":19925,"tokens_out":6824,"duration_ms":68488,"significance":"If the method worked as described, it would be a practically attractive training-free intervention: it is cheap, model-agnostic in principle, backed by a released code repository, and evaluated more broadly than many prior contrastive-decoding papers (five backbones, three decoding schemes, POPE/CHAIR/MMHal-Bench, and five VQA benchmarks). The runtime/memory measurements and the out-of-domain head-stability analysis are useful additions. However, the paper's central mechanistic claim is not supported by the supplied algorithm, and the headline CHAIR gains are obtained after tuning hyperparameters on CHAIR. The empirical evaluation therefore needs substantial revision before the central claim can be accepted.","major_comments":[{"comment":"Step 1 of Algorithm 2 specifies the positive-steering update as A(l)_h <- A(l)_h + alpha_pos |A(l)_h|, followed by normalization. If A(l)_h is the post-softmax attention matrix, all entries are nonnegative and the update is exactly a row-wise multiplication by (1+alpha_pos), which is cancelled by row normalization; the positive branch is then identical to the unsteered branch, and Eq. (2) reduces to (1+alpha)p_orig - alpha p_neg. If A(l)_h instead denotes pre-softmax scores, the update is not row-normalization invariant, but nothing in the update specifically increases attention to visual tokens; for nonnegative scores it again scales all entries by the same factor, and for negative scores it moves them by alpha|A| toward zero, which is not the 'increase vision / decrease text' behavior claimed in Fig. 2 and Section 4.2. Because both branches in Algorithm 2 either leave attention unchanged or suppress visual tokens, the contrastive subtraction's success is unexplained by the stated attention-steering mechanism. The authors should clarify which quantity A(l)_h denotes and provide a version of the positive steer that provably increases relative visual attention, together with an attention-measurement verification.","section":"Methodology, Algorithm 2 (Positive Steering Pass)"},{"comment":"The appendix states that the hyperparameters were chosen 'to obtain the best result on CHAIR,' and Table 1 reports the CHAIR improvements, including the abstract's headline 'up to 38.2%.' This means the main hallucination-reduction claim is evaluated on the same benchmark used for hyperparameter selection, so the CHAIR numbers are overoptimistic and are not a valid test of the method. In addition, no standard errors or confidence intervals are reported for nucleus-sampling runs or for CHAIR, whose random image sampling introduces variance; the POPE/CHAIR tables as printed do not allow the reader to assess whether the gains are significant.","section":"Appendix (Hyperparameters); Table 1"},{"comment":"The reproduced VCD and ICD baselines are inconsistent with published results: in multiple rows VCD increases CHAIR relative to the original model (e.g., LLaVA-1.5 greedy: CHAIRs 56.8 vs 53.2; LLaVA-NeXT greedy: 37.2 vs 31.6), whereas the original VCD paper reports reductions. This suggests the reproduction may not match the reference implementations, and it weakens the claim that ASCD outperforms prior contrastive-decoding methods. The authors should report the original published numbers, state which exact repositories and versions were used, and release or verify the baseline code.","section":"Experiments, Table 1"},{"comment":"The motivating claim that VCD and ICD reduce hallucination 'largely attributable' to attention redistribution is inferred from correlations in Figure 1; no controlled experiment separates the attention shift from the logit-subtraction step. Because the entire design of ASCD depends on this causal reading, the authors should perform an intervention test (e.g., applying the observed attention redistribution without logit subtraction, or steering attention in the opposite direction) to establish that the attention shift is causally responsible. As it stands, Figure 1 only shows a correlation.","section":"Figure 1 and Section 'Proactive Steering of Attention'"},{"comment":"MMHal-Bench is one of the three hallucination benchmarks used to support the abstract's claim, but the results are presented only as radar charts with no numeric values, no aggregate scores, and no error estimates. The reader cannot verify that ASCD's apparent advantage is real or compare it with the baselines quantitatively. Numeric results should be added for every sub-dimension and for the overall score.","section":"Experiments, Figure 6"}],"minor_comments":[{"comment":"Equation (2) is written with p, but the text calls these logits; please clarify whether the contrast is applied to probabilities, log-probabilities, or logits, since this affects the truncation step in Algorithm 2.","section":"Preliminaries, Eq. (2)"},{"comment":"The panel labeling in Figure 3 is confusing: the text refers to '(a) center,' '(b) bottom left,' and '(c) bottom right,' but the layout is not self-explanatory; please add explicit panel order and a color scale for the heatmaps.","section":"Figure 3"},{"comment":"The hyperparameter ablation reports only POPE; since CHAIR was the tuning target, the authors should also include CHAIR results or state why CHAIR was not swept on a held-out split.","section":"Appendix, Table 6"},{"comment":"Table 7 reports dashes for ASCD with SDPA and FlashAttention, yet the text claims overhead is measured under three attention backends; please clarify that the latency/memory comparison for ASCD is for the eager backend only.","section":"Appendix, Table 7"},{"comment":"The definitions of textAttn and visAttn should specify whether attention is averaged over layers, heads, or tokens, and whether image patch tokens include all vision tokens; this is needed to reproduce the head-selection procedure.","section":"Appendix, Eq. (4) and Algorithm 1"},{"comment":"The claim that the ASCD caption contains only entities that are actually present is based on a single qualitative example; please label it as illustrative rather than as quantitative evidence.","section":"Figure 8"}],"recommendation":"major_revision","confidential_remarks":"The manuscript has a public code link and a broad evaluation, but the central algorithm issue and the CHAIR tuning circularity are serious. If the authors can fix the positive-steering update, verify that it actually changes attention in the intended direction, and redo the CHAIR evaluation with a held-out or nested validation scheme, the paper could become publishable. I would not require a complete re-derivation, but the current version should not be accepted without these changes."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know this about the ASCD paper: the evaluation is extensive and the paper makes a genuinely interesting empirical observation, but the core mechanism as written in Algorithm 2 does not do what the text claims. The positive-steering step either leaves attention unchanged or sharpens the text bias, so the 'attention steering' story is not supported by the algorithm.\n\nCredit where due: the paper reproduces VCD and ICD and shows their hallucination reduction coincides with attention redistribution; the head-stability analysis is a nice empirical contribution; and the experiments span five backbones, three decoding schemes, and several benchmarks. The GitHub link is a plus.\n\nNow the problem. Algorithm 2 says for text-centric heads, A <- A + alpha_pos|A|, then normalize. If A is post-softmax (nonnegative rows summing to 1), this is scalar multiplication, and normalization removes the effect completely. If A is pre-softmax scores, as the paper defines, then adding |A| amplifies entries with larger magnitude; in text-centric heads those are the text tokens, so the update sharpens the existing text bias rather than increasing visual attention. Either way, the positive branch does not 'increase vision attention / decrease text attention' as Figure 2 and the prose claim. This is not a minor typo: the whole paper is framed as direct attention steering, with positive and negative branches acting in opposite directions. If both branches actually suppress visual attention, the contrastive subtraction's success is unexplained by the stated mechanism.\n\nOther soft spots are secondary but worth noting: the headline CHAIR improvement comes from hyperparameters explicitly tuned on CHAIR, there are no error bars for stochastic decoding, and the reproduced VCD baseline is weaker than the original paper's, which complicates comparisons.\n\nWho is this for? Practitioners looking for training-free hallucination reduction might still find the method useful; empirically it seems to help. But as written, the paper cannot support its own explanation. It deserves a serious referee because the experiments are substantial and the flaw is fixable—either change the update to actually steer attention, or reframe the contribution. I would not cite it as is.\n\nRecommendation: send to peer review, but the authors should be asked to reconcile Algorithm 2 with the claimed mechanism and to add error bars.","headline":"The paper's experiments are broad and the attention-shift observation is useful, but Algorithm 2's positive-steering step does not implement the claimed mechanism, which undercuts the central narrative.","tokens_in":20430,"tokens_out":4477,"would_cite":false,"duration_ms":45652,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that directly steering attention inside a multimodal LLM—boosting text-centric heads and damping critical visual tokens—reduces object hallucination by up to 38.2 percent without training and while improving VQA accuracy.","keywords":["attention steering","contrastive decoding","multimodal large language models","hallucination mitigation","text-centric attention heads","object hallucination","training-free decoding"],"falsifier":"Run VCD or ICD while holding the model's attention distribution artificially fixed; if hallucination still drops, attention redistribution is not the causal mechanism and the motivation for direct attention steering collapses.","tokens_in":19388,"feed_emoji":"🎯","tokens_out":8221,"duration_ms":72522,"temperature":0.7,"pith_summary":"Multimodal large language models often invent objects that are not in the image. The paper claims that the known hallucination-reducing effects of Visual and Instruction Contrastive Decoding (VCD, ICD) come from how those methods redistribute attention, steering it away from visual tokens and toward textual tokens. If that is right, the natural fix is to steer attention directly rather than perturb the image or prompt. The proposed ASCD does exactly that, amplifying a model's own text-centric attention heads while suppressing only the most attended visual tokens, and it reports lower hallucination on POPE, CHAIR, and MMHal-Bench by up to 38.2 percent while preserving or improving accuracy on five standard VQA benchmarks. No extra training is needed, and the added runtime and memory cost is small.","feed_headline":"Steering attention directly cuts MLLM hallucination by up to 38.2%","feed_subtitle":"A training-free decoding tweak amplifies text-centric heads and dampens critical visual tokens, improving VQA too.","key_machinery":"The central mechanism is the attention-steerable contrastive decoding equation, $p^{\\text{final}}_\\theta = (1+\\alpha)p^{\\text{pos-steered}}_\\theta - \\alpha p^{\\text{neg-steered}}_\\theta$, where both branches are produced by editing attention matrices instead of logits. Positive steering adds $\\alpha_{\\text{pos}}|A^{(l)}_h|$ to the attention of a small set of text-centric heads selected offline by voting over a reference image set using the ratio of text attention to visual attention. Negative steering computes per-token head-averaged attention $s(v)=\\frac{1}{H}\\sum_h A^{(L)}_h(v)$, selects the top-$\\kappa_{\\text{vis}}$ visual tokens as critical, and subtracts $\\alpha_{\\text{neg}}|A^{(l)}_h(v)|$ from every attention entry to those tokens in all heads. The text-centric-head selection is the load-bearing component, since the paper argues these heads are stable within a model across prompts, image sets, domain shifts, and generation lengths.","core_discovery":"ASCD's central claim is that attention redistribution is the operative mechanism behind contrastive-decoding hallucination fixes, and that deliberately manipulating attention is sufficient to reproduce and improve on those fixes. The paper first measures that VCD and ICD, and their combination, systematically lower attention on visual tokens and raise it on textual tokens, with stronger perturbations producing larger shifts. It then constructs two steered decoding branches: a positive branch that boosts the attention weights of automatically selected text-centric heads (heads whose text-to-visual attention ratio is consistently high for that model) and a negative branch that down-weights only the top-attended visual tokens. Subtracting the negative branch's logits from the positive branch's logits, with truncation, yields the final distribution. Across five backbones and three decoding schemes, the paper reports the lowest CHAIR scores and highest POPE accuracy among compared methods, plus gains on MMHal-Bench, while the reproduced VCD and ICD degrade on standard VQA benchmarks.","pith_inferences":["If the causal reading of Figure 1 is correct, then any decoding intervention that changes the visual-to-text attention balance, not just contrastive decoding, should shift hallucination rates, which could turn attention balance into a general diagnostic for hallucination propensity.","A testable extension would be to profile text-centric heads on a model whose hallucination rate is artificially raised or lowered by training interventions; if the same heads stay text-centric, head identity is a fixed architectural bias, while if they change, profiling may need to be repeated after fine-tuning.","Attention steering could combine with training-time methods: using ASCD at inference may reduce the amount of hallucination-specific data or reward modeling needed during training, since the mechanism already counteracts over-commitment to textual priors."],"forward_implications":["Hallucination mitigation can be treated as an attention-editing problem rather than a logit-subtraction or input-perturbation problem, so new contrastive-decoding variants can be designed as cheap attention edits.","One offline profiling pass per model, running a reference image set and voting on high text-to-visual attention heads, is enough to transfer the method across domains because the identified heads are stable within a model.","ASCD can be applied at inference time on top of existing MLLMs without retraining, and it works across greedy, nucleus, and beam search, so it could be dropped into deployed systems with negligible latency increase.","Unlike VCD and ICD, which the paper reproduces as degrading on MMMU, MM-VET, ScienceQA, TextVQA, and GQA, ASCD is reported to match or improve those benchmarks, meaning hallucination reduction and general VQA ability need not trade off.","The gains persist at larger scale (LLaVA-1.5 13B) and on a newer architecture (Qwen2.5-VL-Instruct), suggesting the mechanism is not tied to one training recipe."],"supporting_citations":[{"why":"Introduces Visual Contrastive Decoding (VCD), the image-perturbation method whose attention redistribution the paper measures in Figure 1 and whose logit branch ASCD replaces with positively steered logits.","marker":"[24]"},{"why":"Introduces Instruction Contrastive Decoding (ICD), the negative-prompt method whose attention redistribution is measured and which ASCD is compared against.","marker":"[41]"},{"why":"Defines POPE, the polling-based benchmark used for object-level hallucination evaluation across all backbones and decoding schemes.","marker":"[26]"},{"why":"Defines CHAIR, the caption hallucination metric where ASCD reports its largest percentage reductions.","marker":"[35]"},{"why":"Introduces MMHal-Bench and factually augmented RLHF, providing both a hallucination benchmark and a trained baseline.","marker":"[39]"},{"why":"PAI, an attention-steering baseline that appears in the comparison tables and represents the existing inference-phase attention family ASCD extends.","marker":"[31]"},{"why":"OPERA, an earlier attention-based hallucination method whose over-trust penalty motivates the paper's focus on cross-modal attention.","marker":"[19]"}],"fun_headline_variants":["ASCD steers attention to cut MLLM hallucination by 38.2%","Training-free attention steering cuts MLLM hallucination by 38.2%","Steer attention to slash MLLM hallucinations by 38.2%","ASCD: attention steering cuts hallucination by 38.2%, no training"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that the attention shifts seen with VCD and ICD are the cause of their hallucination reduction, not just a correlated side effect, and that directly editing attention therefore transfers the benefit.","fun_headline_variants_meta":{"raw":{"variants":["ASCD steers attention to cut MLLM hallucination by 38.2%","Training-free attention steering cuts MLLM hallucination by 38.2%","Steer attention to slash MLLM hallucinations by 38.2%","ASCD: attention steering cuts hallucination by 38.2%, no training"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001081,"raw_usage":{"total_tokens":4536,"prompt_tokens":977,"completion_tokens":3559,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":593,"completion_tokens_details":{"reasoning_tokens":3471}},"tokens_in":593,"tokens_out":3559,"duration_ms":26590,"temperature":1.0,"reasoning_tokens":3471,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T19:46:48.732438+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run VCD or ICD while holding the model's attention distribution artificially fixed; if hallucination still drops, attention redistribution is not the causal mechanism and the motivation for direct attention steering collapses.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces Instruction Contrastive Decoding (ICD), the negative-prompt method whose attention redistribution is measured and which ASCD is compared against."}],"review_version":2}