{"id":"a995d7da-b38f-4ad5-85d2-ba6fe3bf2a67","arxiv_id":"2412.13949","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Hallucination in LVLMs correlates with low sensitivity of attention heads to image input, and scaling up the most image-sensitive heads reduces it without training.","lead":"This paper introduces a metric, Vision-aware Head Divergence, that measures how much each attention head in a vision-language model changes when the image is removed. The authors then amplify the most image-sensitive heads at generation time, reducing hallucinated objects in captions without retraining.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"VHD conflates image removal with positional shift: dropping image tokens changes text-token positions, so the metric may not isolate visual awareness; the mechanism claim needs a position-controlled test.","rationale":"The reader's weakest assumption identified the counterfactual validity of Eq. 4, and this is indeed the most load-bearing point. I sharpen it: the removal of image tokens not only removes visual content but also changes the positional encoding of every text token, making the two runs differ in a controlled nuisance variable. This is not merely a philosophical worry; it directly determines whether VHD measures what the paper claims, and whether VHR's head-selection step is well-founded. The reader's concern about internal state distribution is related but less specific; my formulation gives a concrete, testable mechanism for why the counterfactual may fail.\n\nOther potential issues are secondary. Hyperparameter tuning on the evaluation benchmark is a generalizability concern but does not threaten the internal validity of the mechanism claim. The proof of Proposition 1, while algebraically compressed, appears to hold except in degenerate collinear cases and is not central to the empirical contribution. The absence of variance on POPE and LLaVA-Bench is a reporting issue, not a logical flaw. The most fundamental requirement for the central claim—'hallucinated words correspond to lower T-VHD' and 'amplifying vision-aware heads reduces hallucination'—is that VHD isolates visual awareness. My proposed test would settle whether this requirement holds.\n\nI keep the verdict CONDITIONAL rather than moving to REJECT because the empirical method may still work even if the interpretability story is wrong; however, the paper's central mechanistic claim is not established until the positional confound is addressed. This is a stronger and more precise condition than the reader's original counterfactual worry, but it does not overturn the reader's overall conditional acceptance.","tokens_in":15066,"tokens_out":4738,"duration_ms":46591,"concrete_test":"On a held-out set of 100 COCO images with LLaVA-1.5, recompute VHD as in Eq. 4 and a position-controlled variant where the image tokens are replaced by an equal number of zero-embedding (or fixed blank-image) tokens so sequence positions match the image-conditioned run. Compute the Spearman correlation between the two VHD matrices across heads/layers at the first generation step. If the correlation is weak (e.g., < 0.5), the original VHD is substantially confounded by position. Then run VHR using head selection based on the position-controlled VHD; if the CHAIRs/CHAIRi improvements over greedy decoding disappear or shrink sharply, the claimed vision-awareness mechanism is not supported and the gains of VHR stem from a different effect.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"Equation 4 defines VHD as d(A_{l,i}(y_t|y_{<t}, x_V, x_T), A_{l,i}(y_t|y_{<t}, x_T)). In standard LVLMs such as LLaVA-1.5, image tokens are prepended to the text sequence, so removing x_V shortens the sequence and shifts the positional indices of all text tokens (x_T and y_{<t}). The two forward passes therefore differ not only in the presence of visual content but also in the positional encodings (RoPE) of every token. As a result, VHD conflates 'sensitivity to visual context' with 'sensitivity to absolute position', and the identification of 'vision-aware heads' in Section 3.1 and Figure 2 is confounded. This is load-bearing because VHR selects the heads to reinforce based on VHD computed at t=0 (Algorithm 1, Step 3); if those heads are selected due to positional artifacts rather than visual awareness, the mechanistic interpretation of the method—reinforcing vision-aware heads to overcome language bias—is unsupported, even if the empirical gains on CHAIR/POPE are real. The T-VHD correlation with hallucination in Figure 3 is equally exposed: hallucinated words appear at varying sequence positions, so position-dependent divergence could masquerade as language-bias signal. The paper does not control for sequence-length or position alignment in the counterfactual, and its theoretical Proposition 1 concerns scaling attention heads, not the validity of the VHD selector.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper studies object hallucination in large vision-language models (LVLMs) at the level of individual attention heads. It introduces Vision-aware Head Divergence (VHD), the Euclidean distance between a head's output with and without the image input at a given decoding step, and Token-VHD (T-VHD), an aggregation of top-k VHD scores across layers. The authors report that only a small number of heads show large VHD, that hallucinated words and sentences tend to have lower T-VHD than grounded ones, and that this reflects reliance on language priors. They then propose Vision-aware Head Reinforcement (VHR), a training-free decoding intervention that selects the most vision-sensitive heads at the first decoding step and scales their outputs by a factor alpha, with outlier filtering, applied layer by layer. The method is evaluated on InstructBLIP-7b, LLaVA-1.5-7b, and LLaVA-NeXT-7b using CHAIR, POPE, and LLaVA-Bench (In-the-Wild), with ablations and a theoretical proposition (Proposition 1) claiming that scaling a head's output reorients the post-attention representation toward that head's direction.","tokens_in":15351,"tokens_out":21680,"duration_ms":185234,"significance":"If its claims hold, this paper provides a concrete, per-sample, interpretable handle on hallucination: a diagnostic metric (T-VHD) that flags tokens with weak visual grounding, and a training-free intervention (VHR) that improves grounding by amplifying a small set of vision-sensitive heads. The empirical design is a genuine strength: results are reported on three LVLM families, CHAIR numbers are averaged over five random splits with standard deviations, ablations cover adaptive selection, outlier removal, reinforced layers, and the scale factor, the inference-time overhead is analyzed, and a code link is provided. The central hypothesis is falsifiable and the proposed correlation, if confirmed with proper statistics, would be a useful probing tool. The main gaps are that the VHD counterfactual is not position-controlled, that the key hyperparameters are selected on the same CHAIR benchmark used for the headline results, and that the proof of Proposition 1 is not rigorous as written; all three are addressable within the scope of a revision.","major_comments":[{"comment":"The counterfactual in Eq. (4) removes the image tokens x_V entirely, but in the three evaluated LVLMs the image tokens occupy the first positions of the input sequence, so the text-only forward pass places the prompt tokens, y_<t, and the BOS token at different absolute positions and removes the image-token keys/values that text tokens attend to; VHD therefore measures sensitivity to visual content jointly with sensitivity to sequence position and length. Because RoPE-based text-to-text attention is shift-invariant, the positional effect is weaker than a purely absolute-position analysis would suggest, but it is not zero, as it affects the BOS boundary and any text-to-image attention, and the two runs also differ in total sequence length. This confound is load-bearing: Algorithm 1 (step 3) selects the heads to reinforce from VHD, and Eq. (5) with Figure 3 builds the hallucination analysis on the same quantity; in addition, y_<t is generated under image-conditioned decoding, so the text-only pass is queried with an out-of-distribution prefix. Please add a position-controlled counterfactual that keeps the same number of tokens in both passes, for example by replacing the vision tokens with zeroed or neutral embeddings, and report whether the selected head sets and the T-VHD distributions in Figure 3 are preserved; also state how the prefix-distribution issue is handled.","section":"§3.1, Eq. (4); Algorithm 1"},{"comment":"The free parameters of VHR—the scale factor alpha, the reinforced layer set L_r, the top-k in Eq. (5) (whose value is never reported), and the outlier thresholds in Eq. (6)—are chosen by ablating on the CHAIR benchmark (Figure 5 and Tables 5 and 6), and the same CHAIR benchmark is then used for the headline results of Table 1; with no held-out validation split, the reported CHAIR improvements are inflated by selection on the test set. This is the main threat to the empirical claim that VHR outperforms the baselines. Please either fix the hyperparameters on a validation split of the COCO/CHAIR data before computing Table 1, or report the full sensitivity grid and demonstrate that the ordering against baselines is stable across it; the POPE and LLaVA-Bench results, which were not used for tuning, should be foregrounded as the out-of-sample evidence.","section":"§4.4, §4.6; Tables 1, 5, 6"},{"comment":"The proof of Proposition 1 in Appendix B is not valid as written: the step bounding the difference from below replaces the negative term -<x, y_hat>/(||x|| ||y_hat||) with '+1', but that term is at least -1, not at least +1, so the displayed lower bound (||y_hat||-||x||)/||x+y_hat|| + 1 does not follow; moreover, the statement is false in degenerate cases such as x = -mu*y with 0 < alpha-1 < mu/||y||, where the cosine similarity remains at -1 (and x + y_hat can be the zero vector). The intended geometric claim is nevertheless plausible: for x not parallel to y, the function lambda -> cos(x+lambda*y, y) is non-decreasing in lambda >= 0 by Cauchy-Schwarz, with strict increase unless x and y are parallel. Please restate the proposition with the required non-degeneracy conditions and replace the faulty inequality with a correct argument.","section":"§3.3, Appendix B, Eq. (13)"},{"comment":"Algorithm 1 and the surrounding text leave unclear whether the text-only pass (the one without the image) is also reinforced layer by layer. The sentence 'the previous layers are already reinforced when calculating the VHD scores for a given layer' suggests both passes are modified, but Eq. (4) defines VHD on the original unmodified model, so for layers after the first, the quantity computed at inference would not be the VHD defined in Eq. (4); if instead the text-only pass is left untouched, the two passes are asymmetric and the selection at layer l mixes visual sensitivity with the effect of prior-layer reinforcement. Please specify exactly which forward passes are modified, and provide a small experiment comparing the layer-by-layer head selections with the selections from the clean two-pass computation defined by Eq. (4).","section":"§3.2, Algorithm 1"}],"minor_comments":[{"comment":"The POPE results are reported as F1 means over the three splits without standard deviations or significance tests, and several margins over the strongest baseline are very small (e.g., 85.47 vs. 85.45 for LLaVA-1.5); the claim in §4.5 that VHR outperforms all other decoding methods across all LVLMs needs variance or paired-test support.","section":"Table 2"},{"comment":"The LLaVA-Bench results come from a single GPT-4o evaluation without per-item scores or variance, and the prompt template in Table 7 fixes the order of Assistant 1 and Assistant 2; please describe the order randomization and report item-level statistics.","section":"Table 3"},{"comment":"The T-VHD comparison in Figure 3 is described as providing statistical evidence, but no test statistic, sample size, or effect size is reported; please add a two-sample test (e.g., Mann-Whitney U) for both the sentence-level and word-level comparisons.","section":"Figure 3"},{"comment":"The value of k in the top-k operator of Eq. (5) is never stated in the implementation details; please report it.","section":"Eq. (5)"},{"comment":"The alpha=4 row (CHAIRS 3.64, CHAIRI 2.01, length 144.54) is dismissed as an anomaly, but no explanation is given; please clarify what the model generates in that regime (e.g., repetitive or degenerate text) so that it is clear why the metric improvement is not meaningful.","section":"Table 6"},{"comment":"The text alternates between 'the second layer' and 'layer1' for the same setting; please make the layer indexing consistent.","section":"Appendix C, Table 5"},{"comment":"The model names appear both as 'LLaVA-1.5' and 'LLaV A-1.5' (with an internal space) in different places; please unify the notation.","section":"Throughout"},{"comment":"The claim that only a few heads are vision-aware is illustrated with a single generation step in Figure 2; reporting the distribution of the number of high-VHD heads across samples and steps would make the claim quantitative.","section":"Figure 2, §3.1"}],"recommendation":"major_revision","confidential_remarks":"To the editor: the paper is within scope for an NLP/vision-language venue, and the empirical apparatus is solid in design (three model families, five-split CHAIR with standard deviations, ablations of each component, timing analysis, and a code link). The two substantive risks are (1) the VHD counterfactual is confounded by token-position changes when the image is removed, and (2) the headline CHAIR gains are measured on the same benchmark used to tune the method's hyperparameters; both are fixable, and I have asked for targeted controls in the major comments. The proof flaw in Proposition 1 is real but local, and the Algorithm 1 ambiguity is a specification gap rather than a demonstrated failure. I have no concerns about citation practices: the relationship to EAH and to Chen et al. is discussed explicitly, and the baseline comparisons are standard. If the authors supply the position-controlled experiment, re-do the hyperparameter selection on a validation split, and correct the proof, I would be happy to see a revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: this is a solid, if imperfect, method paper. The VHD metric is simple and the observed link between low T-VHD and hallucinated tokens is a real empirical contribution. VHR gives consistent gains over strong decoding baselines on three model families. The Proposition 1 proof checks out; scaling a head's output does rotate the FFN input toward that head's direction. I also like that they zero out negative-sensitivity outliers and adaptively select heads per sample; the ablation shows it matters.\n\nThe soft spots are real but not fatal. Hyperparameters (alpha, layer set, k) are selected using CHAIR, and then CHAIR is the headline benchmark. That makes the CHAIR numbers a bit optimistic. POPE and LLaVA-Bench have no error bars, and LLaVA-Bench is a single GPT-4o pairwise comparison. They cite DAMRO without including it in the comparison; for an attention-intervention paper, that is a gap. None of this undercuts the core finding that amplifying heads with high VHD reduces hallucination.\n\nThe position-shift concern in the stress-test is the sharpest critique, but it does not land as hard as it first seems. In RoPE-based transformers, shifting every token position by a constant leaves the relative positions among text tokens unchanged, so the text-text attention weights do not change due to position alone. What changes is the absence of image tokens and the resulting hidden-state differences, which is exactly the visual sensitivity the authors want to measure. That said, the authors never run a control where they remove the image but insert dummy tokens to preserve position, so the cleanest version of the claim is not directly tested. It is a fair referee request, not a fatal flaw.\n\nWho is this for? Anyone working on LVLM hallucination or token-level interpretability of attention heads. It goes a step beyond the usual logit-contrast decoding papers because it names head-level sensitivity and uses it for a training-free intervention. I would cite it, and I would bring it to a reading group with the caveat that the CHAIR tuning should be disclosed more fully.","headline":"A solid training-free decoding fix for LVLM object hallucination, with a clean metric and real gains; the main caveat is tuning transparency, not the mechanism.","tokens_in":15965,"tokens_out":6097,"would_cite":true,"duration_ms":54750,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that hallucinated tokens in LVLM outputs correspond to low Vision-aware Head Divergence scores, and that amplifying the most visually-sensitive attention heads during decoding reduces hallucination more effectively than…","keywords":["hallucination","large vision-language models","attention heads","vision-aware head divergence","language bias","decoding-time intervention","object hallucination","training-free mitigation"],"falsifier":"A direct test would be to compute VHD on a model whose image encoder has been deliberately blinded or whose vision tokens are shuffled: if hallucination rates stay unchanged while VHD scores remain high, the metric is not tracking visual content. Conversely, a stronger test is to take tokens labeled hallucinated in a held-out benchmark and check whether their T-VHD scores are consistently below the median of correct tokens; showing even one model or benchmark where hallucinated tokens have high T-VHD would break the claimed correspondence.","tokens_in":14828,"feed_emoji":"🖼️","tokens_out":5105,"duration_ms":41330,"temperature":0.7,"pith_summary":"The paper claims that hallucinations in large vision-language models can be traced to a small set of attention heads that carry visual information, and that scaling up those heads during decoding reduces hallucination without retraining. It introduces Vision-aware Head Divergence (VHD), the Euclidean distance between an attention head's output with and without image input, as a way to identify which heads are visually sensitive. Aggregating the top VHD scores per token produces Token-VHD (T-VHD), which the paper finds is lower for hallucinated words and sentences than for grounded ones. Based on this, the Vision-aware Head Reinforcement (VHR) method selects the most visually sensitive heads per sample and amplifies their outputs by a fixed factor, improving hallucination benchmarks over existing training-free decoding methods. A sympathetic reading is that this establishes a causal, internal-mechanism route to reducing hallucination, not just an output-level patch.","feed_headline":"Scaling vision-sensitive heads cuts LVLM hallucinations","feed_subtitle":"Amplifying the few heads that track the image beats seven decoding baselines on hallucination benchmarks, with no retraining.","key_machinery":"The load-bearing object is the Vision-aware Head Divergence (VHD) score, defined as the Euclidean distance between an attention head's output when the image is included and when it is removed, holding the text prefix and generation history fixed. Token-VHD (T-VHD) aggregates the top-k VHD scores across layers for one prediction step, serving as a token-level proxy for visual grounding. Vision-aware Head Reinforcement (VHR) uses these scores to pick, per layer, the half of heads with the highest VHD and multiplies their outputs by α=2, after zeroing out heads whose high VHD reflects an activation surge when the image is removed. The theoretical support is a proposition that amplifying one head's output increases the cosine similarity between the FFN input and that head's component, so the scaling operation reorients the layer output toward the vision-aware direction.","core_discovery":"The paper's central claim is that a minority of attention heads in LVLMs are genuinely vision-aware, while most heads rely on language priors, and that hallucinated tokens are exactly those for which the language-prior heads dominate. The evidence is the T-VHD distribution: on a 500-image CHAIR sample, hallucinated object words and sentences containing them have significantly lower T-VHD scores than correct ones. The paper further claims that intervening on this mechanism—selecting the top half of heads by VHD per layer, zeroing out outliers that respond negatively to image removal, and multiplying their outputs by a factor α=2—reorients the attention output toward the vision-aware component. Across InstructBLIP-7b, LLaVA-1.5-7b, and LLaVA-NeXT-7b, this reduces CHAIR and improves POPE F1 compared with greedy, beam, DoLa, VCD, OPERA, CODE, and EAH. The stated implication is that language bias, not insufficient visual encoding, is a primary driver of hallucination, and that a per-sample, training-free head-scaling intervention is sufficient to counteract it.","pith_inferences":["The paper leaves implicit whether VHR's improvement compounds with contrastive decoding methods, since VHR modifies head states while contrastive methods modify logits, and the two could operate on different failure modes.","The counterfactual in the VHD definition compares image-present with image-absent decoding, which may conflate the effect of the image on the current step with the effect of the image on the KV cache of earlier tokens; an alternative text-only-prefix baseline could isolate the per-step contribution.","If the vision-aware heads identified by VHD generalize across datasets but not across image distributions, the per-sample adaptation could be replaced by a cheap classifier predicting which heads to amplify from the image representation alone."],"forward_implications":["If the claim holds, hallucination can be reduced by a per-sample rescaling of a small set of attention heads, with no training, no external knowledge, and negligible decoding overhead.","The T-VHD score could serve as a token-level diagnostic for when a model is about to generate ungrounded content, enabling early stopping or contrastive adjustment at the step where grounding drops.","The finding suggests that vision-language alignment is concentrated in a few heads across layers, which may guide future fine-tuning or pruning toward those heads rather than treating all parameters uniformly.","Because VHR is applied layer-by-layer, its reinforcement changes the internal state of deeper layers; the paper's ablations indicate that both the second layer and the last layers contribute, possibly through distinct mechanisms."],"supporting_citations":[{"why":"Defines the CHAIR caption-hallucination benchmark that supplies the main evaluation metric and the image annotations used for the T-VHD analysis.","marker":"Rohrbach et al., 2018"},{"why":"Provides the POPE object-hallucination benchmark used as the second evaluation.","marker":"Li et al., 2023"},{"why":"Establishes the in-context and memory head distinction that motivates looking at attention heads, and supplies the per-head manipulation idea.","marker":"Yu et al., 2023"},{"why":"Provides the InstructBLIP model used as one of the three LVLMs in the experiments.","marker":"Dai et al., 2023"},{"why":"Provides the LLaVA-1.5 backbone and the LLaVA-Bench (In-the-Wild) evaluation prompt used for open-ended assessment.","marker":"Liu et al., 2024b"},{"why":"The VCD visual contrastive decoding baseline that the method must beat and whose logits-level intervention contrasts with the head-level intervention.","marker":"Leng et al., 2024"},{"why":"The EAH attention-sink baseline, the closest attention-targeting prior work, also cited for the special role of the second layer.","marker":"Zhang et al., 2024"}],"fun_headline_variants":["Scale vision-aware heads to slash LVLM hallucinations","Language-prior heads cause LVLM hallucinations, not vision gaps","Training-free head boost cuts LVLM hallucinations on benchmarks","Vision-aware heads: the off-switch for LVLM hallucinations"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole metric stands on the assumption that removing the image from the input while keeping the same text prefix is a clean counterfactual that isolates what each attention head learns from the image; if the image-free run produces a different internal state or a different decoding trajectory, the VHD values measure artifacts of the comparison rather than visual awareness.","fun_headline_variants_meta":{"raw":{"variants":["Scale vision-aware heads to slash LVLM hallucinations","Language-prior heads cause LVLM hallucinations, not vision gaps","Training-free head boost cuts LVLM hallucinations on benchmarks","Vision-aware heads: the off-switch for LVLM hallucinations"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000251,"raw_usage":{"total_tokens":1587,"prompt_tokens":1002,"completion_tokens":585,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":618,"completion_tokens_details":{"reasoning_tokens":519}},"tokens_in":618,"tokens_out":585,"duration_ms":6180,"temperature":1.0,"reasoning_tokens":519,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T12:37:19.449130+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A direct test would be to compute VHD on a model whose image encoder has been deliberately blinded or whose vision tokens are shuffled: if hallucination rates stay unchanged while VHD scores remain high, the metric is not tracking visual content. Conversely, a stronger test is to take tokens labeled hallucinated in a held-out benchmark and check whether their T-VHD scores are consistently below the median of correct tokens; showing even one model or benchmark where hallucinated tokens have high T-VHD would break the claimed correspondence.","supporting_citations":[],"review_version":1}