{"id":"9db342f7-5405-458e-ae22-345b66c2e022","arxiv_id":"2411.09968","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A training-free intervention that broadcasts the densest vision-sink attention head's map to all heads in shallow layers lowers hallucination rates on CHAIR across several LVLMs.","lead":"Large multimodal models often describe objects that are not in the image. This paper shows that copying the most visually focused attention head into all heads of the first two layers reduces such hallucinations, with a small recall cost.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Missing control: broadcasting a random or sparse-sink head may match EAH's CHAIR gains, which would refute the claimed causal role of dense vision sinks.","rationale":"I chose the missing control over benchmark-selection bias because it attacks the explanatory claim, not just the statistical reliability. The paper's novelty is the vision-sink mechanism; if any head broadcast works, the phenomenon is an artifact. The reader's weakest assumption (homogenization safety) is adjacent; a random-head control also tests safety by showing whether head identity matters. I partially agree with the reader: the aggressive broadcast is indeed a concern, but the more precise untested premise is that the selected head's density is what drives improvement. The verdict stays CONDITIONAL: the control is easy to run and would settle the question. Benchmark overfitting (hyperparameters chosen on CHAIR, no error bars) is a separate issue that also warrants attention but is secondary to the mechanism test.","tokens_in":17503,"tokens_out":7375,"duration_ms":77932,"concrete_test":"Run EAH on LLaVA-1.5-7B on the CHAIR benchmark under four conditions: (1) broadcast the head with max vision-sink count; (2) broadcast the head with min vision-sink count; (3) broadcast a fixed head (e.g., head 0); (4) average all heads' attention maps in the layer. Use identical layer=2, beta=0.002, decoding settings, and at least 3 seeds. If conditions (2)-(4) match or beat (1) on CHAIR_S/CHAIR_I, the claimed mechanism is unsupported; if (1) is clearly better, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that heads with dense vision sinks are beneficial and that broadcasting the densest such head (Algorithm 1, Eqs. 11-12) alleviates hallucinations. However, every experiment compares EAH to other methods or ablates layer/threshold/head-count; none compares EAH to an equally aggressive control that broadcasts a randomly chosen head, the head with the fewest vision sinks, or the average of all heads. If such a control achieves comparable CHAIR improvements, the gains are attributable to homogenization of early-layer attention (or reduced output diversity), not to the vision-sink property. The paper's own results are consistent with this alternative: recall drops ~4 points and POPE F1 is unchanged, typical of a diversity-reducing regularizer. Without this control, the causal role of dense vision sinks is untested. This is distinct from, and more fundamental than, the benchmark-selection issue: even if the reported numbers are unbiased, they do not discriminate between the proposed mechanism and a trivial 'copy any head' operation.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a training-free, plug-and-play method, Enhancing Attention Heads (EAH), to reduce object hallucinations in large vision-language models (LVLMs). The authors analyze attention maps and observe that shallow layers contain dense 'vision sinks' (columns with high average attention over image tokens) and that hallucination tokens tend to be associated with fewer such heads. EAH identifies, within a chosen shallow layer, the head with the densest vision sink and broadcasts that head's attention map to all other heads in that layer (Algorithm 1, Eq. 12). They report CHAIR and POPE improvements on LLaVA-1.5-7B, plus additional experiments on other LVLMs and LLMs.","tokens_in":17645,"tokens_out":5540,"duration_ms":56007,"significance":"The paper addresses an important problem, hallucination in LVLMs, and proposes a simple, training-free intervention that requires no parameter updates, which is attractive for practical deployment. The observation that attention patterns in shallow layers correlate with hallucination is interesting and potentially useful for future interpretability work. A strength is the breadth of models tested (seven LVLMs and four LLMs). However, the current evidence does not yet establish the paper's causal mechanism: the missing control experiments, the incorrect mask definition, and the lack of statistical rigor all limit the significance of the reported gains. If the control experiments confirm that dense vision sinks—rather than mere attention homogenization—are responsible, the method would be a meaningful contribution; as written, the support is incomplete.","major_comments":[{"comment":"The mask matrix is defined as M = eye(r,c) - diag(1). For a square attention map (r = c), this equals the zero matrix. With M = 0, the left-hand side of Eq. (6) and Eq. (8) is identically 0, so no column can ever exceed any positive threshold β, contradicting the existence of the vision sinks that the paper reports. This is a central definitional error that prevents replication. Please correct the mask definition (likely M = ones(r,c) - eye(r,c)) and verify all subsequent equations.","section":"Section 3.2, Eq. (5)-(6), Eq. (8)"},{"comment":"The central causal claim—that broadcasting the head with the densest vision sink reduces hallucination—is not tested. All ablations vary layer, threshold, or the number of broadcast heads, but none compare EAH to an equally aggressive control that broadcasts a randomly chosen head, the head with the fewest vision sinks, or the average of all heads. Without such controls, the observed CHAIR improvements could be due to the homogenization of early-layer attention (a diversity-reducing operation) rather than the specific dense-sink property. Please add these controls and report their CHAIR and POPE numbers.","section":"Section 3.3, Algorithm 1, Eq. (12)"},{"comment":"The hyperparameters β, layer index, and top-N are selected directly on the CHAIR evaluation benchmark, and the same CHAIR numbers are then reported as the main result in Table 1. This selection procedure risks overfitting to the evaluation set. The paper provides no held-out validation, cross-validation, or confidence intervals. Please report results on a separate validation split or provide statistical significance tests (e.g., repeated runs with different seeds) to show that the improvements are not artifacts of tuning.","section":"Section 4.4.3, Table 6"},{"comment":"Applying EAH to pure text LLMs (LLaMA-3.1, Ministral, Qwen-2, Qwen-2.5) is conceptually unclear: the vision sink definition depends on an image token range [s, e] and an image token count (e.g., 576), but these quantities are undefined for text-only inputs. The paper does not explain how the method is adapted to LLMs, what token range is used, or what the broadcast operation means without image tokens. Please either clarify the adaptation or remove this section, since it currently undermines the paper's focus and reproducibility.","section":"Section 4.4.2, Table 5"},{"comment":"EAH reduces CHAIR CS from 47.0 to 36.4 but also lowers recall from 76.6 to 73.9, and POPE F1 remains unchanged at 85.7. The appendix acknowledges a roughly 4-point recall drop. This pattern is consistent with a diversity-reducing regularizer that makes the model more conservative in its captions, rather than with a mechanism that specifically improves visual grounding. The paper should analyze whether the gains are driven by reduced output diversity (e.g., by measuring caption length, lexical diversity, and recall on a per-category basis) and should report the standard deviations across the evaluation set.","section":"Section 4.2, Table 1 and Appendix A.1"}],"minor_comments":[{"comment":"The symbol \"Pr\" in the equations appears to be a summation (Σ), not a probability. Please replace it with \\sum for clarity.","section":"Equations (1), (6), (8)"},{"comment":"The text refers to \"Table 8\" for the generalization study on other LVLMs, but the corresponding table is numbered Table 4 in the paper. Please fix the cross-reference.","section":"Section 4.4.1"},{"comment":"The row for Layer=32, Threshold=0.008 lists a threshold value that does not appear in the other rows for layer 32; likely a typo for 0.002. Please correct.","section":"Table 7"},{"comment":"The phrase \"layer 1.2\" likely means \"layers 1–2\" or \"layers 1 and 2\". Please clarify.","section":"Appendix A.3.1"},{"comment":"The word \"significant\" is used to describe performance improvements without any statistical significance testing. Please either add significance tests or replace the word with \"substantial\" or similar.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The mask definition error (M = eye - diag) is a serious technical flaw: if the attention map is square, M is the zero matrix, which would make the vision sink definition vacuous. This needs to be corrected before the paper can be evaluated fairly. The missing control experiment is the other load-bearing issue; without a random-head broadcast baseline, the paper's mechanistic interpretation is unsupported. The authors should be encouraged to run these controls; if the random-head baseline performs comparably, the paper's contribution should be reframed as an attention-homogenization method rather than a vision-sink-specific one."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper has a genuinely simple and new inference-time intervention - copy the attention map of the head with the densest vision sink to every other head in layers 0-2 - and it reports large CHAIR improvements across several LVLMs. That is worth taking seriously. But the central mechanistic claim is underevidenced because every experiment compares EAH to other methods or ablates its own hyperparameters; none tests whether broadcasting a random or sparse-sink head would do just as well. If a random-head broadcast gives the same CHAIR drop, the whole 'vision sink' story collapses into 'homogenize early attention,' which is a much weaker and less interesting claim. The stress-test note is on target.\n\nWhat the paper does well: the CHAIR gains are real and replicate across seven models with different projectors (LLaVA, InternVL, Shikra, MiniGPT-4, etc.), and the method is genuinely training-free and plug-and-play. The LLM generalization (GSM8K, TruthfulQA) is a nice bonus. The authors also acknowledge the recall cost in the appendix, which is honest.\n\nSoft spots: (1) Missing control - the big one. (2) The best configuration (layer=2, beta=0.002, top-1) is selected on the same CHAIR benchmark used for the headline table, with no error bars or significance tests. POPE F1 is unchanged and recall drops about 2.7 points, so the net effect on object hallucination is more modest than the CHAIR numbers imply. (3) The mask matrix in Eq. (5) and Algorithm 1 is written as eye(r,c) - diag(1), which is a zero matrix if diag(1) is the identity - that would make the vision-sink condition never fire. Clearly a typo, but as written it breaks the algorithm; needs fixing. (4) Minor: no code, no token-range details for non-LLaVA models, and Table 6's CS of 36.6 doesn't match Table 1's 36.4.\n\nVerdict: conditional accept, not reject. The intervention is simple enough and the CHAIR gains are large enough that a serious referee should see it. Ask the authors for the random-head control, a validation split for hyperparameters, error bars over seeds, and a corrected mask equation. If the control fails, the paper becomes a much more modest empirical observation; if it passes, it's a solid plug-and-play method. Either way it deserves review.\n\nWho it's for: people working on object hallucination in LVLMs, and anyone interested in attention-sink interventions. Bring it to reading group if you want to debate controls.","headline":"A simple training-free attention-broadcast trick that cuts CHAIR scores across LVLMs; the mechanism claim needs a random-head control and the mask equation looks like a zero-matrix typo.","tokens_in":18246,"tokens_out":3277,"would_cite":false,"duration_ms":33781,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that hallucinations in vision-language models track the density of attention sinks over image tokens, and that copying the densest early-layer attention head across its layer reduces hallucination without retraining.","keywords":["hallucination","multimodal large language models","attention sink","vision sink","attention heads","training-free inference","object hallucination","vision-language models"],"falsifier":"A direct test would be to broadcast a randomly chosen head, or the head with the fewest vision sinks, instead of the densest one; if hallucination drops just as much under those alternatives, density is not what matters. A complementary test would be to apply EAH to images deliberately chosen to contain many small, spatially spread objects and check whether recall of those objects falls, which would show the homogenization sacrifices coverage rather than simply sharpening vision.","tokens_in":17250,"feed_emoji":"👁️","tokens_out":4297,"duration_ms":49589,"temperature":0.7,"pith_summary":"The paper tries to establish that hallucinations in multimodal large language models are tied to how attention is distributed across image tokens in the early transformer layers. When early-layer attention heads spread their scores densely over many image tokens, models are less likely to invent objects; when attention collapses into a few sparse columns, hallucination becomes more likely. The proposed intervention, EAH, finds the head with the densest vision sink in a shallow layer and replaces every other head in that layer with that head's attention map. If right, this turns an interpretability observation into a training-free fix that lowers object-hallucination rates across several model families and even improves some general vision-language benchmarks.","feed_headline":"Copying one attention head curbs vision-model hallucinations","feed_subtitle":"Training-free trick: broadcast the densest vision-sink head in early layers so the model keeps attending to the whole image.","key_machinery":"The key machinery is the vision-sink measure combined with a broadcast operation. A vision sink is defined as a column in the image-token range whose average attention score exceeds $\\beta$; the paper counts such columns per head in a shallow layer and classifies heads with many such columns as dense vision sink heads. EAH takes the head with the maximum count and overwrites every other head in that layer with its attention map, forcing the layer to follow one high-density attention pattern. This operation carries the whole argument because the claimed effect depends on whether homogenizing early-layer image attention is safe and beneficial.","core_discovery":"On the paper's own terms, the central discovery is a consistent relationship between the density of vision sinks in shallow-layer attention heads and hallucination, plus an intervention that exploits it. A vision sink is a column within the image-token range of an attention map whose average score exceeds a threshold $\\beta$; a dense vision sink head is one in which many such columns appear. The paper reports that hallucinating outputs activate fewer dense vision sink heads in layers 1 and 2, while non-hallucinating outputs activate more. EAH then selects the head with the highest vision-sink count and broadcasts its attention map to all heads in that layer, using $A[i][j] = A[i][n]$ for every head $j$. The authors claim this dense, homogenized early-layer attention carries image information forward and reduces hallucination on POPE and CHAIR, with additional gains on several general benchmarks and on LLM factuality tasks.","pith_inferences":["If the mechanism is right, the density signal could be computed per token during generation to select which head to broadcast dynamically, rather than fixing the layer and head in advance as the paper does.","The aggressive overwrite implies that multi-head diversity in early layers is largely redundant for vision grounding; a testable extension is whether those heads could be pruned entirely instead of merely copied over.","The reported gains on pure LLM factuality tasks suggest the attention-sink pattern may be inherited from the base language model rather than learned by the vision projector, a distinction the paper leaves open.","A stress test with images containing many small, distributed objects would show whether the homogenized attention pattern trades hallucination reduction for missed visual details."],"forward_implications":["Applying EAH to layer 2 with $\\beta=0.002$ and the top-1 head improves CHAIR$_S$ and CHAIR$_I$ on LLaVA-1.5-7B relative to greedy search while keeping caption length comparable (36.4/9.9 versus 47.0/13.8).","The method transfers across model families including LLaVA, Shikra, MiniGPT-4, InstructBLIP, Qwen-VL, Mini-Gemini, and InternVL, with larger gains reported for models using MLP or Linear projectors.","EAH also produces consistent accuracy gains on GSM8K and TruthfulQA for LLaMA-3.1, Ministral-8B, Qwen-2-7B, and Qwen-2.5-7B, suggesting the underlying LLM pattern is shared.","Ablation results show that broadcasting to all 32 heads works best and that applying EAH in deeper layers (3, 4, 16, 32) degrades performance, supporting the claim that only shallow layers should be modified."],"supporting_citations":[{"why":"Supplies the observation that image-token information flow concentrates in the first two layers, motivating the choice of shallow layers for intervention.","marker":"[5]"},{"why":"Links attention sink and over-trust to hallucination in MLLMs, serving as a conceptual and experimental baseline for the paper's analysis.","marker":"[17]"},{"why":"Shows that only a subset of attention heads plays a significant role, motivating selection of a single head rather than averaging all heads.","marker":"[23]"},{"why":"Provides the information-flow perspective on attention sinks and anchor tokens that the paper adapts to image tokens.","marker":"[35]"},{"why":"Introduces the attention sink concept used throughout the paper's definitions of vision sink and dense vision sink head.","marker":"[37]"},{"why":"Provides the LLaVA-1.5 model used as the primary baseline and testbed for the main hallucination and generalization experiments.","marker":"[29]"},{"why":"Supplies the POPE benchmark used to evaluate object-level hallucination in question answering.","marker":"[24]"},{"why":"Supplies the CHAIR metric used to evaluate object hallucination in captioning.","marker":"[31]"}],"fun_headline_variants":["Copy one attention head, slash vision hallucinations","Training-free fix: reuse the densest vision-sink head","Borrow early-layer attention to stop model seeing ghosts","Attention sink trick cuts LVLM hallucinations","One head's focus fixes the whole layer's blur"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that overwriting every attention head in the first two layers with one selected head's attention map preserves enough information for downstream layers, so that the forced consensus pattern helps more than the destroyed multi-head diversity hurts.","fun_headline_variants_meta":{"raw":{"variants":["Copy one attention head, slash vision hallucinations","Training-free fix: reuse the densest vision-sink head","Borrow early-layer attention to stop model seeing ghosts","Attention sink trick cuts LVLM hallucinations","One head's focus fixes the whole layer's blur"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000552,"raw_usage":{"total_tokens":2659,"prompt_tokens":997,"completion_tokens":1662,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":613,"completion_tokens_details":{"reasoning_tokens":1589}},"tokens_in":613,"tokens_out":1662,"duration_ms":12020,"temperature":1.0,"reasoning_tokens":1589,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T20:06:49.489688+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A direct test would be to broadcast a randomly chosen head, or the head with the fewest vision sinks, instead of the densest one; if hallucination drops just as much under those alternatives, density is not what matters. A complementary test would be to apply EAH to images deliberately chosen to contain many small, spatially spread objects and check whether recall of those objects falls, which would show the homogenization sacrifices coverage rather than simply sharpening vision.","supporting_citations":[{"cited_title":"An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models","cited_arxiv_id":null,"evidence_quote":"Supplies the observation that image-token information flow concentrates in the first two layers, motivating the choice of shallow layers for intervention."},{"cited_title":"Opera: Alleviating hallucination in multi-modal large language models via over-trust penalty and retrospection- allocation","cited_arxiv_id":null,"evidence_quote":"Links attention sink and over-trust to hallucination in MLLMs, serving as a conceptual and experimental baseline for the paper's analysis."},{"cited_title":"Inference-time intervention: Elicit- ing truthful answers from a language model","cited_arxiv_id":null,"evidence_quote":"Shows that only a subset of attention heads plays a significant role, motivating selection of a single head rather than averaging all heads."},{"cited_title":"Visual instruction tuning","cited_arxiv_id":null,"evidence_quote":"Provides the LLaVA-1.5 model used as the primary baseline and testbed for the main hallucination and generalization experiments."}],"review_version":1}