{"id":"d8d41b84-4cc0-492c-87b7-1e8730cfc102","arxiv_id":"2508.03469","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":6.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":2,"one_line_summary":"IKOD reduces hallucination in vision-language models by merging KV states to derive image-focused shorter-sequence logits and combining them with normal decoding, without training.","lead":"This paper claims large vision-language models pay less attention to the image as they generate longer answers, and that this decline links to rising hallucinations. It proposes IKOD, a training-free decoding method that creates image-focused shorter sequences through key-value merging and blends their predictions with normal decoding.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The causal step in the central claim is only hypothesized: unless the full text includes an attention intervention experiment, IKOD's success is not evidence that attention degradation is the key factor.","rationale":"The reader's weakest assumption already identifies the causal status of the measured attention decline as unestablished, and I agree that this is the most load-bearing point in the central claim. The paper presents a correlation between sequence length, lower visual attention, and higher hallucination rates, then builds a decoding method on the hypothesis that the attention decline causes the hallucinations. If that causal link fails, the method's success, even if reproduced, would not support the paper's stated mechanism. The reader also notes the KV-merge distribution-preservation assumption; I regard that as important but secondary, because the causal claim is the scientific contribution and the stated motivation for the method. The supplied full text is corrupted, so I cannot check whether the authors already include an attention intervention or a calibration analysis; my concrete test is designed to settle this directly. The reader's UNVERDICTED verdict with low confidence remains appropriate, so I recommend no change to the verdict rather than accept or reject on the currently unreadable evidence.","tokens_in":744,"tokens_out":812,"duration_ms":88316,"concrete_test":"Once a readable full text is available, run a controlled attention-intervention experiment on the same LVLMs and benchmarks: at each decoding step after a fixed prefix, rescale image-token attention logits by a factor c > 1 while holding sequence length and sampling fixed, and measure hallucination metrics for c = 1, 1.5, and 2. If hallucination does not decrease when image attention is restored, the 'key factor' causal claim is falsified and IKOD's mechanistic explanation is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's strongest claim has three linked parts: (i) visual attention to image tokens declines as generation length grows; (ii) this decline is a key factor causing hallucinations; and (iii) IKOD reduces hallucinations by restoring image attention through KV-merged shorter-sequence logits. The abstract explicitly marks (ii) as a hypothesis: 'which we hypothesize to be a key factor.' That is the load-bearing step. Without a causal test, the observed correlation between attention decay and hallucination rate could be a symptom of other length-dependent mechanisms, such as repetitive context, positional bias, or a degrading language-model distribution. IKOD could then improve scores for reasons unrelated to visual attention, for example because shorter sequences have different length priors or because KV merging acts as a regularizer. A second assumption, noted by the reader, is that logits from KV-merged shorter sequences are commensurable with original decoding logits; if the merge distorts the language distribution, the logit combination is not principled. The supplied full text is a corrupted encoding, so the equations, tables, and any causal analyses cannot be verified. The concern is not that the method is necessarily wrong; it is that the stated mechanism currently rests on a hypothesis that the abstract does not claim to have tested.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes IKOD (Image attention-guided Key-value merging cOllaborative Decoding), a training-free decoding strategy for large vision-language models (LVLMs) that aims to suppress hallucinations by counteracting visual attention degradation. The authors report observing that LVLM attention to image tokens decreases as generated sequences grow longer, hypothesize that this degradation is a key factor behind increased hallucination, and propose to combine logits from the original decoding with logits from KV-merged shorter sequences that exhibit higher image attention. The abstract claims extensive experiments on hallucination and comprehensive benchmarks, with no additional training or external tools. However, the supplied full text is a corrupted encoding and is largely unreadable, and the abstract provides no quantitative results, protocols, or causal evidence for the central hypothesis.","tokens_in":25897,"tokens_out":4039,"duration_ms":54888,"significance":"If the central claims were established, the paper would offer a practically useful, lightweight and model-agnostic intervention for hallucination reduction, and the documented attention-degradation phenomenon would be a valuable empirical finding for the LVLM community. The strengths of the paper are its falsifiable prediction about attention decline, its training-free and tool-free design, and the clarity of the proposed mechanism at the conceptual level. That said, the scientific significance hinges on the causal claim that attention degradation is a key factor behind hallucinations, which the abstract itself labels as a hypothesis; the method's mechanism could work for other reasons, and the supplied full text cannot be verified in its current form.","major_comments":[{"comment":"The abstract states that attention degradation is 'a key factor' behind hallucinations, but the relevant sentence explicitly says only 'which we hypothesize to be a key factor.' Even in the abstract, no causal intervention or mediation analysis is reported. Without an experiment that directly manipulates visual attention while controlling for sequence length, the observed correlation between attention decay and hallucination rate could be explained by other length-dependent mechanisms, such as positional bias, repetitive context, or a degrading language-model distribution. Because the paper's stated mechanism, and not merely its engineering heuristic, is the central claim, this missing causal test is load-bearing. A concrete causal experiment, such as attention intervention or a mediation analysis, should be added.","section":"Abstract (causal claim)"},{"comment":"The supplied full-text file is a corrupted encoding: equations, tables, and most prose are unreadable. As a result, the KV-merging algorithm, the logit combination rule, the hyperparameters (e.g., merging ratio and combination weight), the experimental protocols, and the numerical results cannot be verified. Since these are the core technical and empirical contents of the paper, a readable manuscript is required before the claims can be meaningfully assessed. This issue is not a presentation nitpick; it prevents verification of every load-bearing technical step.","section":"Full text (all sections)"},{"comment":"IKOD selects shorter 'image-focused' sequences by using the same image-attention signal it seeks to restore, and its evaluation is on hallucination benchmarks. There is therefore a risk that the intervention is tuned to its own evaluation criterion. The authors should include control experiments, for example merging or truncating to sequences of the same length without attention guidance, or randomly merging KV pairs, to show that the gains are attributable to restoring image attention rather than to length priors or to the regularizing effect of KV merging. Without such controls, the current experiments would not establish that visual attention is the operative mechanism.","section":"Proposed method (IKOD)"},{"comment":"The method combines logits from the original decoding with logits from KV-merged shorter sequences. Because the merged sequences come from a modified model distribution, their logits may not be commensurable with the original logits. The paper does not appear to provide a calibration analysis or distributional evidence, such as perplexity comparisons or learned scaling factors, that would justify combining the two sets of logits. Without such evidence, the logit combination is an unprincipled heuristic, and the method's success could be due to the merge acting as a regularizer rather than to the proposed attention-restoration mechanism. A distributional analysis or a calibration experiment should be included.","section":"Proposed method (logit combination)"}],"minor_comments":[{"comment":"The phrase 'not incurring too much inference cost' is qualitative; please report the relative latency or FLOPs overhead in the experiments section.","section":"Abstract"},{"comment":"The reference to 'recent research' that found long-term bias should be replaced with explicit citations to those prior works.","section":"Abstract"},{"comment":"Please define precisely how 'visual attention' is measured, for example mean attention weight to image tokens across layers and heads, and report error bars over multiple seeds once the manuscript is readable.","section":"Experimental setup (unreadable in full text)"}],"recommendation":"major_revision","confidential_remarks":"The supplied full text is corrupted, so I could not evaluate the method or experiments beyond the abstract. If this is a submission artifact, the editor should obtain a clean copy before further review. The causal claim is the main scientific contribution and currently appears untested; the paper may be viable as an engineering heuristic, but the evidence as presented does not support the stronger attention-degradation hypothesis. The novelty disclosure relative to prior decoding methods should also be checked once the bibliography is readable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The supplied full text is corrupted — every section after the abstract is mojibake. So this is necessarily an abstract-only read, and my confidence is low.\n\nWhat is actually new: the paper identifies a consistent phenomenon — visual attention to image tokens drops as the sequence grows — and proposes a training-free fix called IKOD that merges KV states to generate shorter, more image-focused sequences and combines their logits with the original decoding. That recipe sits inside the well-trodden KV-merging and logit-combination family, but the particular diagnosis and the shorter-sequence decoding trick have a plausible ring of novelty.\n\nWhat the paper does well: it is honest about its causal claim. The abstract says attention degradation is something they 'hypothesize to be a key factor' behind hallucination, not something they have demonstrated. That is the right level of caution. The method is lightweight, model-agnostic, and has clear practical appeal if the experiments hold up.\n\nWhere it is soft: the causal step is the load-bearing weakness. We only see a correlation between declining image attention and increasing hallucination. IKOD's success could come from other length-dependent effects — shorter sequences carry different length priors, and KV merging can act as a regularizer. Without an attention intervention experiment, the mechanism narrative is unproven. The reader's concern about the same signal being used both to select shorter sequences and as the thing being restored is fair; it is a mild circularity. And because the full text is unreadable, I cannot check whether the tables and ablations address these points.\n\nNo numbers appear in the abstract, which is normal, but it means I have no evidence of effect size.\n\nWho this is for: people working on LVLM hallucination and inference-time decoding. The paper deserves a serious referee if a readable version is on the table. I would not desk-reject it, and I would send it with a request for a causal/ablation analysis that separates attention restoration from other confounds. If the full text stays corrupted, the only fair move is to ask the authors for a clean PDF before reviewing.","headline":"Abstract-only read: promising inference-time hallucination fix, but the causal claim is explicitly a hypothesis and the full text is unreadable.","tokens_in":26364,"tokens_out":2993,"would_cite":false,"duration_ms":34096,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that in large vision-language models, visual attention fades as responses grow longer, that this decline drives the rise in hallucinations, and that a decoding-time logit-blending method called IKOD mitigates both without…","keywords":["large vision-language models","hallucination mitigation","visual attention degradation","decoding-time intervention","key-value cache merging","collaborative decoding","long-sequence bias","visual grounding"],"falsifier":"A clean test is to decode the same prompt twice, once normally and once with attention to image tokens artificially clamped low after the first few tokens; if hallucination rates do not rise in lockstep with the clamp, the paper's causal story is wrong.","tokens_in":25472,"feed_emoji":"👁️","tokens_out":6192,"duration_ms":74389,"temperature":0.7,"pith_summary":"The paper's central claim is that as large vision-language models generate longer responses, the attention they pay to the image steadily declines, and this decline is a key cause of the well-known increase in hallucinations with sequence length. To counter it, the paper proposes IKOD, a decoding-time strategy that builds shorter, more image-focused sequences by merging the model's key-value cache and then combines their output scores with those of the original decoding. This intervention requires no additional training and no external tools, only access to the model's own attention and logits. If the claim is right, a lightweight runtime fix can make long generations more visually grounded and less prone to hallucination, while also improving general benchmark performance.","feed_headline":"Decoding trick rescues image attention and cuts LVLM hallucinations","feed_subtitle":"IKOD blends logits from KV-merged image-focused drafts, with no retraining or external tools.","key_machinery":"The load-bearing mechanism is Image attention-guided Key-value merging cOllaborative Decoding (IKOD), a collaborative decoding strategy that uses attention to image tokens to guide how the key-value cache is merged. The key-value cache stores the key and value representations of all previously generated tokens, and merging it produces a shorter sequence that is more concentrated on visual content; decoding that shorter sequence yields logits from an image-focused branch. The final prediction combines these logits with the logits of the original autoregressive branch, so the model is steered back toward the image as generation proceeds. The measured per-token image-attention signal is what both documents the degradation and guides the merging.","core_discovery":"The paper reports a consistent pattern across current large vision-language models: as a generated response grows longer, the attention directed at image tokens diminishes, and hallucination errors rise in parallel. It argues that this visual-attention degradation is a key mechanism behind the known long-term hallucination bias, not merely a byproduct of length. IKOD counters the degradation by maintaining, alongside the main decoding branch, a branch that merges the key-value cache into a shorter sequence with higher image attention, and the output scores from that image-focused branch are combined with the main branch's scores at each step. On hallucination and comprehensive benchmarks, the paper reports that this combination suppresses hallucinations and improves overall capacities across models. The method needs no additional training and no external tools.","pith_inferences":["The paper leaves causality open: because it does not manipulate attention directly, the observed attention decline and hallucination rise could both be downstream of a third process, such as the model leaning more heavily on language priors as the response grows.","Extending beyond the paper, the optimal KV-merging strength is likely model-dependent; an adaptive recipe that measures image-attention retention per merging ratio could make the method more portable.","A testable extension is to combine the image-focused branch with contrastive decoding, using the merged short sequence as an anchor that keeps the long generation faithful to the image.","The image-attention statistic itself could be repurposed as a real-time hallucination early-warning signal during decoding, something the paper reports but does not explicitly propose as a monitoring metric."],"forward_implications":["If visual-attention degradation is a genuine mechanism, monitoring attention during decoding can flag when a model is about to drift into hallucination.","A retraining-free, tool-free intervention is available for deployment: any autoregressive large vision-language model with accessible attention and key-value cache can use the same idea.","Hallucination suppression and gains on comprehensive benchmarks can appear together, because keeping the model visually grounded helps the whole generation, not just factual accuracy.","The KV-merged branch gives the model a short, image-grounded preview of its own continuation, so longer generations, where attention drift is worst, should benefit most.","IKOD is complementary to other decoding interventions, since it changes neither weights nor prompts."],"supporting_citations":[],"fun_headline_variants":["KV-merge trick keeps LVLM eyes on the image","Short-draft decoding halts LVLM hallucination spiral","No-training fix for LVLM visual attention fade","Attention-guided merging curbs LVLM hallucinations","IKOD: merging keys to stop LVLM visual drift"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the decline in attention to image tokens is a cause of hallucinations, not merely a correlate, and that logits drawn from KV-merged shorter sequences still represent the model well enough to improve the final answer.","fun_headline_variants_meta":{"raw":{"variants":["KV-merge trick keeps LVLM eyes on the image","Short-draft decoding halts LVLM hallucination spiral","No-training fix for LVLM visual attention fade","Attention-guided merging curbs LVLM hallucinations","IKOD: merging keys to stop LVLM visual drift"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000272,"raw_usage":{"total_tokens":1646,"prompt_tokens":975,"completion_tokens":671,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":591,"completion_tokens_details":{"reasoning_tokens":595}},"tokens_in":591,"tokens_out":671,"duration_ms":7864,"temperature":1.0,"reasoning_tokens":595,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T04:24:34.632445+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A clean test is to decode the same prompt twice, once normally and once with attention to image tokens artificially clamped low after the first few tokens; if hallucination rates do not rise in lockstep with the clamp, the paper's causal story is wrong.","supporting_citations":[],"review_version":1}